Snorre Davøen | Frontend Developer | Bergen

Welcome to the new Snorre.io

Wed, 26 Mar 2014 00:00:00 GMT

Snorre.io is the new homepage of Snorre Magnus Davøen, me. As you can see it is not quite a homepage yet. Snorre.io will currently serve as a blog where I can post my experiences, my thoughts on various subjects, tutorials on different technologies, and posts presenting my work.

Snorre.io runs on the Ghost blogging platform. Ghost is currently not very feature rich. As new features are introduced and the Ghost API is stabilized I might add new features to my homepage.

So long.

Mosh (mobile shell)

Sun, 11 May 2014 00:00:00 GMT

It was quite a good while ago that an acquaintance of mine made me aware of a great alternative to SSH called Mosh (short for mobile shell). Mosh was born out of MIT as a result of a few important factors.

Latency on the Internet is on the increase
Roaming has become more common
SSH does not properly support UTF-8

Latency

So why is latency on the Internet an issue when using SSH? Well, if you have ever spent time on a connection with high latency you will know the frustration of using SSH. The problem of course is that SSH is designed so that the user’s key strokes are passed on to the terminal session at the server and the output buffer from the server is passed back to the client’s terminal. On very responsive connections this is of course not an issue. But imagine that there is half a second of delay between your computer and the server. In such an instance it would take a whole second from the time you type a letter till the output of that letter reaches your screen. That kind of delay can be super frustrating. Editing text becomes a chore.

But how does mosh fix this? Well, Mosh uses state synchronisation to synchronise the screen buffer state between the server and the client, and something they call “intelligent local echo”. Essentially the mosh client will just echo out to the client terminal what you are typing. If you start erasing text you can see how you are able to erase the username@hostname# before the server catches up and Mosh corrects the client state. Usually you would type sensibly and so not notice any client state corrections.

High response times and slow connections between the client and server can also pose problems when running scripts through SSH that spews out huge quantities of bloat. This is not a very rare occurrence. To combat this problem Mosh only sends the very last part of the screen buffer to the client. More specifically, it sends just as much of the screen buffer as the client terminal can display at any one time. To get access to the whole screen buffer you need to run something like screen or tmux on the server side to navigate through the buffer. By just sending you the last part of the screen buffer the server saves bandwidth and makes the whole thing feel more responsive.

Roaming

We all know the frustration of loosing our SSH connection because our WIFI-connection broke down, or the VPN server disconnected us, or some other annoying reason. As devices become more portable and we use several connections we’ve also felt anger about the necessity of disconnecting and reconnecting every time we switch from one network to another. SSH doesn’t even care to give us some sort of indication that the connection were lost.

Fortunately Mosh has fixed this as well. Mosh implements a protocol that attempts to reconnect whenever the connection to the Mosh server is lost. Of course security is needed to prevent a random person from hijacking the connection, and this is implemented in the form of a key unique to that session.

UTF-8

According to the Mosh home page SSH, and some terminals, have UTF-8 bugs were they don’t properly support certain UTF-8 characters. They claim that Mosh is implemented to have full UTF-8 support, and that it is built ground up for this character set. I have not personally had problems with UTF-8 encoding in SSH before, but I am sure this is a nice feature for some people.

Conclusion

I have only been using Mosh for a little while now, but it has already made its way onto my must have Ubuntu server application. The responsiveness and roaming support are its key selling features and make remote terminal use much better! Mosh is available for just about every operating system out there (except some mobile OSs). If you run Ubuntu 14.04 you will be fortunate to find Mosh in the default repositories.

Netflix launches second round of kimmification

Sun, 17 Apr 2016 00:00:00 GMT

Netflix recently released the second season of Unbreakable Kimmy Schmidt, one of their original series focused around situational comedy. The first season introduced the titular character Kimmy Schmidt and bootstrapped the series with a rather bizarre premise:

The series follows 29-year-old Kimmy Schmidt (Kemper) as she adjusts to life in New York City after her rescue from a doomsday cult in Indiana where she and three other women were held by Reverend Richard Wayne Gary Wayne (Jon Hamm) for 15 years. - Wikipedia

Now the series is back for yet another humorous season. It picks right up where season one left and Kimmy is as goofy as ever. Where many American sitcoms rely on laugh tracks and predictive social situations to entertain, Unbreakable Kimmy Schmidt will throw out ludicrous plot lines and make you laugh without using as much as an ounce of canned laugh track. The lack of laugh track makes the series’ pacing better and allows the series to have more serious moments.

On the face of it some might reject Unbreakable Kimmy Schmidt on the grounds of being a “girly” show, but they would be missing out on what is a great sitcom. I also commend the show’s creators for writing a comedy with strong female leads as shows in the comedy genre seems to usually center around male characters.

GPG, Web of Trust, and Keybase.io

Thu, 28 Apr 2016 00:00:00 GMT

If you are a developer I’m sure you are familiar with GPG also abbreviated GnuPG which stands for Gnu Privacy Guard. It is an open-source implementation of the PGP (Pretty Good Privacy) algorithm.

Removed image

So what is this privacy stuff all about? Well, PGP/GPG implements what is known as a web of trust. A user creates a private and public key, associates their name and multiple identities (emails) with the public key. The private key is used to decrypt and encrypt the public key so no changes can be done to the GPG public key without the private key. Whenever sending a message or email a user can sign their message with their public key. The receiver (an email client for instance) can then check that the signature is indeed valid by looking at the public key which they obtain from one of several GPG key stores. The public key and message signature verifies that the message was sent by the holder of the private public key pair.

This however does not prove that the message was sent by the person who the key’s identity claims to be. This is where the web of trust concept comes along. Say that you witness John Doe prove possession of a private and public key pair in person, you Jane Doe sign John Doe’s key with your own identity. With this signature you state that you vouch for the legitimacy of John Doe’s public key and that this key does indeed belong to the Mr. John Doe. Now say that some Bob Dobson receives a signed message from Mr. John Doe. He does not know Bob in person, but does know Jane Doe who’s key he trusts (and signed). Since Jane Doe has signed John’s key there is an indirect trust chain from Bob through Jane to John. Bob can be a bit more confident that Bob is the holder of this key, and not some other guy claiming to be John. The more people sign each others keys the more and shorter trust paths exists between individuals in the GPG web of trust.

In theory this works very well, but one weakness with the web of trust model is the need for people to validate a person’s possession of a GPG key before signing it, best done in person (thereby the rise of key signing parties). Setting up a public private key pair is barrier enough for most people outside of the information technology sphere if not requiring them to run around to key signing parties. So in comes keybase.io with an identity verification service built on top of the GPG technology, but with a different take on identity verification.

Removed image

On Keybase you no longer prove your identity by having other (preferably trusted) people signing your key. Instead you register a Keybase.io account, link it to your public GPG key, and are finally prompted to perform a series of proofs. A proof is essentially a message posted to some social service signed with your public GPG key. Services includes Github, Hacker News, Reddit, Twitter, your own domain (through DNS or hosted file), etc. When you’ve successfully accomplished a proof Keybase.io will link that social account to your Keybase.io account and by extension your GPG key. In effect you bind the sum of your social identity across different social services to your keybase.io account and your public key. Now if another Keybase.io user wants to send an encrypted message to your Keybase.io user the Keybase.io server will return a JSON object containing the public key and a list of services with usernames and links to the proofs. The Keybase.io client verifies that these are correct, and you as a user can also verify that John Doe on Keybase.io is the same user that you know from Github, Twitter, etc. This way people can also trust that the John Doe they see on Github is actually the same John Doe they know from Twitter. As an example, here is my Keybase.io profile. If you wanted to send me an encrypted message on Keybase.io, but were unsure if Snorremd was actually me, you could see that my Keybase.io account controls the Twitter and Github Snorremd users and snorre.io (this domain). It is quite unlikely that an attacker would gain entry to all of these services, so you would be quite safe to trust my Keybase.io account.

Keybase.io then solves part of the web identity problem, but not quite the identity verification problem itself. The Mrs. Celebrity of Github and Twitter fame might have the same GPG identity across Github and Twitter, but this does not prove that the GPG key is actually owned by the real Mrs. Celebrity. There is also no indirect trust chain. You can not say that you trust Jane Doe and implicitly trust Bob Dobson who Jane trusts. Keybase.io currently does not claim to implement a web of trust however, so if you are okay with that you should be good. Many people also primarily communicate with each-others web presences. You might not care very much about whether or not you are communicating with the actual Mrs. Celebrity, only that you are communicating with the Mrs. Celebrity as you know her on Github. To allow you some comfort in trusting Keybase.io users Keybase.io implements tracking, a feature in which you are notified upon any change in the public key or proofs of a user. All in all I think Keybase.io is an intriguing concept and might help make GPG keys popular outside the most crypto conscious circles.

isso - a lightweight privacy minded comment system

Tue, 03 May 2016 00:00:00 GMT

I’ve been using Ghost blog for quite some time now and have been very happy with the functionality, performance, and the simplicity of the solution. I use the default theme Casper which I find to be a minimalistic and rather pleasing blog theme. I’ve also avoided analytics software (and ads, not that ads would amount to any money on my little blog) like the plague for two important reasons:

Privacy: Adds and analytics software tend to cobble together visitors’ browsing history to form a web of browsing history. I’m privacy conscious myself and would prefer to not expose my visitors to tracking networks if I can avoid it.
Performance: Adding bloat and unnecessary Javascript to my page would only make the reading experience worse.

That is why if you visit this page adblock plus reports zero blocked elements and Ghostery reports only Gravatar which is used by Ghost to find the avatar of each author. All in all not too shabby if you ask me.

There is however one thing in Ghost that was missing and that I’d really like to have which was a comment system. For some time I have had a few blog posts out, but no way for people to comment on them. This is no good. So I started looking into a few options.

One of the first systems that came to mind were Disqus. It is fairly popular and free to use. It lets you sign in with different SSO providers like Facebook and Google, and offers functionality for moderation etc. The issue for me though is the privacy concern. All comments live on the Disqus servers which is no good from a privacy perspective.

Livefyre is also fairly popular out there, but it costs money and has the same privacy issue as Disqus so I quickly dismissed it.

A very attractive option then seemed to be Discourse which I had read about before and experienced first hand on sites using it. Discourse seems like a really nice solution with some good ideas centred around live discussions through notifications and a flat thread view. The only issue for me was the difficulty in finding an easy to set up Docker image that would fit into my existing docker-compose setup.

So after some consideration and thought I remembered isso a project I had stumbled upon quite some time ago. And to my great fortune the project was alive and well and seemed to have gotten quite a few users since I last had checked it out. Not only was it alive and well, it checked a lot of the requirement boxes I held in my head for a blog comment system:

It is light weight
It’s super simple to use
Hosted on your own server
Requires no SSO login
Provides optional moderation feature
Works very well with Docker and docker-compose with one of two unofficial docker images
Integrates easily into Ghost blog through Handlebar theme modification

I won’t go into how you set isso up in your own Ghost blog because someone already did.

If you want to run isso with Docker check out:

bl4n/isso - Recommended in official isso documentation
wonderfall/isso - Recommended by me for its smaller image size and use of Alpine linux!

You are also encouraged to read the official isso documentation.

Update 19th March 2018

I have since moved to a new comment system, Coral Talk. Read more about that in Coral Talk - An open source comment system from Mozilla.

Jalopy (game) - A trip through Eastern Europe in a Lada

Tue, 26 Jul 2016 00:00:00 GMT

I recently learned about an early access game called Jalopy. For those who does not know jalopy is a term used to describe an old and damaged car in barely functioning state.

I am awaken by an alarm clock early in the morning and greeted by my uncle (in a trippy tweed jacket) who wants to show me some car he has sitting outside his garage workshop. It is a Lada Laika without wheels, engine parts, or a passenger side door. My uncle quickly teaches me the ropes and helps me fit a door, an engine, a fuel tank, a carburetor, an ignition coil, an air filter, and some wheels. I learn about the effects of different engine parts on the fuel consumption, stress on the engine, etc. This tutorial will prove to be of use as I set out in my decrepit Lada with a bit of cash, some tools, an extra fuel tank, oil, and a map to travel across Eastern Europe.

Removed image

As I set out on the highway towards my first destination I listen to the 2-stroke engine struggling along as I try to hit at least 80 km/h. The car handles like a piece of garbage, but I helped fix it up so I still cannot hate it too much. While cruising down the highway I enjoy the view. The graphics in Jalopy is not much to talk about with the game relying on very low polygon-count models, and simple textures. The graphics do however project a very fitting style for a Eastern European setting with the somewhat muted color palette. Driving down the highway is not so terrible at all.

Removed image

The traffic is light, but then again we are in soviet era Eastern Europe. There are other cars on the road in this game, and of course they inevitably catch up to our Lada. This is life in a jalopy car. As I drive along I soon notice smoke emanating from under the bonnet. Shit! I slow down and stop at the side of the road to check what’s going on. As I suspected half my engine parts have gone to shit. Well, it’s a good thing I brought some tools? Or did I forget? Yep, my tweeded uncle forgot to mention how tools work. Oh well. This is one of the great things about Jalopy. There are so many ways you can mess up. I forgot my tools, so I reluctantly close the bonnet, get back in the Lada, and drive on hoping to find a petrol station before my precious Lada breaks completely down. Fortunately I do.

Removed image

Your budget is limited and as you travel you will need to spend money on fuel, oil, tools, overnight stays at motels, and other such things. Your limited budget won’t last long it would seem. But if you pay attention along the road there will be opportunities to make some money as losing cargo along the road seems like a past time for Eastern European citizens. If you are really lucky you might find ditched Lada Laikas from which to loot some engine parts!

Removed image

All in all I’ve had two good hours of fun already with this game, but it is in a very early phase still! If you do decide to buy it be aware that there are still some bugs/glitches, and questionable performance. My fps went from around 160 during daytime to barely 10 - 20 during nighttime. Perhaps something to do with lighting effects. Playing the game in high resolution (I played with 1440p) seemed to present a bit of a challenge as the in game cursor does not scale. For less than a reasonably inexpensive craft beer (where I live) I would say this game provides decent value and hope it will improve, game-play and performance-wise, as the developer puts more work into it.

Clojure - A first encounter

Sun, 24 Aug 2014 00:00:00 GMT

I have been thinking for some time about picking up Clojure, a functional programming language and Lisp dialect that runs on top of the Java Virtual Machine. I have always wanted to learn a proper functional language, and Clojure with its Python and Ruby influences, compatibility with Java and the JVM, and web version (ClojureScript) seemed like a good bet.

My five year long education and short few months of work have so far been spent programming Java, Python and a bit of JavaScript and CoffeeScript. I shared a brief semester struggling with Prolog, a language inspired by first order logic, but never took the scenic route through the backroads of the varying Lisp dialects. It is about time I took the plunge into the vast sea of functional programming.

To that end I bought Clojure Programming and set to it. This blog post will not discuss the book, but it might be the topic of a future post.

Removed image

In what might become a blog series I will attempt to document my journey with Clojure. Without further ado let’s get on with it.

Getting Clojure up and running

Installing Clojure was relatively simple as Clojure is really just a JAR library that runs on the JVM. I ended up installing the newest version of the JVM (8), and installed Leiningen, a build manager for Clojure with the HomeBrew package manager for OS X. I immidiately stumbled into an error after running lein repl, the lein command for running an interactive Clojure repl. The JVM stack trace informed me of the following error:

Error loading namespace; falling back to userNoSuchMethodError

Some googling turned up an issue submitted in the GitHub repo for the HomeBrew distribution of Leiningen. Of course the issue is already fixed in the Leiningen master branch, but the newest version is probably not pushed into the Homebrew package repositories yet. A workaround for the moment appeared to be running the command from inside a Leiningen Clojure-project. So that is was I tried:

lein app helloclojure
cd helloclojure
lein repl

Running lein repl from inside a project changes the context in which the repl is run and alleviates the error, but I can’t say what differences there might be.

Light Table (editor)

While writing and testing Clojure in the OS X terminal in a REPL environment works well enough for learning the basics, it can quickly become cumbersome to write longer function as lack of linting and syntax highlighting makes it harder to read the code. So I set out to get myself an editor. Many programmers are big fans of emacs, an open source editor written in Lisp that is highly extensible. emacs provide almost endless possibilities, such as interpreting Lisp functions inside your code documents, and extending the editor functionality with Lisp code even as the editor is running. While emacs may be an amazing editor I find myself leaning towards editors that hold your hand; Editors that comfort you with a proper GUI interface, easy to use menus when you forget commands, and a good file map that provides an easy way to get an overview of your work.

Light Table is one of those editors. It is a quite recent addition to the editor world of old, and as such does not provide the same extensive feature set of older editors such as the aformentioned emacs, or newer additions such as Sublime Text. What it does provide is excellent Clojure and ClojureScript support, modern visuals, and great promise. It raised over 300.000$ on Kickstarter. Not bad for an open-source editor.

I found that it kickstarted me on my way to Clojure comprehension quite soon as its documentation suggested I try its feature instarepl. Instarepl appears to be a pseudo-repl environment that does not conform to the repl loop of read, evaluate, print, loop. Instead the editor provides a window in which you can enter and edit code at will; The code is interpreted on the fly. Next to each line of code Light Table shows you the data value or values being used. This amazing feature quickly tells you what the input to, and output of a given expression is.

Removed image

I will end the blog post before it becomes a Clojure book. Fare well, and keep on coding.

Setting up a Docker image builder with Gitlab CI Runner

Thu, 21 Apr 2016 00:00:00 GMT

In this post I’ll give you an idea on how to implement a Docker image builder with the Gitlab Runner. Before I do that I’ll briefly introduce Gitlab and Gitlab CI.

Update! See my new post detaling som strategies for building Docker images with Gitlab CI.

Gitlab, if you do not already know, is an open-source git hosting and management solution also offered as an enterprise version (on premise or hosted). In my opinion it is fast becoming one of the best solutions for git hosting and is a welcome competitor to Github.

For quite some time now Gitlab has offered Gitlab CI a continuous integration server, initially stand alone, but later integrated into Gitlab itself. Among the most important features in Gitlab CI we find the concept of dedicated runners (equivalent to slaves for those familiar with Jenkins). The CI server itself only orchestrates the CI build jobs, while the runners do the heavy lifting. As long as the runners run on one or more separate servers the orchestrator-runner pattern results in a safe environment where the build scripts have very little access to the CI server and of course git server itself.

Gitlab CI uses gitlab-ci.yml files defining one or more build jobs to run continuous integration jobs. These are similar to travis.yml and other yaml based CI build script files. It might for example look something akin to this (some bits missing):

types:
  - build
  - test
  - deploy

build_app:
  type: build
  script: ./make build

test_app:
  type: test
  script: ./make test

deploy_app:
  type: deploy
  script: scp ./app deployuser@someserver:/app

In the example we define three build stage types: build, test, and deploy. The order is significant. We also define three jobs, build_app, test_app, and deploy_app, which are executed in the order given by the types to which they belong. If one build stage fail, the others will not be executed. All of this is probably familiar to you if you’ve used other CI services before.

Gitlab Runner executors

With hosted CI solutions you often don’t have a lot of, or need for, choice with regards to how the builds are executed. At Travis CI, a good hosted solution, you either build in a virtual machine or docker container.

Gitlab Runner on the other hand affords you a host of executors:

shell - executes build script on host
docker - executes build inside docker container(s)
virtualbox - executes build in a virtualbox vm
parallels - executes build in a parallels vm
ssh - runs build on a different host via ssh
docker-machine - autoscales docker machine instances and runs builds in containers across these instances

While you might find it tempting to run builds directly on the host with the shell executor it is seldom a good idea to allow script execution directly on your host, especially if this runner is to be shared with several people. The ssh runner will also be problematic in that aspect.

The Virtualbox and Parallels based runners isolate builds as a new VM is cloned and started for each new build. One problem with the vm based executors are their lack of support for some of the caching features of Gitlab Runner. They are also slow to start up for big VMs as the whole virtual machine image has to be copied.

That leaves us with the final two options the docker and docker-machine runners. The docker executor will spawn a container for each build (and even each build stage) and run the build scripts inside the container thereby isolating the build environment from the Gitlab Runner host environment. One of the great benefits of the docker runner is the lightweight nature of spawning docker containers. Once a docker image, say node, is present on the Gitlab Runner host, it takes mere seconds to start it!

Unless you have extremely large and fluctuating build pressure you probably don’t need the docker-machine executor with auto-scaling.

So you install the Gitlab Runner package and Docker on your Ubuntu/Debian/SomeDistro server. Everything is set for your docker enabled build adventure.

So say we want to build a Node.js-application, we can supply a .gitlab-ci.yml file similar to the one below (greatly simplified). Note the image property for the build stage. It specifies which docker image to use for the build. Here we use the Node 5 image.

types:
  - allthethings

build_app:
  type: allthethings
  image: node:5
  script:
    - npm run test
    - npm run build
    - npm run deploy

Great! We’ve managed to get our app tested, built, and deployed from within the Node.js docker container. Though this is not actually the problem we set out to solve. Indeed what we want is for the Gitab Runner to build us a Docker image with our app inside and then deploy that docker image to a registry of our choice.

There’s not a ton of documentation on the subject out there, hence this blog post, but Gitlab has a page on Using Docker Build in their docs. What you will quickly discover, as I once did, is that both their suggested options come with some security warnings. You can either add the gitlab-runner user to the docker group and use the shell executor. This will however essentially give the gitlab-runner user root access to your host server. The docker-in-docker solution is not much better as the docker container must run in privileged mode thereby providing a chance for the build script to perform privilege escalation and all other shenanigans.

So what can we do to get around this you may ask? Use an external docker engine on a more isolated host! There is nothing dictating that the build process be given direct unix socket access to a docker engine. Instead I suggest exposing a docker-engine from another host via tcp. This way your build script can use the remote docker engine without compromising your build server. Granted you set up server client certificate authentication on the remote docker engine only your build scripts will have access. So how might this look you ask?

Well start a docker engine for example via docker-machine. docker-machine create -d virtualbox dockerbuilder. This will create a vm based on boot2docker, configure ssh, and set up docker server and client certificates. For simplicity’s sake you can start it on your Gitlab Runner-server, though the location should not matter much.

Once you’ve started the machine the machine configuration should be available at ~/.docker/machine/machines/dockerbuilder. You should see ca.pem, cert.pem, and key.pem. These are the three certificated needed to connect remotely to the docker engine on the boot2docker machine.

Running docker-machine ls will reveal the IP your dockerbuilder machine is listening to:

NAME            ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
dockerbuilder   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.11.0

So how do we gain access to the docker-engine from our build script running inside a container? Docker Hub provides us with a Docker Docker image. This image container the docker binary we need to communicate with our virtualbox hosted docker engine.

To build a docker image from within the docker docker container we can now simply add our certificates as variables in the Gitlab project settings and create a .gitlab-ci.yml build job like so:

types:
  - dockerbuild

build_docker_image:
  type: dockerbuild
  image: docker:latest
  script:
    - echo "$DOCKER_CERT_PEM" > /root/cert.pem
    - echo "$DOCKER_KEY_PEM" > /root/key.pem
    - echo "$DOCKER_CA_PEM" > /root/ca.pem
    - export DOCKER_TLS_VERIFY="1"
    - export DOCKER_HOST="tcp://192.168.99.100:2376"
    - export DOCKER_CERT_PATH="/root"
    - docker build -t username/imagename:latest .
    - docker login --username="${REGISTRY_USER}" --password="${REGISTRY_PASSWORD}" --email="${REGISTRY_EMAIL}" docker.example.com
    - docker push username/imagename:latest

DOCKER_CERT_PEM is the variable name from the Gitlab project settings and its value is the contents of the cert.pem file from before. The solution is a bit noisy, but could probably be moved into a shell script to reduce clutter. The important takeaway here is that we fetch the certificates that the docker-machine create -d dockerbuilder command generated for us and stored them in the Gitlab project settings variable page. Gitlab CI will send them to Gitlab Runner when orchestrating a build, and Gitlab Runner will make them available as environment variables in our build.

After doing some boilerplate setup by writing certificates to files and setting some docker environment variables we can freely use the docker command to communicate with the engine running at our virtualbox on address 192.168.99.100:2376. We can then login to a docker registry and push our image there.

One note. Our virtualbox based boot2docker instance will keep all images and login info we store on it as long as it is running. Subsequent builds of our docker image will therefore be able to use pre-existing layers from previous builds. Because the build script itself is run in our docker docker container all data we write in it, such as our super secret certificates, will disappear into thin air together with the container when the build is done. The certificates and your connection to the virtualbox docker engine should therefore be safe. Note however that anyone with developer or master access to your Gitlab project has access to the variables on the project settings page.

Setting up (B)ELK stack on your server

Sun, 24 Apr 2016 00:00:00 GMT

I’m a developer by day, but as evening falls I sometimes like to dabble as an amateur sysops. I’ve recently helped out setting up a server as part of a hobby project that I participate in. While I’m no sysops expert I like testing out technologies helpful in server administration. As a side effect it makes me appreciate just how much detail and work sysops people put into managing servers. One of these sysops technologies is the much known ELK stack (ElasticSearch, Logstash, and Kibana) recently joined by the different Beat log ingestion system services. ELK are usually used as a general tool to gather, filter, index, and analyze logs from different sources. In this post I will describe how a BELK stack can be set up, partially in docker and partially on the host, to provide an insight into your systems network traffic, system activity (from system logs), and process and performance activity.

ElasticSearch

The first piece of the puzzle is ElasticSearch a distributed search and analysis technology based on the Lucene library. It can be used to index documents and then search in them. It will index our log and event entries in our BELK stack. We can use the ElasticSearch docker image. If you are not familiar with Docker you might want to read up on both Docker and Docker Compose.

Create an elk and elasticsearch folder somewhere appropriate. When using docker /srv is commonly used.

mkdir -p /srv/elk/elasticsearch

Now create a new docker-compose.yml file either in the /srv/elk folder, or better yet on your own computer. It all depends on whether you have tcp access to the docker-engine on the host where you are setting up the BELK stack. Add the following contents to it:

elasticsearch:
  image: elasticsearch:latest
  restart: always
  volumes:
    - /srv/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins
    - /srv/elk/elasticsearch/data:/usr/share/elasticsearch/data
  ports:
    - "127.0.0.1:9200:9200"
    - "127.0.0.1:9300:9300"

We call the service elasticsearch, use the latest elasticsearch image, and map in ports and volumes from the host into the container. The elasticsearch api should now be available from the host server’s port 9200 on localhost only.

You may start the elasticsearch container to check that everything works by running docker-compose up -d. Run docker-compose stop to stop elasticsearch.

Kibana

Kibana is our second piece of the puzzle. It is simply put a web based data exploration and visualization tool built on top of elasticsearch to make it simple to explore and visualise/graph your elasticsearch data. It is commonly used to visualise logs or just data.

For Kibana we’ll use the official Kibana image to run Kibana in a docker container. We can simply add Kibana as a service in our docker-compose.yml file (the same one as before) by appending the following yaml config:

kibana:
  image: kibana:latest
  restart: always
  volumes:
    - /srv/elk/kibana/config:/opt/kibana/config
  ports:
    - "127.0.0.1:5601:5601"
  links:
    - elasticsearch

and create a kibana folder under our elk folder:

mkdir -p /srv/elk/kibana/config

Kibana relies on a config file kibana.yml to function so you will need to add this under the kibana config folder.

touch /srv/elk/kibana/config/kibana.yml

See Setting Kibana server properties in the Kibana docs for which properties you may configure. For this setup you probably don’t need to change anything and can rely on the default values.

If you want to test that your Kibana instance works you can again start the docker containers as instructed before with docker-compose up -d. You Kibana instance should be accessible from localhost:5601 on your server. Run docker-compose stop to stop the services again.

Logstash

While it is entirely possible to make TopBeat, PacketBeat, and FileBeat send their log and event data straight to ElasticSearch you might want to pass them through Logstash as this affords you Logstash’s filter feature with which you may extract more detailed information from your logs. We will therefore set up Logstash as well, all though this post won’t touch upon using the filter feature.

An official Logstash Docker image is available so again we use Docker to run the service. Create a folder for logstash configs:

mkdir -p /srv/elk/logstash/config

and add the following Logstash service configuration to the existing docker-compose.yml file:

logstash:
  image: logstash:latest
  restart: always
  command: logstash -f /etc/logstash/conf.d/logstash.conf
  volumes:
    - /srv/elk/logstash/config:/etc/logstash/conf.d
  ports:
    - "127.0.0.1:5000:5000"
  links:
    - elasticsearch

As you can see we link in the elasticsearch service (so logstash can gain access to it from elasticsearch:9200, and map in a configuration directory from our host to the logstash container. We’ve also exposed the Logstash container’s port 5000 on the server’s localhost:5000.

You need to provide logstash with a configuration file in order to make it listen to port 5000 and send documents to elasticsearch:

vim /srv/elk/logstash/config/logstash.conf

which should contain an input section for the Beats and an ouput section for the ElasticSearch server:

input {
  beats {
    port => 5000
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

The Beats will now be able to send their data to Logstash at localhost:5000 from the host. We are now done with the Docker based services so you may start them with: docker-compose up -d from the directory where you keep your docker-compose.yml file. We need Logstash and Elasticsearch to be running for the upcoming steps.

Beats

Finally we need a way to collect our log data. While Logstash can read log files from our host the FileBeat package is special made to perform this task. Unlike Elasticsearch, Logstash, and Kibana we will install FileBeat, TopBeat, and PacketBeat on our host.

First add the Beat repository as described here (for the distro of your choice). Then install the beats. On Ubuntu you can use apt:

apt install filebeat packetbeat topbeat

Now we need to configure each Beat to use our Logstash instance, and provide our Elasticsearch server with the relevant Beat templates.

First edit the filebeat config with vim /etc/filebeat/filebeat.yml and make sure that the elasticsearch ouput is commented out:

# elasticsearch:
    # hosts: ["localhost:9200"]

Then comment in the logstash ouput config:

### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["localhost:5000"]

Remember that we configured our Logstash server to listen to port 5000, and mapped port 5000 out to localhost:5000 on our host. This enables Filebeat to log its event to that port.

Then we need to insert the Filebeat template into our Elasticsearch index. The documentation tells us that we can use Curl for this:

curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Finally we can start Filebeat. On Ubuntu you may use service filebeat start.

You can now repeat these steps for PacketBeat and TopBeat. Note that you may want to set the interval of data collection in the Topbeat configuration to avoid generating too much data. What interval is right for you might depend on your disk space, but 5 minutes works for me.

Add Beat indexes to Kibana.

If everything worked out okay you should now be able to go to Kibana and add your indices. Go to localhost:5601 and click the Settings tab. You should be presented with a screen where you can configure an index pattern. Thanks to our Logstash configuration an new index will be created each day for each of our Beat loggers. Our pattern should thus match the configuration "%{[@metadata][beat]}-%{+YYYY.MM.dd}" (from the Logstash config). So to get each of the three index types into Kibana we simply add them as filebeat-*, etc. If your index pattern is matched Kibana should show a green Create button as in the screenshot below.

Removed

After adding the index pattern you should be able to go to the Discover tab and view and search your logs.

Removed

You can now create your own visualizations over the indexed logs or you can download some of the visualizations and dashboards already created by the Kibana team. You can read more about that here and here.

Hope this guide helped you get started. Currently I’ve not yet tried setting up custom filters for things such as fail2ban and ufw in order to better visualize malicious traffic. You can also use a geoip plugin to map log and network traffic events on a world map to see where traffic is coming from. A future blog post might cover these subjects.

Attributions

Image post header:

By Bureau of Land Management (Dean Creek Elk 4 Uploaded by Albert Herring) [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons.

A privacy respecting talk gravatar plugin

Wed, 25 Apr 2018 00:00:00 GMT

Some time ago I started using the Coral Talk comment system for my blog. The system is probably way overkill for a blog my size, and I have so far only received one comment! While Coral Talk is pretty nice it lacks built in avatar support and I could only find a plugin recipe requiring avatars to be hosted by third parties and the links to be updated by some external service.

In my quest for avatar support I decided I would like to implement an avatar plugin that would require minimal effort from users and the maintainer of a Coral Talk instance. Gravatar (i.e. globally recognized avatars) is a popular avatar service which provides avatar URLs for each user which can be embedded by any service. The concept is simple enough. Gravatar creates an avatar URL based on a user’s md5 hashed email address and responds with the user uploaded avatar upon requests. Gravatar is used almost everywhere so why not in a Talk avatar plugin?

There are unfortunately a number of privacy issues to be aware of when considering using Gravatar. Fly.io, a cdn/adn provider, wrote a summary of how gravatar hurts your visitors which goes into these privacy issues and how some of them are solved using their Gravitas proxy server.

This inspired me to create a Talk Gravatar plugin that would solve some of these issues to help safeguard commenters’ privacy. Enter talk-plugin-gravatar.

With talk-plugin-gravatar I’ve implemented a talk plugin that will safely proxy Gravatar requests via the Talk server using randomly generated avatar uuids to avoid leaking indentifible information to browsers. At the same time the Gravatar servers will only know the IP of the talk service from which the avatars are requested, not the specific pages the Gravatar users are commenting on. This is a big privacy win for commenters.

It should be pointed out that unlike the Fly.io Gravitas service which would proxy Gravatar requests for any number of source sites, the Gravatar plugin will only proxy requests for a single Talk instance and therefore not provide the same level of privacy. There is also the issue of styling. Unfortunately the default Talk markup and styling places the so called avatar slot on top of, rather than left of, the comment. You need to supply custom styling via Talk’s administration settings to fix this issue.

In conclusion

The plugin is pretty basic, but does its one function well. It relies on Talk’s redis cache to cache avatar requests. This way the plugin avoids hammering the Gravatar service with requests, while offering about the same response timings as the Gravatar service would have if used directly. There are zero configuration options which makes it incredibly easy to set the plugin up.

You can see the plugin in action in my comment field.

New homepage with Gatsby.js hosted with Cloudflare Workers

Sun, 03 May 2020 00:00:00 GMT

The past couple of years or so I’ve been running a homepage at snorre.io. As many students of IT I wanted to host my website and blog on a virtual machine server under my control. So I set out to find a good VM cloud provider and a simple open source blog CMS. I eventually landed on Ghost which at the time was a welcome alternative to WordPress and other CMS. Ghost served me well over the years being mostly free of distractions and simple to maintain. At some point the Ghost Foundation started complicating the content model and writing experience.

The past few years I’ve worked on projects using React in some form and grown to really like the UI library. React allowed me to write nearly functional and declarative code to express UIs and was a big catalyst in getting me into Clojure (a modern lisp). Additionally React allowed me to write UI as regular old JavaScript (with some light JSX syntax perhaps) with proper error messages and type checking. At the same time I worked a bit with Hugo a static site generator (SSG) on a different project. Working with Hugo showed me how nice it could be to work with SSG, to be able to produce static and fast web pages.

So at some point I decided to build myself new statically generated home page. But was there any static site generators out there that satisfied my interests and requirements? Hugo, Jekyll and others were too opinionated and used template engines focused around html. Previous encounters with magic html templates in Angular had not done much to make me love html templates.

Fortunately I looked at Gatsby a React based framework for building static sites. Where many other SSG are focused on building simple static html pages via html templates, Gatsby offers full flexibility. Gatsby is based around a GraphQL data fetching core and pages based on React to fetch the data and render pages. You can easily build a highly interactive page with React featuring dynamic content.

It is still possible to use markdown files to write your blog via Gatsby’s plugin system and the remark plugin. This makes the writing experience equivalent to other SSG like Hugo and Jekyll for normal blog posts. In the future I will experiment with mixed markdown and react component posts using the mdx plugin. This will allow me to embed more interactive content into the post experience.

Previously I have hosted my Ghost blog in a Docker container running on Ubuntu on a virtual machine. The docker container was proxied by Traefik an ingenious proxy server with automatic Docker load balancing. While I was very satisfied with this setup the complexity and time spent setting it up is undeniable.

So I wanted to find a simpler way of hosting my new home pages. Having already used Next.js a few times I learnt about Vercel’s edge network offering to host sites. Due to a short lived error with Vercel’s GitHub signup integration I could not create an account there. I am also a Cloudflare user and had looked a bit at Cloudflare’s Workers offering. The major benefit of Vercel’s offering is of course their free pricing tier for personal use. I ended up subscribing to and setting up Cloudflare Workers before Vercel had fixed their signup bug, so for now I will stick with them. To Vercel’s benefit they responsed to my support request in a matter of two days confirming the issue and having already applied a fix. At 5USD per month minimum the Cloudflare solution is not that costly however and has an edge server right here in my own country!

Some clever developers will probably point out that I could have used GitHub Pages for free. And while I could certainly have done that I wanted a new learning experience and to use a product with a clear pricing model. Paying for Cloudflare’s Worker service assures me that I’m a customer and that hosting my page is indeed a priority for them.

At my old website I tried out two different comment systems, but never really got many comments. To be compliant with GDPR (just in case) I had to set up a privacy policy to explain users how the data submitted to my comment system would be handled. So for my new pages I probably will not use a comment system. Instead you can reach me at contact@snorre.io. Don’t hesitate to contact me if you have any questions about the setup. I will leave you with this boilerplate Gatsby image from their example site setup:

Caching apollo graphql query results with your own caching hoc

Wed, 01 Aug 2018 00:00:00 GMT

I am currently building a search portal at work and have been using a React, Apollo Client, and GraphQL stack.

The search app so far is pretty simple. It uses apollo link state to store local application state like the users’ search query and selected filters, and then performs graphql queries based on that state.

The relevant part of the React component tree looks something like this:

Removed image

The search state hoc graphql component queries the local app state to fetch the user’s current search. Any time this state changes, the hoc will re-render the search results hoc with new properties (the search to perform).

The search results hoc performs a graphql query on the remote graphql server to perform the user specified search against a search index. It passes on three different properties to the child: error, loading, and results which can be used to determine if the query is finished running or not, which can be handled in the child component.

So when the hoc performs the remote query it will pass down the following properties:

{ loading: true, error: false, results: null }

When the query is complete the hoc re-renders the search presentation component, but now with the following properties, where results is a nested data structure with search results:

{ loading: false, error: false, results: { ... } }

The problem arises when the user refines her search and the state change triggers a re-render of the search state hoc which in turn triggers a re-render of the search results hoc. This triggers a new remote query to be performed, and the search results hoc will re-render the search presentation component with a loading state:

{ loading: true, error: false, results: null }

Because no results are available while loading, any downstream presentation component will now have to be rendered without the previous content. In the search portal I’m developing filters are drawn using the aggregation data contained within the results value! Whenever the user performs an adjustment to her search the UI would then have to draw the filters without any content before the new result would become available.

To fix this I first turned to the Apollo docs to see if there was a solution to keep former results while the component entered a new loading state. Apparently this was not so easy. The only solution I could see would be to read the Apollo client cache directly when loading equals true. This would require the code to know the old query so that I could fetch the correct cached result. Using the optimistic ui feature did not seem to fit nicely into the flow either.

After some thought it occured to me that it would probably be possible to just create my own cache inbetween the search result hoc and search presentation component. So after a quick Google search I came up with the following component:

function cacheSearch (WrappedComponent) {
  return class extends React.Component {
    constructor () {
      super()
      this.state = {
        previousResults: null
      }
    }

    componentWillReceiveProps () {
      if (this.props.results) {
        this.setState(function () {
          return { previousResults: this.props.results }
        })
      }
    }

    render () {
      // Wraps the input component in a container, without mutating it. Good!
      return <WrappedComponent {...this.props} previousResults={this.state.previousResults} />
    }
  }
}

As you can see the cache hoc is implemented as a function that takes the component to wrap as an argument. The cache component uses the React state property on the component instance to hold the previous result (the cache), and the componentWillReceiveProps life cycle hook to update the cache. The render method simply returns the wrapped component with the properties of the cache component as well as the previousResults cache.

Because all React hocs look the same it was easy to integrate into the compose call used to hook up graphql hocs to my presentation component:

export default compose(
  graphql(SEARCH_QUERY, {
    name: 'searchQuery',
    props: searchQueryProps
  }),
  graphql(SEARCH_RESULTS, {
    name: 'searchResults',
    options: searchOptions,
    props: searchProperties
  }),
  cacheSearch // My caching hoc
)(Search)

This solution with a caching higher order component is nothing ground-breaking, but was not obvious to me at first. It might not be the best approach, but it works well enough and is very easy to grok. So tell me what you think of the approach or suggest a better one if you know one!

Running Wireguard on Scaleway Stardust

Tue, 03 Nov 2020 00:00:00 GMT

Photo by Nasa

Scaleway just (3rd of November 2020) released their new tiny cloud instance offering aptly named Stardust. As exploding stars launch tiny atoms into space so does Cloudflare launch tiny instances into the cloud space. The Stardust instance is a very modest virtual private server clocking in at 1 vCPU, 1GB RAM, 1 IPv4 address, 10GB of local NVMe storage and up to 100Mbps bandwidth. And best of all the stardust instance cost clocks in at €0.0025 per hour adding up to under €2 per month, barely the cost of a coffee! If you can make do without the IPv4 address you can get it even cheaper, at only €0.0005 which should not even register in your wallet. Safe to say Stardust is a pretty sweet deal, but is limited to 1 per region per customer, so don’t go thinking that you can set up some sort of swarm on them.

So what can you do with something small like the Stardust instance? Well according to the blog post users can

[…] deploy Stardust instances for use cases such as creating their own workers free from vendor lock-in, configuring triggers and daemons, as well as small Load-Balanced LAMP websites, VPN or Network Bastions. VPN networks they say?

Wireguard

I had always wanted to set up a personal Wireguard VPN server. A VPN is typically used to secure your network traffic when on untrusted networks like at cafés and hotels. Wireguard is a modern VPN implementation implemented largely in kernel space to provide speed and security. While I had originally considered subscribing to a commercially hosted Wireguard host, the Scaleway Stardust launch gave me an excellent excuse to try to host it myself.

The setup proved to be not so complicated at all. I will not repeat the setup instructions here, but you should be aware that Wireguard now ships in Ubuntu’s own repositories so no PPA is required. The most complex configuration in the setup was the post up and post down settings to make sure the correct iptables commands were run. While they are documented and working it is always a good idea to understand what is going on.

So how did the Wireguard server on Scaleway Stardust perform? A quick unscientific test on my desktop on a 750Mbps download and 25Mbps upload speed gave me the following results:

$ speedtest
Retrieving speedtest.net configuration...
Testing from Scaleway (51.x.x.X)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Naitways (Paris) [1.88 km]: 80.411 ms
Testing download speed...........................
Download: 122.54 Mbit/s
Testing upload speed.............................
Upload: 21.11 Mbit/s

$ speedtest
Retrieving speedtest.net configuration...
Testing from Scaleway (51.x.x.X)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by KEYYO (Paris) [1.88 km]: 80.166 ms
Testing download speed...........................
Download: 98.86 Mbit/s
Testing upload speed.............................
Upload: 19.38 Mbit/s

$ speedtest
Retrieving speedtest.net configuration...
Testing from Scaleway (51.x.x.X)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Naitways (Paris) [1.88 km]: 80.891 ms
Testing download speed...........................
Download: 155.69 Mbit/s
Testing upload speed.............................
Upload: 19.60 Mbit/s

On average I would say that Wireguard on the Stardust instance is more than capable of delivering the 100Mbps the instance offers. If you have a few times more speedy Internet, or even gigabit speeds, this setup will obviously not be your daily driver. On the other hand the next time you are sitting at some café or hotel, maybe after the covid-19 pandemic is over, your VPN is likely not going to be the bottleneck. The usual hotel and café networks will be limited by the wi-fi network and internet connection speed.

Before ending the post I would like to point out that this setup affords no extra anonymity on the wider Internet. While you will be masking your own private IP address, you will still be broadcasting your Scaleway instance’s IP address instead. What this setup does afford you is a secure connection to a trusted network if no trusted networks are available where you are. It also hides your traffic from your ISP if that is a concern to you.

This was my first blog post in quite a while and I hope you enjoyed it. If you have any questions or comments feel free to shoot me an email at contact@snorre.io.

Spotifyd and Spotify TUI for a leaner listening experience

Fri, 20 Nov 2020 00:00:00 GMT

This post is for those of you subscribing to Spotify, the mega-popular streaming service. Spotify as many companies before them have released multi-platform apps based on web technologies. The desktop app is quite user friendly and pretty, but as it turns out it uses quite a lot of resources. In this post we will see how the good old terminal user interface and a Rust-based daemon can help!

During a typical work day I will stream music for hours at a time, only being interrupted by video meetings and lunch. For all those hours my Spotify app is barely open, mostly to skip songs I don’t like. For this measly work Spotify fires up five processes consuming more than 300MB of memory. All of this just to mostly stay in the background and playing music. Below you see a screenshot of the processes and their memory consumption as reported by MacOS monitor:

I’m not trying to lampoon the Spotify client. All things considered it is very nicely designed and very user friendly. But if you don’t have a lot of memory to go on and/or you don’t really care about the album covers and nice design there is another option. Enter Spotifyd:

Spotifyd streams music just like the official client, but is more lightweight and supports more platforms. Spotifyd also supports the Spotify Connect protocol, which makes it show up as a device that can be controlled from the official clients.

Spotifyd when set up will allow you to stream music and can be controlled from the official clients (say the Spotify app on your phone) or even other clients with support for the spotify connect protocol. It doesn’t offer a UI of its own and only handles the streaming, playback, and any incoming Spotify connect calls. It does however do this beautifully and with hardly any impact on your resources. In my experience it consumes less than 10 MB of memory and very little CPU. Because it is designed to run in headless mode and consumes very little resources it could very well fit onto an original Raspberry Pi.

Now that you have the playback part of Spotify replaced you still might need a user interface. Sure you can use your cellphone, but having to thumb or face-unlock your phone can get tiresome. A good alternative is spotify-tui.

A Spotify client for the terminal written in Rust.

Before grahical user interfaces (GUIs) there were terminal/text user interfaces (TUIs). While GUIs have mostly displaced terminal user interfaces for consumer apps, TUIs still persist especially as code editors and configuration tools. One big advantage of TUIs today are the fact that they run in your terminal and only output text. This allows them to be quite lean as they only need to work with text output. Gone are the embedded browser engines with giant DOM hierarchies, layout engines, etc.

My favorite thing about spotify-tui is the clean design. There is no disturbing elements and you can use the q, arrow, and enter keys to do most of the navigation in the app. There is support for themes, and you can configure things like the increment of the volume control, etc. And as with the Spotifyd daemon the resource usage is marginal, consuming about 13 MB of memory.

Conclusion

Essentially you get the same listening experience with a little over 20 MB of memory consumed by two lightweight processes as you do with ~300 MB of memory spread over five more CPU intensive processes. That is about an order of magnitude in difference. If you’re running Spotify on a more memory constrained machine and find yourself using all the memory you might consider using the Spotifyd and Spotify TUI combo.

Spotifyd also gives you the option of building your very own open-source Spotify listening device should you have the appropriate hardware laying about.

Lets hope Spotify does not do something to limit playback to first party clients and partners, because this setup is great!

Ngrok like tunnels with Tailscale, a VPS, and Traefik

Wed, 31 Mar 2021 00:00:00 GMT

Sometimes while developing you may want to test a callback or share your work over an https connection. You might have turned to ngrok a tool that allows you to create tunnels to your localhost. Ngrok is a good tool, but to get consistent subdomains you need to pay for a subscription and you only get ngrok domains. In this post we will see how I setup my own domain and tunnel to a dev server on my laptop using Tailscale and a reverse proxy.

Tailscale - A simpler Wireguard setup

Tailscale advertises itself as a “Zero config VPN”. VPN is short for virtual private network and Tailscale implements such a network by building ontop of the Wireguard protocol. In practice you install the Tailscale software on a device, start the daemon, and complete a login process via a URL. Behind the scenes Tailscale will set up keys, distribute them to any other devices via their admin server to join the device into your Tailscale network. This ensures that your new device is accessible from any existing devices in your Tailscale network. There is of course nothing preventing you from setting up Wireguard on your laptop and server and then handling key creation and IPs yourself.

What makes Tailscale useful is that it sets up an overlay network of sorts by meshing all devices and handling physical IP address changes. For N devices you dont have to manually set up pairwise connections between all of them, Tailscale handles this for you. If you run regular Wireguard and wanted a connection from your VPS to your laptop you’d have to update your config everytime your ISP modem changes your IP.

Another nice feature of Tailscale is their magic DNS feature that automatically assigns domains to each of your devices based on their hostnames. These DNS entries are then available on your Tailscale network interfaces and can be used when you want to access a device in place of their Tailscale IPs. Say your laptop hostname is Bobs-MBP, then Tailscale’s magic DNS will make the laptop available as bobs-mbp across your devices.

Each device in your Tailscale service can expose “services” which means any network exposed port running on that device. Essentially if you have an http server listening to port 3000 on your dev machine, the service will be exposed under bobs-mbp:3000 on your Tailscale network. If you join both your laptop and a virtual private server in the cloud to your Tailscale network, the VPS would then have access to your laptop’s services.

Exposing your services to the Internet

With the Tailscale explanation out of the way we can move onto the part where I go into the ngrok like setup. The chart below represents the simplified flow of an HTTPS request to your public domain all the way down to your laptop.

graph TB
    browser[Alice's Browser]
    dns[Cloudflare DNS]
    vps[Cloud VPS]
    traefik[Traefik LB]
    tailscaleVPS[Tailscale VPS]
    magicDNS[Tailscale Magic DNS]
    tailscaleMBP[Tailscale MBP]
    service[Web Service]

    browser -- What is the IP of app.example.com? --> dns
    dns -- It is X.X.X.X --> browser
    browser -- GET https://app.example.com --> vps
    vps -- GET https://app.example.com --> traefik
    traefik -- What is IP of bobs-mbp:3000 --> magicDNS
    magicDNS -- It is 100.X.X.X --> traefik
    traefik -- GET http://100.X.X.X:3000 --> tailscaleVPS
    tailscaleVPS -- tunnel GET http://100.X.X.X:3000 --> tailscaleMBP
    tailscaleMBP -- GET http://local-ip:3000 --> service
    service -- 200 OK --> tailscaleMBP
    tailscaleMBP -- tunnel 200 OK --> tailscaleVPS
    tailscaleVPS -- 200 OK --> traefik
    traefik -- 200 OK --> vps
    vps -- 200 OK --> browser

So essentially what happens is that Alice’s browser performs an HTTPS request against some domain you have setup, e.g. app.example.com. This domain resolves to the public IP address of a server you control, commonly some kind of VPS. As the request reaches your VPS it is routed to a load balancer you are running on the VPS, for example Traefik. The load balancer is then configured to proxy the request to the domain that was generated by Tailscale’s magic DNS. So in the example Traefik resolves bobs-mbp to the Tailscale network IP address of the other device and performs a request with that IP. The Tailscale daemon then handles tunneling that request over its udp tunnel to Bob’s MacBook Pro to the web service you’re running.

When running Traefik on your VPS you might create a rule such as this:

# http routing section
[http]
  [http.routers]
     # Define route to match requests for your domain to a service
     [http.routers.yourapp-dev]
      rule = "Host(`yourapp-dev.example.com`)"
      service = "yourapp-dev"
      # If you want https you need to first configure a certificate resolver which can be used as below
      [http.routers.yourapp-dev.tls]
        certResolver = "letsencrypt"

  # Define a service to handle forwarding the request to your Tailscale domain
  [http.services]
    # Define how to reach an existing service on our infrastructure
    [http.services.yourapp-dev.loadBalancer]
      [[http.services.yourapp-dev.loadBalancer.servers]]
        url = "http://bobs-mbp:3000"

In conclusion

While the setup might look complex from the diagram the process of setting everything up took about an hour of work. The Tailscale service is simple to install and configure, and you’ll get going in almost no time. Once both my VPS and laptop was enrolled in the Tailscale network and magic DNS had been enabled setting up the proxy was the only thing left to do.

Tailscale, being built on top of the Wireguard protocol, performs really well and more than good enough for a dev environment. With this approach you can expose your services to the internet in a safe manner on a domain you own. Shutting down the public access to your service can be done in many ways including:

disconnecting your laptop from Tailscale, thus shuttign down the tunnel
removing the proxy in your proxy server config, the tunnel is still there
using Tailscale ACL config to deny your VPS access to the service, tunnel is there but connections stopped by ACL

In sum I think the setup works well enough that I won’t need ngrok. Additionally Tailscale does more than ngrok and can be used as a traditional VPN when you are on unsecure networks.

Shared mutable app state in Actix Web

Thu, 23 Aug 2018 00:00:00 GMT

Not too long ago, and in this very galaxy, I decided to learn the Rust programming language. I went through each chapter in the Rust Book and discovered a lot of things I like about Rust. When the chapters ran out I set out to build something with Rust, a little pet project that would put my newly acquired Rust knowledge to the test. I decided to implement an authorization server using the actix web framework.

A central component of the Actix web framework is the App struct that is used to create an Actix application. The App struct has a generic type that is used as a placeholder for whatever struct you create to hold the application state to be used when handling requests. So far so good. I implemented my struct to hold my cache:

pub struct AppState {
    pub cache: Mutex<TimedCache<String, CodeItem>>,
}

As you can see the Appstate struct’s cache property uses a Mutex type to hold the cache which would make it possible to do thread safe mutations on the cache via locking. Then I created the actix app:

pub fn new_app() -> App<AppState> {
    let state = AppState {
        cache: Mutex::new(TimedCache::with_lifespan(3600 * 3)),
    };

    App::with_state(state)
        .middleware(Logger:default())
        // More app-building here
}

My main.rs module then instantiated a server with an app like so:

fn main() {
    let mut server = server::new(|| app::new_app());
}

In each request handler were I needed access to the cache I would simply look up the cache property in the app state and create a lock:

let mut cache = match state.cache.lock() {
    Ok(cache) => cache,
    Err(error) => {
        return Ok(HttpResponse::from_error(error::ErrorInternalServerError(
            format!("Failed to get lock on cache: {}", error),
        )))
    }
};

And then add or get item:

let item = match cache.cache_get(&params.code) {
    Some(item) => item,
    None => {
        return Ok(HttpResponse::from_error(error::ErrorForbidden(
            "Code is no longer valid",
        )))
    }
};

In my /auth handler I would verify some authentication request, generate a code, cache the code, and finally return the response. In my /token handler I would try to read the value associated with the code from the cache, as the code above shows. My less than profound knowledge of Rust and the Actix framework left me surprised when it turned out that the cache was properly set in the /auth handler, but then suddenly empty in the /token endpoint. As it turns out the Actix documentation offered an answer and a clue to fix this.

Note: http server accepts an application factory rather than an application instance. Http server constructs an application instance for each thread, thus application state must be constructed multiple times. If you want to share state between different threads, a shared object should be used, e.g. Arc.

Aha! The server::new(|| app::new_app()) call will actually use the anonymous function (the application factory) to create an application for each thread it spawns. How does Arc solve this? So I read the Arc documentation and could surmise that Arc is a thread safe reference count pointer and that invoking clone on such a pointer creates a new reference to the same value in the heap. Well, that sure sounds useful to share the cache between the threads using my application. Let’s update the app struct:

pub struct AppState {
    pub cache: Arc<Mutex<TimedCache<String, CodeItem>>>,
}

Now the cache property would hold an Arc style pointer to a cache value, and the value would only be deallocated once all the references to it was gone. Finally it was necessary to move the initialization of the cache to someplace outside the application factory. So for the time being I placed it right before the code constructing the server:

// In the app module
pub fn new_app(cache: Arc<Mutex<TimedCache<String, CodeItem>>>) -> App<AppState> {
// App initialization code
}

// In the main module
fn main() {
    let cache = Arc::new(
        Mutex::new(
            TimedCache::with_lifespan(3600 * 3)
        )
    );
    let mut server = server::new(move || app::new_app(cache.clone()));
}

Seasoned Rust developers and people familiar with the Actix framework probably think most of this is quite obvious, but to me this took some thinking to figure out. I hope this might help others starting out. Rust is pretty cool and I would recommend it to anyone wanting to learn a more low level language.

Coral Talk - An open source comment system from Mozilla

Sat, 27 Jan 2018 00:00:00 GMT

Coral Talk is a comment and moderation system from Mozilla. It is part of the Coral project which proclaims that it wants to improve journalism. I’m no journalist, but I found their comment system interesting for four particular reasons:

It is open source and backed by mozilla
It is self hosted which means

I own my own data
Users won’t have their comments become ad targeting food

It has a simple admin and moderation dashboard
It is based on a plugin architecture which allows anyone including me to easily extend its functionality

I had previously set up isso on my blog, but the lack of a moderation system not relying on smtp meant I would have to trust Internet users not to post obscene content or spam in my comment section. That said, isso is an excellent little comment server. If you have an smtp server set up the built in moderation feature might work for you.

Talk checks some of the same checkboxes isso does. It is open source, self hosted, etc. It does however come at a greater resource cost using both mongodb and redis to store and cache content where isso just uses sqlite. It does not allow anonymous comments either, which is something that might stop people from commenting all together. I did receive some comments in my old isso installation. It will be interesting to see if anyone bothers creating a user to comment on any of my posts.

Running Coral Talk with Docker

As you might have surmised I’m a fan of Docker. Docker allows you to quite easily run software without worrying about dependencies and process managers. The Talk project provides an Installation from Docker guide which will get you started. Essentially you can host it using a docker-compose setup where talk, mongodb, and redis is configured. My current setup looks like this:

version: "3"

services:
  talk:
    build: ./talk
    restart: unless-stopped
    ports:
      - 80:3000
    depends_on:
      - mongo
      - redis
    networks:
      - talk
    environment:
      - TALK_MONGO_URL=mongodb://mongo/talk
      - TALK_REDIS_URL=redis://redis
      - TALK_ROOT_URL=https://talk.<yourdomain>.tld
      - TALK_PORT=3000
      - TALK_JWT_SECRET=<your-super-secret-jwt-secret>
    logging:
      driver: "json-file"
      options:
        max-size: "512k"
        max-file: "10"

  mongo:
    image: mongo:latest
    restart: unless-stopped
    volumes:
      - mongo:/data/db
    networks:
      - talk
    logging:
      driver: "json-file"
      options:
        max-size: "512k"
        max-file: "10"

  redis:
    image: redis:latest
    restart: unless-stopped
    volumes:
      - redis:/data
    networks:
      - talk
    logging:
      driver: "json-file"
      options:
        max-size: "512k"
        max-file: "10"

volumes:
  mongo:
  redis:

networks:
  talk:

Note the build: ./talk section under the talk service. Because I’m not a big fan of Facebook login and setting up a Facebook app and api key just to use a comment system I decided to modify the default Talk docker image and not load the Facebook authentication plugin. My Dockerfile to accomplish this is included below. The command simply removes the config line containing the facebook auth plugin.

FROM coralproject/talk:4-onbuild

RUN sed -i '/talk-plugin-facebook-auth/d' ./plugins.default.json

This allows me to build an image where the Facebook authentication plugin is not activated and thus I do not have to setup a Facebook application just to use the Talk comment system. Setting up Talk using Docker proved to be quite easy.

In conclusion I think Talk will serve me quite nicely. There are still features I have yet to try like the toxic-comments plugin, authoring my own plugins, and of course creating a custom style. But everything in due time.

Building Docker images with Gitlab CI

Wed, 14 Feb 2018 00:00:00 GMT

Different approaches to building Docker images with Gitlab CI, drawbacks, and solutions.

My most popular blog post so far is Setting up a Docker image builder with Gitlab CI Runner. The focus of that post was how you could set up a relatively secure Docker build environment using Gitlab CI and a host with a Docker engine. It also touches on the subject of how to design your .gitlab-ci.yml pipeline to build a Docker image. I wrote that piece almost two years ago. Since then I have learned a few neat things about both Docker and Gitlab CI. There has also been some welcome updates in both Gitlab CI and Docker that has improved my docker building pipelines. In this post I will try to show you what my early implementations looked like, which problems they caused, and how later features allowed me to create better implementations. In the end this post should give you an idea about how you could go about setting up Gitlab pipelines to build your own Docker images using Gitlab CI and Docker.

Early days

In the early days of Docker and Gitlab CI Runner. Before Docker multistage builds and Gitlab job artifacts building applications and putting them in clean production Docker images was not straight forward. At least not with the docker executor. My solution involved using the cache feature to transfer build files from a build stage into a containerize stage and referencing them in the Dockerfile used for the docker build command. For a Node app written in Typescript using this solution might give you a .gitlab-ci.yml file which looks like the one below.

stages: build
  containerize

cache:
  key: $CI_BUILD_REF
  paths:
    - dist
    - server.js
    - package.json

build:
  stage: build
  image: node:8
  script:
    - npm install
    - npm run build

containerize:
  stage: containerize
  image: docker:17
  script:
    -  # A few lines to set DOCKER_HOST and certificates and log in.
    - docker build -t user/my-typescript-image:latest
    - docker push user/my-typescript-image:latest

The Dockerfile would reference those files:

FROM node:8-alpine
WORKDIR /usr/src/app
COPY ./package.json /usr/src/app/
RUN npm install --production
COPY ./dist /usr/src/app/
COPY ./server.js /usr/src/app/
CMD ["node" "server.js"]

My approach worked relatively well, but has some problems.

The cached build files is not guaranteed to be there for the containerize stage.
The build files cache for various projects will build up until the cache is cleared.
Clearing all the cache will cause other builds using cache for actual caching purposes to slow down.
Depending on the cache key used, cache collisions could occur thus causing potential race conditions where a containerize job ends up using build files from an unrelated build job.

All in all this was not a very elegant solution, and using cache for build output is not recommended today!

Enter Gitlab Job Artifacts

Gitlab job artifacts allows you to define the artifacts produced by a Gitlab CI job and have those files and folders archived and persisted in Gitlab. The artifacts are made available to subsequent jobs in the pipeline which makes them ideal for pipelines where the output of one job should be used in later stages. Gitlab eventually added the expire_in option to the artifact feature which allows us to automatically clean up old build artifacts. Armed with this new feature I devised a new .gitlab-ci.yml pipeline:

stages: build
  containerize

build:
  stage: build
  image: node:8
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist
      - server.js
    expire_in: 1 hour

containerize:
  stage: containerize
  image: docker:17
  script:
    -  # A few lines to set DOCKER_HOST and certificates and log in.
    - docker build -t user/my-typescript-image:latest
    - docker push user/my-typescript-image:latest

Here we use the artifacts configuration directive in the build job to instruct the CI runner to create an archive of our build files and to persist it in Gitlab for 1 hour. When the runner executes the containerize job it will download and extract the artifacts archive and make the files available. After 1 hour Gitlab will delete the files. Because job artifacts are stored for their specific pipelines collisions will not occur. As you might have surmised artifacts fix a lot of the issues present with the cache based approach.

Multistage docker builds

The artifact approach above works quite well for many build scenarios. I still use it for some projects, but it does have some issues. Because the runner has to archive and upload the build files to Gitlab at the end of the build job, and then download and extract the files at the start of the containerize job, large builds could take some time to run. Because artifacts are unique per pipeline they will not help you with dependencies and build cache. For this you would have to use the Gitlab CI Runner’s cache feature.

When Docker 17.05 was released it brought about the Docker multistage build feature, promising a new and simpler way to build clean production images. This feature has simplified and improved how I build Docker images with Gitlab CI as the build stage and job can be removed in favor of a single containerize step using a multistage Dockerfile. In essence you would change your old Dockerfile to use a build and production stage.

FROM node:8 as builder
WORKDIR /usr/src/app
COPY package.json
RUN npm install
COPY ./src /usr/src/app/
RUN npm run build

FROM node:8-alpine
WORKDIR /usr/src/app
COPY package.json
RUN npm install --production
COPY --from=builder /usr/src/app/dist /usr/src/app/
COPY --from=builder /usr/src/app/server.js /usr/src/app/

CMD ["node" "server.js"]

The builder stage in the Dockerfile, not to be confused with a Gitlab CI stage, will build the app using your regular Dockerfile commands. When the builder stage is complete Docker will have made image layers for the builder stage. The final stage which produces your actual Docker image will start from the desired base image, here node:8-alpine. We can then instruct the Dockerfile to copy files from the previous stage instead of our local file system. The result is a clean production image without build dependencies.

You could then change your .gitlab-ci.yml file to only use one CI stage:

stages: containerize

containerize:
  stage: containerize
  image: docker:<tag>
  script:
    -  # A few lines to set DOCKER_HOST and certificates and log in.
    - docker build -t user/my-typescript-image:latest
    - docker push user/my-typescript-image:latest

There are some major benefits to this approach. First of all you eliminate the overhead associated with the artifact handling in Gitlab. Second of all the image layers for your Dockerfile’s builder stage will be cached by Docker. If you have structured your builder stage correctly Docker might already have the layers for the correct dependencies and build cache, and Docker will happily start on the RUN command associated with your build (e.g. RUN npm run build) thus saving a lot of time. Finally the multistage Dockerfile provides a much more consistent and portable build solution. In the artifact based approach you might end up with one build setup for local use and one for Gitlab CI. A multistage Dockerfile will happily work in both environments without any differences. This makes builds and build errors more reproducible as you can simply run docker build -t my-container . locally on your dev machine if your build fails in Gitlab.

Two Dockerfiles approach

Readers with lots of Docker experience might have seen a different solution that I so far did not mention. Docker has a copy feature which allows you to copy files out of a container. A solution would thus be to have two Dockerfiles, one to produce build output and one to run your application. In a single containerize CI job you would first build a Docker image of your project producing the build output inside the image. Then you would run a container from the image and copy out the necessary files. That container could then be stopped and you would build your production image. This solution was not apparent to me at the time, and would in any case be considered quite complex. The multistage feature works on the same principle, but is far simpler to implement.

Conclusion

We have seen how building Docker images with Gitlab CI has gone from being a somewhat involved process with workarounds involving the Gitlab CI cache to being a first class experience. Using multistage Dockerfiles allows you to run your whole build process with the docker build command, and reduce the complexity of your Gitlab CI pipelines. This post does not give you all the details, but should point you in the right direction.

Embedding Coral Talk in your Ghost Blog

Thu, 23 Aug 2018 00:00:00 GMT

Today I changed how I embed the Coral Talk comment system in my blog posts. Previously I had downloaded the Casper theme and edited the post.hbs file and inserted the Coral embed script. Then I re-uploaded the customized theme and enabled it in the admin panel. This works of course, but also mean you don’t get to enjoy any of the new Casper updates for free. Instead you need to regularly re-customize new versions of Casper. So today I decided to be a bit smarter about it and created this little embed script to add to the Ghost footer (via the admin dashboard):

<script
  src="https://talk.snorre.io/static/embed.js"
  async
  onload="
  var articles = document.getElementsByClassName('post-full post');
  if (articles.length === 1) {
    var coral = document.createElement('div');
    coral.setAttribute('id', 'coral_talk_stream');
    articles[0].appendChild(coral);
    
    Coral.Talk.render(document.getElementById('coral_talk_stream'), {
      talk: 'https://talk.snorre.io/'
    });
  }
  
"
></script>

Removed image

Code Injection Footer settings screen capture So essentially I now first get all elements in the dom with the class name “post-full post”. Only if such an element is found do I create a div to mount Coral in and render Coral. As long as the Casper theme developers don’t change the class names for the article element on the post page this script will continue to do its job.

This embed will only work for Ghost themes using the “post-full post” class name the same way Casper does you would need to change the script to fit your needs.

Note that this setup will cause Ghost to load the Coral Talk embed.js source file on the homepage of your Ghost blog, but simply not mount the app. This is acceptable as your user most likely will navigate to one of your posts and have to load the embed.js file anyway.

Generating opengraph images for your Astro page

Fri, 08 Sep 2023 00:00:00 GMT

Facebook introduced the Open Graph protocol in 2010. It allows you to specify metadata about your page, such as the title, description, and image. This metadata is then in turn used by social media platforms to display a preview of your page when it is shared. The open graph image is especially important as it is the first thing people see when they see a link to your page. The image should either be an eye-catching image or an image resembling your site brand in some way. If you are anything like me and running an Astro site, you probably don’t want to manually create an image for every page you create. Luckily, there is a way to automate this process.

Satori - A simple opengraph image generator

If you’ve browsed the Internet a bit you might have come over web site image previews with a nice background and some text. I wanted to have something similar for my own site, and remembered something about a Vercel project that did just that. After a bit of searching I found Satori, a simple opengraph image generator. In short Satori allows you to write HTML and CSS that it then renders to an SVG. It does this using a custom rendering engine that handles layouting, typograhpy, and other things.

There are limitations however. It only supports a limited flexbox layout, has some font limitations, and only ltr languages. But for my use case it was perfect.

Using Satori with Astro

I did what any old developer would do and started searching for a pre-existing solution. I found some options, but none of them were quite what I wanted.

One blog post suggested using astro-og-canvar and generating static routes for each image./ This would work, but astro-og-canvas was too limited as it had a predefined layout. I wanted to be able to use my own layout and have more control over the image.

astro-opengraph-image seems like a promising effort. It is implemented as an Astro middleware using Satori to generate the image. Astro middleware run as part of the build process and can modify the output. Unfortunately it is not yet ready as it requires changes to Astro before it will work as intended.

Implementing my own solution

I decided to try and implement my own solution as an Astro plugin instead. To build a plugin you simply need to create a JavaScript function that optionally takes some options and returns an object with a name and hooks property. The name property is the name of the plugin, and the hooks property is an object of the hooks the plugin wants Astro to call. For example the astro:config:done hook is called when Astro is done configuring itself.

My plugin would run after Astro has built the site and generate an opengraph image for each page. To do this I would need to use the astro:build:done hook. The plugin function is called with an object containing the routes (a.k.a pages) Astro has generated.

/**
 * Satori plugin for Astro, returns Astro plugin
 */
export function satoriPlugin(): AstroIntegration {
  let astroConfig: AstroConfig;
  return {
    name: "satori-plugin",
    hooks: {
      "astro:config:done": ({ config }) => {
        console.log("Store config for satori-plugin");
        astroConfig = config;
      },
      "astro:build:done": async ({ routes }) => {
        console.log("Build done, generate og:image for each route");

        const profilePic = (await readFile("./public/images/profile.jpeg")).toString("base64");
        const wavesSvg = (await readFile("./public/graphics/layered-waves-dark.svg")).toString("base64");
        
        const ogImages = routes
          .filter((route) => route.distURL?.pathname?.endsWith("index.html")?? false)
          .map((route) =>
            generateOgImage({ route, site: astroConfig.site ?? "", profilePic, wavesSvg }),
          );

        await Promise.all(ogImages);
      },
    },
  };
}

The plugin function first run when Astro’s config done hook is done. It reads the Astro config and stores it in a variable we can access later.

Then when the Astro build is done the astro:build:done hook is called and we can start generating the opengraph images. We begin by reading the profile picture and the waves SVG file (my site uses an svg to add wavy gradients). This is to avoid having Satori load the files from remote URLs and every time we generate an image.

To find the routes to generate images for I simply filter the routes to only include the ones that ended with index.html. Other routes are not pages, but assets such as images and JavaScript files. The generateOgImage function is where the magic happens.

/**
 * Generate an og:image for a given route
 */
async function generateOgImage({ site, route, profilePic, wavesSvg }: GenerateOgImageProps) {
  // First we get the html for the route and parse out the title and description
  const html = await readFile(route.distURL!, { encoding: "utf-8" });
  const root = parse(html.toString());
  const title =
    root.querySelector("meta[name='title']")?.attributes["content"] ?? "";
  const author = "Snorre Magnus Davøen";

  // Then we generate the URL for the og:image
  const template = OgImageTemplate({ site, title, author, profilePic, wavesSvg });


  try {
    const svg = await satori(template, {
      width: 1200,
      height: 600,
      fonts: [
        {
          // Same font as used on the site headers
          name: "shortstack",
          data: await readFile("./public/fonts/shortstack-regular.woff"),
          style: "normal",
        },
      ],
    });

    // calculate write path by replacing index.html with index-og.svg
    const writePath = route.distURL!.pathname.replace(
      "index.html",
      "index-og.png",
    );

    // Convert svg to png 
    const resvg = new Resvg(svg, {})
    const png = await resvg.render()
    const pngBuffer = await png.asPng()
    await writeFile(writePath, pngBuffer);
  } catch (e) {
    console.error("Could not generate og:image for route", route.route, e);
  }
}

Okay so there is a lot going on here, so let me explain each part in some more detail.

First we read the html for the route and parse it using parse5. This allows us to find the title and description for the page using the meta tags. We have to do this as the title and description are not available in the astro:build:done hook data.
Then we generate an SVG opengraph image using Satori. We pass in the title we found in the previous step, as well as the site name and author. The OgImageTemplate function is a simple function that returns a Satori template object structure. This HTML string is then passed to Satori to generate the SVG. We use the same font as the site headers to keep the style consistent.
After we have generated the SVG we convert it to a PNG. Generally opengraph consumers will not support SVGs, so PNGs are a better choice. We use resvg to convert the SVG to a PNG. It is a fast XML to PNG converter with a focus on correctness.
Finally we write the PNG to the file system in a location that matches the route. This is done by replacing index.html with index-og.png in the route path. For example the route /blog/astro-opengraph-image/index.html would be written to /blog/astro-opengraph-image/index-og.png.

Using the plugin

To use the plugin I simply add it to my astro.config.mjs file.

import { satoriPlugin } from "./src/plugins/satori-plugin"

export default {
  integrations: [
    satoriPlugin(),
  ],
}

The result

In the end I got exactly what I wanted. A simple way to generate opengraph images for my Astro site. If you want to see the result you can check out the open graph image for this page. The full source code for my site is available on GitHub. Feel free to use it as a starting point for your own site.

process-bonk a small utility to watch and kill a process

Tue, 16 Jan 2024 00:00:00 GMT

It was a lovely day, I was at work doing my regular frontend development. I was testing out the spotify_player a beautiful TUI client for spotify on MacOS. As anyone testing out a new app I was curious to see how much resources it was using so I fired up the activity monitor. spotify_player was using a modest amount of resources and I was happy. But to my dismay I discovered that a completely different process was using a lot of resources. In fact it was using 100% of my CPU and had been doing so for quite a few hours! It was a process that ran as a service and it was one that was mandated by my company.

So I killed the process, it restarted, and from then on seemed to behave normally. But I was curious, why did it start using so much resources? Apparently it was a known issue that this process could sometimes go haywire, leak memory, and start hogging the CPU. If there were shorter periods of high CPU usage it might be fine, but this process had been running for hours at 100% CPU usage. That seems excessive for most processes, and I was reasonably sure the process was in fact buggy and restarting it was the way to go.

Now the fun thing is that the new MacBook Pros are insanely quiet, and so the normal fan noise was not enough to alert me to the fact that my CPU was being used at 100%. So how would I know if it started acting up again? I could check the activity monitor every few minutes, but that would be borderline insane. So I did what any sane person would do, I wrote a small utility to watch the process and allow me to kill it if it started acting up again.

process-bonk

Enter process-bonk, a small utility to watch and kill a process. It’s a small utility written in Rust that allows you to watch a process and kill it if it starts using too much CPU over time. process-bonk is about a 100 lines of code, and a silly project, but it was fun to write and it is useful to me. Basically the application works by reading the process name to monitor from a config file. It then starts a monitoring loop that first identifies the process corresponding to the name. Then it checks the CPU usage and if over a certain threshold starts a timer. Now if the CPU usage is still over the threshold when the timer expires the app notifies me via a dialog and gives me the option to kill the process.

use std::{env, fs};
use std::time::{Duration, Instant};
use std::thread::sleep;
use sysinfo::{System, Signal};
use toml::Value;
use native_dialog::{MessageDialog, MessageType};

fn main() {
    let mut system = System::new_all();
    let mut monitored_process: Option<&sysinfo::Process> = None;
    let mut high_cpu_usage_start: Option<Instant> = None;

    // Get the config file path from the command-line arguments
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        println!("Please provide a config file path");
        std::process::exit(1);
    }

    let config_file_path = &args[1];

    println!("Loading config file from {}", config_file_path);

    // Read the config file
    let config_file_content = fs::read_to_string(config_file_path)
        .expect("Could not read config file");

    // Parse the TOML content
    let config: Value = config_file_content.parse()
        .expect("Could not parse config file");

    // Get the process name from the config
    let process_name = config.get("process_name")
        .and_then(Value::as_str)
        .expect("Could not find 'process_name' in config file");

    // Log that we are monitoring the process
    println!("process-bonk is now monitoring the {} process", process_name);
    system.refresh_all();

    loop {
        while monitored_process.is_none() {
            // Find the process
            let processes = system.processes();
            for (pid, process) in processes.iter() {
                if process.name() == process_name {
                    println!("Found {} process with PID {}", process_name, pid);
                    monitored_process = Some(process);
                    break;
                }
            }

            // If we didn't find the process, sleep for a while before trying again
            if monitored_process.is_none() {
                println!("{} process not found, sleeping for 60 seconds", process_name);
                sleep(Duration::from_secs(60));
            }
        }

        if let Some(process) = monitored_process {
            let cpu_usage = process.cpu_usage();
            if cpu_usage >= 98.0 {
                match high_cpu_usage_start {
                    Some(start_time) if start_time.elapsed() > Duration::from_secs(300) => {
                        // If the process has been using 100% CPU for more than 5 minutes, kill it
                        println!("{} has been using lots of CPU for more than 5 minutes, bonking it", process_name);
                        let yes = MessageDialog::new()
                            .set_type(MessageType::Info)
                            .set_title("Process Bonk")
                            .set_text(&format!("{} is misbehaving, kill it?", process_name))
                            .show_confirm()
                            .unwrap();
                        if yes {
                            println!("User chose to kill {}", process_name);
                            process.kill_with(Signal::Kill);
                        } else {
                            println!("User chose not to kill {}", process_name);
                        }
                        // User has been notified, reset the tracking
                        high_cpu_usage_start = None;
                    }
                    None => {
                        // Start tracking when the process started using 100% CPU
                        println!("{} started using lots of CPU at {}%, will bonk if not behaving", process_name, cpu_usage);
                        high_cpu_usage_start = Some(Instant::now());
                    }
                    _ => {
                        println!("{} is still using lots of CPU at {}% for {} minutes", process_name, cpu_usage, high_cpu_usage_start.unwrap().elapsed().as_secs() / 60);
                    }
                }
            } else {
                // If the process is not using 100% CPU anymore, reset tracking
                println!("{} is currently using {}% CPU", process_name, cpu_usage);
                high_cpu_usage_start = None;
            }
        }
        // Sleep for a while before the next iteration
        sleep(Duration::from_secs(60));
    }
}

But how to run this thing? I didn’t want to have to manually start it every time I rebooted my computer. Well fortunately I remembered having previous experience with Brew formulas implemented as services. Homebrew allows formula creators to create application services that gets installed as MacOS launchd services. I also had some experience with creating Homebrew formulas, so I decided to create a formula for process-bonk.

process-bonk Homebrew formula

The Homebrew formula is pretty simple, it compiles the Rust application using cargo build --release and then installs it as a launchd service. The formula is available in the process-bonk Homebrew tap and can be installed using the following command:

brew install snorremd/tap/process-bonk

Again there is not that many lines of code to achieve my goal, but it was my first service!

class ProcessBonk < Formula
  desc "BONK process, don't eat all my Mac's CPU"
  homepage "https://github.com/snorremd/process-bonk"
  url "https://github.com/snorremd/process-bonk/archive/refs/tags/v0.1.0.tar.gz"
  sha256 "6a6bedc6b2e0b17badd068cc99c720de7aaa5492a218ead7a750469345234f38"
  license "MIT"

  bottle do
    root_url "https://ghcr.io/v2/snorremd/tap"
    sha256 cellar: :any_skip_relocation, ventura:      "907b82f4dbae39a595fb75fe197a0792cce270329d9b97d0fb0c8280ebbe4f26"
    sha256 cellar: :any_skip_relocation, x86_64_linux: "fd68dccc7bdc3aa415c5513784acd27f8dea524afe58ffafc0a57f70b45091f4"
  end

  depends_on "rust" => :build

  def install
    system "cargo", "install", *std_cargo_args
  end

  service do
    run [opt_bin/"process-bonk", "#{etc}/process-bonk/process-bonk.toml"]
    log_path var/"log/process-bonk.log"
    error_log_path var/"log/process-bonk.log"
  end

  def post_install
    # Create a configuration file if it doesn't exist
    (etc/"process-bonk").mkpath
    unless (etc/"process-bonk/process-bonk.toml").exist?
      (etc/"process-bonk/process-bonk.toml").write <<~EOS
        process_name = ""
      EOS
    end
  end

  def caveats
    <<~EOS
      The config file is located at:
      #{pkgetc}/process-bonk.toml

      Please edit this file to suit your needs.
      E.g. to monitor a process with name "my-process"
      specify the following in the config file:
      process_name = "my-process"
    EOS
  end

  test do
    system "true"
  end
end

As you can see it uses the rust toolchain dependency in Homebrew as a dependency. It compiles the application using cargo and installs it as a service. A default config file is also created if it doesn’t exist, and the Formula prints a message to the user to edit the config file.

Conclusion

So far I’ve only tested the service on my own computer, and the process in question have ironically so far behaved itself. Maybe it is scared of being bonked? But I’m happy to have learned a bit more about Rust, Homebrew, and MacOS services. And I’m happy to have a small utility that can help me monitor and kill a process if it starts acting up.

If you want to try out process-bonk feel free to read the code and install it using the Homebrew formula. If you have any feedback or suggestions feel free to open an issue or a pull request on the process-bonk repository.

A comment system on top of ATProto and Bluesky

Sat, 19 Aug 2023 00:00:00 GMT

In this post you will learn about how I built a comment interface on top the ATProto protocol and Bluesky social network. The post is relatively technical, but I also try to tell a story of how I got to this point. Read on if you are interested in social networks, decentralized protocols, open standards, and Bluesky.

Some history first

This is not the first time I’ve tried to run a comment system on my blog. I was determined to create a nice blog, and any nice blog needs a comment system right? Disqus was out of the question for reasons of privacy and tracking.

Research suggested that Isso was a good option. Isso solved a lot of the problems I had with Disqus. It was self hosted, stored everything in a SQLite database, and was open source. Additionally it did not require users to sign up for an account in order to comment. Isso is probably still a good option for many people, but alas I eventually wanted something with more features.

Along came Coral Talk. It was open source, backed by Mozilla (at the time), and had a lot of features. The implementation supported plugins and was written in Node.js, meaning I could read the source code. I even built a more privacy friendly Gravatar plugin. Unlike Isso though it was rather heavy, complex to setup, and required a database. It slowly dawned on me that I was not a news site and probably not in need of all the features Coral Talk provided.

The Fediverse and the micro community trap

Meanwhile a lot of interesting things where happening in the social network space. People were exploring and building decentralized or federated social network protocols. One of the more successful attempts at the time were ActivityPub. Mastodon is perhaps the most recognizable project using ActivityPub. As the experience of Twitter became ever more toxic Mastodon seemed like a promising alternative. One of the things that these new social networks promised was data ownership and portability.

An idea started to form in my head, what if I use Mastodon as a comment system? Then people could comment on my posts on Mastodon and I could fetch the comments and display them on my blog. I joined one of the “official” Mastodon instances in the hope that a lot of people would be there. But with time it became clear that the fediverse was pretty small and fragmented. It would be difficult to get the reach I wanted for my blog as the right people would have to share my posts for it to reach multiple instances. The protocol is also quite complex and implementing a client capable of adding comments to the fediverse seemed a bit daunting.

Cryptobros and web3 enthusiasts

Later Elon Musk bought Twitter and again I wanted to find a new home for my social media presence. Someone posted about Nostr and after reading about the protocol I decided to give it a try. Nostr is a true decentralized social network protocol, and as such does not present the same issues as Mastodon in terms of fragmentation. Running a server seemed simpler, and content would be more discoverable across servers. But to my big dismay the network seemed full of cryptobros, web3 enthusiasts, and other people I did not desire to engage with. Back when I was in university, Web 3.0, the future of the web, was all about the semantic web and linked data. Not at all about crypto currencies, smart contracts and pyramid schemes. But I digress.

ATProto and Bluesky

I was about to give up on the idea of having a comment system when I came across the AT Protocol and Bluesky. ATProto saw its infancy inside Twitter, but was later spun out as an independent project and company. Like Mastodon the protocol is federated, but not focused on keeping instances separate. Instead ATProto is based on global discovery services and freedom to choose your algorithms. No matter what instance you are on you can see content from other instances, perfect!

So after receiving an invite to the Bluesky server I started looking at the protocol and client libraries. I won’t go into too much details about the protocol here, but I will try to summarise the important bits.

Every user has a repository of data records, each record being some unspecified type of data. Each ATProto data server also has a lexicon of record types that enumerate the types . For Bluesky data types could be things like post, like, follow, profile, etc. Additionnaly the lexicon describes methods, i.e. things you can do on the server. For Bluesky these include fetching a post thread or creating a new post.

For my comment system I essentially only needed three methods from the Bluesky server:

app.bsky.feed.getPostThread to fetch all the replies to a post I use as a comment thread
app.bsky.feed.post to add a new comment to a blog post or reply to an existing comment
app.bsky.feed.like to allow users to like a comment

Implementation

I had a few goals for the implementation of my Bluesky-based comment system:

Keep it reasonsbly light weight
Fit seamlessly into my blog design
Simple enough implementation
Don’t gather any data unless necessary

My choice fell on a client-side only solution using the Astro islands pattern. The islands pattern allows you to integrate client-side components at specific points in your static site. Astro also supports Solid, a relatively young client-side library for building user interfaces. Conceptually it is similar to React, but escews the virtual DOM and instead compiles to native DOM operations. It also has a very small run time footprint, but still manages to be fast and provide the features I need.

I would also need to talk to the Bluesky server somehow. While I could have implemented the lexicon methods from scratch we are fortunate enough that there is an ATProto client. The ATProto client libraries are written in TypeScript and are quite easy to use in a typical Type/JavaScript project. It provides a simple API for talking to the server and also handles authentication.

You can find the source code for the comment system here.

Authentication

Users authenticate using their Bluesky handle and app password. To keep the authentication live I cache the authentication token in local storage so that it can be reused on page reloads. The token is also used to fetch the user’s profile information, which is used to display the user’s avatar and name.

Fetching and displaying comments

The first thing I do when the comment component is mounted and a user is logged in is to fetch the comments for the current post. I do this by calling the app.bsky.feed.getPostThread. The reponse is a tree shaped structure of posts, with an optional parent and a list of one or more replies. The parent is the post that the current post is a reply to, and the replies are the posts that are replies to the current post.

I initially thought that the simplest way to display the comments would be to use a recursive component. This would make the rendering code a bit more complex, but it would allow me to render without having to do data transformations. With this approach I would display the thread using the old school nested comments view where each reply is indented. But how would this handle very long reply chains? And would users familiar with the modern flat comments view be confused by the nested view?

I decided to go with a flat view mimicing the Bluesky app, but I was stumped on how they flattened the tree! Fortunately Bluesky is open source, so I could just go and look at the source code. It turns out that two functions are essential in determining the view. The PostThreadItemModel.assignTreeModels method determines for each post if reply lines should be shown. The flattenThread function then flattens the tree into a list of posts, first walking up any parent posts and then walking down the replies. There is also a sorting step, but I decided to skip that as I don’t expect to have a lot of comments.

Should we show reply lines?

I simplified the assignTreeModels method, but the basic idea is the same. When you fetch a post thread you can specify a highlightedPost which is the post that the user is currently viewing. This puts the outermost post in the response in the middle of the tree! After checking the first parent we recursively check the parent’s parent until we reach the original post.

Next we start enumerating all the replies and for each reply check if we should add reply lines. Reply lines are added if the reply has replies of its own.

function addThreadUIData(
  threadViewPost: ThreadViewPostUI,
  walkChildren = true,
  walkParent = true,
): ThreadViewPostUI {
  let parent = threadViewPost.parent;
  if (walkParent && AppBskyFeedDefs.isFeedViewPost(threadViewPost.parent)) {
    // Recursively add UI data to parent
    const newParent = {
      ...threadViewPost.parent,
      showParentReplyLine: false,
      showChildReplyLine: false,
      isHighlightedPost: false,
    } satisfies ThreadViewPostUI;
    threadViewPost.parent = newParent;
    addThreadUIData(newParent, (walkChildren = false), (walkParent = true));
  }

  let replies: ThreadViewPostUI[] = [];
  if (walkChildren && threadViewPost.replies?.length) {
    replies = threadViewPost.replies
      .map((reply) => {
        if (AppBskyFeedDefs.isThreadViewPost(reply)) {
          // Recursively add UI data to children
          return addThreadUIData({
            ...reply,
            showParentReplyLine: threadViewPost?.isHighlightedPost
              ? false
              : true,
            showChildReplyLine:
              (reply?.replies?.length ?? 0) > 0 ? true : false,
            isHighlightedPost: false,
          } satisfies ThreadViewPostUI);
        }
        return undefined;
      })
      .filter((x): x is ThreadViewPostUI => x !== undefined);
  }

  return { ...threadViewPost, parent, replies };
}

Flattening the tree

Here the Bluesky app code used a fairly smart trick. JavaScript has a concept called generators. Generators allow you to write functions that can be paused and resumed. The Bluesky app ingeniously uses this to flatten the tree. The flattenThread function is a generator that yields posts as it walks up the tree and then down the replies. By using a recursive generator function we can produce a flat list without producing a lot of intermediate data structures.

My flatten function is more or less the same as theirs. It first recursively walks up the tree, yielding the result of calling flatten on the parent. Once we reach the top of the tree we yield the post being flattened. This ensures the root parent is yielded first, then the second parent, and so forth. Note that the tree does not contain the siblings of the parents as this data is not interesting in context of the highlighted post.

The flatten function then yields the highlighted post as it has now yielded all parents.

Finally we walk down the replies yielding the result of recursively calling flatten on each reply. This causes the recursive calls to be evaluated in a pre order traversal of the tree. It begins with the left most child, then the left most child of the left most child, and so forth.

export function* flatten(
  thread: ThreadViewPostUI,
): Generator<ThreadViewPostUI, void> {
  if (thread.parent) {
    if (isKnownType(thread.parent)) {
      if (AppBskyFeedDefs.isThreadViewPost(thread.parent)) {
        yield* flatten(thread.parent as ThreadViewPostUI);
      }
    }
  }

  yield thread;

  if (thread.replies && thread.replies.length > 0) {
    for (const reply of thread.replies) {
      if (isKnownType(reply)) {
        if (AppBskyFeedDefs.isThreadViewPost(reply)) {
          yield* flatten(reply as ThreadViewPostUI);
        }
      }
    }
  }
}

If you think about the order in which the posts are yielded you will recognise the Bluesky app view. It shows a chain of posts down to the highlihted post without any of the sibling replies. From the highligted post it shows the replies in a flat list, showing nested replies as a chain. This is exactly what we want!

Styling and layout

I wanted the comment system to fit seamlessly into my blog design. So I just used the same styling setup as the rest of my blog. It uses Tailwind CSS and PostCSS to style content. Tailwidn has seen its fair share of criticism, but I find it to be a very pleasent and flexible way to style content.

Essentially Tailwind is a set of CSS utility classes that represent common CSS properties. What makes Tailwind so special is that the utility classes are built on top of a design system. Unlike using regular CSS, where you have to decide on the values for each property, Tailwind provides a set of values that work well together.

Tailwind defines a list of colors, whitespace sizes, font sizes, border radii, etc. You can then use these values to get consistent styling across your site. For example, instead of deciding on a specific font size for a heading you can use the text-2xl class. If you always use the text-2xl class for h1 headings you can easily change the font size for all headings by changing the value of text-2xl.

You could of course define a design system using CSS Custom Properties (aka CSS variables). But then you’d have to write css classes for each type of layout and component in your markup. Tailwind frees you from having to artifically separate style from markup and grouping css attributes into classes. Instead you can just use the utility classes directly in your markup.

With Tailwind you’d write something like this:

<div class="bg-gray-100 rounded-lg p-4">
  <h1 class="text-2xl">Hello world</h1>
  <p class="text-lg">This is a paragraph</p>
</div>

Instead of this:

<div class="my-component">
  <h1 class="my-component__heading">Hello world</h1>
  <p class="my-component__paragraph">This is a paragraph</p>
</div>

.my-component {
  background-color: var(--color-gray-100);
  border-radius: var(--border-radius);
  padding: var(--spacing-4);
}

.my-component__heading {
  font-size: var(--font-size-2xl);
}

.my-component__paragraph {
  font-size: var(--font-size-lg);
}

Privacy and data collection

I wanted to avoid collecting any data about my users. The comment system is entirely handled client-side, so I don’t have to worry about storing any data. When users comment on my blog all communication goes directly to the Bluesky server. Users can easily check that no data is sent to my server by inspecting the network requests. I never touch any data and never see my readers’ passwords. All in all I consider this a big win. Of course my readers now need to trust a third party, but if you are on Bluesky you already trust them.

Conclusion

I am very happy with the result. The comment system is light weight, easy to use, and fits seamlessly into my blog design. This is a one time integration, so I don’t have to worry about maintaining a server or database. I also don’t have to worry about privacy as I don’t collect any data. The only downside is that users need to have a Bluesky account, but as it becomes more available I hope this will be less of an issue.

The comment system is currently in a minimal viable product state. It works and looks pretty good, but there are missing features and imperfections. Among other things I would like to support clicking a reply to highlight it, just like the Bluesky app. It would also be nice to extract it into a separate component that can be reused on other sites. But for now I am happy with the result and I hope you will enjoy using it.

Building beer recipe pages with Astro and Brewfather

Mon, 18 Dec 2023 00:00:00 GMT

Christmas is coming up and I took an early vacation to spend some time on my side projects. One of my precious hobbies is home brewing which I get to do every so often. Me and my friend like to experiment with different recipes that we make ourselves. For recipes I use a tool called Brewfather.

Brewfather is a great tool for creating and managing your recipes and brew sessions. As a home brewer I have often found help in reading and learning from other people’s recipes. So I wanted a way to share my recipes with the home brewing world. There is a function in Brewfather to share individual recipes via links, but I wanted to have a place where I could share all my recipes in one place. As I already have this site I thought it would be a good idea to add a section for my recipes here.

Brewfather API

Fortunately for me Brewfather has an official API that I can use to fetch my recipes. The API is provided as a REST-like API with endpoints for fetching recipes, batches, and inventory. So I set out to build a helper script that would download each recipe and save it as a JSON file. Because I’m in love with Clojure from my old job, I decided to use babashka to build it.

The script is pretty simple, it essentially boils down to a few steps:

Define some global variables
Fetch the API secret using the 1Password CLI
Fetch the list of recipes not already downloaded
Fetch each recipe and save it as a JSON file

The main function then looks like this:

(defn main [& args]
  (op-available?)
  (as-> (op-secret!) $
    (basic-auth client $)
    (list-remote-recipes! $ (last-recipe!))
    (map :_id $)
    (doseq [id $]
      (println "Fetcing recipe" id)
      (-> (fetch-remote-recipe! (basic-auth client (op-secret!)) id)
          (write-recipe!)))))

I keep the secret stored in 1Password and fetch it directly from 1Password using their CLI to avoid leaking it. Leaking the secret can happen if you pass it as an environment variable or as a command line argument. Brewfather uses basic authentication so I create a basic auth header using the hard-coded client id and the secret. Then I fetch the list of recipes that I haven’t already downloaded. Because I order the recipes by date I can use the last recipe I downloaded as a starting point. I then map the list of recipes to their ids and fetch each recipe and save it as a JSON file.

Astro

Now that I have all my recipes downloaded I need to build a page to display them. Astro has great support for data collections. This allows me to define a collection type for my beer recipes and strongly type them via zod schemas. So I created a config.ts file in src/content/config.ts and defined my collection type:

// Sub schemas elided for brevity

const beerRecipesCollection = defineCollection({
  type: "data",
  schema: z.object({
    name: z.string(),
    slug: z.string(),
    _timestamp_ms: z.number(),
    abv: z.number(),
    ibu: z.number(),
    color: z.number(),
    fg: z.number(),
    og: z.number(),
    batchSize: z.number(), // liters
    style: style.optional().nullable(),
    mash: mashAndFerment,
    fermentables: z.array(fermentable),
    yeasts: z.array(yeast),
    hops: z.array(hop),
    equipment
  }),
});

export const collections = {
  beers: beerRecipesCollection,
}

So now I have a collection of beer recipes that I can use in my pages. I then defined two pages, one for the list of recipes and one for the individual recipes.

List page

The list page, /src/pages/beers.astro, is pretty simple. It just lists all the recipes in the collection.

---
import { getCollection } from "astro:content";
import BaseLayout from "../layouts/BaseLayout.astro";
import BeerGlass from "../components/BeerGlass.astro";
import { SITE_TITLE, SITE_DESCRIPTION } from "../config";

// Use Astro.glob() to fetch all posts, and then sort them by date.
const recipes = (await getCollection("beers")).sort((a, b) => {
  return b.data._timestamp_ms - a.data._timestamp_ms;
});
---


{/* Simplified layout example */}
<BaseLayout title={SITE_TITLE} description={SITE_DESCRIPTION}>
  <h1>Beer recipes</h1>
  <ul>
    {recipes.map((recipe) => (
      <li>
        <a href={`/recipes/${recipe.data.slug}`}>{recipe.data.name}</a>
      </li>
    ))}
  </ul>
</BaseLayout>

Recipe page

The recipe page, /src/pages/beers/[slug].astro, is a bit more involved. Astro requires you to define a getStaticPaths function that returns a list of paths to render. For each path you return you specify any parameters and properties that should be passed to the page.

---
import { getCollection } from 'astro:content';
import { SITE_TITLE, SITE_DESCRIPTION } from '../../config';
import BaseLayout from '../../layouts/BaseLayout.astro';
import BeerGlass from '../../components/BeerGlass.astro';

export async function getStaticPaths() {
  const beers = await getCollection('beers')

  return beers.map(beer => {
    return {
      params: {
        slug: beer.data.slug
      },
      props: {
        beer: beer.data
      }
    }
  })
}

const { slug } = Astro.params;
const { beer } = Astro.props;
---

{/* Simplified layout */}
<BaseLayout title={SITE_TITLE} description={SITE_DESCRIPTION}>
  <h1>{beer.name}</h1>
  <BeerGlass beer={beer} />
</BaseLayout>

Some design choices

For the beer recipe design I wanted to keep it simple like the rest of the site. However, I wanted to add some color to the page to make it stand out a bit. Also it is common to show which color the beer will have based on the recipe. I ended up using ChatGPT to help generate a simple vector like image of a tulip style beer glass. This was then vectorized using Vectorizer.ai to convert the webp raster image to an SVG. The result from that was a promising SVG with only three different colors used for the beer liquid. I then edited away artifacts using Krita which has some simple SVG support.

From the SVG I then created an Astro component which accepts a beer color (SRM). It then calculates the primary beer color in RGB and two darker shades of the same color. I do this by simply incrementing the SRM value by 2 and 4 respectively and using the same converter as with the base color.

The end result turned out to be pretty good, but don’t take my word for it, see for yourself:

Conclusion

I’m really happy with the results so far. There is still some work to be done for the recipe page. It lacks some details around mash temperatures, total boil time, and sparge water volume among other things. I do believe that this is a good start though and I’m looking forward to adding more recipes to my site. You can find all the recipes at /beers.

Bluesky feed generators giving users algorithmic choice

Sun, 01 Oct 2023 00:00:00 GMT

Earlier in august I wrote about creating a Bluesky based comment system. This came about after I had joined the Bluesky community and was looking for a way to allow other Bluesky users to comment on my blog posts. The comment system should also be so simple that it could be used by other bloggers who wanted to allow Bluesky users to comment on their posts. Since then I’ve enjoyed using Bluesky and have been looking for other ways to contribute to the community. One of the things I wanted to do was to build a feed generator that would allow users to browse Norwegian language posts.

The Bluesky feed generator

So what is a feed generator? One of Bluesky’s main objectives is to provide users algorithmic choice over how they browse the social network. This stands in contrast to other social networks where the algorithm is chosen for you. For example on Twitter (now X) you can’t choose to browse your network in chronological order.

The feed generator is part of the federated nature of the Bluesky social network. Instead of all algorithms being hosted by Bluesky, they can be hosted by anyone. Simple put:

Feed Generators are services that provide custom algorithms to users through the AT Protocol. Source: feed-generator.

Feed generator specification and API

So how do you go about creating a feed generator? A feed generator is an HTTP server that implements a specific protocol using JSON as its data exchange format. The server exposes three endpoints.

`/.well-known/did.json`

This endpoint is used to discover the feed generator. It returns a JSON document that contains the feed generator’s DID (Decentralized Identifier).

{
  "@context": [
    "https://www.w3.org/ns/did/v1"
  ],
  "id": "did:web:norsky.snorre.io",
  "service": [
    {
      "id": "#bsky_fg",
      "serviceEndpoint": "https://norsky.snorre.io",
      "type": "BskyFeedGenerator"
    }
  ]
}

The id field is the feed generator’s DID. Most feeds use the web identifier which means that the DID points to a hostname. The service field contains the feed generator’s HTTP root endpoint. type specifies that this is a Bluesky feed generator.

`/xrpc/app.bsky.feed.describeFeedGenerator`

This endpoint is used to describe the feed generator and available feeds. Bluesky will use this endpoint to check which feeds are available after you have registered the feeds. If one of the feeds you’ve registered is not listed here it will not be available to users. Each feed is identified by a URI on the form:

at://did:web:{hostname}/app.bsky.feed.generator/{feed-id}

at:// tells Bluesky that this is an AT Protocol URI. did:web:{hostname} is the feed generator’s DID. app.bsky.feed.generator is the lexicon id for the feed generator. {feed-id} is the feed’s id.

A service would then typically return something like this:

{
  "did": "did:web:norsky.snorre.io",
  "feeds": [
    {
      "uri": "at://did:web:norsky.snorre.io/app.bsky.feed.generator/bokmaal"
    },
    {
      "uri": "at://did:web:norsky.snorre.io/app.bsky.feed.generator/nynorsk"
    },
    {
      "uri": "at://did:web:norsky.snorre.io/app.bsky.feed.generator/sami"
    },
    {
      "uri": "at://did:web:norsky.snorre.io/app.bsky.feed.generator/all"
    }
  ]
}

`/xrpc/app.bsky.feed.getFeedSkeleton`

This endpoint is used to get a feed skeleton. Feed generators do not return feeds containing the full post data. Instead it returns a list of post ids that Bluesky apps can then flesh out. This has many advantages including saving the feed generator bandwidth and making the PDSs (Personal Data Stores) the single source of truth for post data.

The endpoint takes two parameters feed and cursor, cursor being optional. It takes the form:

{serviceEndpoint}/xrpc/app.bsky.feed.getFeedSkeleton?feed={feed-uri}&cursor={cursor}

An example request URL would look like this:

https://norsky.snorre.io/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:web:norsky.snorre.io/app.bsky.feed.generator/bokmaal

This yields a JSON response containing a cursor (if there are more posts to load) and list of post ids:

{
  "feed": [
    {
      "post": "at://did:plc:2ouk4ptm336l2lcce6qxs3ar/app.bsky.feed.post/3kaoxdbnvwv2y"
    },
    {
      "post": "at://did:plc:y5zzasijmapifytqvhxnwsrm/app.bsky.feed.post/3kaoxbtnw7n2x"
    },
    {
      "post": "at://did:plc:jqdbfqiyyne7jkxpqlfacray/app.bsky.feed.post/3kaoxal4n3n2m"
    },
    {
      "post": "at://did:plc:hlxobkvuv64wghyltgbgz6f3/app.bsky.feed.post/3kaoxafag2y2s"
    },
    {
      "post": "at://did:plc:b66g2f6utk25ppjzsujhhqgt/app.bsky.feed.post/3kaox5gojov2q"
    }
  ],
  "cursor": "2482"
}

Note that the cursor is an opaque string that is not meant to be parsed by clients. It can contain any data and is only meant to be used by the feed generator to keep track of where it is in the feed. In the above case the cursor relies on the feed generator having a database of post ids that work in a sequential manner.

There are three primary ways to get posts for a feed generator:

Hardcode the post ids in the feed generator (not very interesting)
Make a request to get posts from one or more users’ repositories
Subscribe to the firehose, store the posts, and filter out posts that are not relevant to the feed

Depending on the type of feed you are building all of these are valid options.

Implementing the Norsky feed generator

With the specification in hand I set out to implement the feed generator. I had looked at some starter projects and found several.

All of these looked like good starting points, but I wanted something more lightweight. I started looking into Rust Bluesky clients and found Atrium. After working a bit with Atrium I couldn’t really get a Firehose feed to work. Not all of the library was published to crates.io either, and so the project did not seem quite ready for prime time.

With Rust off the table I decided to look at the next best thing: Go. The Bluesky team was already building an official, albeit experimental, Go client Indigo. Let’s go!

First steps, subscribing to the firehose

For my feed generator it seemed like the best option was to subscribe to the firehose. I couldn’t reliably list all norwegian users anyway, and many of their posts would likely be in English. To get started I googled for an example of subscribing to the firehose in go and found blueskyfirehose. Blueskyfirehose is a command line app that subscribes to the firehose and prints posts to stdout. Having read the source code I found that the indigo library had their own event scheduler setup for processing events from the Firehose.

Subscribing to the firehose then boils down to the following code:

// Setup a websocket dialer
d := websocket.DefaultDialer
con, _, err := d.Dial( "wss://bsky.social/xrpc/com.atproto.sync.subscribeRepos", http.Header{})
if err != nil {
    log.Fatal(err)
}

// Setup a callback event handler function to process events
rscb := &events.RepoStreamCallbacks{
			RepoCommit: func(evt *comatproto.SyncSubscribeRepos_Commit) error {
        // Process a repository commit here
        // This is a bit tricky as you have to do some marshalling and unmarshalling
        // From cbor to json and back to go structs
      }
}

// Setup event scheduler and start processing events
seqScheduler := sequential.NewScheduler(con.RemoteAddr().String(), rscb.EventHandler)
events.HandleRepoStream(ctx, con, seqScheduler)

See full processing example in my repository.

Storing posts

Now that I had a way to subscribe to the firehose I needed a way to store posts. Too keep things simple I wanted an in-process database that would store files on disk. Running things in-process would make it easier to deploy the feed generator and reduce the latency of serving requests. The database should also support some form of querying so that I could filter out posts that were not relevant to a given feed. Based on these requirements I decided to go with a classic database, SQLite. It is known for being fast and reliable and supports my use case well.

I quickly ended up in some trouble though. SQLite is a C program and so would require me to use cgo to interface with it. This is not a problem in itself, but it would make cross-compiling the feed generator more difficult. Fortunately I found a pure Go SQLite implementation called modernc.org/sqlite.

I then created a simple database schema that would allow me to store posts and their languages. While ORM libraries like gorm exist for Go I’ve become a bit skeptical of them. I prefer to model my data in SQL first and then make models in the application code that maps to SQL query results. This allows you to write more efficient queries and makes it easier to reason about the data.

CREATE TABLE posts (
  id INTEGER PRIMARY KEY AUTOINCREMENT, -- We need an id to implement deterministic cursor based pagination
  uri TEXT, -- The URI of the post
  created_at INTEGER NOT NULL -- The time the post was created as a Unix timestamp
);

CREATE TABLE post_languages (
  post_id INTEGER NOT NULL,
  language TEXT NOT NULL,
  PRIMARY KEY (post_id, language),
  FOREIGN KEY (post_id) REFERENCES posts(id) ON DELETE CASCADE
);

To ensure the database is always migrated I use go-migrate. This allows me to write migrations in SQL and have them run automatically when the application starts.

Reading posts from the database

The feed generator needs to be able to read posts from the database to serve the feed skeleton endpoints. Initially I started out by using only the official barebones sql module to write queries. Writing queries by hand and using scanning to map query results to structs is a bit tedious though. So the hunt began for a good query builder library. I found go-sqlbuilder. Using go-sqlbuilder I could write queries like this:

sb := sqlbuilder.NewSelectBuilder()
sb.Select("id", "uri", "created_at", "group_concat(language)").From("posts")
if postId != 0 {
  sb.Where(
    sb.LessThan("id", postId),
  )
}
if lang != "" {
  sb.Where(sb.Equal("language", lang))
}
sb.Join("post_languages", "posts.id = post_languages.post_id")
sb.GroupBy("posts.id")
sb.Limit(limit).OrderBy("id").Desc()

sql, args := sb.BuildWithFlavor(sqlbuilder.Flavor(sqlbuilder.SQLite))

rows, err := reader.db.Query(sql, args...)

This if of course much more verbose than using an ORM, but it is also much more explicit. I can see exactly what the query is doing and how it is being built. It is also possible to log the resulting SQL query and run it manually against the database to debug issues.

Writing posts to the database

It turns out go-sqlbuilder also supports insert statements. This allowed me to write a simple function to insert posts into the database.

insertPost := sqlbuilder.NewInsertBuilder()
sql, args := insertPost.InsertInto("posts").Cols("uri", "created_at").Values(post.Uri, post.CreatedAt).Build()

// Spread args
res, err := db.Exec(sql, args...)
if err != nil {
  log.Error("Error inserting post", err)
  return err
}

// Get inserted id
id, err := res.LastInsertId()
if err != nil {
  log.Error("Error getting inserted id", err)
  return err
}

// Post languages insert query
insertLangs := sqlbuilder.NewInsertBuilder()
insertLangs.InsertInto("post_languages").Cols("post_id", "language")
for _, lang := range post.Languages {
  insertLangs.Values(id, lang)
}
sql, args = insertLangs.Build()

_, err = db.Exec(sql, args...)

Observant readers will notice that I perform the insert of the post and the languages in two separate transactions. This is purely because it is easier to do it this way, and allows me to get a reference to the inserted post’s id.

Serving the feed skeleton

Now that I had a way to subscribe to the firehose and store posts in a database I needed to serve the feed skeleton. Fortunately this is not the first time I have written an HTTP API in Go. So I quickly landed on using fiber. Fiber is a fast HTTP framework for Go that is conceptually similar to Express.js and other Node.js frameworks. You define middleware and routes as callback functions on the server. The middleware is executed in the order they are registered and can be used to do things like logging, authentication, and error handling. Routes are defined by a path and a callback function that is executed when a request matches the path.

app := fiber.New()

// Middleware to track the latency of each request
app.Use(func(c *fiber.Ctx) error {
  start := time.Now()
  err := c.Next()
  stop := time.Now()

  // Diff
  log.WithFields(log.Fields{
    "method":  c.Method(),
    "route":   c.Route().Path,
    "latency": stop.Sub(start),
  }).Info("Request")
  return err
})


app.Get("/", func(c *fiber.Ctx) error {
  return c.SendString("This is the Norsky feed generator for listing Norwegian posts on Bluesky.")
})

// Well known
app.Get("/.well-known/did.json", func(c *fiber.Ctx) error {
  // Return the DID document, using regular map[string]interface{} for now

  return c.JSON(map[string]interface{}{
    "@context": []string{"https://www.w3.org/ns/did/v1"},
    "id":       "did:web:" + config.Hostname,
    "service": []map[string]interface{}{
      {
        "id":              "#bsky_fg",
        "type":            "BskyFeedGenerator",
        "serviceEndpoint": "https://" + config.Hostname,
      },
    },
  })
})

// Rest of the handlers ...

Putting it all together

Now that I had all the pieces I needed to put them together. I wanted the feed generator to be run as a single command in one main process. Having to launch multiple processes would make it more difficult to deploy and manage. I also wanted there to be some way to publish the feed, migrating the database, and other administrative tasks.

Command line interface

To make the feed generator easy to use I decided to create a command line interface. I first used cobra that allows you to create a CLI with subcommands and flags. However I also wanted to be able to configure the feed generator using environment variables. I tried using viper, but it turns out these two libraries don’t play well together.

At the same time while building the publish command I found that there was no client api in the indigo library to authenticate with Bluesky or publish my feed generator. Writing a client api for indigo would be a lot of work! Fortunately Kagi search came to the rescue revealing that the Furry community had already solved this issue. They had built a simple client api for Bluesky. Coincidentally they also used a different command line library urfave/cli. This library was both a bit simpler than Cobra and also supported environment variables as fallback for flags.

Communicating processes

With a way to run commands I set out to build my serve command. It would need to start the HTTP server, subscribe to the firehose, make a process to write posts to the database. Essentially the serve command would run three separate processes in parallel and somehow allow them to communicate. Go is a language that is known for its concurrency primitives. It supports communicating Sequential Processes (CSP) in the form of channels. Channels are typed and can be used to send and receive data between goroutines (go’s coroutines).

So in my serve command I created a channel for each process:

// Channel for subscribing to bluesky posts
postChan := make(chan interface{})

// Setup the server and firehose
app := server.Server(&server.ServerConfig{
  Hostname: hostname,
  Reader:   db.NewReader(database),
})
fh := firehose.New(postChan, ctx.Context)
dbwriter := db.NewWriter(database, postChan)

// Graceful shutdown via wait group
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
var wg sync.WaitGroup

go func() {
  <-c
  fmt.Println("Gracefully shutting down...")
  app.ShutdownWithTimeout(60 * time.Second)
  defer wg.Add(-3) // Decrement the waitgroup counter by 2 after shutdown of server and firehose
  fh.Shutdown()
}()

go func() {
  fmt.Println("Subscribing to firehose...")
  if err := fh.Subscribe(); err != nil {
    // Use signal to shutdown all the goroutines
    log.Error(err)
    c <- os.Interrupt
  }
}()

go func() {
  fmt.Println("Starting server...")
  if err := app.Listen(fmt.Sprintf("%s:%d", host, port)); err != nil {
    log.Error(err)
    c <- os.Interrupt
  }
}()

go func() {
  fmt.Println("Starting database writer...")
  dbwriter.Subscribe()
}()

// Wait for both the server and firehose to shutdown
wg.Add(3)
wg.Wait()

log.Info("Norsky feed generator stopped")

return nil

Esentially I first instantiate each service I need to run. A post channel is created to allow the firehose to send posts to the database writer. Finally a goroutine is started for each service which allows them to run in parallel. The wg.Wait() call blocks until the waitgroup counter is zero to tell the app to wait for all goroutines to finish. postChan is passed to both the firehose and the database writer. It is untyped so that it can be used to send any type of firehose event. The database writer then uses type assertions to check if the event is a post and then writes it to the database.

I find the go concurrency primitives to be very powerful and easy to use. They allow you to write code that is easy to reason about and that is also fast. Readers familiar with Clojure will recognize the similarities between go channels and Clojure’s core.async channels. The similarities are not a coincidence as both languages base their concurrency primitives on CSP.

Conclusion

I’ve had a lot of fun building the Norsky feed generator. It has been a great way to learn more about the Bluesky protocol and the AT Protocol. I also learned a lot about Go and how to build concurrent applications.

The go Bluesky library still feels a bit immature. If you want to build a feed generator you should consider using the more mature TypeScript library. But if you are looking for performance Go is a great choice to get started.

Check out Norsky in action on the Norwegian languages feed. The source code is available on GitHub.

Make some confetti with JavaScript and Canvas

Fri, 19 Jul 2024 00:00:00 GMT

This post features interactive content not available in RSS. View on snorre.io for the full experience.

I have been working on Code Monkey Clash for a while now. It’s a game where you compete against other players to write code as fast as possible. In one of the latest released I wanted to display a winner page with the top three players. A winner page deserves some cheerfulness and what’s more cheerful than confetti? After searching for confetti libraries I decided I would try to implement it myself.

Having not worked that much with animated graphics in the browser before I decided to search for some tutorials. I found a few, but most of them just used a library like js-confetti to do the heavy lifting. So I decided to ask ChatGPT for help. I know using ChatGPT is a bit controversial, but for exploratory coding tasks like this it can be quite helpful. Rather than ask for the whole solution up front I tried prompting it for some ideas on how to get started.

All artists need a canvas

Essentially the solution would require creating a canvas element and drawing on it. To get it out of the way lets make a reusable function for setting up the canvas and context. That way we don’t need to repeat the same code in every example.

<script>
  function setupCanvas(id) {
    const canvas = document.getElementById(id);
    const ctx = canvas.getContext('2d');
    canvas.width = 1600/2;
    canvas.height = 900/2;
    return { canvas, ctx };
  }
</script>

I dream of rectangles

Say you wanted to draw a red square on position 0,0 with a size of 100x100. You would set the fill style to red and then call fillRect(0, 0, 100, 100) on the canvas context.

<canvas id="canvas-rect"></canvas>
<script>
  function render() {
    const { canvas, ctx } = setupCanvas('canvas-rect');
    ctx.fillStyle = 'red';
    ctx.fillRect(0, 0, 100, 100);
  }
  render();
</script>

Paths and strokes for confetti

This is a good start, we have a canvas element and have drawn something on it. After some more prompting it seemed that drawing paths using strokes would be a good way to create the confetti. Mozilla Developer Network has good documentation on strokes. The idea is that you configure the stroke style (width and color), and then start drawing a path. To make something a little more confetti like we can draw a slightly diagonal line.

<canvas id="canvas-strokes"></canvas>
<script>
function render() {
    const { canvas, ctx } = setupCanvas('canvas-strokes');
    
    // First we set up the stroke style
    ctx.strokeStyle = 'red';
    ctx.lineWidth = 8;

    ctx.moveTo(30, 30); // Move to where we want to start the line
    ctx.lineTo(40, 50); // Draw a line to the new position
    ctx.stroke(); // Fill the path according to the stroke style
  }
  render();
</script>

Now this is starting to look like confetti already!

Colors of the Tailwind rainbow

But currently we’re just drawing one line, and have hardcoded the color to red. Let us add some more colors to use in our examples! I let GitHub Copilot suggest the colors based on the Tailwind CSS color palette. Not sure if they actually match those in Tailwind, but they look good to me!

<script>
  // For brevity I will only show a simplified example of the color function
  // If you inspect my code I account for dark mode preference as my blog
  // supports both light and dark mode
  let colors = [
    "#10b981",
    "#7c3aed",
    "#fbbf24",
    "#ef4444",
    "#3b82f6",
    "#22c55e",
    "#f97316",
    "#ef4444",
  ]
</script>

With our new-found colors lets try to draw a few more confetti pieces. We can use the colors array to pick a random color for each piece. Drawing a row of confetti in various colors should just require a loop!

<canvas id="canvas-colors"></canvas>
<script>
function render() {
    const { canvas, ctx } = setupCanvas('canvas-colors');
    ctx.lineWidth = 8;

    for (let i = 0; i < 20; i++) {
      ctx.strokeStyle = colors[Math.floor(Math.random() * colors.length)];
      ctx.moveTo(30 + i * 30, 30); // For each iteration move further to the right
      ctx.lineTo(40 + i * 30, 50); // Draw a line to the new position
      ctx.stroke(); // Fill the path according to the stroke style
    }
  }
  render();
</script>

But what the heck is this?! All our confetti pieces have the same color. As it turns out we’ve been drawing sub-paths on the same path for each iteration. So while the color changes for each iteration, it is set for all sub paths. Fortunately beginPath is here to save the day! Now we have a row of confetti pieces in various colors!

<canvas id="canvas-colors-fixed"></canvas>
<script>
function renderColors() {
    const { canvas, ctx } = setupCanvas('canvas-colors-fixed');
    ctx.lineWidth = 8;

    for (let i = 0; i < 20; i++) {
      ctx.beginPath(); // Start a new path
      ctx.strokeStyle = colors[Math.floor(Math.random() * colors.length)];
      ctx.moveTo(30 + i * 30, 30); // Move to where we want to start the line
      ctx.lineTo(40 + i * 30, 50); // Draw a line to the new position
      ctx.stroke(); // Fill the path according to the stroke style
    }
  }
  renderColors();
</script>

Randomness is the spice of life

But what is confetti without some randomness? Right now all our confetti pieces are the same diagonal line. We can introduce some randomness by adding a random radius and tilt for each piece. This will simulate 3D space a bit as smaller pieces will appear further away. Tilting the pieces various directions will also make them look more natural and random.

<canvas id="canvas-random"></canvas>
<script>
  function render() {
    const { canvas, ctx } = setupCanvas('canvas-random');

    for (let i = 0; i < 40; i++) {
      // Make the radius of the path random so confetti are different sizes
      const radius = Math.floor(Math.random() * 50) - 10
      // Ensure the paths tilt in various directions
      const tilt = Math.floor(Math.random() * 10) - 10

      // Distribute the confetti randomly on the canvas
      const x = Math.random() * canvas.width;
      const y = Math.random() * canvas.height;

      ctx.beginPath(); // Start a new path
      ctx.lineWidth = radius / 2;
      ctx.strokeStyle = colors[Math.floor(Math.random() * colors.length)];
      ctx.moveTo(x + tilt + radius / 4, y);
      ctx.lineTo(x + tilt, y + tilt + radius / 4);
      ctx.stroke(); // Fill the path according to the stroke style
    }
  }
  render();
</script>

Weightless confetti is no fun

Up until now the confetti has just been hanging in the air. No party is complete without some gravity! We can simulate some gravity by introducing some delta velocity for each piece. Then we need to update the position of the confetti pieces each frame. To do this we can use requestAnimationFrame to call a function that updates the position and redraws the canvas. We also need to clear the canvas each frame to avoid a mess of confetti pieces.

<canvas id="canvas-gravity"></canvas>
<script>
  function renderFallingConfetti() {
    const confetti = []
    const start = performance.now() // To keep track of time
    const { canvas, ctx } = setupCanvas('canvas-gravity');

    for (let i = 0; i < 40; i++) {
      const radius = Math.floor(Math.random() * 50) - 10
      const tilt = Math.floor(Math.random() * 10) - 10
      const x = Math.random() * canvas.width;
      const y = Math.random() * canvas.height;

      confetti.push({
        x,
        y,
        radius,
        tilt,
        velocity: Math.random() * 2 + 1
      })
    }

    function update() {
      // Run for at most 10 seconds
      if (performance.now() - start > 10000) return
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      confetti.forEach((piece) => {
        piece.y += piece.velocity;
        ctx.beginPath();
        ctx.lineWidth = piece.radius / 2;
        ctx.strokeStyle = colors[Math.floor(Math.random() * colors.length)];
        ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
        ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
        ctx.stroke();
      })
      requestAnimationFrame(update);
    }
    update();
  }
  renderFallingConfetti();
</script>

Confetti don’t fall straight down

Right now our confetti animation is pretty boring. All the pieces just fall straight down as if there was no atmosphere. A party without some oxygen will quickly become a disaster! Usually confetti pieces will scatter a bit as they fall going every which way.

Each confetti piece should thus have some random horizontal velocity as well as the vertical velocity. As gravity is mostly always pointing down the vertical velocity should be positive. However, the horizontal velocity can be both positive and negative as air resistance scatter the pieces.

<canvas id="canvas-scattering"></canvas>
<script>
  function renderScatteringConfetti() {
    const confetti = []
    const start = performance.now() // To keep track of time
    const { canvas, ctx } = setupCanvas('canvas-scattering');

    for (let i = 0; i < 40; i++) {
      const radius = Math.floor(Math.random() * 50) - 10
      const tilt = Math.floor(Math.random() * 10) - 10
      const x = Math.random() * canvas.width;
      const y = Math.random() * canvas.height;

      confetti.push({
        x,
        y,
        radius,
        tilt,
        color: colors[Math.floor(Math.random() * colors.length)],
        yVelocity: Math.random() * 3,
        xVelocity: Math.random() * 2 - 1
      })
    }

    function update() {
      // Run for at most 10 seconds
      if (performance.now() - start > 10000) return
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      confetti.forEach((piece) => {
        piece.y += piece.yVelocity;
        piece.x += piece.xVelocity;
        ctx.beginPath();
        ctx.lineWidth = piece.radius / 2;
        ctx.strokeStyle = piece.color;
        ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
        ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
        ctx.stroke();
      })
      requestAnimationFrame(update);
    }
    update();
  }
  renderScatteringConfetti();
</script>

Flutter like a butterfly

I still thought the confetti was a bit boring. How can we make it look more lively? Well, ChatGPT has a good suggestion:

Cosine for Vertical Movement:

When Math.cos is used for vertical movement, it creates a smooth oscillation that starts at a maximum value and goes through a complete cycle (down to a minimum value and back to the maximum). This can simulate a fluttering effect where the object moves up and down smoothly.

Sine for Horizontal Movement:

When Math.sin is used for horizontal movement, it creates a smooth oscillation that starts at zero and goes through a complete cycle (left and right movement). This can simulate a swaying effect where the object moves side to side smoothly.

So essentially because both the sine and cosine functions in JavaScript oscillate between -1 and 1 we can use them to create smooth movements. For each confetti piece we make a phase offset that is unique to the piece. Then we make a time counter to gradually increase the phase offset. The quickler the time counter increases the quicker the oscillation will be. We also introduce some speed constants for the x and y direction to control the speed and magnitude of the movement.

<canvas id="canvas-fluttering"></canvas>
<script>
  
  function renderFlutteringConfetti() {
    const timeDelta = 0.05;
    const xAmplitude = 0.5;
    const yAmplitude = 1;

    let time = 0;
    const confetti = []
    const start = performance.now() // To keep track of time
    const { canvas, ctx } = setupCanvas('canvas-fluttering');

    for (let i = 0; i < 40; i++) {
      const radius = Math.floor(Math.random() * 50) - 10
      const tilt = Math.floor(Math.random() * 10) - 10
      const x = Math.random() * canvas.width;
      const y = Math.random() * canvas.height;

      confetti.push({
        x,
        y,
        radius,
        tilt,
        color: colors[Math.floor(Math.random() * colors.length)],
        phaseOffset: i, // Randomness from position in list
      })
    }

    function update() {
      // Run for at most 10 seconds
      if (performance.now() - start > 10000) return
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      confetti.forEach((piece, i) => {
        piece.y += (Math.cos(piece.phaseOffset + time) + 1) * yAmplitude;
        piece.x += Math.sin(piece.phaseOffset + time) * xAmplitude;
        ctx.beginPath();
        ctx.lineWidth = piece.radius / 2;
        ctx.strokeStyle = piece.color;
        ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
        ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
        ctx.stroke();
      })
      time += timeDelta;
      requestAnimationFrame(update);
    }
    update();
  }
  renderFlutteringConfetti();
</script>

Feel free to play around with the amplitude and time delta to get different effects!

Horizontal amplitude

Vertical amplitude

Time delta

Tying up the pieces

We have come a long way from a single red square to a fluttering confetti party! But we’re not quite there yet. When adding the flutter we completely removed the sideways movement. On average the confetti pieces move straight down! And as you might have noticed, annoyingly the confetti pieces fall off the canvas. How about we re-introduce some sideways momentum and make the confetti pieces wrap around the canvas?

<canvas id="canvas-wrapping"></canvas>
<script>

  const { canvas, ctx } = setupCanvas('canvas-wrapping'); 
  function renderWrappingConfetti() {
    const timeDelta = 0.05;
    const xAmplitude = 0.5;
    const yAmplitude = 1;
    const xVelocity = 2;
    const yVelocity = 3;

    let time = 0;
    const confetti = []

    for (let i = 0; i < 100; i++) {
      const radius = Math.floor(Math.random() * 50) - 10
      const tilt = Math.floor(Math.random() * 10) - 10
      const xSpeed = Math.random() * xVelocity - xVelocity / 2
      const ySpeed = Math.random() * yVelocity
      const x = Math.random() * canvas.width;
      const y = Math.random() * canvas.height - canvas.height;

      confetti.push({
        x,
        y,
        xSpeed,
        ySpeed,
        radius,
        tilt,
        color: colors[Math.floor(Math.random() * colors.length)],
        phaseOffset: i, // Randomness from position in list
      })
    }

    function update() {
      // Run for at most 10 seconds
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      confetti.forEach((piece, i) => {
        piece.y += (Math.cos(piece.phaseOffset + time) + 1) * yAmplitude + piece.ySpeed;
        piece.x += Math.sin(piece.phaseOffset + time) * xAmplitude + piece.xSpeed;
        // Wrap around the canvas
        if (piece.x < 0) piece.x = canvas.width;
        if (piece.x > canvas.width) piece.x = 0;
        if (piece.y > canvas.height) piece.y = 0;
        ctx.beginPath();
        ctx.lineWidth = piece.radius / 2;
        ctx.strokeStyle = piece.color;
        ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
        ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
        ctx.stroke();
      })
      time += timeDelta;
      requestAnimationFrame(update);
    }
    update();
  }
</script>

Conclusion

If you made it this far I hope you have learned something new! With the help of Kagi Search, MDN documentation, and ChatGPT I was able to make quite the confetti party. The final result is quite lively and has some interesting movement! By using the sine and cosine functions we were able to create smooth oscillations for the confetti pieces.

The confetti example can be extended even further. For example the pieces don’t rotate as they fall. You could also make the confetti pieces have different shapes, and different colors on each side. There are tons of possibilities! Some libraries offer more advanced animation than what we have here. For me though half the fun was in the exploration and learning. In any case, feel free to play around with the code and use it for your own needs.

Addendum

This post makes no attempt at optimizing the code! Observant and knowledgeable readers will notice that the code could be optimized in various ways. I will leave that as an excersize for the reader! A great tip is looking at TypedArrays for performance improvements.

Make some confetti with JavaScript and Canvas - Part Two

Sat, 17 Aug 2024 00:00:00 GMT

This post features interactive content not available in RSS. View on snorre.io for the full experience.

In a previous post we started making some confetti using JavaScript and the canvas element. The implementation was quite simple, but the end result looked quite nice. In the conclusion I alluded to performance improvements. In this post we will explore how to improve peformance.

The old solution

First I want to recap what the old solution looked like. We made a helper function to set up the canvas element.

<script>
  function setupCanvas(id) {
    const canvas = document.getElementById(id);
    const ctx = canvas.getContext('2d');
    canvas.width = 1600/2;
    canvas.height = 900/2;
    return { canvas, ctx };
  }
</script>

Then we added some colors:

<script>
  // For brevity I will only show a simplified example of the color function
  // If you inspect my code I account for dark mode preference as my blog
  // supports both light and dark mode
  let colors = [
    "#10b981",
    "#7c3aed",
    "#fbbf24",
    "#ef4444",
    "#3b82f6",
    "#22c55e",
    "#f97316",
    "#ef4444",
  ]
</script>

This time around I’ll add a helper function to set up the particles. That way we can easily change the number of particles as the animation is running. It takes a reference to an array of particles and the number of particles to create. If the number of particles is less than the length of the array it will remove particles from the end. If the number of particles is greater than the length of the array it will add particles to the end.

<script>
  function setupParticles(particles, numParticles) {
    if(numParticles < particles.length) {
      particles.length = numParticles;
    } else if(numParticles > particles.length) {
      for (let i = particles.length; i < numParticles; i++) {
        particles.push({
          x: Math.random() * canvas.width,
          y: Math.random() * canvas.height - canvas.height,
          radius: Math.floor(Math.random() * 50) - 10,
          tilt: Math.floor(Math.random() * 10) - 10,
          xSpeed: Math.random() * 2 - 1,
          ySpeed: Math.random() * 3,
          color: colors[Math.floor(Math.random() * colors.length)],
          phase: i, // Randomness from position in list
        })
      }
    }
  }
</script>

Now we can wire together all the parts to implement the old solution:

<canvas id="canvas-slow"></canvas>
<script>

  const { canvas, ctx } = setupCanvas('canvas-slow'); 
  const slowConfetti = [];

  function renderSlow() {
    const timeDelta = 0.05;
    const xAmplitude = 0.5;
    const yAmplitude = 1;
    const xVelocity = 2;
    const yVelocity = 3;

    let time = 0;

    setupParticles(canvas, slowConfetti, 100);

    function update() {
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      slowConfetti.forEach((piece, i) => {
        piece.y += (Math.cos(piece.phase + time) + 1) * yAmplitude + piece.ySpeed;
        piece.x += Math.sin(piece.phase + time) * xAmplitude + piece.xSpeed;
        // Wrap around the canvas
        if (piece.x < 0) piece.x = canvas.width;
        if (piece.x > canvas.width) piece.x = 0;
        if (piece.y > canvas.height) piece.y = 0;
        ctx.beginPath();
        ctx.lineWidth = piece.radius / 2;
        ctx.strokeStyle = piece.color;
        ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
        ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
        ctx.stroke();
      })
      time += timeDelta;
      requestAnimationFrame(update);
    }
    update();
  }
</script>

Number of confetti: 100

FPS: 0

Frame: 0

Particle: 0

As you increase the number of confetti you will notice the performance starts to degrade. My MacBook Pro with the M1 processor sees about these numbers:

Number of confetti	Time to draw frame	Frames per second
100	0.1ms	120
10 000	2.4ms	120
100 000	25ms	40
250 000	60ms	16

As you can see the performance is good enough for practical purposes on powerful hardware. When rendering 100 000 confetti you can hardly tell where one particle ends and the next begins. However, slower hardware will likely struggle much earlier than this. And we’re seeing significant dips in FPS when rendering 250 000 confetti, likely due to GC. So let us try to squeeze out some more performance!

Move computations out from the hot path

As it turns out we’re doing some expensive computations in the hot path. The Math.cos and Math.sin functions are called for each particle in each frame. These are only computed for each particle as the phase offsets are unique for each particle. Each particle also have a complete unique x and y velocity and amplitude.

Let us try to instead make 100 unique combinations to compute for each frame. This should be much faster than computing 100 000+ unique combinations. At the start of the function we create arrays for each property:

<script>
  // Setup a helper function to create the phases
  function particlePhases() {
    return Array.from({ length: 500 }, (i) => {
      return {
        xPhase: Math.cos(i),
        yPhase: Math.sin(i) + 1,
        xAmplitude: Math.random() * 2 - 1,
        yAmplitude: 1,
        x: Math.cos(i) * 
      }
    });
  }

  // Before the animation loop we can build the phases
  const phases = particlePhases();

  // Then for each frame we can update the particles
  for (let i = 0; i < phases.length; i++) {
    phases[i].x = Math.sin(i + time) * phases[i].xAmplitude;
    phases[i].y = Math.cos(i + time) + 1 * phases[i].yAmplitude;
  }

  // Inside the loop we can use the phases to update the particles
  phaseConfetti.forEach((piece, i) => {
    piece.y += phases[i % phases.length].y * piece.ySpeed;
    piece.x += phases[i % phases.length].x * piece.xSpeed;
  })  
</script>

Number of confetti: 100

FPS: 0

Frame: 0

Particle: 0

With this new improvement we’re already seeing significant performance improvements, but only at 250 000 confetti. The 250 000 confetti now runs closer to 20 frames per second, but with the same drops as before.

Number of confetti	Time to draw frame	Frames per second
100	0.1ms	~120
10 000	2.3ms	~120
100 000	20ms	~40
250 000 *	50ms	~20

Draw more efficiently

One of the more expensive operations in the loop is drawing each particle. We start a new path for each particle and draw a line. If we can reduce the number of paths we start we can reduce the number of operations. Each path can only hold one color, so we need to group or sort the particles by color.

I’m elliding most of the existing code for brevity, but the important part is the sorting and drawing of the particles.

<script>
    // Before we start the animation we can sort the particles by color 
    pathConfetti.sort((a, b) => a.color.localeCompare(b.color));

    // Then for each frame we can draw the particles in groups by color
    let prevColor = ''

    pathConfetti.forEach((piece, i) => {
      if(piece.color !== prevColor) {
        ctx.stroke();
        ctx.beginPath();
        ctx.strokeStyle = piece.color;
        prevColor = piece.color;
      }
      ctx.lineWidth = piece.radius / 2;
      ctx.moveTo(piece.x + piece.tilt + piece.radius / 4, piece.y);
      ctx.lineTo(piece.x + piece.tilt, piece.y + piece.tilt + piece.radius / 4);
    })
</script>

Number of confetti: 100

FPS: 0

Frame: 0

Particle: 0

Now we’re getting somewhere! The 250 000 confetti now runs at about 40 frames per second, twice as fast as before. There are still occasional drops in FPS, but they are less severe. At 100 000 confetti it jumps between 60 and 120 frames per second. You’ll notice now that with such a high number of confetti the canvas appears almost uniformly colored. This is because the particles are sorted by color and drawn in groups. For this post though we’re mostly interested in testing performance, not making pretty pictures.

Number of confetti	Time to draw frame	Frames per second
100	0ms	~120
10 000	2ms	~120
100 000	12ms	~60
250 000	27ms	~30

Typed arrays

Up until now we’ve been using plain JavaScript objects and arrays to represent the particles. These are often sufficient for everyday UIs and APIs that push small amounts of data around. When you have a lot of data, like 250 000 particles, you might want to consider using something more efficient. JavaScript arrays are actually objects with some special properties. Thus they are always stored on the heap and you potentially miss out on some performance benefits.

Let us see if we can improve performance by using typed arrays. Typed arrays are arrays where each element is of the same type. This allows them to be stored in a contiguous block of memory and cached in CPU cache. This can lead to faster access times and fewer cache misses

Because this is a larger change I will show the entire implementation. This time we will use a Float32Array to store the particles. We don’t need the full precision of double precision floating point numbers to animate confetti.

As before we keep the phases in a separate list to avoid recomputing them for all particles. 7 properties at 1 byte each means each confetti particle is 7 bytes. With 250 000 particles that’s 1.75MB of data which is well within most L2 caches.

<script>
  // Setup a helper function to create the phases
  function linearTransform(value, inMin, inMax, outMin, outMax) {
    return Math.round((value - inMin) * (outMax - outMin) / (inMax - inMin) + outMin);
  }

  // Setup a helper function to create the phases
  function setupTypedPhases() {
    const phases = new Float32Array(500 * 4);
    for (let i = 0; i < 10; i++) {
      phases[i] = Math.random() * 2 - 1;       // xAmplitudes
      phases[500 + i] = 1;                      // yAmplitudes
      phases[1000 + i] = Math.sin(i);            // xPhases
      phases[1500 + i] = Math.cos(i) + 1;        // yPhases
    }
    return phases;
  }

  function setupTypedParticles(canvas, numParticles) {
    const numProperties = 7; // x, y, radius, tilt, xSpeed, ySpeed, color
    const particles = new Float32Array(numParticles * numProperties);
    const numColors = colors.length;

    // Initialize particles
    for (let i = 0; i < numParticles; i++) {
      const index = i * numProperties;
      particles[index] = Math.random() * canvas.width;       // x
      particles[index + 1] = Math.random() * canvas.height - canvas.height; // y
      particles[index + 2] = Math.floor(Math.random() * 50) - 10;        // radius
      particles[index + 3] = Math.floor(Math.random() * 10) - 10;        // tilt
      particles[index + 4] = Math.random() * 2 - 1;          // xSpeed
      particles[index + 5] = Math.random() * 3;              // ySpeed
      particles[index + 6] = linearTransform(i, 0, numParticles, 0, numColors); // color
    }

    return { particles, numProperties };
  }
</script>

With the helper functions to build typed arrays of particles and phases we can now implement the animation loop.

<canvas id="canvas-typed"></canvas>
<script>
  let typedNumConfetti = 100;

  function renderTyped() {
    const { canvas, ctx } = setupCanvas('canvas-typed');
    let time = 0;
    let typedConfetti = setupTypedParticles(canvas, typedNumConfetti);
    const phases = setupTypedPhases();

    function update(animationTime) {
      const {particles, numProperties} = typedConfetti;
      ctx.clearRect(0, 0, canvas.width, canvas.height);

      // Calculate the phase values for the frame
      for (let i = 0; i < 500; i++) {
        phases[1000 + i] = Math.sin(i + time) * phases[i];
        phases[1500 + i] = (Math.cos(i + time) + 1) * phases[500 + i]; // yPhases updated with yAmplitudes
      }


      ctx.beginPath();

      let prevColor = -1
      
      for (let i = 0; i < typedNumConfetti; i++) {

        const index = i * numProperties;
        
        particles[index] += phases[20 + (i % 10)] + particles[index + 4];  // x + xSpeed
        particles[index + 1] += phases[30 + (i % 10)] + particles[index + 5]; // y + ySpeed

        // Wrap around the canvas
        if (particles[index] < 0) particles[index] = canvas.width;
        if (particles[index] > canvas.width) particles[index] = 0;
        if (particles[index + 1] > canvas.height) particles[index + 1] = 0;

        const radius = particles[index + 2];
        const tilt = particles[index + 3];
        const colorIndex = particles[index + 6];

        if (colorIndex !== prevColor) {
          ctx.stroke();
          ctx.beginPath();
          ctx.strokeStyle = colors[colorIndex];
          prevColor = colorIndex;
        }

        ctx.lineWidth = radius / 2;
        ctx.moveTo(particles[index] + tilt + radius / 4, particles[index + 1]);
        ctx.lineTo(particles[index] + tilt, particles[index + 1] + tilt + radius / 4);
      }

      ctx.stroke();

      time += timeDelta;
      typedTimeToDrawFrame = performance.now() - typedTimeToDrawFrame;

      requestAnimationFrame(update);
    }
      
    update(0);
  }
</script>

Number of confetti: 100

FPS: 0

Frame: 0

Particle: 0

We managed to eek out just a bit more performance, but at this point we’re starting to hit the limits of what we can do with CPU and Canvas. The 250 000 confetti now runs at above 40 frames per second, a good improvement over the previous implementation. Rendering up to 50 000 confetti should be no problem on most hardware with this implementation.

Number of confetti	Time to draw frame	Frames per second
100	0ms	~120
10 000	2ms	~120
100 000	9ms	~90
250 000	22ms	~40

There are further room for improvements. One could conceivably parallellise the computation of the phases and the particle updates, but I won’t cover that here. Also the Canvas is not the most efficient way to render large amounts of particles. For that you might want to look into WebGL or WebGPU.

Conclusion

We started with a simple implementation of confetti animation and iterated on it to improve performance. Moving expensive computations out of the hot path provided a significant performance boost. Drawing particles in groups by color also improved performance as we could reduce the number of paths we started. Finally we used typed arrays to store the particles and phases to improve memory locality and cache performance.

The final implementation can render 250 000 confetti particles at 40 frames per second on my M1 MacBook Pro. This is a significant improvement over the original implementation which struggled at 100 000 particles. While you should really consider using WebGL or WebGPU for this many particles, the Canvas API can still be performant with some clever optimizations.

I hope you found this post interesting and maybe even learned something new.

Running Bluesky PDS with Traefik proxy on Scaleway

Mon, 22 Sep 2025 00:00:00 GMT

Bluesky is a decentralized social network built on the AT Protocol. If you’ve followed me for a while you probably know I’m a big fan. While most users sign up on the main bsky.social PDS, you can run your own PDS instance to maintain full control over your data. I decided to self-host my PDS to keep my social data sovereign and experiment with the protocol’s decentralized nature. This post walks through my setup hosting a PDS on Scaleway using Traefik as a reverse proxy.

Part 1: Personal Data Servers

The AT Protocol Personal Data Server (PDS for short) is the service that stores user records and file uploads (blobs). Whenever you post or repost, like a post, follow a user, or do anything else on the network you create a record on your PDS. As of this writing (February 2025) almost every user in the ATProto network is on a Bluesky PDS. Are we decentralized yet is a useful tracker that measures user concentration on different federated social networks.

I am a big believer in the AT Protocol, even having built a Feed Generator server. But as long as the vast majority users remain on the official PDSs the decentralized nature of Bluesky is only theoretical. To help the network move in a more decentralized direction you can host your own Personal Data Server, and that is exactly what I did (pun not intended).

I did a bit of research and decided to go with the official PDS which runs on Node.js on top of SQLite.

Part 2: Choosing a hosting provider

Part of the reason for hosting my own PDS was to move my data away from the US and closer to home. My apartment isn’t really equipped for running a home lab, so I needed to find a hosting provider. I decided to go with a hosting provider I was familiar with, Scaleway. There are a couple of reasons for this:

They are european and have servers in Amsterdam, not too far away from Norway
They offer cheap small VPS instances with unlimited traffic
Storage is networked and can easily be scaled up or down
You can easily scale up to a more powerful instance if needed
You can create snapshots of your instance-attached storage and restore them
They have a good S3-compatible block storage offering for hosting files

I ended up going with the DEV1-S instance which is a 2 vCPU, 2GB RAM, 20GB block storage, and 200Mbps bandwidth. This instance is quite cheap while allowing me to run the PDS, a reverse proxy, and some supporting services.

Part 3: Setting up the PDS

The official repository is pretty opinionated on how to run the PDS. It provides an install script that:

Installs Docker
Sets up environment variables in specific location
Creates a Caddy proxy config
Installs a unit file for systemd to run Caddy and the PDS
Starts the services

Now I like Docker, but I feel Caddy perhaps is a bit too commercial these days. Also I’d like to have some say over how everything is set up. The official installer assumes you use disk storage for the PDS blobs. I’d rather use the S3-compatible block storage offered by Scaleway. So I decided to take a step back and do the configuration myself, after all how hard could it be?

Part 4: Docker setup

I mostly run everything I do in Docker. It allows me to easily set up and tear down services without worrying about dependencies and lingering config and files. One of the easier ways to expose Docker services I’ve found is to use Traefik. They support connecting to a Docker daemon and automatically set up a reverse proxy for labeled Docker Compose services.

Before we look at the services you need to configure some networks and volumes. I prefer to use docker volumes over bind mounts for data, but using bind mounts is also possible. Observe that we only need one network, traefik, as the PDS will only be accessed via the Traefik proxy.

volumes:
  bluesky-pds:
  letsencrypt:

networks:
  traefik:
    name: traefik

Let’s start with the Traefik service.

services:
  traefik:
    image: "traefik:v3"
    container_name: "traefik"
    restart: always
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik"
      # Web sites entrypoint
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.http3.advertisedport=443"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
      # Certificate/ACME stuff
      - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.cloudflare.acme.email=${ACME_EMAIL}"
      - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json"
    environment:
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
      - CLOUDFLARE_EMAIL=${CLOUDFLARE_EMAIL}
    networks:
      - traefik
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - "letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

Traefik is configured to listen on port 80 and 443 for http and https requests. We configure the Traefik to redirect http requests to https, a simple security measure. Usually when running Traefik I use the http challenge for ACME, but this does not work for wildcard certificates. Instead we have to use a DNS challenge, in my case via Cloudflare DNS. Traefik supports automatic handling of DNS challenges for lots of DNS providers. We also need to map the docker socket to the Traefik container to allow it to read from the Docker daemon socket.

Now let’s take a look at the Bluesky PDS service.

  bluesky-pds:
    container_name: bluesky-pds
    image: ghcr.io/bluesky-social/pds:0.4
    restart: unless-stopped
    networks:
      - bluesky-pds
      - traefik
    labels:
      - traefik.enable=true
      - traefik.http.middlewares.bluesky-pds-header.headers.customrequestheaders.Host="{host}"
      - traefik.http.routers.bluesky-pds.rule=Host(`pds.snorre.io`) || HostRegexp(`[a-zA-Z0-9-]+.pds.snorre.io`)
      - traefik.http.routers.bluesky-pds.entrypoints=web,websecure
      - traefik.http.routers.bluesky-pds.tls=true
      - traefik.http.routers.bluesky-pds.tls.certresolver=cloudflare
      - traefik.http.routers.bluesky-pds.tls.domains[0].main=pds.snorre.io
      - traefik.http.routers.bluesky-pds.tls.domains[0].sans=*.pds.snorre.io
      - traefik.http.routers.bluesky-pds.middlewares=bluesky-pds-header 
      - traefik.http.services.bluesky-pds.loadbalancer.server.port=3000
      - traefik.docker.network=traefik
    env_file:
      - ./pds/pds.env
    volumes:
      - bluesky-pds:/pds

You’ll probably notice the many labels on the PDS service. These are used to configure Traefik to request SSL certificates from Cloudflare and to proxy requests to the PDS service. The traefik.http.middlewares.bluesky-pds-header.headers.customrequestheaders.Host="{host}" label is especially important. It allows the PDS to know the original hostname of the request which tells it which user the request was made for.

The PDS service expects configuration in the form of environment variables. However the pdsadmin helper scripts expects the configuration to be in a .env file. To keep things simple we can reuse the same .env file for both by using the env_file option in Docker Compose. The .env file looks like this:

# PDS specific variables
PDS_HOSTNAME=pds.snorre.io
PDS_JWT_SECRET=<my-jwt-secret>
PDS_ADMIN_PASSWORD=<my-admin-password>
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX=<my-plc-rotation-key>
PDS_DATA_DIRECTORY=/pds
#PDS_BLOBSTORE_DISK_LOCATION=/pds/blocks

PDS_BLOBSTORE_S3_BUCKET=<my-s3-bucket-name>
PDS_BLOBSTORE_S3_REGION=<my-s3-region>
PDS_BLOBSTORE_S3_ENDPOINT=<my-s3-endpoint>
PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=false
PDS_BLOBSTORE_S3_ACCESS_KEY_ID=<my-s3-access-key-id>
PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY=<my-s3-secret-access-key>

PDS_BLOB_UPLOAD_LIMIT=52428800
PDS_DID_PLC_URL=https://plc.directory
PDS_BSKY_APP_VIEW_URL=https://api.bsky.app
PDS_BSKY_APP_VIEW_DID=did:web:api.bsky.app
PDS_REPORT_SERVICE_URL=https://mod.bsky.app
PDS_REPORT_SERVICE_DID=did:plc:ar7c4by46qjdydhdevvrndac
PDS_CRAWLERS=https://bsky.network

# SMTP for mail sending
PDS_EMAIL_SMTP_URL=smtps://<my-smtp-user>:<my-smtp-password>@<my-smtp-domain>/
PDS_EMAIL_FROM_ADDRESS=<my-smtp-user>@<my-smtp-domain>

# Logging
LOG_ENABLED=true

As you can see I’m configuring the PDS to use S3 storage for blobs. This is not officially documented in the PDS repository, but it seems to work. Using S3 storage means I don’t need to worry about disk storage running out too fast. The S3 bucket should be created before running the PDS and should be configured as private. The PDS handles requests for blobs so they don’t need to be publicly accessible by default.

With all this out of the way I could start the services, no systemd unit files needed.

docker compose up -d

Part 5: Running admin commands

As we’ve forgone the whole official installer we need to download the psdadmin scripts. The pdsadmin script is not bundled in the PDS Docker image. Thus we need to clone the PDS repository to get access to it.

git clone https://github.com/bluesky-social/pds
cd pds
chmod +x pdsadmin.sh

Then we can run the pdsadmin script.

sudo ./pdsadmin.sh help

For some reason they require sudo to run every command, even if only the update command needs it depending on which permissions you’ve set on the env file. If you wanted to create an invite code you can do so with the create-invite-code command. Because it expects the environment file to be under /pds/pds.env we need to override the path to the environment file. You can forgo that if you’ve put the environment file in the expected location.

sudo su
PDS_ENV_FILE=<path-to>/pds.env ./pdsadmin.sh create-invite-code

You can then use the invite code to create a new account on the PDS.

Part 6: Being discovered by Bluesky Relay

The Bluesky Relay is the service that actually makes content on PDSs discoverable on the Bluesky network/app. To make your new PDS discoverable you need to register it with the Bluesky Relay.

sudo su
PDS_ENV_FILE=<path-to>/pds.env ./pdsadmin.sh request-crawl bsky.network

Hurray, you’re now discoverable on the Bluesky network!

Part 7: Migrating your Bluesky account to the new PDS

Now this part was the most scary part for me. The documentation makes it out to be a bit scary:

Account migration is a potentially destructive operation. Part of the operation involves signing away your old PDS’s ability to make updates to your DID. If something goes wrong, you could be permanently locked out of your account, and Bluesky will not be able to help you recover it.

Fortunately I found that Bluesky has a tool that among other things can migrate your account to a new PDS. It is called goat. I won’t go into too many details here, instead I’ll just link to this excellent guide by bryan newbold. It lists all the relevant steps and commands you need to follow. Suffice to say my account is now on my new PDS! I hope this post can inspire you to host your own PDS and take back control of your social data.

Ansible - An elegant provisioning tool for a more civilized age

Tue, 24 Dec 2024 00:00:00 GMT

It has been a while since I’ve written a blog post. Life’s been busy, and so time has sort of slipped away. These past few weeks I’ve been setting up some new servers and wanted to standardise the setup. I recalled the old, trusty CLI tool Ansible and decided to give it a try.

Server provisioning

So before the advent of serverless computing, people would manage and host applications on physical or virtual servers. Now there is something about the feeling of ssh-ing into a server and getting your hands dirty that I really like. You can apt install stuff, manage services with systemctl, shoot your own foot with rm -rf /, and generally just have full control over your own little corner of the internet. But with great power comes great responsibility. If you don’t want your server to be a target for hackers, you need to configure the sshd service, set up a firewall, add intrusion detection, and so on. You also might want to monitor the resource usage of your servers and much more. Doing this manually for each new server is a pain, and remembering what you’ve done previously is even worse.

Server provisioning tools

But fret not, for the system administrators of old did not sit around doing this manually! Initially sysadmins would write shell scripts to automate the setup of servers, but this was not scalable. So they started writing tools to manage the setup of servers. Generally there are two models of server provisioning tools:

The pull model
The push model

In the pull model, you install an agent on each server you want to manage, and the agent will pull the configuration from a central server. The agent then applies any changes required to the server it is running on. This models incurs some overhead on each server, and as soon as you update the configuration on the central server, the agents will pull the new configuration and apply it.

In the push model, you run a central server that pushes the configuration to the servers you want to manage. This model negates the need for an agent on each server reducing the overhead. However, as the central server or machine have to apply the configuration on each server, this model is often slower to apply. Further more you need to occassionally check the configuration on target servers to detect any drift that might have occurred.

Ansible

Ansible is a tool that implements the push model, sending python scripts to the servers you want to manage. Each python script runs on the target server and can be used to install packages, configure services, and so on. The only requirement for each server is thus a python interpreter, which is generally available on most Linux distributions.

The tool was released way back in February 2012, and has since then become a very popular tool for server provisioning. Some might argue that running servers and provisioning them is a bit of an old-school thing to do. Why not just use managed cloud services? Well, managed cloud services are great, but they can be expensive, and you don’t have full control over your own infrastructure.

So what does it look like to provision a server with Ansible?

The inventory

First you want to define an inventory of servers you want to manage. This can be done in a file called inventory.yml:

db-servers:
  hosts:
    server1.example.com:
      ansible_host: 192.168.1.100
      ansible_user: hunter
    server2.example.com:
      ansible_host: 192.168.1.101
      ansible_user: hunter

This inventory file defines two servers, server1.example.com and server2.example.com, that we want to manage, their respective IP addresses, and the user we want to use to ssh into them. They are both grouped under the db-servers group which we can reference in our playbooks.

Modules

Modules are the building blocks of Ansible. They are small reusable python modules that can be used to perform specific tasks. For example, the package module is used to install and remove packages on a server using the distribution’s package manager. Another example is the shell module, which allows you to run shell commands on a server.

Actions

Actions are the tasks that are executed on the target server(s). They are defined in the playbooks and are executed by the Ansible agent on the target server(s).

Roles

Roles are a way to organize tasks into reusable units. The Ansible community has created a lot of roles for common tasks, such as installing and configuring specific services like Docker and CrowdSec.

You can also define your own roles, which can be reused in multiple playbooks.

The playbook

With modules, actions, roles and inventories defined we can finally get into the fun part: writing playbooks. A playbook is a yaml file that describes one or more plays, which are a set of tasks and/or roles that are executed on the target server(s). Each task defines some provisioning logic to be executed on the target server(s).

For example, to install a package on a server, you can use the built-in package module:

- name: Install packages
  hosts: db-servers
  tasks:
    - name: Install ufw
      package:
        name: ufw
        state: present

Simple, right?

If you want to do more than just install packages, you can run multiple tasks in a playbook.

- name: Install packages
  hosts: db-servers
  become: true
  tasks:
    - name: Install ufw
      package:
        name: ufw
        state: present
    - name: Enable ufw
      shell:
        cmd: echo "y" | ufw enable

Let say you want to install and configure a PostgreSQL server on your db servers. You can use the geerlingguy.postgresql role to do this.

# site.yml
- name: Provision postgresql on db-servers
  hosts: db-servers
  roles:
    - role: geerlingguy.postgresql
      become: yes
      vars:
        postgresql_unix_socket_directories:
            - /var/run/postgresql
        postgresql_service_state: started
        postgresql_service_enabled: true

You can of course combine roles and tasks in a playbook. But I won’t go into details here.

Conclusion

Ansible is a powerful tool for provisioning servers. It allows you to standardise the setup of your servers, and to automate the provisioning of new servers. It is a bit of a learning curve, but once you get the hang of it, you’ll wonder how you ever managed servers without it. Not only does it help automate the provisioning of new servers, your playbooks also act as documentation as code for your server setup.

Review of Kerbal Space Program 2 - Part 1

Thu, 18 Jan 2024 00:00:00 GMT

You’ve probably heard it, the famous countdown as the Apollo 11 rocket is about to launch:

…2, 1, zero, all engine running…
LIFT-OFF! We have a lift-off, 32 minutes past the hour. Lift-off on Apollo 11.

It must have been a very exciting moment for those who were alive to follow the launch back then. The Apollo 11 program was the culmination of years of hard work by scientists and engineers. Humankind’s first trip to the surface of the Moon, in a time when computers were still the size of a room. For most of us being a part of such a momentous mission, to further mankinds knowledge of the universe, will just be a dream. Rocket science is hard, so for most of us rocket launches and space missions are just something we get to follow from afar.

Photo by NASA in the public domain. Downloaded from Wikimedia.

But what if you could? What if you could design your own rockets? Run a space agency? Explore the solar system? What if you could be the one to make the countdown, and then press the button to launch the rocket? With Kerbal Space Program 2 you can!

In the beginning

In the beginning there was Kerbal Space Program, an ambitious space simulation game originally released in 2011. It was a passion project of Felipe Falanghe, a Brazilian game developer working for a media company, Squad. With Kerbal Space Program Felipe explored the idea of making realistic space exploration accessible to the masses. The alpha was very rudimentary featuring only a single planet, limited physics and game play. But the game showed promise, and little by little it gained advanced physics, a solar system to explore, and a fullfledged career mode. I had always been in awe of the space exploration missions of NASA, ESA, and other space agencies. So when I discovered Kerbal Space Program I bought it and had some fun doing some simple missions. But for some reason I never really got into the game, and my space agency felt more like a satellite company.

Earth orbited the sun eight more times, 2019 rolled around, and a successor was announced. An early access version of Kerbal Space Program 2 was to take off in 2020, and it would be developed by Intercept Games. We orbited the sun once more, 2020 came about, but there were no signs of a release on the event horizon. It would take three more orbits before the game would be released in February 2024.

But the proverbial rocket crashed on the launch, much to the chagrin of the Kerbal Space Program community. The game received negative reviews on Steam, many complaining about the lack of gameplay content, bugs, and poor performance. I was dissapointed, but the Apollo program wasn’t built in a day, so I decided to wait for the game to mature.

Then 2023 came about, the northern hemisphere was leaning away from the sun, and Intercept Game launched the game’s firt big update “For Science!”. With love and care the developers had added some much needed content in the form of a science progression and mission system. Performance issues and bugs had also been adressed. Kerbal Space Program was shaping up to look like a proper game. I decided to put on the space suit and give it a try.

A space agency

I arrive at the Kerbal Space Center, the headquarters of the Kerbal Space Program. For some reason I, a total stranger with no rocket experience, have been put in charge of the space program. Fortunately the space program already has all the buildings and facilities you’d need to get started. I am lead through a series of tutorials teaching me the basics of rocket design and rocket control. Kerbal Space Program 2’s tutorials quickly move you from firing rockets at the sky to orbital mechanics. They stop short of the more advanced topics like interplanetary travel, but they give you a good foundation to build on.

What goes up must come down

My fledgling space program is ready to launch its first rocket. I don’t have that many rocket parts yet, so I’m tasked with some simple missions. As a lesson in the futility of reaching orbit I’m told to launch a rocket up to a certain height. I build what amounts to an oversized firework, launch it straight up, and watch it fall back down again. My fearless Kerbalnaut Tim Kerman does not seem all too entused about the dangers, but he’s a professional. As I watch the rocket fall down I learn a valueable lesson about the importance of parachutes. Not to mention how Kerbin’s (the planet) rotation keeps moving my rocket away from the launch pad. When you launch high enough you’ll find the earth moves away from under you, and your landing site will be a few kilometers away from the launch pad. These small progressive lessons are a great way to gain some intuition about how rocket physics and orbital mechanics work.

Orbiting a green rugged marble

I complete the first few missions, and I’m rewarded with science points that I use to unlock new rocket parts. My budding space agency is shaping up to be a real do-getter as I quickly achieve orbit around Kerbin. The first few missions are fairly simple, yet they give you a sense of accomplishment! As you orbit Kerbin (the planet) for the first time you can’t help but feel a sense of wonder as you admire the green marble below you. Your space rocket is hurtling through space a couple of thousand meters per second, yet you feel calm and at peace.

Kerbin is one of the many planets and moons in the Kerbal system, the solar system of Kerbal Space Program 2. The Kerbal system is a scaled down version of our own solar system, so Kerbin is the equivalent of Earth, and Mun is the equivalent of the Moon. To make space travel a bit easier the entire solar system is scaled down, thus reducing the gravity of the planets, and the distances between them. This allows your rockets to reach celestial bodies without too much fuel and mass. You’ll still find the distances cosmic in nature though, so a fast forward button is available to speed up time. As the game will teach you, space travel is mostly the silent waiting between the exciting storms of rocket burns to adjust your trajectories.

Yet, the game is not all quiet. First of all the game forgoes the realistic silence of space and blasts the engine noise of your rockets into your ears. Even as you watch your rocket from outside in the cold vacuum of space you’ll hear the roar of the engines. The game also has a beautiful soundtrack that plays as you explore the solar system. As you near your planned maneuver nodes the music will swell into an excited crescendo. This is a nice touch that connects the soundtrack to the gameplay, and it makes the game feel more in touch with you the player.

Freedom to explore and solve things your way

I complete a couple of more missions and find that increasingly I’m left to figure out my rocket designs on my own. Getting anywhere in the game is simply a matter of strapping enough explosive fuel and some engines to a command capsule. Kerbalnauts are fortunately very brave, so volunteering them for a mission is not a problem. For my first Mun mission I find myself overcompensating a bit, and I end up with a rocket that would probably bankrupt my space agency. Fortunately for me Kerbal Space Program 2 does not yet have an economy system, so I can build rockets with reckless abandon.

The game gives you the option of manned and unmanned probes. My agency begins it journey sending Tim Kerman into space and onto the two moons. But as I progress my space agency has shifted a bit toward unmanned probes (satellites) and rovers. An advantage of unmanned probes is that you don’t risk stranding your poor Kerbalnauts in space.

Sunrises and Sunrises

As I orbit the various celestial bodies I get to experience the magic of the sunrise and sunset from space. A big development from the first game is the improved graphics and graphical style. Gone are the mostly matte textures and lighting of the first game, and in comes a sense that the sun actually shines! With proper volumetric cloud rendering and other graphical effects space looks better than ever. As my rocket orbits a planet and I slowly see the sun rise over the horizon I forget my mission for a moment and just enjoy the view.

Rocket science is hard

The game really shines in the early game. when you are still exploring the close vicinity of Kerbin. But as the missions progress I miss some of the guidance you get in the early game. Going interplanetary is almost impossible without leaving the game and doing some research on the Internet.

There are also the occassional bugs and performance issues. For example the game sometimes fail to render an orbital trajectory for your rocket in the map view. If you can’t see the trajectory you can’t plan your rocket burns, and you can’t get anywhere. The game’s map UI also has some usability issues. Spaceship icons and labels often get in the way of your maneuver nodes (a way to create burn maneuver plans). If you zoom out the map view maneuver nodes become hard to manipulate, so you’ll find yourself zooming in and out.

Even so the game manages to be a lot of fun, and I’ve had a blast exploring Kerbin, its moons, and beyond. My agencies latest accomplishment (as of this writing) was a Rover mission to the surface of the moon. Another triumph for my agencies was a successful Duna (Mars) satellite. My satellite did not suffer the catastrophic fate of the real life Mars Climate Orbiter, and it is happily orbiting Duna (Mars) as we speak. I guess it helps that the game does not fail to account for different units of measurement, using only the metric system. Take that American Imperial system! Triumphs like these are what makes the game fun, and I’m looking forward to exploring the rest of the solar system.

The future

The game is still in the early stages, and there are many features to come. Upcoming features include colonization, interstellar travel, multiplayer, and more. For now though I’m declaring Lift-off! If you like simulation games and space exploration I recommend you give Kerbal Space Program 2 a try. The game is already very playable, and it can only get better from here!

See you around, space cowboy!

Protecting Traefik proxy with CrowdSec

Sun, 25 Jan 2026 00:00:00 GMT

I’ve been using Traefik as a reverse proxy for many years now. It’s an open-source proxy with powerful features and a great community. They were even kind enough to send me a t-shirt once for one of my blog posts, so I’m a bit biased. Recently I’ve been running Crowdsec as a firewall and intrusion detection system. Until now I’ve been content protecting against SSH brute force attacks and relying on their IP blocklists to protect against malicious actors. But now that I host more of my own services on VMs behind Traefik, I decided it was time to protect against HTTP attacks as well. Fortunately, Crowdsec has facilities for handling Traefik HTTP traffic, and in this post I’ll show you my setup.

The Traefik access log

To protect against common HTTP brute force attacks, we need to be able to detect them. Crowdsec is largely based around parsing and analysing logs, tracking the number of events over time to determine the IP addresses that are most likely to be malicious actors. Traefik has a built-in access log that can be used to feed into Crowdsec. Some of the most relevant fields in the access logs are shown below. By extracting these fields Crowdsec can determine the IP address of the malicious actor and take action based on the patterns it finds.

{
    "ClientHost": "<client-ip>",
    "RequestMethod": "GET",
    "RequestPath": "/some/path",
    "RequestHost": "<external-domain>",
    "DownstreamStatus": 400,
    "RequestProtocol": "HTTP/2.0",
    "time": "2026-01-25T16:18:38Z"
}

To configure the access log, we need to set the accessLog section in the Traefik configuration file.

accessLog:
  filePath: /opt/traefik/logs/access.log
  format: json

If you run Traefik inside Docker, you’ll want to mount the log file to the host filesystem so it can be read by Crowdsec.

volumes:
  - /opt/traefik/logs/access.log:/opt/traefik/logs/access.log

Log rotation

Traefik only supports rotation of its own service logs, not the access log. In order to rotate the access log you can use the logrotate service to rotate the log file. Here is an example configuration for the logrotate service:

/opt/traefik/logs/access.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 root root
    sharedscripts
    postrotate
        # Send USR1 signal to Traefik to reopen log file
        docker kill --signal="USR1" $(docker ps | grep traefik | awk '{print $1}') 2>/dev/null || true
    endscript
}

I use the postrotate directive to send a USR1 signal to Traefik to reopen the log file. This is necessary because logrotate will move the log file to access.log.1, access.log.2, etc. Traefik would append to its open file descriptor indefinitely, causing it to write to the moved file under access.log.1, access.log.2, etc. Sending the USR1 signal to Traefik causes it to reopen the log file and start appending to the new file.

Setting up Crowdsec to analyze Traefik access logs

Crowdsec has a collection that provides the log parser for Traefik access logs and includes base HTTP scenarios to protect against. These scenarios include:

aggressive crawl detection
scanning/probing detection
bad user-agent detection
path traversal detection
sensitive data access attempts detection
SQL injection detection
administration interface probing detection

You can install it using the following command:

cscli collections install crowdsecurity/traefik

For Crowdsec to analyse the logs, you need to set up an acquisition target for the logs, for example at /etc/crowdsec/acquis.d/traefik.yaml:

filenames:
  - /opt/traefik/logs/access.log
labels:
  type: traefik

The scenarios should be enabled by default when installing the collection. You can verify this by checking the cscli scenarios list command.

After enabling the scenarios on multiple VMs of mine I’ve already seen a few malicious actors get banned:

A word of caution

The default http-probing scenario is quite aggressive, banning IPs that generate too many 400-range responses in a short period of time. I managed to get my own IP banned by this scenario after pushing Docker images to my Forgejo registry. This was caused by docker push making multiple requests to check for the existence of layers, resulting in a cascade of 404 responses that triggered the scenario.

To avoid this, you can either disable the specific scenario or create a custom configuration to exclude certain paths or adjust the thresholds. You could define a custom /etc/crowdsec/scenarios/http-probing-override.yaml file that excludes the /v2/ path from the scenario and allows up to 100 400-range requests per IP within 30 seconds:

type: leaky
name: crowdsecurity/http-probing-override
description: "Override for http-probing scenario with much higher threshold to reduce false positives"
filter: "evt.Meta.service == 'http' && evt.Meta.http_status in ['404', '403', '400'] && evt.Parsed.static_ressource == 'false' && !(evt.Parsed.http_path matches '^/v2/')"
leakspeed: 30s
capacity: 100
groupby: "evt.Meta.source_ip + evt.Meta.http_target_fqdn"
distinct: "evt.Parsed.http_path"
blackhole: 5m
labels:
  service: http
  type: scan

Conclusion

With a few quick steps, I was able to set up Crowdsec to ban malicious actors from accessing my VMs. Note that this setup is not a replacement for a Web Application Firewall (WAF), as it does not block attacks at the application layer. Instead, it analyses the logs and takes action based on the patterns it finds. This makes it a good complement to a WAF, providing additional protection against common HTTP attacks. Another positive aspect of this setup is that Crowdsec shares the IP of the attackers with the entire Crowdsec network. This way other Crowdsec users can benefit from the information and take action against the same malicious actors without first having to identify them.

If you need a WAF, you can check out Crowdsec WAF, which sits as a component inside your Traefik proxy. This allows you to block attacks at the application layer and block malicious actors in the proxy layer. Note that blocking at the network layer via your firewall is better, as traffic never reaches the proxy layer and thus reduces computational overhead. Fortunately, Crowdsec WAF can be used in conjunction with your firewall to provide additional protection.

Code Monkey Clash - A Code Competition Workshop Game

Tue, 02 Jul 2024 00:00:00 GMT

Code Monkey Clash is an open source code competition workshop game I’ve been working on. A little more than a decade ago I first played the Extreme Startup workshop game. I liked it so much that I also hosted an Extreme Startup workshop of my own during my time at university. Later a colleague hosted a session at work, and I started thinking it would be fun to renew the experience. Code Monkey Clash is my take on the game with new questions and an implementation from scratch in TypeScript.

The concept and gameplay

The game is a workshop where participants are challenged to solve questions by writing code. Each player creates an HTTP server that accepts a GET request with a question, and responds with the answer. The game master runs a server that sends questions to the players and evaluates the answers. Based on the answer, or the lack thereof, the game server assigns points to the players, and adjusts the question interval. The game is played in rounds, and the player with the most points at the end of the game wins.

The questions ranges from simpler word problems to more complex algorithmic challenges. They are expressed in a relatively natural language way, requiring users to do some string parsing. The questions are designed to be solvable in a few minutes, but get progressively more difficult as the game progresses. The game randomizes the input to questions to avoid players hard-coding answers. A question might be something like:

What is the sum of the numbers: 1, 5, 12, 13

In order to keep the feel of the original Extreme Startup game I’ve borrowed two key algorithms:

The sliding window algorithm for question selection
The linear adjustment algorithm for question rate

The sliding window algorithm is fairly simple:

const windowEnd = round * 2 - 1;
const windowStart = Math.max(windowEnd - 4, 0);

The algorithm produces this sequence of windows:

Round	Start	End
1	0	1
2	0	3
3	1	5
4	3	7
5	5	9

This has some key benefits:

The game starts slowly the first three rounds, gradually increasing from two to four questions in the window. After that it slides the window by two questions each round to not overwhelm the players and allow them to keep earning points.
Players who lost out on a question in earlier rounds can still catch up in later rounds as the window slides. Later rounds are worth more points, so it’s possible to catch up even if you’re behind.
The game master can adjust the difficulty of the game by how fast each round progresses. If the players are struggling, the game master can slow down the window progression to give them more time to catch up.

Question rate and difficulty

The game loop for each player essentially consist of the following steps:

Game server immediately schedules the next question (to avoid drift)
Game server picks a random question from the window and gets random input
Game server sends the question to the player
The player responds with the answer, if they have one
Based on the answer, or the lack thereof, the game server assigns points to the player
Game server adjusts the question interval based on the player’s answer
When the next question is due, it starts from step 1.

The question interval uses the linear adjustment algorithm from the original game. It adjusts the question rate by 100 ms deltas based on the player’s last answer. Successful players will see their question rate increase, while struggling players will see it decrease.

A more advanced technique looking at score averages were tried out, but did not feel as good during simulation. The linear adjustment technique is simple and works well in practice. I might revisit the question interval adjustments in the future, but for now it’s good enough.

The technology

The Code Monkey Clash game is implemented from scratch as I did not want to just fork the original Extreme Startup game. My objective was to learn more about some of the new frontend technologies I had not used before. The game is essentially a clone of the original game, but with a new set of questions and a new implementation.

Bun - The all-in-one JavaScript runtime & toolkit

Bun is ”[…] an all-in-one JavaScript runtime & toolkit designed for speed, complete with a bundler, test runner, and Node.js-compatible package manager.” As Bun supports TypeScript out of the box, it seemed like a nice fit for the project as I would not have to compile the code or use ts-node to run my code. It also supports bundling and minification which meant I could write client side code in TypeScript as well without having to worry about the build pipeline.

ElysiaJS - Ergonomic Framework for Humans

As I was writing this project in TypeScript I wanted to use a web framework that supports TypeScript well. I had read a bit about ElysiaJS and decided to give it a try. ElysiaJS’ tagline is “Ergonomic Framework for Humans” and is developed by a single developer in Thailand. The framework somewhat resembles Express and Koa, but focuses on performance and TypeScript inference to improve developer experience. Where other frameworks often require manual typing for middleware and request data, Elysia infer types from your schemas and middleware handlers. If you structure the code correctly you don’t have to typecast or manually type almost anything, which is a pretty big win! As an added benefit you also get a type-safe HTTP client library for free, but I did not use it in this project as I’m returning HTML.

HTMX - High power tools for HTML

I’ve often used the RESTish backend paired with a single page application frontend pattern when developing web applications in the past. In the event that I needed server side rendering for SEO I would often use Next.js or my own custom React server side rendering. This works well for most applications, and having React at your fingertips in the client allows for some pretty dynamic applications.

For this project I wanted to get away from the heavy client side frameworks and try a more traditional server side rendered application. I still needed some client side interactivity for the game, but I wanted to keep it as simple as possible. A buddy of mine was talking about HTMX and how simple it made the mental model for web development. I decided to give it a try and I must say I’m pretty impressed. HTMX allows you to add dynamic behavior to your HTML with attributes. Your server only needs to return HTML or HTML fragments, and you mostly don’t need to write any JavaScript. Instead HTMX uses the hx-* attributes to add behavior to your HTML elements, making server requests and updating the DOM as needed.

In the signup form above, only the signup form HTML with errors would be returned in the event of a validation error. HTMX fetches the HTML fragment and replaces the form on the page with the new form without the browser having to do a page load. This makes the application feel very snappy and responsive, while keeping most of the logic on the server side.

Tailwind CSS and DaisyUI

I’ve been using Tailwind CSS for a while now and I really like it. It’s a utility-first CSS framework that allows you to build complex designs without having to write custom CSS. The utility-first approach feels like functional programming. Each utility composes neatly with other utilities, and you can build complex designs by combining simple utilities. I’ve not had this feeling with any other way of writing CSS, and I’m really happy with the results I’ve gotten using Tailwind CSS in the past.

For this project I decided to try DaisyUI, a plugin for Tailwind CSS that adds components and adjusts the default theme. By using DaisyUI I was able to quickly build a nice looking UI without having to write much custom CSS for buttons, and other native HTML elements. While I have been wary of using component libraries like Bootstrap, DaisyUI seems to fit well with the utility first approach of Tailwind CSS. Where you lack components you can quickly build them with Tailwind CSS utilities. And because DaisyUI is built on top of Tailwind CSS it is easily customizable with Tailwind’s own configuration system.

Server sent events for live updates

The nature of the game means that events happen pretty much all the time. Players sign up, questions are sent, answers are received, and points are assigned. I wanted to surface this liveness to the game master and players so that they do not have to refresh the page to see updates. In theory one could poll the backend for changes and have it re-render HTMX fragments. This however would have been unecessarily chatty, not to mention taxing on the server!

Instead I decided to use Server Sent Events to push updates to the clients. Fortunately for me there is a server sent events extension for HTMX that adds support. This allowed me to push updates to the clients as they happen, and the clients would update the UI as needed. As with everything else in HTMX, with SSE the HTML (fragments) are rendered on the server, and in this case pushed to the client as events. This way when a player joins, the relevant player table row is rendered server side, then pushed to the client’s admin dashboard via the SSE connection. On the client HTMX inserts the row into the table and the UI is updated accordingly!

For the Chart.js based scoreboard I needed to hack a bit as HTMX does not support updating the chart data directly. Instead I added a hidden span that initiates the SSE connection for the chart data, and then added client side logic to listen for that event.

On the server side:

// Use a hidden element to swap the chart data, don't actually swap json into the DOM
<span 
  class="hidden"
  sse-swap={`player-chart-${player.nick}`}
  hx-swap="none"
/>

Then on the client:

// On SSE messages do DOM-manipulation where necessary
htmx.on("htmx:sseMessage", (evt) => {
	// If the event is a player-chart event, update the chart with the new data
	if (
		evt instanceof CustomEvent &&
		evt.detail.type.startsWith("player-chart-")
	) {
		const nick = evt.detail.type.replace("player-chart-", "");
		const data = JSON.parse(evt.detail.data);
		const dataset = chart.data.datasets.find((d) => d.label === nick);

		if (dataset) {
			dataset.data.push(data);
			chart.update();
		}
	}
});

In the end I was able to push updates to the clients as they happened, and the clients would update the UI as needed. The scoreboard looks really nice and Chart.js animates new data points as they are added. All in all it was a bit tricky to figure out how to do this, but once I got it working it was pretty neat!

Conclusion

I’ve had a lot of fun building Code Monkey Clash. The Bun, ElysiaJS, HTMX and Tailwind CSS stack has been a joy to work with. If you’re familiar with TypeScript and JSX and want to build a server rendered application you can consider a similar stack. You might want to consider using a more supported framework than ElysiaJS, but it’s been a fun experiment. In terms of developer experience I’ve been pretty happy with this stack. I’ve also had a lot of fun coming up with questions for the game, and I’m looking forward to hosting a workshop with my colleagues.

Demystifying Docker – A cursory look beneath the magic

Sat, 21 Feb 2026 00:00:00 GMT

This post features interactive content not available in RSS. View on snorre.io for the full experience.

Containers might feel like lightweight VMs, but under the hood they’re something else entirely. Images, Containers, Volumes, networks and more are all built on top of fundamental concepts in Linux. This post attempts to demystify these concepts and reveal what is actually going on under the hood.

Containers vs VMs

If your frame of reference is virtual machines, it’s tempting to think of containers the same way—but they’re not. The diagram below illustrates the difference between the two concepts.

Virtual machines

Physical hardware

CPU, RAM, disk, NIC

Host OS / Kernel

Host kernel

Hypervisor

Virtualises hardware for guests (each VM)

VM 1

Virtual hardware

OS / kernel

Processes / apps

VM 2

Virtual hardware

OS / kernel

Processes / apps

VM 3

Virtual hardware

OS / kernel

Processes / apps

Containers (Docker)

Physical hardware

CPU, RAM, disk, NIC

Host OS / Kernel

Single shared kernel

Docker

Daemon, image layers, runtime

Container A

Process(es)

Filesystem isolation

Network isolation

Container B

Process(es)

Filesystem isolation

Network isolation

Container C

Process(es)

Filesystem isolation

Network isolation

Left: each VM gets emulated hardware and a full guest OS. Right: one kernel; containers are “jailed” processes with isolated filesystem, namespaces, and cgroups—no separate OS.

Virtual machines

Physical hardware runs a host kernel and OS, plus a hypervisor that emulates hardware for each guest VM. Each VM runs its own kernel and OS and is isolated from the host and from other VMs. From the guest OS’s perspective it’s running on hardware; it doesn’t know about the hypervisor or the host unless the hypervisor is configured to expose the host machine.

Containers

A container is just a regular Linux process (possibly with child processes) running on the host kernel, sharing the same OS. Isolation comes from kernel primitives like namespaces, cgroups, chroot, and capabilities:

Namespaces — pid (own process tree only), mount (own view of the filesystem), user (e.g. root-in-container → unprivileged on host), network (own interfaces, no visibility of host/others).
cgroups — limits and accounting for CPU, memory, disk I/O, and network.
chroot — the process’s root is a chosen directory; it cannot see or reach paths outside that subtree.
Capabilities — fine-grained privileges instead of full root.

Together these form a sandbox. The process is separated from the rest of the host without a separate kernel or OS.

Linux filesystem and permissions (primer)

Before we look at chroot and volume mounts, here is a quick primer on how Linux organizes files.

Linux filesystem and permissions (primer)

Paths start at / (root). Each file has permissions (owner/group/others) and ownership. Docker uses this when setting up container roots and volume mounts.

File tree with permissions

d rwx r-x r-x = dir owner group others

root of filesystem

drwxr-xr-x root:root

home/

user home dirs

drwxr-xr-x root:root

etc/

config files

drwxr-xr-x root:root

var/

variable data (logs, caches)

drwxr-xr-x root:root

tmp/

temporary, world-writable

drwxrwxrwx root:root

app/

your app (common in containers)

drwxr-xr-x myappuser:myappuser

r read w write x execute (dirs: enter)

Linux mounts (primer)

The single directory tree is built from many filesystems attached at mount points. The kernel keeps a mount table: what is mounted where. For example the root directory / is mounted from one filesystem on disk A, the home directory /home is mounted from a disk partition on disk B. Both /etc and /home are in the file tree next to each other, but they are actually on different filesystems on different disks.

Linux mounts (primer)

Mount points attach filesystems into the tree; the kernel's mount table records what is mounted where. Bind mount = same filesystem at another path.

Mount table (conceptual)

path ← mounted from (source)

rootfs (e.g. ext4 on sda1)

/home

ext4 on sda2

/mnt/backup

ext4 on sdb1

/mnt/data

bind from /data (same filesystem, other path)

chroot — setting a new root directory for a process

The process’s ”/” is a chosen directory on the host. It cannot see or reach anything above it.

Host filesystem

/
├── home/
├── etc/
├── var/
│   └── lib/
│       └── docker/
│           └── containers/
│               └── abc123…/  ← container root
│                   ├── app/
│                   ├── etc/
│                   └── bin/

What the process sees (inside container)

/  ← container root
├── app/
├── etc/
└── bin/

Paths like /etc or /app refer only to files under the container root.

Mount namespace — isolated mount table

The container has its own mount table: it only sees the mounts Docker set up (overlay for /, plus any volume mounts). It cannot see or unmount the host’s mounts. Volume mapping is done by adding entries to this isolated table.

Host mount table

/ — rootfs

/home — disk partition

/var/lib/docker/... — volumes, overlay

Container mount table (isolated)

/ — overlay (image + RW layer)

/app/data — volume (bind or named)

PID namespace — isolated process tree

The container only sees its own processes. PID 1 inside the container is a different process on the host; host PIDs are invisible.

Host view

PID 1 — init

PID 100 — systemd

PID 200 — sshd

PID 5000 — dockerd

PID 5001 — app (container)

PID 5002 — worker

PID 5003 — worker

Container view (same processes)

PID 1 — app

PID 2 — worker

PID 3 — worker

Only these processes exist from the container’s perspective. Host PIDs 1, 100, 5000 are invisible. This is why you can’t see host processes or kill host processes from inside a container.

Docker images

Imagine you run the following command to build a new image from a Dockerfile:

docker build -t my-image:1.0.0 .

Docker creates a new image with the name my-image and tags it with the version 1.0.0. It uses your Dockerfile adding the files and folders you’ve specified and running the commands you’ve specified. A Docker image is not like a VM image, it is not a single file system snapshot with all your files and folders. Instead it is made up of layers. Dockerfile instructions like RUN, COPY, and ADD create new layers. Each layer is a read only archive that contains the files added, modified, or deleted since the previous layer. Other instructions like ENV, WORKDIR, and CMD do not create new layers, instead they add metadata to the image.

After running the docker build command you’ve now got an image entry that points at the last layer in the image, sha256:000080bf7f36. That layer points to the second last layer in the image (sha256:00006e913295), and so on until it reaches the first layer in the image (sha256:00007bedb7f3).

Because each layer is immutable and is identified by its instruction input, Docker can efficiently reuse layers when building images. It is important not to confuse this with the idea of reproducible builds, which is a different concept. Docker does not guarantee the same Dockerfile instruction will always produce the same output, this is up to the author of the Dockerfile to ensure. Adding a single COPY or CMD (or changing one line) means every layer before that line in the image is unchanged. This allows Docker to reuse the existing layer archive file for those layers and only rebuild the changed layer and any below.

FROM instruction

The FROM instruction is special in that it essentially links your first image layer to a previous image’s layers. This way you can easily build on top of existing images like FROM ubuntu:20.04 or FROM node:18-alpine. This is also what enables the idea of multi-stage builds, where you define multiple images in a single Dockerfile which inherit from each other.

Starting a container from an image

First, the image’s read-only layers are merged into a single coherent read-only filesystem (e.g. via overlay). This is done by overlaying the layers on top of each other.

When you run a container, Docker then creates a thin read-write layer on top of that, sets up namespaces and cgroups, and runs the entrypoint. The container is that writable layer plus the image layers—ephemeral unless you commit or use volumes.

Volume mapping (bind mounts and named volumes)

To persist or share data, you mount a path from the host (or a Docker-managed volume) into the container. Bind mounts and named volumes are implemented by adding mounts in the container’s mount namespace. With -v /host/path:/container/path, the host directory appears in the container’s mount table at the chosen path; writes in the container go directly to the host path. With -v myvol:/container/path, Docker creates a directory under e.g. /var/lib/docker/volumes/ and mounts it; the data survives docker rm and can be shared between containers. Use the animation below to step through no volume, bind mount, and named volume.

User mapping

When a container writes to a bind-mounted host path, files are created with the same UID/GID the process has—and by default that UID is the same inside and outside the container.

Host user list (examples)

On the host, each user has a numeric UID. The kernel uses these IDs for file ownership and process privileges. Containers share the same kernel, so the same UID list applies unless you use user namespaces.

0root

1daemon

……

1000alice

1001bob

No user mapping (default)

By default, Docker does not use a user namespace. The container process runs with the same numeric UID on the host as inside the container. What matters is the number: root (0) in the container is root on the host; UID 1000 in the container corresponds to host UID 1000 (e.g. alice). The username inside the container (from the image’s /etc/passwd) does not have to match the host—same UID, different names.

Container runs as

root (UID 0)

→ Host sees: root (0)

Container runs as

appuser (UID 1000)

→ Host sees: alice (1000)

Same UID; names can differ (container "appuser" ≠ host "alice").

With user mapping (user namespace)

You can enable user namespace remapping (e.g. --userns-remap=alice in the daemon, or /etc/subuid and /etc/subgid). Container UIDs are then mapped into a range on the host (e.g. 100000–165535). Root inside the container is no longer root on the host.

Example mapping (e.g. alice:100000:65536 in /etc/subuid)

Container 0 → Host 100000

Container 1 → Host 100001

Container 1000 → Host 101000

Container runs as

root (UID 0)

→ Host sees: 100000 (unprivileged)

Container runs as

UID 1000

→ Host sees: 101000

Docker networking

Network isolation for containers comes from the network namespace: the container gets its own view of the network stack. We start with what that stack looks like on Linux, then how Docker uses it—port mapping, container-to-container, and host mode.

The Linux network stack (primer)

A typical Linux host has physical interfaces (e.g. eth0), a loopback device (lo), and the kernel’s networking stack (routing, iptables/nftables, etc.). Docker’s networking builds on these same building blocks.

Linux network stack — how the pieces fit together

A packet flows through the stack. Incoming: interface → routing → iptables → app. Outgoing: app → (DNS for hostnames) → routing → iptables → interface.

Network namespace — own interfaces only

The container has its own network stack: loopback and a virtual interface (e.g. veth) paired with the host. It cannot see the host’s physical NICs or other containers’ interfaces.

Host

eth0 (physical)

docker0 (bridge)

veth0abc ←→ container's eth0

Container

lo (127.0.0.1)

eth0 (e.g. 172.17.0.2)

Only these interfaces exist. No visibility of host eth0 or other containers.

Port mapping (host :8080 → container :8080)

When you publish a port with -p 8080:8080, the host listens on 8080 and uses iptables/nftables DNAT to rewrite the destination to the container’s IP and port. The packet is then forwarded via docker0 and the veth pair into the container. Use the animation below to step through the path.

Networking between containers

When two containers are on the same user-defined network (e.g. docker network create mynet), they each get a veth pair onto the same bridge on the host. The bridge is outside the containers—it lives in the host network namespace. Packets between containers go from one container’s eth0 through its veth to the bridge, then through the other veth to the other container’s eth0; no NAT, just bridge forwarding. On user-defined networks, Docker runs an embedded DNS so containers can reach each other by name (e.g. “ping app” resolves “app” to that container’s IP). The animation below shows the layout (which interface belongs to which container, and that the bridge is on the host), then DNS resolution, then the packet path.

Host network mode

With --network host, the container does not get its own network namespace. It uses the host’s network stack directly—no veth pair, no bridge, no NAT. The container sees the host’s eth0 and lo; if it listens on port 8080, it binds to the host’s 8080. Port mapping (-p) is ignored. This mode is useful when you need maximum performance or direct access to host network interfaces (e.g. for certain monitoring or high-throughput workloads), but it removes network isolation.

Why Docker? The daemon and the engine

If containers are “just” processes plus namespaces, cgroups, and chroot, why do we need Docker at all? You could in principle create a container by hand: use unshare to create new namespaces, chroot into a rootfs, configure cgroups, and run your process. The reason Docker exists is that it orchestrates all of these primitives and adds a consistent workflow around images, networking, and storage.

When you run docker run or docker build, the docker command-line tool does not perform the work itself. It talks to the Docker daemon (often called the Docker engine)—a long-running process (e.g. dockerd) that runs on the host, usually with root privileges. The daemon is responsible for:

Images and layers — Storing and resolving image manifests and layer blobs, building images from Dockerfiles (running each instruction in a temporary container, committing layers), and managing the content-addressable layer store (e.g. under /var/lib/docker).
Containers — Creating the container’s filesystem (e.g. overlay merge of image layers plus a writable layer), creating the namespaces (pid, mount, network, user, etc.), applying cgroup limits, setting up the rootfs and mounts, and starting the entrypoint process. On shutdown, it cleans up namespaces, mounts, and the writable layer unless you use volumes or commit.
Networking — Creating and attaching virtual interfaces (veth pairs), bridges (e.g. docker0), and iptables/nftables rules for port publishing and container-to-container communication. It also runs an embedded DNS for user-defined networks so containers can resolve each other by name.
Volumes — Creating and mounting bind mounts or named volumes into the container’s mount namespace and managing volume lifecycle.

So the engine is the daemon plus the low-level plumbing (containerd, runc, etc., depending on your Docker version). The CLI is a client; the daemon is the single place that owns image and container state and ties all the Linux primitives together. Without it, you’d be manually creating namespaces, mounting overlays, and wiring cgroups and networks every time. Docker gives you a single API, a portable image format, layer caching, and declarative Dockerfiles so you don’t have to script those steps yourself.

Alternatives

Because containers are just processes plus Linux primitives, the same workflow — images, layers, run, build — can be implemented by different tools. Podman is a prominent alternative. It is daemonless: there is no long-running podman process. When you run podman run or podman build, the Podman CLI (or its subprocesses) sets up namespaces, overlay mounts, and cgroups directly and then exits or hands the container to a short-lived conmon process for monitoring. That avoids a single root-owned daemon and can make it easier to run containers as an unprivileged user (rootless mode). Podman is CLI-compatible with Docker for many commands (podman run accepts the same flags as docker run), and it uses the same image format (OCI) and can pull from the same registries. So existing Docker images and Dockerfiles usually work as-is with Podman.

Summary

Containers are not lightweight VMs. They are processes on the host kernel, isolated by namespaces (pid, mount, user, network), cgroups, chroot, and capabilities. The Docker daemon (engine) is what ties it all together. The CLI talks to the daemon, which owns images, layers, container lifecycle, networking, and volumes. It orchestrates the Linux primitives so you get a single API and portable image format instead of scripting namespaces and mounts by hand. An image is a stack of read-only layers; a running container adds a thin writable layer and uses overlay (or similar) to present a single filesystem. Volumes and port mapping are just bind mounts and NAT rules in the container’s mount and network namespaces. Once you see Docker as a tidy wrapper around these Linux primitives, the “magic” becomes predictable: same kernel, same building blocks, just a different view of the system.

A likes feature with Cloudflare D1 and Astro API endpoints

Mon, 06 May 2024 00:00:00 GMT

I’ve always had a goal of keeping Snorre.io a privacy-friendly website. However, this privacy-friendly approach does come with a trade-off. It means that I have a limited amount of insight into what content people enjoy the most. This is because I prioritize your privacy over extensive data collection. Cloudflare provides some statistics for my domain, including the number of unique visitors the domain receives per day, week, and month. If Cloudflare is to be believed, at the time of this writing I’ve had about 3.3 thousand unique visitors in the past month. I have yet to learn which blog posts they read. To help me gauge the interest in my content, I needed some form of like system, so why not build one?

The likes feature

You have undoubtedly seen them, the claps emoji adorning a blog post on Medium.com or a heart or thumbs up on some blog. The number of likes indicates how many people might have found the content worth reading, watching, or listening to. I had a few requirements I wanted to satisfy when building my like system. It should:

be easy to use, but not in your face
not require user registration or collection of personal information
prevent like-spam
allow liking blog posts, recordings, beer recipes, etc

Based on these requirements, it was clear I needed some form of backend capable of storing the likes. Each like would be associated with a collection (e.g. blog), slug (content identifier), and session ID. The session ID would tie a like to a user session, but not allow for the identification of a specific user.

Implementation

As I am already running on Cloudflare, I found Cloudflare D1 a compelling tool to persist the likes. D1 is a global “serverless” relational database built by Cloudflare. It uses the SQLite dialect for its SQL queries making it easy to use SQLite for local development. They automatically handle creating read replicas based on your Cloudflare Pages or Worker DB usage. This way, reads should always be reasonably fast, while writes must be routed to the write instance.

D1 has a low-level client library for JavaScript, but I wanted something to help me with migrations and query building. I’m normally very skeptical of ORMs, but I recalled having read good things about Drizzle in the past. Their tag-line really resonated with me:

Drizzle is a good friend who’s there for you when necessary and doesn’t bother when you need some space.

This is exactly what I want from an ORM. It should not force me to do things in a specific way, but it should provide me helpers when I need them.

DB schema

Given the aforementioned requirements, I ended up with a simple schema for the likes table:

import { primaryKey, sqliteTable, text, unique } from 'drizzle-orm/sqlite-core';

export const likes = sqliteTable('likes', {
  sessionId: text('session_id'),
  collection: text('collection'),
  slug: text('slug'),
}, likes => {
  return {
    pk: primaryKey({ columns: [likes.sessionId, likes.collection, likes.slug] }),
  }
});

This schema defines a likes table with three columns:

sessionId is a string that identifies a user session
collection is a string that identifies the collection of content the like is associated with
slug is a string that identifies the specific content the like is associated with

The primary key is a composite key of all the columns to ensure that a user can only like a piece of content once per session. I could have added a createdAt column to store the time of the like, but this seems unnecessary for now.

API endpoints

I did not want to change away from static site generation, so I needed a way to interact with the D1 database from my Astro site. Fortunately Astro provides a concept called adapters that allows you to run Astro in various serverless environments. The Cloudflare adapter is a perfect fit for my use case. It supports a hybrid rendering approach where the site is mostly static, but with some on-demand rendered pages. I could then keep my site as is, but add a few dynamic API page routes to handle likes!

I ended up with a /pages/api/likes.ts file with a GET and POST function to fetch and add/remove likes.

The post function loads the JSON body from the request. Then checks if the user has a session ID cookie, and creates one if not. Finally, it toggles the like for the given slug and collection based on the session ID before returning a success response.

export async function POST(context: APIContext) {
  const runtime = context.locals.runtime;
  const d1 = runtime.env.DB;
  const db = drizzle(d1, { schema: { likes } });

  const { slug = "", collection = "" } = (await context.request.json()) as {
    slug?: string;
    collection?: string;
  };

  if (!context.cookies.has("sessionId")) {
    context.cookies.set("sessionId", crypto.randomUUID(), {
      httpOnly: true,
      secure: true,
      sameSite: "strict",
      path: "/",
      maxAge: 60 * 60 * 24 * 365,
    });
  }

  const sessionId = context.cookies.get("sessionId")!.value;

  const liked =
    ((
      await db
        .select({ likes: count() })
        .from(likes)
        .where(and(eq(likes.slug, slug), eq(likes.sessionId, sessionId)))
    ).at(0)?.likes ?? 0) > 0;

  if (liked) {
    await db
      .delete(likes)
      .where(
        and(
          eq(likes.slug, slug),
          eq(likes.collection, collection),
          eq(likes.sessionId, sessionId),
        ),
      )
      .execute();
  } else {
    await db.insert(likes).values({ slug, sessionId, collection }).execute();
  }

  return new Response(JSON.stringify({ success: true }));
}

The GET functions is no more complex. It fetches the slug and collection from the query parameters. Then it fetches the session ID from the cookie. It then queries the database for the number of likes and whether the user has liked the content before returning the result. Because we don’t want to unecessarily add cookies to a user’s browser, we never set a cookie for a data retrieval request.

export async function GET(context: APIContext) {
  const runtime = context.locals.runtime;
  const d1 = runtime.env.DB;
  const db = drizzle(d1, { schema: { likes } });

  // Get slug and collection from query parameters
  const slug = context.url.searchParams.get("slug") ?? "";
  const collection = context.url.searchParams.get("collection") ?? "";
  // Get the session ID from the cookie
  const sessionId = context.cookies.get("sessionId")?.value;

  console.log("Slug", slug, "Collection", collection);

  const likesCount = (
    await db
      .select({ likes: count() })
      .from(likes)
      .where(and(eq(likes.slug, slug), eq(likes.collection, collection)))
  ).at(0)?.likes;
  let liked = false;
  if (sessionId) {
    const sessionCount =
      (
        await db
          .select({ likes: count() })
          .from(likes)
          .where(and(eq(likes.slug, slug), eq(likes.sessionId, sessionId)))
      ).at(0)?.likes ?? 0;
    liked = sessionCount > 0;
  }

  return new Response(JSON.stringify({ likes: likesCount, liked }), {
    headers: {
      "Content-Type": "application/json",
    },
  });
}

Client side Like counter and button

To allow dynamic rendering of likes count and allowing users to like content I added a simple Likes component to my Astro setup. Astro supports client side rendering using something called Astro Islands. I decided to use Solid.js for the client side rendering as I was already using it for my comments system. The whole implementation ended up being 35 lines of code:

import { type Component, createResource } from "solid-js";
import { VsHeart, VsHeartFilled } from "solid-icons/vs";

interface LikesProps {
  slug: string;
  collection: string;
}

const fetchLikes = async ({slug, collection}: LikesProps) => {
  const res = await fetch(`/api/likes?slug=${slug}&collection=${collection}`);
  const data = await res.json() as { likes: number, liked: boolean };
  return data;
}

export const Likes: Component<LikesProps> = ({ slug, collection }) => {
  const [likes, { refetch }] = createResource(() => ({ slug, collection }), fetchLikes);

  return (
    <div class="flex flex-row gap-2 text-stone-800 dark:text-stone-400">
      <span>{likes()?.likes}</span>
      <button
        class=""
        onClick={async () => {
          await fetch(`/api/likes`, {
            method: "POST",
            body: JSON.stringify({ slug, collection }),
          });
          refetch()
        }}
      >
        {likes()?.liked ? <VsHeartFilled /> : <VsHeart />}
      </button>
    </div>
  );
}

The whole thing ends up compiling to less than 20 kilobytes of JavaScript (compressed) having a negligible impact on the performance of the site.

Tooling, local environment and Cloudflare

To develop and test this locally was a bit more complex than I had hoped. Fortunately I found an informative blog post by Kevin Kipp that helped me get started. Note that the Astro config for the Cloudflare adapter has changed since Kevin’s post. As of Astro Cloudflare adapter version 10.2.5 you want to use the following configuration:

{
  adapter: cloudflare({
    platformProxy: {
      enabled: true,
      persist: true,
    }
  })
}

Feel free to check out my source code for a complete example of how to use Cloudflare D1 with Astro.

Conclusion

I now have a simple like system that allows me to gauge the interest in my content. It is privacy-friendly, easy to use, and does not require user registration. The like system is a bit vulnerable to abuse as you can like content multiple times by clearing your cookies. But this is a risk I’m willing to take to keep the site privacy-friendly.

If you enjoyed this blog post, please give it a like!

Musical recordings on Snorre.io

Sun, 14 Apr 2024 00:00:00 GMT

I bought a Focusrite Scarlett 4i4 Gen 4 as a christmas present to myself last year. I’ve always enjoyed improvising music on my synthesizers, but I never really had the gear to record something properly. The situation changed with the Scarlett soundcard and I’ve now started to practice recording my synths with the GarageBand app.

So far I’ve recorded one track and I’m planning to record more in the future. I do not predict a regular schedule or high musical talent for these recordings, but I hope you’ll enjoy them anyway. If you find them interesting or enjoyable, please let me know and I’ll be more motivated to continue recording and sharing them.

Distributing my music

For now I’ll be sharing my music on two platforms: Soundcloud and my own website. Soundcloud gives me a good platform to get a bit of reach and allows people to provide feedback. It is however not privacy friendly and I want to make sure that my music is available to everyone! That’s why I’ll also be sharing my music on my own website as well.

Currently I’m using the standard HTML5 audio player to allow playback of the music straight from my website. There are also download links to the music files in various formats if you want to listen to them in a player of your choice. The web audio player will select the best format for your device if you choose to play the music directly from my website.

I convert my recordings to the various formats using a simple ffmpeg script. I’ve added a function to my shell configuration to make this easier.

# Convert wav files to other audio formats
function convert_audio() {
    local input="$1"
    local basename="${input:r}"
    
    ffmpeg -i "$input" \
           -c:a flac "${basename}.flac" \
           -c:a libmp3lame -qscale:a 2 "${basename}.mp3" \
           -c:a libvorbis -qscale:a 4 "${basename}.ogg" \
           -c:a aac -b:a 192k "${basename}.m4a"
}

# Create an alias to the function
alias convert_audio='convert_audio'

Including an audio player on a web page is not so complicated. My code for achieving this with Astro boils down to the following:

<audio controls class="w-full" preload="none">
  {track.downloads.map((download) => {
    return (
      <source
        src={`/downloads/music/${download.url}`}
        type={`audio/${download.format}`}
      />
    );
  })}
</audio>

Here I’m using the track object that I pass to the page to render the audio player. A track can contain multiple downloads, essentially different formats of the same recording. For each track I include a source element with the URL to the music file and the format of the file. The browser will then select the best format to play based on the capabilities of the device, neat!

In the future I might build a Soundcloud like player for my website using [Wavesurfer][https://wavesurfer.xyz/]. This is a bigger task and I’ll have to see if I can find the time to do it. For now I hope you enjoy the music as it is provided.

Licensing

See privacy and license page for information on how you can use my music. I’ve used the Creative Commons Attribution 4.0 International License for the other content on my website. For my music I’ve chosen to keep the rights to myself, but I’m happy to share it with you for personal listening. All songs can be listened to and downloaded for free on my website in various open DRM-free formats. If you want to use my music in any other way please contact me and we can discuss it further.