Everybody loves stuff that just work 🤟🏽. But getting there is a long journey!
Discussing with some people at the Nextcloud Conference in Berlin last week, a recurrent pain is app compatibility
with each new server version. Nextcloud releases new versions every 4 months. Then, each individual app's maintainer
needs to test if they work with the new version, fix any possible breakage, and manually self-report as compatible.
A common reason for an app to break is that it uses APIs that are not marked as stable. So I tried to help the
discussion by
bringing some data with the ANEBA prototype 🦠.
It's nice when some classes of bugs cannot be fabricated. For example, some languages like Kotlin and Rust bake the
concept of "empty" especially in the language and standard library to remind and force you (the developer) to deal with
the case when something is missing.
This post is about a library I wrote that brings a similar compile-verified
assurance to URL construction when writing web servers in Rust. 🦀
The problem it solves is: in a web server, you usually declare some "handlers" (like web pages, assets or
api endpoints) and hook them to a global path router. It works well, but what about when you want to link from one page
to another? How to do that correctly and let the compiler help you keep all links correct as your app evolves?
To illustrate, consider this basic example using the library axum in Rust to create a website with a home page and a
dynamic "posts" page, with the home linking to a post:
use axum::extract::Path;
use axum::response::Html;
use axum::routing::get;
#[tokio::main]
async fn main() {
let app = axum::Router::new()
.route("/", get(home))
.route("/post/{id}", get(about));
let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
.await
.unwrap();
axum::serve(listener, app).await.unwrap();
}
async fn home() -> Html<String> {
Html("Hello! <a href='/post/17'>See post</a>".to_string())
}
async fn about(Path(id): Path<u32>) -> Html<String> {
Html(format!("Thanks for visiting post {}", id))
}
This year I'm taking a sabbatical pause to explore the world of tech, especially in Europe. So I've searched for a way
to keep a record of what I'm up to: the cities I visit, the events I attend, the people I talk to, the tech I learn...
This way, my family can follow my adventures, and I can help my forgetful brain to keep these memories for longer!
The obvious answer is a shared photo album. But none of the apps that I've tested had quite all that I wanted:
I don't want to use a cloud service like Photo Circle, Google Photos, iCloud, etc because I want to own my data and
host it in my homelab
I use Nextcloud and Memories for my private photos, and they work well. But not so much for sharing albums with people
outside the instance that do not have an account. Also, the UI/UX don't work well for my intended usage.
Immich has shared albums with comments, which I wanted, but it doesn't seem to notify users on new comments.
In the past, I've worked with a system that produced hundreds of thousands of compressed JSON files every day, each
weighting around 1MiB uncompressed. We wanted to ingest all that data into datalake for debug and analytics, and for
that we've used Spark to append to a delta table. Conceptually, the flow is very straight forward:
a job runs every day
it lists the new files that need to be ingested
those files are read
they are decompressed, parsed and encoded as parquet
these new parquet files are written to the table
In my last blog post,
I've dissected the delta table format and
showed
that it's basically just a bunch of JSON and parquet files. Part of that knowledge will be useful for this post, so
go take a look there if you want. I'll wait here :)
At its core, the Spark script to do this ETL is (in Python):
In Rust, the tokio's ecosystem has a fundamental crate called bytes that abstracts
and helps dealing with bytes (you don't say!). I've indirectly used it a billion times and I thought that I had a good
mental model of how it worked.
So, in the spirit of the "decrusting" series
by the excellent Jon Gjengset, I've decided to peek behind the curtains to understand more what axum, tokio, hyper and
the kind do to them bytes! The code is well written, but surprisingly complex. I understand now what it does, but I
still don't fully grasp why it does some things in a certain way.
I'm ready to share with you my discoveries. I hope that you are sitting, laying or squatting comfortably. This is the
first post in a small series. I'm legally required by my marketing department to remind you that you can subscribe to
my low-traffic newsletter, so that you'll know when new posts are up!
A quick note before we start: this posts is based on the current bytes version 1.11.1.
This blog is written in Rust, and I wanted a way to reload the web pages automatically while I change the posts'
contents, styles, etc. This is common-place with JavaScript frameworks, but not automatic in the Rust land. So I've
embarked on a side quest to achieve just that: the "type and auto-reload" experience. In the end, I was surprised to
learn a bit more about sockets and processes in Linux.
This post is a note to myself about these nuggets that I've learned and to share the solution. It may be helpful for
future me and I hope for someonelse out there.
