This year I'm taking a sabbatical pause to explore the world of tech, especially in Europe. So I've searched for a way
to keep a record of what I'm up to: the cities I visit, the events I attend, the people I talk to, the tech I learn...
This way, my family can follow my adventures, and I can help my forgetful brain to keep these memories for longer!
The obvious answer is a shared photo album. But none of the apps that I've tested had quite all that I wanted:
I don't want to use a cloud service like Photo Circle, Google Photos, iCloud, etc because I want to own my data and
host it in my homelab
I use Nextcloud and Memories for my private photos, and they work well. But not so much for sharing albums with people
outside the instance that do not have an account. Also, the UI/UX don't work well for my intended usage.
Immich has shared albums with comments, which I wanted, but it doesn't seem to notify users on new comments.
In the past, I've worked with a system that produced hundreds of thousands of compressed JSON files every day, each
weighting around 1MiB uncompressed. We wanted to ingest all that data into datalake for debug and analytics, and for
that we've used Spark to append to a delta table. Conceptually, the flow is very straight forward:
a job runs every day
it lists the new files that need to be ingested
those files are read
they are decompressed, parsed and encoded as parquet
these new parquet files are written to the table
In my last blog post,
I've dissected the delta table format and
showed
that it's basically just a bunch of JSON and parquet files. Part of that knowledge will be useful for this post, so
go take a look there if you want. I'll wait here :)
At its core, the Spark script to do this ETL is (in Python):
In Rust, the tokio's ecosystem has a fundamental crate called bytes that abstracts
and helps dealing with bytes (you don't say!). I've indirectly used it a billion times and I thought that I had a good
mental model of how it worked.
So, in the spirit of the "decrusting" series
by the excellent Jon Gjengset, I've decided to peek behind the curtains to understand more what axum, tokio, hyper and
the kind do to them bytes! The code is well written, but surprisingly complex. I understand now what it does, but I
still don't fully grasp why it does some things in a certain way.
I'm ready to share with you my discoveries. I hope that you are sitting, laying or squatting comfortably. This is the
first post in a small series. I'm legally required by my marketing department to remind you that you can subscribe to
my low-traffic newsletter, so that you'll know when new posts are up!
A quick note before we start: this posts is based on the current bytes version 1.11.1.
This blog is written in Rust, and I wanted a way to reload the web pages automatically while I change the posts'
contents, styles, etc. This is common-place with JavaScript frameworks, but not automatic in the Rust land. So I've
embarked on a side quest to achieve just that: the "type and auto-reload" experience. In the end, I was surprised to
learn a bit more about sockets and processes in Linux.
This post is a note to myself about these nuggets that I've learned and to share the solution. It may be helpful for
future me and I hope for someonelse out there.
