Sitegui's Blog

Dissecting a delta table: just a bunch of JSON and parquet files

2026-05-06T00:00:00+00:00

Today I'll share some points that I've learned while playing with Spark, Parquet and the Delta format. Even if you don't use these technologies, I hope you can spot some neat ideas to reuse somewhere else later.

I like to picture in my head that the most important architectural distinction between a datalake table and a typical database (like Postgres) is that compute and storage are handled very separately: the table's data is stored in one distributed system (typically a cloud object storage), while another distributed system (or even multiple ones) read and write to those files that compose the table.

There are competing formats to represent these tables, with distinct trade-offs of course: Delta, Iceberg, Hudi. But from my research, I don't think there is anything fundamentally different between then. This post will focus on Delta, but most of it should be easily transposable for the others.

I like to understand technical solutions by framing the fundamental problems that they aim to solve best, so I'll present it like that. Just remember that, although I have read the specification and have used Spark with Delta tables for years, I didn't invent any of this: I'm just an outside observer who can be wrong. If you spot a misconception, please tell me in the comments!

How it solves its main challenges

Very large tables: pruning files

The Delta format (and datalakes in general) aims to work effectively with very large tables, with petabytes of data over trillions of rows. To allow for this, the data is divided into multiple files which are themselves divided in chunks. The format then has ways to drastically reduce how many files and how much of these files need to be accessed in order to answer queries.

To cut early which files are read, Delta keeps track of some simple statistics about each column in each file: number of nulls, maximum and minimum value. To give a concrete example, let's explore a very simple Delta table made of 3 files:

my-table/_delta_log/00000000000000000000.json
my-table/part-00000-0d56d77a-f779-46e3-adf2-7eaa14656c20-c000.zstd.parquet
my-table/part-00000-a6ac1d2e-304a-465e-b549-ccc442aa3855-c000.zstd.parquet

The .parquet files contain the actual rows' data, but put them aside for now: we'll talk about them in a minute.

The _delta_log/00000000000000000000.json file is a JSON-lines file that describes the table itself. It has a funny name (not judging!), but you can imagine that it's somehow linked to history and evolution of the table. In it, each parquet file that constitutes the table has a record like this:

{
  "add": {
    "path": "part-00000-0d56d77a-f779-46e3-adf2-7eaa14656c20-c000.zstd.parquet",
    "partitionValues": {},
    "size": 1040477,
    "modificationTime": 1778102420000,
    "dataChange": true,
    "stats": "{\"numRecords\":10,\"minValues\":{\"id\":0,\"name\":\"gz ovkfgoconqkwxf alpdjfuk rqvvb\",\"edition\":12},\"maxValues\":{\"id\":9,\"name\":\"zulxlljs fizl qtivsko dkb\",\"edition\":90},\"nullCount\":{\"id\":0,\"name\":0,\"edition\":0,\"days\":0,\"games\":0,\"editors\":0,\"visitors\":0}}"
  }
}

Look who's there: stats! It's a JSON inside a JSON, why not... Here, I'll format it so we can take a closer look:

{
  "numRecords": 10,
  "minValues": {
    "id": 0,
    "name": "gz ovkfgoconqkwxf alpdjfuk rqvvb",
    "edition": 12
  },
  "maxValues": {
    "id": 9,
    "name": "zulxlljs fizl qtivsko dkb",
    "edition": 90
  },
  "nullCount": {
    "id": 0,
    "name": 0,
    "edition": 0,
    "days": 0,
    "games": 0,
    "editors": 0,
    "visitors": 0
  }
}

Cool! So it has 10 rows, id is between 0 and 9, name is between "gz ovkfgoconqkwxf alpdjfuk rqvvb" and "zulxlljs fizl qtivsko dkb", no column has any null.

This is a silly example, but in a more realistic data and query it can be very handy. Say you're doing

select *
from my_table
where created_at > '2026-01-01'

A Delta-compatible engine will read the 00000000000000000000.json file to extract the list of all files, then use the statistics to decide which files are even worth considering.

But to be frank, I was somewhat shocked that the statistics are so simple (just max, min and null count)! If the column you're searching on has naturally a similar range in all files (for example, it's a UUID v4, product name or event kind), it seems to me that this file pruning will be much less effective. The Delta format has no such concept of "indexes" as other databases have. Also, it surprised me that columns with array of structs don't get statistics at all!

Very large tables: reading less data

Delta uses the parquet format to store the rows. Parquet is pretty neat. If you have time to spare, go and read the summary on the project's page. For our discussion right now, you only have to know that parquet divides the table horizontally into "row groups" and that, in each row group, the data for each column is stored sequentially. The end of the file contains a footer with the metadata and offsets for each column in each row group:

So instead of reading each file in full, the Delta engine first reads only the footer; then depending on the queried columns it reads only the segments of the file that are of interest. This is very important for datalake tables, since the files are usually stored in a separate cloud service.

Small parenthesis here: this separate cloud service is usually a "dumb" object storage, but it's extra cool when you realise that it can be something much more refined! For example, it can be a service that implements authorisation over which columns the requesting user can access to protect sensitive information.

Let's take a train back to the statistics of the columns. While the Delta format keeps some very simple statistics as part of the JSON file that links each parquet file to the table, the parquet itself is more sophisticated! First, because it does so per row group. Second, because it can store the list of unique values (if it's small) or a bloom filter (if there are many distinct values).

Very large tables: distributed reading

That's easy: the table is split into multiple parquet files, each one may be further broken down in row groups. Each row group can be handled independently, so a distributed system (say a cluster with many machines and cores) can divide the work to reduce total processing time.

Consistent data schema and schema evolution

A typical use case for datalake tables is to keep a history of multiple months or even years of data. Just like a SQL database like Postgres, each table has a schema (set of column names and types), which has to be migrated as the underlying data evolves. However, unlike Postgres, the allowed changes in the schema are designed so that historical data (that is, old parquet files) don't need to be rewritten.

In practice, in the example table above the 00000000000000000000.json file also contains a record like:

{
  "metaData": {
    "id": "6a8b2481-493a-4796-a77d-ed5e1b70a1e1",
    "format": {
      "provider": "parquet",
      "options": {}
    },
    "schemaString": "{\"type\":\"struct\",\"fields\":[...]}",
    "partitionColumns": [],
    "configuration": {},
    "createdTime": 1778102418300
  }
}

Yes, another JSON in a JSON in schemaString.

Any evolution to the schema will produce a new metaData record with the full updated schema.

So that no previous parquet file needs to rewritten, Delta only allows:

adding a new column: reading a previous parquet file will assume null for it
removing a column: the data is left in previous parquet files, but no longer visible for the engine
changing the order of a column: it's just superficial
some specific change of types, called type "widening": readers have to convert on the fly as needed
renaming a column: requires "column mapping"

This last one is the most complex one and has to be enabled in more recent Delta versions. With this, the name of the column in the parquet files is just a UUID. The Delta metadata in the JSON file then maps each UUID to a user-visible column name.

The schema string is quite large, and it has to be repeated in its entirety every time it evolves. I guess for most tables that's negligible, but we can imagine a pathological case for tables with thousands of columns that evolve frequently.

Append data

Many use cases of Delta tables require adding more data to it in batches, usually by some automatic ingestion pipeline every hour or day. Delta deals with this by versioning the table and splitting the operation into two distinct steps:

write new parquet files
commit the new version

The first step can take an arbitrarily long time, and multiple concurrent writers can do this at the same time. Crucially, these new files are "dangling" so far and no reader can see them.

The second step has to be done atomically by writing a new versioned file with the next version number. In our examples above, it would be 00000000000000000001.json. All these JSON files have to be read and merged in sequence by future readers.

Delta uses "optimistic" concurrency. To illustrate, when two writers work in parallel they can do step 1 at their own pace, but will have to serially do step 2. In the example below, writer 1 commits first, so writer 2 has to check that what writer 1 has done does not conflict with its work, then commit. For appending data that's usually fine, but for updates it may require writer 2 to start over.

However, note how this architecture has one great weakness: each addition requires writing a new parquet file then commiting a new version. Also, every future reader has to read all the small JSON files to glue them all together. So Delta is a catastrophic format if you want to do multiple updates every second!

To mitigate the ever-increasing number of JSON files to read, the writer may decide to compact the history. For example:

my-table/_delta_log/00000000000000000000.json
my-table/_delta_log/00000000000000000001.json
my-table/_delta_log/00000000000000000002.json
my-table/_delta_log/00000000000000000003.json
my-table/_delta_log/00000000000000000004.json

can be compacted as

my-table/_delta_log/00000000000000000004.json
my-table/_delta_log/00000000000000000004.checkpoint.parquet

The .checkpoint.parquet file containing the add entries of all versions so far, and the .json file containing only the last metaData entry.

Delete and update data

Delta represents "removed" data without actually removing them 😜. It uses "delete vector": a new binary file that describes which rows in the parquet files should be considered removed.

Again, this produces a new version with a delete entry, and the same optimistic concurrency model and compaction logic described above apply.

Delta can update data by either producing a delete followed by an add in the same version, or using a dedicated cdc (which stands for change data capture). I could not learn enough about the CDC feature, so I'll not try to explain it here.

Optimisation and clean-up

As you may imagine, as the table evolves due to writes, the data may become too much fragmented and removed data no longer accessible by readers still taking space, etc. A Delta engine implements a VACCUM and OPTIMIZE commands to rewrite recently added and modified parquet files.

There's a sweet spot: too many parquet files bring a lot of metadata overhead, too few don't allow enough parallelism.

Also, newer implementations sort and divide the rows in the row groups and parquet files to improve the relevance of the column statistics for file and row-group pruning. They've named this feature "liquid clustering", which my brain finds too much marketing-y.

That's all for now, thanks for reading!

Delta is a pretty neat format that solves some hard problems not with magic, but with a bunch of JSON and parquet files. At the same time, its design also brings some major pain points... I hope that you've learned something new, 'cause I certainly did.

My homelab: an old laptop behind the fridge

2026-03-30T00:00:00+00:00

This very website is served from my home by an Ubuntu Server running on old hardware behind my fridge:

I've talked in a local dev meetup about my setup, you can check the slides here:

My homelab is at the same time a "production" environment for my digital life and "staging" environment for all sort of crazy project that I want to play with. It's amazing what 10-year-old hardware is capable of!

Maybe someday I'll invest into another second-hand to create separate environments.

All my homelab config is available this repo in my git forge (also itself hosted behind the fridge, as you may have guessed 😜).

In a future post, I want to take the layers of my setup apart and write about the different problems that I've solved and solutions that I've found. Until next time: go and play! The world is large and there are a lot of very good self-hostable projects around. You can start simple.

Performance advice nugget 01: use flat representations

2026-03-21T00:00:00+00:00

This is my last week on my current job, and I was reflecting about some of the things that I've learned in the past 7 years there. There's a lot of course! As a staff data engineer, I worked with many teams and codebases and learned a couple of tricks at scale. So I'm starting a new blog series "performance advice nugget" (or PAN for short), in which I'll share some insights of what worked quite well in practice.

So welcome to PAN 01: use flat representations.

I'll try to make these posts quite short. As usual with everything related to "performance", you should always measure and benchmark with your real workload, and always balance whether additional complexity is worth the performance gains.

Nice, forewords are said and out of the way. Let's focus on the matter: imagine that you are handling a data that has multiple levels, for example, a paragraph, that is made of sentences, each made of words, each made of characters:

One straight way to model this is with lists of lists:

class Paragraph:
    sentences: list[Sentence]


class Sentence:
    words: list[Word]


class Word:
    chars: list[str]


data = Paragraph([
    Sentence([
        Word(["H", "e", "l", "l", "o"]),
        Word(["W", "o", "r", "l", "d"]),
    ]),
    Sentence([
        Word(["B", "y", "e"]),
    ]),
])

Note: don't overthink Word in this synthetic example! In a more realistic setup, Word could probably just be a str. But here I'm using it to illustrate the "layered" modeling.

Today's PAN is: don't nest the lists! Instead, keep the leaf information (here the chars) flat in a single list. Then use other lists to index into the flat representation:

class FlatParagrah:
    characters: list[str]
    words: list[WordInfo]
    sentences: list[SentenceInfo]


class WordInfo:
    num_chars: int


class SentenceInfo:
    num_chars: int
    num_words: int


data = FlatParagrah(
    characters=["H", "e", "l", "l", "o", "W", "o", "r", "l", "d", "B", "y", "e"],
    words=[
        WordInfo(5),
        WordInfo(5),
        WordInfo(3),
    ],
    sentences=[
        SentenceInfo(num_chars=10, num_words=2),
        SentenceInfo(num_chars=3, num_words=1),
    ],
)

Why? Two main reasons:

CPU caching: CPUs are more efficient when accessing data that is packed closer together
less allocations: each list has to be allocated and deallocated individually. The flat design has a fixed and small number of lists

Is it true?

Well... as it's usual with data structures, it will highly depend on what you actually do with the data! In our case, it worked wonders.

I've prepared a micro benchmark in Rust to illustrate, you can check the source code for Paragraph and FlatParagraph.

Operation	Paragraph	FlatParagraph	Gain
Create	2100 ns	370 ns	5.7 x
Deallocate	460 ns	39 ns	12 x
Iterate over words	91 ns	100 ns	0.91 x
Iterate over chars	110 ns	32 ns	3.4 x
Create with inserted word	2400 ns	350 ns	6.9 x

The last operation is the one that our codebase did the most: create a new Paragraph from an existing one by inserting a new word in the middle of it. Something like:

class Paragraph:
    def with_inserted_word(
            self,
            paragraph_i: int,
            word_i: int,
            word: list[int],
    ) -> 'Paragraph':
        pass

And we've used a lot Rust's type system to provide an ergonomic and safe API for the rest of the codebase to handle the data, allowing it to cut and slice the "paragraph" however necessary.

Cooperative Buffon π

2026-03-14T00:00:00+00:00

Happy pi day! (not to be confused with pie day, which has more pastry but is less sweat).

I need your help: please refresh this page to throw more matches to the ground!
I'll use Comte de Buffon's result to estimate π.

The underlying relation is:

                          2 * match_size
P(match_crosses_a_line) = --------------
                           π * line_gap

So π can be estimated as:

    2 * match_size
π ~ --------------
     P * line_gap

In the image above match_size/line_gap = 1/2, so this reduces to

    1
π ~ -
    P

You can check the source code here.

Update: a friend told me that it would be nice to know which of the matches you've just thrown into the floor. So I've added a circle around it :)

Understanding Bytes in Rust, one bit at a time

2026-02-27T00:00:00+00:00

In Rust, the tokio's ecosystem has a fundamental crate called bytes that abstracts and helps dealing with bytes (you don't say!). I've indirectly used it a billion times and I thought that I had a good mental model of how it worked.

So, in the spirit of the "decrusting" series by the excellent Jon Gjengset, I've decided to peek behind the curtains to understand more what axum, tokio, hyper and the kind do to them bytes! The code is well written, but surprisingly complex. I understand now what it does, but I still don't fully grasp why it does some things in a certain way.

I'm ready to share with you my discoveries. I hope that you are sitting, laying or squatting comfortably. This is the first post in a small series. I'm legally required by my marketing department to remind you that you can subscribe to my low-traffic newsletter, so that you'll know when new posts are up!

A quick note before we start: this posts is based on the current bytes version 1.11.1.

What the Bytes?

The bytes crate does many things, but its public API is quite small: it basically has two structs and two traits. Quoting from the documentation:

struct Bytes: a cheaply cloneable and sliceable chunk of contiguous memory
struct BytesMut: a unique reference to a contiguous slice of memory
trait Buf: read bytes from a buffer
trait BufMut: a trait for values that provide sequential write access to bytes

In this series, I'll focus on the Bytes struct, because yeah, it's not as simple as one may be led to believe!

A compelling example

I would like to start concrete, with an example of what the problem that Bytes helps solve: zero-copy parsing. Imagine that you're parsing HTTP requests and your code receives these bytes from the network:

GET /post/2026/some-post HTTP/2
Host: sitegui.dev
Accept: text/html
Referer: https://sitegui.dev/
Connection: keep-alive

You would like to efficiently parse it into a struct like this:

struct Request {
    method: Something,
    path: Something,
    version: Something,
    /// Note: headers can repeat, so I'm using a
    /// `Vec` not a `HashMap` to represent them
    headers: Vec<(Something, Something)>,
}

This post is not about the parsing bit, instead it's about the bytes themselves. What should we choose as Something above?

If you chose a type like Vec<u8>, your parser would need to allocate and copy the information into many multiple instances of Vec<u8>. But with Bytes, all these instances are cheap to produce as they share the same storage (your initial bytes are represented only once) and each instance then has a "window" to this buffer. The underlying buffer will only be deallocated when all the instances sharing it are dropped.

As usual in computing, using Bytes (or any "shared" construction) vs Vec<u8> (or any "owned" construction) is a matter of trade-offs. For example, if your parser code only produces output that references a small part of the input, it's probably better to copy over just that. Otherwise, the Bytes instances will keep the whole input around in memory.

An initial mental model

As I said before, I had a useful mental model before embarking in this exploration: a Bytes instance simply has a reference-counted shared reference to the actual bytes and some information to where in the buffer those bytes are. The fundamental operations available are:

trait BytesStorage {
    /// Create a new instance with some data
    /// It should "adopt" the data, not copy it
    fn new(data: Vec<u8>) -> Self;
    /// Create a new instance that references some part of this data
    /// Constant time, and it should not copy the actual data
    fn slice(&self, range: Range<usize>) -> Self;
    /// Return the bytes slice
    /// Very cheap
    fn as_ref(&self) -> &[u8];
}

We could draft a similar-in-spirit implementation using Arc<Vec<u8>>:

pub struct SharedBytes {
    range: Range<usize>,
    data: Arc<Vec<u8>>,
}

impl BytesStorage for SharedBytes {
    fn new(data: Vec<u8>) -> Self {
        Self {
            range: 0..data.len(),
            data: Arc::new(data),
        }
    }

    fn slice(&self, range: Range<usize>) -> Self {
        // Translate the "local" range to a "global" range
        let root_range = range.start + self.range.start..range.end + self.range.start;
        Self {
            range: root_range,
            data: self.data.clone(),
        }
    }

    fn as_ref(&self) -> &[u8] {
        &self.data[self.range.clone()]
    }
}

Note how new() does not clone the data, instead the Vec is adopted into the SharedBytes instance. Also, slice() clones the Arc<_>, which uses reference counting and avoids cloning the actual data in the Vec.

However, as_ref() is not very good for two reasons: Rust will execute bound checks to ensure that the range is valid at every access, and the actual buffer is behind two memory references Arc (in SharedBytes) -> Vec -> buffer.

The actual Bytes implementation avoids these two problems by storing the slice components (pointer to first byte and length) as two fields directly in the struct, trading off some bloat in the struct and a small performance hit in the slice() operation for a simpler as_slice():

pub struct Bytes {
    ptr: *const u8,
    len: usize,
    // ... more fields ...
}

impl Bytes {
    #[inline]
    fn as_slice(&self) -> &[u8] {
        unsafe { slice::from_raw_parts(self.ptr, self.len) }
    }
}

Let's test our mental model

In Rust, you can set a different global memory allocator like this:

use std::alloc::{GlobalAlloc, Layout, System};

/// A global allocator that can track all allocations made
struct TrackingAllocator;

// Ask Rust to use our allocator as the global allocator
#[global_allocator]
static ALLOCATOR: TrackingAllocator = TrackingAllocator;

unsafe impl GlobalAlloc for TrackingAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        // Do stuff
        unsafe { System.alloc(layout) }
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        // Do stuff
        unsafe { System.dealloc(ptr, layout) };
    }
}

There is a Rust working group to have a more fine control of the allocator, allowing us to overwrite it only for some parts of the code. But for this reduced example, it's okay to capture all allocations.

I used this mechanism to track exactly which allocations and deallocations the code was doing for Vec<u8>, our own SharedBytes and bytes::Bytes and where in the source code they happened. You can check the full code here, if you are curious or want to reproduce my work.

To simulate the HTTP parser, I'm using this simple code:

fn parse_request<B: BytesStorage>(data: Vec<u8>) -> Request<B> {
    let data = B::new(data);
    Request {
        method: data.slice(0..3),
        path: data.slice(4..24),
        version: data.slice(25..31),
        headers: vec![
            (data.slice(33..37), data.slice(39..50)),
            (data.slice(52..58), data.slice(60..69)),
            (data.slice(71..78), data.slice(80..100)),
            (data.slice(102..112), data.slice(114..124)),
        ],
    }
}

The following table shows what each line of the code above allocates for Vec<u8>, SharedBytes and bytes::Bytes, in bytes.

But before looking at the results, take a while to run this code against your mental model and try to predict how many allocations will occur for each implementation and where that will happen.

Code source	`Vec<u8>`	`SharedBytes`	`bytes::Bytes`
`data = B::new(data)`		40
`method: data.slice(0..3)`	3		24
`path: data.slice(4..24)`	20
`version: data.slice(25..31)`	6
`vec![...]`	192	192	256
8x `data.slice(...)`	57
Total	298	232	280

As I expected, Vec<u8> allocates a new independent buffer in every call to .slice(), with the necessary capacity for the data piece then copy it. SharedBytes allocates the Arc<Vec<u8>> once at the start, and later the Vec<(B, B)> with 8 × 24 bytes.

I want to explore in a separate blog post the whys and the hows Arc<Vec<u8>> has 40 bytes, and also compare it against Arc<[u8]>. Stay tuned!

The behavior of Bytes was interesting to me for two reasons:

it somehow defers the first allocation: B::new() doesn't allocate (in this case), but the first call to .slice() does. Spoiler alert: it's pretty neat but complex. I'll explore it in details later on.
it's big! 32 bytes each instance. So Vec<(B, B)> has 8 × 32 bytes

No `Arc` in sight

Armed with the above initial mental model, it's not a surprise that the documentation and comments in the file implementing Bytes mention Arc 6 times. What is a bit eyebrow-raising is that the code itself does not use Arc at all!

Maybe it's related to the defered allocation we saw before? Maybe it's related to the 32 bytes needed for each instance? Well, kind of...

I'll use memory diagrams like this one below to represent what the structs (in grey) are made of. Simple numerical fields are in yellow and pointers are in green. In parentheses, I've put the memory size in bytes, assuming a 64-bit system.

For example, in Rust you can represent a growable owned sequence with Vec<T> and a fixed owned sequence with Box<[T]>. The physical difference is that Vec may have allocated more space than it is currently using, to amortize buffer grow on inserts, so it uses an additional capacity field. A Box uses exactly the bufffer's size, so length and capacity are the same. Converting from a Vec that is "full" (that is, for which capacity is the same as length) is cheap because the underlying buffer is reused:

Here's a Box. Give-me Bytes

The implementation of Bytes::from(Vec<u8>) will first check if it's a "full" Vec to convert it into a Box<[u8]> and build a Bytes instance like below. When the Vec<u8> it not "full", a different logic is used, but let's put that aside for now.

That's a lot to unpack. I'll explain what I've understood of the design and the trade-offs of this representation:

First, note that the buffer is reused, as expected.

The start and length fields simply represent the slice information. It seems a bit redundant with the data field, that also points to the buffer. However, this distinction will be useful when we slice this data: the start of our "window" will be different from the buffer's start, and we need to keep the buffer's start around so that we can deallocate it when the moment comes.

The vtable field implements the "dynamic dispatch" pattern. Rust has "trait objects" to implement this pattern with &dyn SomeTrait or Box<dyn SomeTrait>, but I guess the designers of the crate didn't use them because this pattern has restrictions that are a deal-breaker for this use. If you know more why, please tell me!

Note that vtable points to a statically declared instance of VTable called "promotable". Aha! This name is related to how Bytes deferr allocation as we observed earlier.

The data field is funny-looking (no judgements!): it's an AtomicPtr<_>, that unlike a normal pointer, uses atomic operations to read/write to it. But why do we need atomic operations here? Spoiler alert: it's the deferred allocation again. Somehow (we'll see next), this pointer will point to something else in the future.

Continuing on the data field, it's not just a pointer. It uses the pattern of "tagged pointer" to store one bit of information there. I think you can already guess it: it's the deferred allocation again. This trick avoids having a separate field for this, but it requires the code to ensure and check that the pointer to the buffer is always a multiple of 2, so that the least bit carries no useful information and can be co-opted as a tag.

The other side of the river

Let's spring straight into action: what happens when we slice our newly-created Bytes? I've tried my best to convey everything that happens in the diagram below without making it too busy. Take some time to navigate it first.

First, note that calling .slice(&self) the first time actually modifies the original Bytes: the previous state of the instance is represented in a dashed countour, the new state uses red on the changed field: data. Naturally, only this field can be modified, because .slice(&self) takes a shared reference, so only internal mutation is possible, through the AtomicPtr<_>.

As we've observed earlier, a new piece of data is allocated: an instance of the Shared struct with 24 bytes. It carries information about the original buffer that will be used to deallocate (buffer and capacity). I was surprised to learn that in Rust the original size of the buffer is actually necessary to deallocate it. I was used to C's free(ptr) that clearly doesn't need it.

The Shared struct also carries an atomic counter with the number of Bytes instances referencing this buffer. The implementation doesn't use Rust's native Arc. Instead the designers preferred re-implementing it. I could not find a definitive answer to why, but I guess that's because they wanted to avoid the overhead in Arc related to the implementation of "weak references", that is not necessary for Bytes' use case. To implement it, Arc uses an additional atomic counter and executes some extra atomic instructions when cloning and droping.

Finally, note how the vtable pointer does not change: it's still the same "promotable" method list. The code uses the tag in the data pointer to distinguish a not-yet-promoted data (tag = 1) from a promoted data (tag = 0).

The deferred allocation feature is a trade-off: the code is more complex because we need an AtomicPtr and a tag to implement it. However, for code that creates a Bytes without actually slicing and sharing them, it avoids the Shared allocation entirely.

It's a lie

Let's step back a bit: the behavior above only happens when we create a Bytes from a Box<[u8]> or "full" Vec<u8>. When we start with a non-full Vec<u8>, the code will not use the deferred mechanism and instead allocates a Shared struct right away. In this case, there's no tagging of the data field and a dedicated vtable is used that will not check for the tag:

This is very similar to how the simpler SharedBytes work, except that Bytes has 8 more bytes for the vtable and Shared has 8 fewer bytes than Arc<Vec<u8>> for the lack of weak references.

Bytes are versatile

The dynamic dispatch implemented by Bytes with the vtable and data fields effectively allows it to use custom implementation for different backing storages of the data.

We saw what happens to Box and Vec. Here's what happens when you create a Bytes from an existing statically allocated slice of bytes (in Rust represented as &'static [u8]):

Note that there is no reference counting, because in this mode, Bytes does not own the data: instead it borrows data that outlives it, as indicated by the 'static lifetime.

Another interesting usage of the dynamic dispatch is to create Bytes from anything else that somehow owns a buffer. For example, from a memmap2::Mmap instance that represents a memory-mapped region:

Note that the owned struct is copied over into Onwed that, just like Shared, acts like an Arc<_> without the weak reference feature. The major distinction between Owned and Shared is that Bytes does not know how to take ownership of the buffer, as it only requires that the given type implements AsRef<[u8]> to produces a borrowed slice of bytes.

Micro benchmarks

Never trust a benchmark you see online. So here's one more for you to ignore:

fn test<B: BytesStorage>() {
    let data = black_box(example_request_package());
    let request = parse_request::<B>(data);
    assert_example_request(&request);
}

	`Vec<u8>`	`SharedBytes`	`bytes::Bytes`
Allocated (bytes)	298	232	280
Execution time (ns)	115	115	150

So Bytes seems both chunkier and slower than my half-backed SharedBytes on this synthetic benchmark 🫣. But of course, my implementation does not offer the same versatility.

I searched for public benchmarks that justified some design decisions on the bytes crate, but could not find them. If you know any, please tell me!

Web server auto-reload in Rust

2026-02-14T00:00:00+00:00

This blog is written in Rust, and I wanted a way to reload the web pages automatically while I change the posts' contents, styles, etc. This is common-place with JavaScript frameworks, but not automatic in the Rust land. So I've embarked on a side quest to achieve just that: the "type and auto-reload" experience. In the end, I was surprised to learn a bit more about sockets and processes in Linux.

This post is a note to myself about these nuggets that I've learned and to share the solution. It may be helpful for future me and I hope for someonelse out there.

TL;DR

You can check the solution here: https://git.sitegui.dev/sitegui/axum-web-auto-reload-example/src/branch/main/src/main.rs. The README in that repo has some nice diagrams as well.

Shopping list

To reload the browser page on a source file change, you will need:

your server: I'm developing mine with axum in Rust
a tool to listen to a port, pass down the socket to the server, detect file changes, and restart the server. I'm using watchexec
a browser

To run it all, I use this command:

watchexec \
  --socket 8080 \
  --restart \
  --stop-signal SIGINT -- \
  cargo run

Accepting the passed socket

This part is very important for smooth reloads: when the browser reloads, it will try to restablish a connection with your server. However, at the same time your server is reloading and probably not yet available on the localhost port, causing the browser to fail immediately with a passive-aggressive message telling it was ghostest by its dearest friend localhost.

The solution is a bit complex but quite brilliant: let's leave the socket connection to the watchexec process, which will stay alive through the session. When the browser tries to connect, the connection will not fail immediately because no process was listening. Of course, watchexec has no idea what to do with that incomming connection: it is your server that knows. So watchexec spawns your server and pass it the socket, so that it can do its serverly stuff, like accepting connections and spitting HTML or whatever. To "pass the socket", watchexec uses two environment variables:

LISTEN_FDS: the number of sockets being passed
LISTEN_FDS_FIRST_FD: the file-descriptor id of the first socket being passed

The sockets are created with ReuseAddr and ReusePort so that the server can listen to it again.

To make it work, your server should detect that it is called by something like watchexec and that it has received a socket. I'm using the crate listenfd to help with that:

use listenfd::ListenFd;
use std::error::Error;
use tokio::net::TcpListener;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Use the crate `listenfd` to get the socket passed by `watchexec`
    let inherited_socket = ListenFd::from_env().take_tcp_listener(0)?;

    let (auto_refresh, listener) = if let Some(inherited_socket) = inherited_socket {
        println!("Listening on inherit socket");
        inherited_socket.set_nonblocking(true)?; // required by TcpListener::from_std()
        (true, TcpListener::from_std(inherited_socket)?)
    } else {
        // Fallback to the typical way of listening to a new port
        let listener = TcpListener::bind(("127.0.0.1", 8000)).await?;
        println!("Listening on http://{}", listener.local_addr()?);
        (false, listener)
    };
}

Add a page script to reload

When auto_refresh is true in the code above, it means that we're in "let's reload guys!" territory.

In my server, I'm using minijinja to render Jinja2 templates, so I do something like this:

{% if AUTO_REFRESH %}
<script>
  // Connect to the server and reload when a new message is received
  const eventSource = new EventSource(`/auto_refresh`)
  eventSource.addEventListener("message", () => location.reload())
</script>
{% endif %}

and in my main:

fn main() {
    jinja_env.add_global("AUTO_REFRESH", auto_refresh);
}

To tell that the browser should reload, I'm using server-sent events (SSE) which are pretty cool in fact! My use case here is very simple: whenever the server sends an event, reload.

Please call me

The last piece of the puzzle is to implement the /auto_refresh SSE endpoint in the server:

use axum::response::sse::{Event, KeepAlive};
use axum::response::Sse;
use std::convert::Infallible;
use tokio::signal;
use tokio::sync::mpsc;
use tokio_stream::Stream;
use tokio_stream::wrappers::UnboundedReceiverStream;

/// Create a server-sent event stream that will send a single "goodbye" event when the server is
/// stopped.
pub async fn get_auto_refresh() -> Sse<impl Stream<Item=Result<Event, Infallible>>> {
    println!("GET /auto_refresh");

    let (tx, rx) = mpsc::unbounded_channel();
    tokio::spawn(async move {
        wait_ctrl_c().await;
        println!("SSE: sending goodbye");
        let event = Event::default().data("goodbye");
        let _ = tx.send(Ok(event));
    });

    Sse::new(UnboundedReceiverStream::new(rx)).keep_alive(KeepAlive::new())
}

async fn wait_ctrl_c() {
    let _ = signal::ctrl_c().await;
}

And that's it! For sure, the JavaScript ecosystem has perfected this integration of different pieces into a better experience, but now I can enjoy it also. Maybe as a next step I can package all the different pieces into a single crate?

The 15-game blew my mind

2026-02-11T00:00:00+00:00

I've just watched the calmly titled "The 15-game" in the Numberphile Youtube channel and boy oh boy, I cannot stop thinking about it!

I will not spoil the end, but if you ever got intrigued by games, maths, brain-teasers, just take a 15-minute break and watch it. At school, I loved to solve the same problem from different angles, not only to increase my chances of getting the good answer, but also because it was fun. This video hit right there in my heart.

One of my first programs was a game of tic-tac-toe. If only someone had shown me the 15-game back then!

I choose 5. Your turn!

Welcome Framework Laptop

2026-01-29T00:00:00+00:00

Around 3 years ago my phone's screen broke and changing the screen would cost close to a half the device's original price. It was cheaper to throw away and buy a new one.

This is clearly wasteful, but I guess this is a typical experience with well-known consumer brands. But hey, I don't want to move to a new home because my sink broke!

In Europe, new regulations like Right-to-Repair Directive represent a smart step to force the hand of manufacturers. It requires them, for example, to keep spare parts stocked for at least 10 years. In France, I always check the repairability index before buying. I'm more than willing to pay a bonus price for good product design and engineering, that respects the resources and costumers.

So I've begrudgingly replaced my phone. But this time I wanted to break the wasteful cycle, so I've bought a Fairphone, which promises 10 year software and hardware support. Also, if something breaks I can just order a replacement and repair it myself. So far, I'm pretty happy with the experience! It works, it just does. I feel soon I'll replace my battery, and that's it: it will probably stick with me a bunch more years.

Since my old laptop became my home server, I needed a new one for hacking on the loose. Following a similar strategy, I'm going with Framework for my new rig.

They send the machine in a very simple DIY kit. It's a bit too easy for those used to tinker: snap, screw, click, boom! But it's a genius brand move to prove less experienced people that they can do it too.

The final result is pretty good. Come back in 3 years and I'll share my thoughts.

sent from my framework

Nice people don't play board games for 27 straight hours

2026-01-22T00:00:00+00:00

They play for 28 hours! We did it and it was pretty cool o/

Well... not all those hours were spent actually playing, we also had to decide what to play next! With more than 500 games available at the event, this sometimes took a while :)

You can picture 3 rooms like the one below, full of meeples, cards, coins and fun.

It was the 16th edition of the festival 24h Chaud les Jeux! in Cholet. The festival name (and the association behind it) is a well-chosen self-referential pun! It translates to something like "hot for games", and the chaud les sounds exactly the same as the city's name. French people, especially hairdressing owners, love their jeu de mots.

The festival is made not only for hardcore gamers: it's open for anyone that passes by. I always find it heart-warming to see friends and family sitting around, just enjoying their company on a Sunday afternoon, while they fight furiously for the control of Tokyo, or accuse each other of being a Moriarty malfeasant ready to explode the Big Ben, or just casually arranging their azulejos the best they can.

For my part, I've managed to discover 9 new games and play other 2 that I already knew. The big ones were Endeavor: age of sail, Coming of age, Nemesis. I like them and would happily play another time.

Playing Nemesis for the first time at 08:00 was a difficult task hhaha! After 3 hours, we all exploded in space before coming back to Earth because one of the players decided that it HAD to auto-destroy the spaceship in order to content their desire for chaos! It was a nice play though, we were all dead and happy (and ready for lunch).

We made good choices of lighter party games to cool down: Decrypto, Nosferatu and Cabanga!.

A big Thank You! for the game club Chaud les jeux for all the time and effort you've put into this event. I'll be happy to see you again next time.

Hello world 2026

2026-01-01T00:00:00+00:00

I used to blog as a kid back in the 2010s, and for a long time I wanted to get back to it. Now it's the time! This is my personal space on the web, it's not the shiniest, not the most visited, but it is mine :)

For now there isn't much to see here, this first post is mostly a hack to get me going.

See you next time!