Well... There are trade-offs, but don't trash the idea just yet :)
Be assured that nobody is coming to take your beloved MongoDB, Postgres or other database away. They are
marvellous piece of software that solve complex problems quite well. However, in this post I invite you to think what
do you need them for, which problems they solve for your apps and, crucially, which problems they create on their own.
I got tangled in these questions while
developing nidimages, a self-hosted app to share
photos with family and friends. So I decided to write a synthetic load test to compare from
different perspectives Postgres and a file-backed variable.
A file-backed variable
That's a fancy name for a rather dangerously-simple concept: you parse a file on startup, then you rewrite the file from
time to time. Let's implement the gist of it in Python:
... continue reading ...
This is my last week on my current job, and I was reflecting about some of the things that I've learned in the past 7
years there. There's a lot of course! As a staff data engineer, I worked with many teams and codebases and learned a
couple of tricks at scale. So I'm starting a new blog series "performance advice nugget" (or PAN for short), in which
I'll share some insights of what worked quite well in practice.
So welcome to PAN 01: use flat representations.
I'll try to make these posts quite short. As usual with everything related to "performance", you should
always measure and benchmark with your real workload, and always balance whether additional complexity is worth the
performance gains.
Nice, forewords are said and out of the way. Let's focus on the matter: imagine that you are handling a data that has
multiple levels, for example, a paragraph, that is made of sentences, each made of words, each made of characters:
... continue reading ...
