Data-Eng on Max Halford

Lower your warehouse costs via DuckDB transpilation

maxhalford25@gmail.com (Max Halford) — Thu, 12 Mar 2026 00:00:00 +0000

Many people seem to admire DuckDB. But most of us are stuck with our traditional warehouses, because they’re entrenched in our data stacks and IT landscape. This is with good reason: BigQuery, Snowflake, ClickHouse and co. are great software. But they’re not cheap, and keeping a warehouse’s monthly bill under control is non-trivial.

What if you could get the best of both worlds? Tables could be kept in your warehouse of choice, but computed with DuckDB. There’s been some discussion on multi-engine warehouses – see this article by Julien Hurault and Sung Won Chung. But their proposition is to assign each table to one engine. Multi-engine stacks can be useful, and SQLMesh provides support for it. But to me it sounds too sophisticated, and I’m not convinced it’s what most practitioners want/need.

Row level lineage at Carbonfact

maxhalford25@gmail.com (Max Halford) — Fri, 09 Jan 2026 00:00:00 +0000

Thoughts on DuckLake

maxhalford25@gmail.com (Max Halford) — Mon, 09 Jun 2025 00:00:00 +0000

DuckLake is the new data lake/warehouse from the makers of DuckDB. I really like the direction they’re taking. I’m hopeful it has the potential to streamline the data engineering workflow for many people, vastly reducing costs along the way.

I’m a bit of a nut and don’t use SQLMesh or dbt. Instead, I built lea a few years ago, and we still use it at Carbonfact. I would probably pick SQLMesh if I had to start over, but lea allows me to explore new ideas, so I’m sticking to it for now.

Minimizing the runtime of a SQL DAG

maxhalford25@gmail.com (Max Halford) — Sat, 08 Feb 2025 00:00:00 +0000

I recently looked into reducing the runtime of Carbonfact’s SQL DAG. Our DAG is made up of roughly 160 SQL queries. It takes about 10 minutes to run with BigQuery, using on-demand pricing. It’s decent. However, the results of our DAG feed customer dashboards, and we have the (bad) habit of refreshing the DAG several times a day. Reducing the runtime by a few minutes can be a nice quality-of-life improvement.

A training set for bike sharing forecasting

maxhalford25@gmail.com (Max Halford) — Thu, 04 Apr 2024 00:00:00 +0000

Last night I went to a Toulouse Data Science meetup. The talks were about generative AI and information retrieval, which aren’t topics I’m knowledgeable about. However, one of the speakers was a friend of mine, so I went to support him. Toulouse used to be my hometown, so I bumped into a few people I knew. It was a nice evening.

Efficient ELT refreshes

maxhalford25@gmail.com (Max Halford) — Fri, 01 Dec 2023 00:00:00 +0000

A tenant of the modern data stack is the use of ELT (Extract, Load, Transform) over ETL (Extract, Transform, Load). In a nutshell, this means that most of the data transformation is done in the data warehouse. This has become the de facto standard for modern data teams, and is epitomized by dbt and its ecosystem. It’s a great time to be a data engineer!

We at Carbonfact fully embrace the ELT paradigm. In fact, our whole platform is powered by BigQuery, which acts as our single source of truth. We have a main BigQuery dataset where we materialize several SQL views that power what our customers see.

Sh*t flows downhill, but not at Carbonfact

maxhalford25@gmail.com (Max Halford) — Mon, 16 Oct 2023 00:00:00 +0000

I’m writing this after watching the talk Joe Reis gave at Big Data LDN. It’s called Data Modeling is Dead! Long Live Data Modeling! It’s an easy-to-watch short talk that calls out on a few modern issues in the data world.

I’d like to bounce off one of Joe’s slides:

I’m aligned with Joe that many issues stem from the lack of unison between engineering and data teams. A fundamental aspect of the Modern Data Stack is to replicate/copy production data into an analytics warehouse. For instance, copying the production PostgreSQL database into BigQuery.

For analytics, don't use dynamic JSON keys

maxhalford25@gmail.com (Max Halford) — Thu, 11 May 2023 00:00:00 +0000

I love the JSON format. It’s the kind of love that grows on you with time. Like others, I’ve been using JSON everywhere for so many years, to the point where I just take it for granted.

I suppose the main thing I like about JSON is its flexibility. You can structure your JSONs without too much care. There will always be a way to consume and manipulate it. But I have discovered a bit of anti-pattern, which I believe is worth raising awareness about. Especially when you’re doing analytics.

A rant against dbt ref

maxhalford25@gmail.com (Max Halford) — Tue, 28 Jun 2022 00:00:00 +0000

Disclaimer

Let me be absolutely clear: I think dbt is a great tool. Although this post is a rant, the goal is to be constructive and suggest an improvement.

dbt in a nutshell

dbt is a workflow orchestrator for SQL. In other words, it’s a fancy Make for data analytics. What makes dbt special is that it is the first workflow orchestrator that is dedicated to the SQL language. It said out loud what many data teams were thinking: you can get a lot done with SQL.

Dashboards and GROUPING SETS

maxhalford25@gmail.com (Max Halford) — Fri, 10 Sep 2021 00:00:00 +0000

Motivation

At Alan, we do almost all our data analysis in SQL. Our data warehouse used to be PostgreSQL, and have since switched to Snowflake for performance reasons. We load data into our warehouse with Airflow. This includes dumps of our production database, third-party data, and health data from other actors in the health ecosystem. This is raw data. We transform this into prepared data via an in-house tool that resembles dbt. You can read more about it here.

An overview of dataset time travel

maxhalford25@gmail.com (Max Halford) — Wed, 07 Apr 2021 00:00:00 +0000

TLDR

You’re a data scientist. The engineers in your company overwrite data in the production database. You want to access overwritten data to train your models. How?

I thought time travel only existed in the movies

You’re probably right, expect maybe for this guy.

I want to discuss a concept that’s been on my mind for a while now. I like to call it “dataset time travel” because it has a nice ring to it. But the association of “time travel” and “data” has already been used elsewhere. It’s not something I’m pulling out from thin air. Essentially, what I want to discuss is the ability to view a dataset at any given point in the past. Having this ability is powerful, as it allows answering important business questions. As an example, let’s say we have a database table called users. We might ask the following question:

A few intermediate pandas tricks

maxhalford25@gmail.com (Max Halford) — Mon, 17 Aug 2020 00:00:00 +0000

I want to use this post to share some pandas snippets that I find useful. I use them from time to time, in particular when I’m doing time series competitions on platforms such as Kaggle. Like any data scientist, I perform similar data processing steps on different datasets. Usually, I put repetitive patterns in xam, which is my personal data science toolbox. However, I think that the following snippets are too small and too specific for being added into a library.

Finding fuzzy duplicates with pandas

maxhalford25@gmail.com (Max Halford) — Mon, 16 Sep 2019 00:00:00 +0000

Duplicate detection is the task of finding two or more instances in a dataset that are in fact identical. As an example, take the following toy dataset:

	First name	Last name	Email
0	Erlich	Bachman	eb@piedpiper.com
1	Erlich	Bachmann	eb@piedpiper.com
2	Erlik	Bachman	eb@piedpiper.co
3	Erlich	Bachmann	eb@piedpiper.com

Each of these instances (rows, if you prefer) corresponds to the same “thing” – note that I’m not using the word “entity” because entity resolution is a different, and yet related, concept. In my experience there are two main reasons why data duplication may occur:

A smooth approach to putting machine learning into production

maxhalford25@gmail.com (Max Halford) — Sat, 13 Jul 2019 00:00:00 +0000

Putting machine learning into production is hard. Usually I’m doubtful of such statements, but in this case I’ve never met anyone for whom everything has gone smoothly. Most data scientists might agree that there is a huge gap between their local environment and a live environment. In fact, “productionalizing” machine learning is such a complex topic that entire companies have risen to address the issue. I’m not just talking about running a gigantic grid search and finding the best model, I’m talking about putting a machine learning model live so that it actually has a positive impact on your business/project. Off the top of my head: Cubonacci, H2O, Google AutoML, Amazon Sagemaker, and DataRobot. In other words people are making money off businesses because data scientists and engineers are having a hard putting their models into production. In my opinion if a data scientist can’t put her model into production herself then something is wrong. Life should be simpler.

Skyline queries in Python

maxhalford25@gmail.com (Max Halford) — Tue, 21 May 2019 00:00:00 +0000

Imagine that you’re looking to buy a home. If you have an analytical mind then you might want to tackle this with a quantitative. Let’s suppose that you have a list of potential homes, and each home has some attributes that can help you compare them. As an example, we’ll consider three attributes:

The price of the house, which you want to minimize
The size of the house, which you want to maximize
The city where the house if located, which you don’t really care about

Some houses will be objectively better than others because they will be cheaper and bigger. However, for some pairs of houses the comparison will not be as clear. It might be that house A is more expensive than house B but is also larger. In data analysis this set of best houses which are incomparable with each other is called a skyline. As they say, a picture is worth a thousand words, so let’s draw one.