Sql on Max Halford

Lower your warehouse costs via DuckDB transpilation

maxhalford25@gmail.com (Max Halford) — Thu, 12 Mar 2026 00:00:00 +0000

Many people seem to admire DuckDB. But most of us are stuck with our traditional warehouses, because they’re entrenched in our data stacks and IT landscape. This is with good reason: BigQuery, Snowflake, ClickHouse and co. are great software. But they’re not cheap, and keeping a warehouse’s monthly bill under control is non-trivial.

What if you could get the best of both worlds? Tables could be kept in your warehouse of choice, but computed with DuckDB. There’s been some discussion on multi-engine warehouses – see this article by Julien Hurault and Sung Won Chung. But their proposition is to assign each table to one engine. Multi-engine stacks can be useful, and SQLMesh provides support for it. But to me it sounds too sophisticated, and I’m not convinced it’s what most practitioners want/need.

Graph components with DuckDB

maxhalford25@gmail.com (Max Halford) — Sat, 03 Jun 2023 00:00:00 +0000

Introduction

Graph problems are quite common. However, it’s rare to have access to a database offering graph semantics. There are graph databases, such as Neo4j and GraphX, but it’s difficult to justify setting one of those up. One could simply use networkx in Python. But that only works if the graph fits in memory.

From a practical angle, the fact is that people are querying data warehouses in SQL. There are many good reasons to write graph algorithms in SQL. And anyway, one may argue that graphs are a special case of the relational model.

For analytics, don't use dynamic JSON keys

maxhalford25@gmail.com (Max Halford) — Thu, 11 May 2023 00:00:00 +0000

I love the JSON format. It’s the kind of love that grows on you with time. Like others, I’ve been using JSON everywhere for so many years, to the point where I just take it for granted.

I suppose the main thing I like about JSON is its flexibility. You can structure your JSONs without too much care. There will always be a way to consume and manipulate it. But I have discovered a bit of anti-pattern, which I believe is worth raising awareness about. Especially when you’re doing analytics.

Online gradient descent written in SQL

maxhalford25@gmail.com (Max Halford) — Tue, 07 Mar 2023 00:00:00 +0000

Edit – this post generated a few insightful comments on Hacker News. I’ve also put the code in a notebook for ease of use.

Introduction

Modern MLOps is complex because it involves too many components. You need a message bus, a stream processing engine, an API, a model store, a feature store, a monitoring service, etc. Sadly, containerisation software and the unbundling trend have encouraged an appetite for complexity. I believe MLOps shouldn’t be this complex. For instance, MLOps can be made simpler by bundling the logic into your database.

A rant against dbt ref

maxhalford25@gmail.com (Max Halford) — Tue, 28 Jun 2022 00:00:00 +0000

Disclaimer

Let me be absolutely clear: I think dbt is a great tool. Although this post is a rant, the goal is to be constructive and suggest an improvement.

dbt in a nutshell

dbt is a workflow orchestrator for SQL. In other words, it’s a fancy Make for data analytics. What makes dbt special is that it is the first workflow orchestrator that is dedicated to the SQL language. It said out loud what many data teams were thinking: you can get a lot done with SQL.

Dashboards and GROUPING SETS

maxhalford25@gmail.com (Max Halford) — Fri, 10 Sep 2021 00:00:00 +0000

Motivation

At Alan, we do almost all our data analysis in SQL. Our data warehouse used to be PostgreSQL, and have since switched to Snowflake for performance reasons. We load data into our warehouse with Airflow. This includes dumps of our production database, third-party data, and health data from other actors in the health ecosystem. This is raw data. We transform this into prepared data via an in-house tool that resembles dbt. You can read more about it here.

Computing cross-correlations in SQL

maxhalford25@gmail.com (Max Halford) — Tue, 17 Nov 2020 00:00:00 +0000

Introduction

I’m currently working on a problem at work where I have to measure the impact of a growth initiative on a performance metric. Hypothetically, this might answer the following kind of question:

I’ve spent X amount of money, what is the impact on the number of visitors on my website?

Of course, there are many measures that can be taken to answer such a question. I decided to measure the correlation between the initiative and the metric, with the latter being shifted forward in time. This measure is called the cross-correlation. It’s different from serial correlation, which is the correlation of a series with a shifted version of itself.

SQL subquery enumeration

maxhalford25@gmail.com (Max Halford) — Mon, 06 May 2019 00:00:00 +0000

I recently stumbled on a rather fun problem during my PhD. I wanted to generate all possible subqueries from a given SQL query. In this case an example is easily worth a 1000 thousand words. Take the following SQL query:

SELECT *
FROM
 customers AS c,
 purchases AS p,
 shops AS s
WHERE
 p.customer_id = c.id AND
 p.shop_id = s.id
AND
 c.nationality = 'Swedish' AND
 c.hair = 'Blond' AND
 s.city = 'Stockholm'

Here all the possible subqueries that can be generated from the above query.