Python on Max Halford

Text classification with Python 3.14's zstd module

maxhalford25@gmail.com (Max Halford) — Fri, 06 Feb 2026 00:00:00 +0000

Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms.

I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it will maintain an internal state. It’s particularly well suited for compressing small data. It’s perfect for the classify text via compression trick, which I described in a previous blog post 5 years ago.

Row level lineage at Carbonfact

maxhalford25@gmail.com (Max Halford) — Fri, 09 Jan 2026 00:00:00 +0000

Scraping Google Calendar events

maxhalford25@gmail.com (Max Halford) — Sun, 12 Oct 2025 00:00:00 +0000

At my day job we deal with enterprise customers. They pay us a subscription fee, and in return we help them in various ways to reduce their carbon footprint. To keep the boat afloat, we need to make some money. We shouldn’t spend more money than we make. So we need to keep track of our revenue and costs. Our gross margin is (revenue - cost) / revenue, where the cost is mostly the salaries of our employees.

Minimizing the runtime of a SQL DAG

maxhalford25@gmail.com (Max Halford) — Sat, 08 Feb 2025 00:00:00 +0000

I recently looked into reducing the runtime of Carbonfact’s SQL DAG. Our DAG is made up of roughly 160 SQL queries. It takes about 10 minutes to run with BigQuery, using on-demand pricing. It’s decent. However, the results of our DAG feed customer dashboards, and we have the (bad) habit of refreshing the DAG several times a day. Reducing the runtime by a few minutes can be a nice quality-of-life improvement.

Introducing icanexplain @ PyData Paris 2024

maxhalford25@gmail.com (Max Halford) — Thu, 26 Sep 2024 00:00:00 +0000

@daily_cache implementation in Python

maxhalford25@gmail.com (Max Halford) — Tue, 27 Aug 2024 00:00:00 +0000

I spend a lot of time at Carbonfact working on datasets shared by our customers. We typically set things up so that our customers can export data automatically. They usually deposit files to a GCP bucket, with a script, once a day. We then have an ETL script for each customer that runs afterwards to fetch their latest data and process it.

During development, I load customer data to my laptop and work on it. The datasets can be quite heavy, and it takes time to fetch them, so I cache them to save some time. Python has something for this in its standard library:

LCA software: exit the matrix

maxhalford25@gmail.com (Max Halford) — Sun, 09 Jun 2024 00:00:00 +0000

Measuring the environmental impact of a product is done using life cycle assessment (LCA). This is a methodology that breaks down a product’s life cycle into stages (LCI), and measures the impact of each stage on the environment (LCIA).

There are a few pieces of LCA software to choose from. The leading ones are SimaPro, GaBi, openLCA, and Umberto. These are all proprietary software, and they’re expensive. But there’s a free and open source alternative: Brightway.

Fast Poetry and pre-commit with GitHub Actions

maxhalford25@gmail.com (Max Halford) — Tue, 27 Feb 2024 00:00:00 +0000

This is a short post to share a GitHub Actions pattern I use to setup Poetry and pre-commit. These two tools cover most of my Python development needs. I use Poetry to manage dependencies and pre-commit to run code checks and formatting. The setup is fast because it caches the virtual environment and the .local directory.

I like to use custom actions for this type of stuff. These are base actions that can be re-used in multiple workflows. I have a custom action to install the Python environment. Here’s the action file:

Measuring the carbon footprint of pizzas

maxhalford25@gmail.com (Max Halford) — Sun, 25 Jun 2023 00:00:00 +0000

Making environmentally friendly decisions can only be done with the right information. At Carbonfact, we’ve realized a big challenge is the lack of information about industrial processes. We tackle that slowly but surely by gathering data from various sources, and making it available to our customers.

Regarding food, the French government has a great initiative called Agribalyse. It’s a free database of environmental footprints for various food products. It includes raw ingredients straight out from the farm, as well as ready to eat dishes from the supermarket. It’s a great initiative, as it allows anyone to do their own research and make informed decisions.

Using SymPy in Python doctests

maxhalford25@gmail.com (Max Halford) — Wed, 15 Feb 2023 00:00:00 +0000

A program which compiles and runs without errors isn’t necessarily correct. I find this to be especially true for statistical software, both as a developer and as a user. Small but nasty bugs creep up on me every week. I keep sane in the membrane by writing many unit tests 🐛🔨

I make heavy use of doctests. These are unit tests which you write as Python docstrings. They’re really handy because they kill two birds with one stone: the unit tests you write for a function also act as documentation.

Fuzzy regex matching in Python

maxhalford25@gmail.com (Max Halford) — Mon, 04 Apr 2022 00:00:00 +0000

Fuzzy string matching in a nutshell

Say we’re looking for a pattern in a blob of text. If you know the text has no typos, then determining whether it contains a pattern is trivial. In Python you can use the in function. You can also write a regex pattern with the re module from the standard library. But what about if the text contains typos? For instance, this might be the case with user inputs on a website, or with OCR outputs. This is a much harder problem.

Comic book panel segmentation

maxhalford25@gmail.com (Max Halford) — Sat, 05 Mar 2022 00:00:00 +0000

Edit (2023-05-26) – I’ve learnt about the Kumiko project, which is exactly devoted to slicing comic book panels. There’s even a live tool. I discovered it thanks to being pinged on this issue.

Motivation

I’ve recently been reading some comic books I used to devour as a kid. Especially those from the golden era of francophone comics: Thorgal, Lanfeust, XIII, Tintin, Largo Winch, Blacksad, Aldebaran, etc.

It’s not easy to get my hands on many of them. Luckily enough I found a website called ReadComicOnline which is delightfully profuse. It gives access to comics for free under the murky “fair use” copyright doctrine. I’m very doubtful about the legality of the website, but I’m still using it for lack of a better option.

Weighted sampling without replacement in pure Python

maxhalford25@gmail.com (Max Halford) — Fri, 24 Dec 2021 00:00:00 +0000

I’m working on a problem where I need to sample k items from a list without replacement. The sampling has to be weighted. In Python, numpy has random.choice method which allows doing this:

import numpy as np

n = 10
k = 3

np.random.seed(42)
population = np.arange(n)
weights = np.random.dirichlet(np.ones_like(population))

np.random.choice(population, size=k, replace=False, p=weights)

array([0, 9, 8])

I’m always wary of using numpy without thinking because I know it incurs some overhead. This overhead is usually meaningful when small amounts of data are involved. In such a case, a pure Python implementation may be faster.

Focal loss implementation for LightGBM

maxhalford25@gmail.com (Max Halford) — Sun, 20 Sep 2020 00:00:00 +0000

Edit (2021-01-26) – I initially wrote this blog post using version 2.3.1 of LightGBM. I’ve now updated it to use version 3.1.1. There are a couple of subtle but important differences between version 2.x.y and 3.x.y. If you’re using version 2.x.y, then I strongly recommend you to upgrade to version 3.x.y.

Motivation

If you’re reading this blog post, then you’re likely to be aware of LightGBM. The latter is a best of breed gradient boosting library. As of 2020, it’s still the go-to machine learning model for tabular data. It’s also ubiquitous in competitive machine learning.

Server-sent events in Flask without extra dependencies

maxhalford25@gmail.com (Max Halford) — Mon, 04 May 2020 00:00:00 +0000

Server-sent events (SSE) is a mechanism for sending updates from a server to a client. The fundamental difference with WebSockets is that the communication only goes in one direction. In other words, the client cannot send information to the server. For many usecases this is all you might need. Indeed, if you just want to receive notifications/updates/messages, then using a WebSocket is overkill. Once you’ve implemented the SSE functionality on your server, then all you need on a JavaScript client is an EventSource. Trust me, it’s very straightforward.

Target encoding done the right way

maxhalford25@gmail.com (Max Halford) — Sat, 13 Oct 2018 00:00:00 +0000

When you’re doing supervised learning, you often have to deal with categorical variables. That is, variables which don’t have a natural numerical representation. The problem is that most machine learning algorithms require the input data to be numerical. At some point or another a data science pipeline will require converting categorical variables to numerical variables.

There are many ways to do so:

Label encoding where you choose an arbitrary number for each category
One-hot encoding where you create one binary column per category
Vector representation a.k.a. word2vec where you find a low dimensional subspace that fits your data
Optimal binning where you rely on tree-learners such as LightGBM or CatBoost
Target encoding where you average the target value by category

Each and every one of these method has its own pros and cons. The best approach typically depends on your data and your requirements. If a variable has a lot of categories, then a one-hot encoding scheme will produce many columns, which can cause memory issues. In my experience, relying on LightGBM/CatBoost is the best out-of-the-box method. Label encoding is useless and you should never use it. However if your categorical variable happens to be ordinal then you can and should represent it with increasing numbers (for example “cold” becomes 0, “mild” becomes 1, and “hot” becomes 2). Word2vec and others such methods are cool and good but they require some fine-tuning and don’t always work out of the box.