Machine-Learning on Max Halford

Text classification with Python 3.14's zstd module

maxhalford25@gmail.com (Max Halford) — Fri, 06 Feb 2026 00:00:00 +0000

Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms.

I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it will maintain an internal state. It’s particularly well suited for compressing small data. It’s perfect for the classify text via compression trick, which I described in a previous blog post 5 years ago.

A training set for bike sharing forecasting

maxhalford25@gmail.com (Max Halford) — Thu, 04 Apr 2024 00:00:00 +0000

Last night I went to a Toulouse Data Science meetup. The talks were about generative AI and information retrieval, which aren’t topics I’m knowledgeable about. However, one of the speakers was a friend of mine, so I went to support him. Toulouse used to be my hometown, so I bumped into a few people I knew. It was a nice evening.

Text classification by data compression

maxhalford25@gmail.com (Max Halford) — Tue, 08 Jun 2021 00:00:00 +0000

Edit – I posted this on Hackernews and got some valuable feedback. Many brought up the fact that you should be able to reuse the internal state of the compressor instead of recompressing the training data each time a prediction is made. There’s also some insightful references to data compression theory and its ties to statistical learning

Edit (2025-06-29) – *Python 3.14 introduces compression.zstd, which implement’s Facebook’s Zstandard compression algorithm, as discussed in the comments section below.

Reducing the memory footprint of a scikit-learn text classifier

maxhalford25@gmail.com (Max Halford) — Sun, 11 Apr 2021 00:00:00 +0000

Context

This week at Alan I’ve been working on parsing French medical prescriptions. There are three types of prescriptions: lenses, glasses, and pharmaceutical prescriptions. Different information needs to be extracted depending on the prescription type. Therefore, the first step is to classify the prescription. The prescriptions we receive are pictures taken by users with their phone. We run each image through an OCR to obtain a text transcription of the image. We can thus use the text transcription to classify the prescription.

Unsupervised text classification with word embeddings

maxhalford25@gmail.com (Max Halford) — Sat, 03 Oct 2020 00:00:00 +0000

Edit – since writing this article, I have discovered that the method I describe is a form of zero-shot learning. So I guess you could say that this article is a tutorial on zero-shot learning for NLP.

Edit – I stumbled on a paper entitled “Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings” which proposes something very similar. The paper is rather well written, so you might want to check it out. Note that they call the tech -> technology trick “label enrichment”.

Focal loss implementation for LightGBM

maxhalford25@gmail.com (Max Halford) — Sun, 20 Sep 2020 00:00:00 +0000

Edit (2021-01-26) – I initially wrote this blog post using version 2.3.1 of LightGBM. I’ve now updated it to use version 3.1.1. There are a couple of subtle but important differences between version 2.x.y and 3.x.y. If you’re using version 2.x.y, then I strongly recommend you to upgrade to version 3.x.y.

Motivation

If you’re reading this blog post, then you’re likely to be aware of LightGBM. The latter is a best of breed gradient boosting library. As of 2020, it’s still the go-to machine learning model for tabular data. It’s also ubiquitous in competitive machine learning.

Speeding up scikit-learn for single predictions

maxhalford25@gmail.com (Max Halford) — Tue, 31 Mar 2020 00:00:00 +0000

It is now common practice to train machine learning models offline before putting them behind an API endpoint to serve predictions. Specifically, we want an API route which can make a prediction for a single row/instance/sample/data point/individual (call it what you want). Nowadays, we have great tools to do this that take care of the nitty-gritty details, such as Cortex, MLFlow, Kubeflow, and Clipper. There are also paid services that hold your hand a bit more, such as DataRobot, H2O, and Cubonacci. One could argue that deploying machine learning models has never been easier.

Global explanation of machine learning with sensitivity analysis @ MASCOT-NUM

maxhalford25@gmail.com (Max Halford) — Tue, 10 Mar 2020 00:00:00 +0000

Bayesian linear regression for practitioners

maxhalford25@gmail.com (Max Halford) — Wed, 26 Feb 2020 00:00:00 +0000

Motivation

Suppose you have an infinite stream of feature vectors $x_i$ and targets $y_i$. In this case, $i$ denotes the order in which the data arrives. If you’re doing supervised learning, then your goal is to estimate $y_i$ before it is revealed to you. In order to do so, you have a model which is composed of parameters denoted $\theta_i$. For instance, $\theta_i$ represents the feature weights when using linear regression. After a while, $y_i$ will be revealed, which will allow you to update $\theta_i$ and thus obtain $\theta_{i+1}$. To perform the update, you may apply whichever learning rule you wish – for instance most people use some flavor of stochastic gradient descent. The process I just described is called online supervised machine learning. The difference between online machine learning and the more traditional batch machine learning is that an online model is dynamic and learns on the fly. Online learning solves a lot of pain points in real-world environments, mostly because it doesn’t require retraining models from scratch every time new data arrives.

Under-sampling a dataset with desired ratios

maxhalford25@gmail.com (Max Halford) — Tue, 17 Dec 2019 00:00:00 +0000

Introduction

I’ve just spent a few hours looking at under-sampling and how it can help a classifier learn from an imbalanced dataset. The idea is quite simple: randomly sample the majority class and leave the minority class untouched. There are more sophisticated ways to do this – for instance by creating synthetic observations from the minority class à la SMOTE – but I won’t be discussing that here.

I checked out the imblearn library and noticed they have an implementation of random under-sampling aptly named RandomUnderSampler. It contains a sampling_strategy parameter which gives some control over the sampling. By the default the observations are resampled so that each class is equally represented:

A smooth approach to putting machine learning into production

maxhalford25@gmail.com (Max Halford) — Sat, 13 Jul 2019 00:00:00 +0000

Putting machine learning into production is hard. Usually I’m doubtful of such statements, but in this case I’ve never met anyone for whom everything has gone smoothly. Most data scientists might agree that there is a huge gap between their local environment and a live environment. In fact, “productionalizing” machine learning is such a complex topic that entire companies have risen to address the issue. I’m not just talking about running a gigantic grid search and finding the best model, I’m talking about putting a machine learning model live so that it actually has a positive impact on your business/project. Off the top of my head: Cubonacci, H2O, Google AutoML, Amazon Sagemaker, and DataRobot. In other words people are making money off businesses because data scientists and engineers are having a hard putting their models into production. In my opinion if a data scientist can’t put her model into production herself then something is wrong. Life should be simpler.

Target encoding done the right way

maxhalford25@gmail.com (Max Halford) — Sat, 13 Oct 2018 00:00:00 +0000

When you’re doing supervised learning, you often have to deal with categorical variables. That is, variables which don’t have a natural numerical representation. The problem is that most machine learning algorithms require the input data to be numerical. At some point or another a data science pipeline will require converting categorical variables to numerical variables.

There are many ways to do so:

Label encoding where you choose an arbitrary number for each category
One-hot encoding where you create one binary column per category
Vector representation a.k.a. word2vec where you find a low dimensional subspace that fits your data
Optimal binning where you rely on tree-learners such as LightGBM or CatBoost
Target encoding where you average the target value by category

Each and every one of these method has its own pros and cons. The best approach typically depends on your data and your requirements. If a variable has a lot of categories, then a one-hot encoding scheme will produce many columns, which can cause memory issues. In my experience, relying on LightGBM/CatBoost is the best out-of-the-box method. Label encoding is useless and you should never use it. However if your categorical variable happens to be ordinal then you can and should represent it with increasing numbers (for example “cold” becomes 0, “mild” becomes 1, and “hot” becomes 2). Word2vec and others such methods are cool and good but they require some fine-tuning and don’t always work out of the box.

Subsampling a training set to match a test set - Part 1

maxhalford25@gmail.com (Max Halford) — Mon, 19 Jun 2017 00:00:00 +0000

Edit – it’s 2022 and I still haven’t written a part 2. That’s because I believe this problem is easily solved with adversarial validation.

Some friends and I recently qualified for the final of the 2017 edition of the Data Science Game competition. The first part was a Kaggle competition with data provided by Deezer. The problem was a binary classification task where one had to predict if a user was going to listen to a song that was proposed to him. Like many teams we extracted clever features and trained an XGBoost classifier, classic. However, the one special thing we did was to subsample our training set so that it was more representative of the test set.

Predire la disponibilité des Velib' @ Toulouse Data Science Meetup

maxhalford25@gmail.com (Max Halford) — Wed, 30 Mar 2016 00:00:00 +0000

The Naïve Bayes classifier

maxhalford25@gmail.com (Max Halford) — Thu, 10 Sep 2015 00:00:00 +0000

The objective of a classifier is to decide to which class (also called label) to assign an observation based on observed data. In supervised learning, this is done by taking into account previous classifications. In other words if we know that certain observations are classified in a certain way, the goal is to determine the class of a new observation. The first group of observations on which the classifier is built is called the training set.

An introduction to genetic algorithms

maxhalford25@gmail.com (Max Halford) — Sun, 02 Aug 2015 00:00:00 +0000

The goal of genetic algorithms (GAs) is to solve problems whose solutions are not easily found (ie. NP problems, nonlinear optimization, etc.). For example, finding the shortest path from A to B in a directed graph is easily done with Djikstra’s algorithm, it can be solved in polynomial time. However the time to find the smallest path that joins all points on a non-directed graph, also known as the Travelling Salesman Problem (TSP) increases exponentially as the number of points increases. More generally, GAs are useful for problems where an analytical approach is complicated or even impossible. By giving up on perfection they manage to find a good approximation of the optimal solution.