Text-Processing on Max Halford

Text classification with Python 3.14's zstd module

maxhalford25@gmail.com (Max Halford) — Fri, 06 Feb 2026 00:00:00 +0000

Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms.

I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it will maintain an internal state. It’s particularly well suited for compressing small data. It’s perfect for the classify text via compression trick, which I described in a previous blog post 5 years ago.

Parsing garment descriptions with GPT-3

maxhalford25@gmail.com (Max Halford) — Sun, 20 Nov 2022 00:00:00 +0000

The task

You’ll have heard of GPT-3 if you haven’t been hiding under a rock. I’ve recently been impressed by Nat Friedman teaching GPT-3 to use a browser, and SeekWell generating SQL queries from free-text. I think the most exciting usecases are yet to come. But GPT-3 has a good chance of changing the way we approach mundane tasks at work.

I wrote an article a couple of months ago about a boring task I have to do at work. I got a few interesting suggestions by email. Raphaël suggested a question-answering model here. I’ve slowly become convinced that GPT-3 might be the perfect tool for the job. It’s cold and miserable where I am, so I thought it would be opportune to take GPT-3 for a spin 🧙

NLP at Carbonfact: how would you do it?

maxhalford25@gmail.com (Max Halford) — Tue, 06 Sep 2022 00:00:00 +0000

The task

I work at a company called Carbonfact. Our core value proposal is computing the carbon footprint of clothing items, expressed in carbon dioxide equivalent – $kgCO_2e$ in short. For instance, we started by measuring the footprint of shoes – no pun intended. We do these measurements with life cycle analysis (LCA) software we built ourselves. We use these analyses to fuel higher-level tasks for our clients, such as carbon accounting and sustainable procurement.

A life cycle analysis is essentially a recipe, the output of which is a carbon footprint assessment. Like any recipe, an LCA necessitates ingredients. In a cradle-to-gate scenario, this includes everything that is needed to make the product: the materials, the mass, the manufacturing methods, the transport between factories, etc. In our experience, the biggest impact on a product’s footprint come from the materials which it is made of.

Fuzzy regex matching in Python

maxhalford25@gmail.com (Max Halford) — Mon, 04 Apr 2022 00:00:00 +0000

Fuzzy string matching in a nutshell

Say we’re looking for a pattern in a blob of text. If you know the text has no typos, then determining whether it contains a pattern is trivial. In Python you can use the in function. You can also write a regex pattern with the re module from the standard library. But what about if the text contains typos? For instance, this might be the case with user inputs on a website, or with OCR outputs. This is a much harder problem.

OCR spelling correction is hard

maxhalford25@gmail.com (Max Halford) — Sun, 06 Mar 2022 00:00:00 +0000

I recently saw SymSpell pop up on Hackernews. It claims to be a million times faster than Peter Norvig’s spelling corrector. I think it’s great that there’s a fast open source solution for spelling correction. But in my experience, the most challenging aspect of spelling correction is not necessarily speed.

When I worked at Alan, I mostly wrote logic to extract structured information from medical documents. After some months working on the topic, I have to admit I hadn’t cracked the problem. The goal was to process >80% of documents with no human interaction, but when I left we had only reached 35%. However, I developed a good understanding of what made this task so difficult.

Homoglyphs: different characters that look identical

maxhalford25@gmail.com (Max Halford) — Thu, 19 Aug 2021 00:00:00 +0000

A wild homoglyph appears

For instance, can you tell if there’s a difference between H and Η? How about N and Ν? These characters may seem identical, but they are actually different. You can try this out for yourself in Python:

>>> 'H' == 'Η'
False

>>> 'N' == 'Ν'
False

Indeed, these all represent different Unicode characters:

>>> ord('H'), ord('Η')
(72, 919)

>>> ord('N'), ord('Ν')
(78, 925)

Η in fact represents the capital Eta letter, while Ν is a capital Nu. In fact, entering H or Η in Google will produce different results. The same goes for N and Ν.

Automated document processing at Alan

maxhalford25@gmail.com (Max Halford) — Thu, 10 Jun 2021 00:00:00 +0000

Text classification by data compression

maxhalford25@gmail.com (Max Halford) — Tue, 08 Jun 2021 00:00:00 +0000

Edit – I posted this on Hackernews and got some valuable feedback. Many brought up the fact that you should be able to reuse the internal state of the compressor instead of recompressing the training data each time a prediction is made. There’s also some insightful references to data compression theory and its ties to statistical learning

Edit (2025-06-29) – *Python 3.14 introduces compression.zstd, which implement’s Facebook’s Zstandard compression algorithm, as discussed in the comments section below.

Reducing the memory footprint of a scikit-learn text classifier

maxhalford25@gmail.com (Max Halford) — Sun, 11 Apr 2021 00:00:00 +0000

Context

This week at Alan I’ve been working on parsing French medical prescriptions. There are three types of prescriptions: lenses, glasses, and pharmaceutical prescriptions. Different information needs to be extracted depending on the prescription type. Therefore, the first step is to classify the prescription. The prescriptions we receive are pictures taken by users with their phone. We run each image through an OCR to obtain a text transcription of the image. We can thus use the text transcription to classify the prescription.

Converting Amazon Textract tables to pandas DataFrames

maxhalford25@gmail.com (Max Halford) — Thu, 14 Jan 2021 00:00:00 +0000

I’m currently doing a lot of document processing at work. One of my tasks is to extract tables from PDF files. I evaluated Amazon Textract’s table extraction capability as part of this task. It’s very well documented, as is the rest of Textract. I was slightly disappointed by the examples, but nothing serious.

I wanted to write this short blog post to share a piece of code I use to convert tables extracted through Amazon Textract to pandas.DataFrames. I’ll be using the following anonymised image as an example:

Unsupervised text classification with word embeddings

maxhalford25@gmail.com (Max Halford) — Sat, 03 Oct 2020 00:00:00 +0000

Edit – since writing this article, I have discovered that the method I describe is a form of zero-shot learning. So I guess you could say that this article is a tutorial on zero-shot learning for NLP.

Edit – I stumbled on a paper entitled “Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings” which proposes something very similar. The paper is rather well written, so you might want to check it out. Note that they call the tech -> technology trick “label enrichment”.