Slides on Max Halford

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

Incremental machine learning

## Online machine learning with creme

### Max Halford

#### 11th of May 2019, Amsterdam

</div>

???

Hello!

---

### Outline

.bullets[
1. You're not doing ML the correct way
2. Cool kids do online learning
3. Introducing `creme`, the fresh kid on the block
4. Forecasting League of Legends match durations
5. Future work
]

???

My goal is to convince you that the way most people do machine learning isn't the way to go in production scenarios. Online learning is more suitable and nullifies many issues. I'll introduce a new library me and some friends have been working on, and I'll show you we're using for a pet project
trying to forecast the duration of League of Legends matches.

---

### Batch learning

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```

???

The most common form of machine learning is called batch learning. It's what you do when you use scikit-learn and do Kaggle competitions. I won't dwell on this too long as I'm pretty sure you're all acquainted with it. In short: load, fit, predict.

---

### Batch machine learning in production

???

There isn't a clear pattern as to how to put into production a batch learning system. Typically this involves training a model periodically, and then serializing it before saving it somewhere. API calls can then be issues to make predictions. This is such a difficult thing to get right that startups and tech giants are building products around it.

---

### Batch machine learning in production is hard

???

Works fine for Kaggle cases and when you have a clear split between train and test. But things are not so smooth in production.

---

background-color: #2ac380
class: center, middle, white

# Models have to be retrained from scratch with new data

---

background-color: #dbaaa8
class: center, middle, white

# Ever increasing computational requirements

---

background-color: #1f282d
class: center, middle, white

# Models are static and rot faster than bananas 🍌

---

background-color: #e69138
class: center, middle, white

# Features that you develop locally might not be available in real-time

???

This is more subtle. Sometimes it isn't possible to recreate features for past instances. If you don't store information for an instance at every particular point in time, then it isn't possible to reproduce the true state during your feature extraction process. In other words you need a solid data engineering team that cares about time.

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

background-color: #607bd4
class: middle, white

## Online learning

.bigbullets[
- Data comes from a stream
- Models learn 1 observation at a time
- Observations do not have to be stored
- Features and labels are dynamic
]

---

background-color: #FF7F50
class: middle, white

## Different names, same thing

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Out-of-core learning
]

---

background-color: #008080
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

### Online learning in a nutshell

---

background-color: #e66868ff
class: middle, white

## Why is batch learning so popular?

.bigbullets[
- Taught at university
- (Bad) habits
- Hype
- Kaggle
- Library availability
- Higher accuracy
]

---

.bullets[
- Python library for doing online learning
- API heavily inspired from `sklearn` and easy to pick up
- Focus on feature extraction, not just learning
- First commit in January 2019
- Version `0.1.0` released earlier this week
]

---

### Observations

Representing an observation with a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Python's standard library plays nicely with `dict`s

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 height: 29.7,
 width: 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator** and consumes a tiny amount of memory
- The point is that we want to **handle data points one by one**
- Source depends on your use case (can be Kakfa producer, CSV file, HTTP requests, etc.)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
 model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred_before = model.predict_one(x)
 model.fit_one(x, y)
* y_pred_after = model.predict_one(x)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Monitoring performance

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

* metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need to do any cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

```python
from creme import linear_model
from creme import metrics
from creme import preprocessing
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

scale = preprocessing.StandardScaler()
lin_reg = linear_model.LogisticRegression()

* model = scale | lin_reg # Pipeline shorthand

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
 metric.update(y, y_pred)
 print(metric)
```

---

### `creme`'s current modules

.left-column[
- `cluster`
- `compat`
- `compose`
- `datasets`
- `dummy`
- `ensemble`
- `feature_extraction`
- `feature_selection`
- `impute`
- `linear_model`
]
.right-column[
- `model_selection`
- `multiclass`
- `naive_bayes`
- `optim`
- `preprocessing`
- `reco`
- `stats`
- `stream`
- `tree`
- `utils`
]

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$ (running sum of squares)
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

With the mean and the variance, we can scale incoming data so that it has zero mean and unit variance.

```python
>>> scaler = creme.preprocessing.StandardScaler()

>>> for x in [2, 3, 4, 5]:
... print(x, 'becomes', scaler.fit_one(x)['x'])
2 becomes 0.0
3 becomes 0.9999999999999996
4 becomes 1.224744871391589
5 becomes 1.3416407864998738

```

---

### Linear regression (1)

Model is $y_t = \langle w_t x_t \rangle + b_t$. The weights $w_t$ can be learnt with any online gradient descent algorithm, for example:

- Stochastic gradient descent (SGD)
- Adam
- RMSProp
- Follow the Regularized Leader (FTRL)

```python
from creme import linear_model
from creme import optim

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01)
)
```

---

### Linear regression (2)

The intercept term $b_t$ is difficult to learn

Some people (Léon Bottou, scikit-learn) suggest to use a lower learning rate for the intercept than for the weights (heuristic but okay)

`creme` uses any running statistic from the `creme.stats` module, which is a powerful trick

```python
from creme import linear_model
from creme import optim
from creme import stats

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01),
 intercept=stats.RollingMean(42)
)
```

---

### Online aggregations

```python
>>> import creme

>>> X = [
... {'place': 'Taco Bell', 'revenue': 42},
... {'place': 'Burger King', 'revenue': 16},
... {'place': 'Burger King', 'revenue': 24},
... {'place': 'Taco Bell', 'revenue': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='revenue',
... by='place',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'revenue_mean_by_place': 42.0}
{'revenue_mean_by_place': 16.0}
{'revenue_mean_by_place': 20.0}
{'revenue_mean_by_place': 50.0}
```

---

### Bagging (1)

Bagging is a popular and simple ensemble algorithm:

1. Pick $m$ base models (usually identical copies)
2. Train each model on a sample with replacement
3. Average the predictions of each model on the test set

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

---

### Bagging (2)

`ensemble.BaggingClassifier` is very simple:

```python
def fit_one(self, x, y):

for estimator in self.estimators:
 for _ in range(self.rng.poisson(1)):
 estimator.fit_one(x, y)

return self

def predict_proba_one(self, x):
 y_pred = statistics.mean(
 estimator.predict_proba_one(x)[True]
 for estimator in self.estimators
 )
 return {
 True: y_pred,
 False: 1 - y_pred
 }
```

---

### League of Legends match duration forecasting (1)

---

### League of Legends match duration forecasting (2)

---

### Architecture

---

### Django model

```python
from django.db import models
from picklefield.fields import PickledObjectField

class CremeModel(models.Model):
 name = models.TextField(unique=True)
 pipeline = PickledObjectField()

class Meta:
 db_table = 't_models'
 verbose_name_plural = 'models'

def fit_one(self, x, y):
 self.pipeline.fit_one(x, y)
 return self

def predict_one(self, x):
 return self.pipeline.predict_one(x)
```

---

### Code for predicting

```python
match = fetch_match(match_id)

model = models.CremeModel.objects.get(name='My awesome model')

duration = model.predict_one(match.raw_info)

match.predicted_ended_at = match.started_at + predicted_duration
match.predicted_by = model
match.save()
```

---

### Code for training

When the match ends, it has a `true_duration` property.

```python
model = models.CremeModel.objects.get(id=match.predicted_by.id)

model.fit_one(
 x=match.raw_info,
 y=match.true_duration.seconds
)

model.save()
```

---

### Code for calculating performance

Just some Django black magic.

```python
duration = ExpressionWrapper(
 Func(F('predicted_ended_at') - F('ended_at'), function='ABS'),
 output_field=fields.DurationField()
)

agg = models.Match.objects.exclude(ended_at__isnull=True)\
 .annotate(duration=duration)\
 .aggregate(Avg('duration'))

avg_error = agg['duration__avg']
```

---

### Benefits of online learning

.bullets[
- No need to schedule model training
- Easy to debug and to monitor
- You're not "far" from production
- Way more fun than batch learning
]

---

### Future work

.left-column[
.bullets[
- Decision trees (nearly there!)
- Gradient boosting
- Bayesian linear models
- More feature extraction
- More models
- More benchmarks
- Many issues [on GitHub](https://github.com/creme-ml/creme/issues)
]
]
.right-column[

]

---

### If you want to contribute

- [creme-ml.github.io](https://creme-ml.github.io/)
- [github.com/creme-ml](https://github.com/creme-ml/)
- You can shoot emails to [maxhalford25@gmail.com](mailto:maxhalford25@gmail.com)
- Get in contact if you want to try `creme` and want advice
- Spread the word!

---

# Thanks for listening!

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

Machine learning incrémental: des concepts à la pratique

## Machine learning incrémental: des concepts à la pratique

### Max Halford

#### 28 mai 2019

#### Toulouse Data Science Meetup

</div>

???

Hello!

---

### Outline

.bullets[
1. You're doing machine learning the wrong way 😱
2. Cool kids do online learning 😎
3. Introducing `creme`, a Python lib for online learning 🐍
4. Bike stations forecasting demo 🚲 🔮
]

---

### PyData Amsterdam 2019 🐍 🇳🇱 🧀

---

### Batch learning

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```
---

### Batch machine learning in production

---

background-color: #2ac380
class: center, middle, white

# Models have to be retrained from scratch with new data ➰➰➰

---

background-color: #663399
class: center, middle, white

# Models needs increasing amounts of power 🔌

---

background-color: #1f282d
class: center, middle, white

# Models are static and "rot" faster than bananas 🍌

---

background-color: #e69138
class: center, middle, white

# Models that work locally don't always work in production 😭

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

background-color: #607bd4
class: middle, white

## Online learning

.bigbullets[
- Data comes from a stream
- Models learn 1 observation at a time
- Features and labels are dynamic
]

---

## Everything changes 💥

</div>
</div>

---

background-color: #008080
class: middle, white

## Different names, same thing 🤷

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Continuous learning
- Out-of-core learning
]

---

background-color: #FF7F50
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

### Online learning in a nutshell

---

background-color: #e66868ff
class: middle, white

## Why is batch learning so popular?

---

# Questions?

---

.bullets[
- Online machine learning library for Python 🐍
- Easy to pick up API inspired by `sklearn`
- Written with production scenarios in mind
- First commit in January 2019
- Version `0.2.0` released yesterday
]

---

#### scikit-learn

```python
from sklearn import datasets
from sklearn import linear_model

X, y = datasets.load_boston(return_X_y=True)
model = linear_model.LinearRegression()

model.fit(X, y)
```

#### creme

```python
from creme import linear_model
from creme import stream
from sklearn import datasets

X_y = stream.iter_sklearn_dataset(datasets.load_boston)
model = linear_model.LinearRegression()

for x, y in X_y:
 model.fit_one(x, y)
```

---

### Features

Representing a set of features using a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Python's standard library plays nicely with `dict`s

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 height: 29.7,
 width: 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator** and consumes a tiny amount of memory
- The point is that we only need one data point at a time
- Source depends on your use case (CSV file, Kafka consumer, HTTP requests)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred = model.predict_one(x)
 model.fit_one(x, y)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Progressive validation 💯

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need for cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

```python
from creme import compose
from creme import linear_model
from creme import preprocessing

scale = preprocessing.StandardScaler()
lin_reg = linear_model.LogisticRegression()

# You can do this...
model = compose.Pipeline([
 ('scale', scale),
 ('lin_reg', lin_reg
])

# Or this...
model = scale | lin_reg
```

---

# Questions?

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

Using the mean and the variance, we can rescale incoming data.

```python
>>> scaler = creme.preprocessing.StandardScaler()

>>> for x in [2, 3, 4, 5]:
... features = {'x': x}
... scaler.fit_one(features)
... new_x = scaler.transform_one(features)['x']
... print(f'{x} becomes {new_x})
2 becomes 0.0
3 becomes 0.9999999999999996
4 becomes 1.224744871391589
5 becomes 1.3416407864998738

```

---

### Linear regression (1)

Model is $y_t = \langle w_t x_t \rangle + b_t$. The weights $w_t$ can be learnt with any online gradient descent algorithm, for example:

- Stochastic gradient descent (SGD)
- Adam
- RMSProp
- Follow the Regularized Leader (FTRL)

```python
from creme import linear_model
from creme import optim

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01)
)
```

---

### Linear regression (2)

Some people (Léon Bottou, scikit-learn) suggest to use a lower learning rate for the intercept than for the weights (heuristic but okay)

`creme` uses any running statistic from the `creme.stats` module, which is a powerful trick

```python
from creme import linear_model
from creme import optim
from creme import stats

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01),
 intercept=stats.RollingMean(42)
)
```

---

### Online aggregations

```python
>>> import creme

>>> X = [
... {'meal': 🍕, 'sales': 42},
... {'meal': 🍔, 'sales': 16},
... {'meal': 🍔, 'sales': 24},
... {'meal': 🍕, 'sales': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='sales',
... by='meal',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'sales_mean_by_meal': 42.0}
{'sales_mean_by_meal': 16.0}
{'sales_mean_by_meal': 20.0}
{'sales_mean_by_meal': 50.0}
```

---

### Bagging (1)

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

This leads to a simple and efficient online algorithm.

---

### Bagging (2)

`ensemble.BaggingClassifier` is very simple:

```python
def fit_one(self, x, y):

for estimator in self.estimators:
 for _ in range(self.rng.poisson(1)):
 estimator.fit_one(x, y)

return self

def predict_proba_one(self, x):
 y_pred = statistics.mean(
 estimator.predict_proba_one(x)[True]
 for estimator in self.estimators
 )
 return {
 True: y_pred,
 False: 1 - y_pred
 }
```

---

### Decision trees 🌳

- A version of Hoeffding trees is being implemented
- Basic idea:
 - Start with a leaf 🍃
 - Find the leaf where an observation belongs 🔎
 - Update the leaf's sufficient statistics 📊
 - Measure information gain every so often 🔬
 - Split when the information gain is good enough 🍂
- Mondrian trees 👨‍🎨 are another possibility but they only work for continuous attributes

---

# Questions?

---

### `creme`'s current modules

<div style="display: flex; justify-content: space-around;">
 <ul>
 <li><pre>cluster</pre></li>
 <li><pre>compat</pre></li>
 <li><pre>compose</pre></li>
 <li><pre>datasets</pre></li>
 <li><pre>dummy</pre></li>
 <li><pre>ensemble</pre></li>
 <li><pre>feature_extraction</pre></li>
 </ul>
 <ul>
 <li><pre>feature_selection</pre></li>
 <li><pre>impute</pre></li>
 <li><pre>linear_model</pre></li>
 <li><pre>model_selection</pre></li>
 <li><pre>multiclass</pre></li>
 <li><pre>naive_bayes</pre></li>
 <li><pre>optim</pre></li>
 <li><pre>plot</pre></li>
 </ul>
 <ul>
 <li><pre>preprocessing</pre></li>
 <li><pre>proba</pre></li>
 <li><pre>reco</pre></li>
 <li><pre>stats</pre></li>
 <li><pre>stream</pre></li>
 <li><pre>tree</pre></li>
 <li><pre>utils</pre></li>
 </ul>

</div>

---

### Cool stuff in `creme` we skipped 😢

.bullets[
- Clustering
- Factorization machines
- Feature selection
- Passive-aggressive models
- Recommender systems
- Histograms
- Skyline queries
- Fourier transforms
- Imputation
- Naive Bayes
]

---

### Alternative frameworks

---

### Benefits of online learning

.bullets[
- No need to schedule model training
- Easy to monitor
- You're very close to production
- Way more fun than batch learning
]

---

### Current work

.bullets[
- Decision trees (nearly there)
- Gradient boosting (easy)
- Bayesian linear models (part of my PhD)
- Latent Dirichlet Allocation (ask Raphael)
- Many issues [on GitHub](https://github.com/creme-ml/creme/issues)
]

---

### What next?

- [creme-ml.github.io](https://creme-ml.github.io/)
- [github.com/creme-ml](https://github.com/creme-ml/)
- You can send emails to [maxhalford25@gmail.com](mailto:maxhalford25@gmail.com)
- Get in touch if you want help and/or advice
- Starring us on GitHub helps a lot 🌟

---

# Thanks for listening!

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

The Benefits of Online Learning

## The Benefits of Online Learning

### (and other shenanigans)

#### Max Halford

</div>

???

Hello!

---

## Outline

---

### A bit about me

.right-column[
- 3rd year PhD student in Toulouse
- PhD on Bayesian networks applied to database cost models
- Topics of interest:
 - Online machine learning
 - Systems for machine learning
 - Machine learning for systems
 - Competitive machine learning
 - Fair and explainable learning
- Into opensource (mostly Python)
- Kaggle Master (rank 247)
]

---

### Batch learning in a nutshell

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```

---

## Lambda architecture

---

background-color: #e66868ff
class: middle, white

## Batch learning in production has issues

1. Models are retrained from scratch with new data 🕒
2. Models require powerful machines 💰
3. Models are static and "rot" faster than bananas 🍌
4. Models that work in development don't always work in production 🤷
]

???

- [Continuum: a platform for cost-aware low-latency continual learning](https://blog.acolyer.org/2018/11/21/continuum-a-platform-for-cost-aware-low-latency-continual-learning/)
- [Applied machine learning at Facebook: a datacenter infrastructure perspective](https://blog.acolyer.org/2018/12/17/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/)

As we looked at last month with Continuum, the latency of incorporating the latest data into the models is also really important. There’s a nice section of this paper where the authors study the impact of losing the ability to train models for a period of time and have to serve requests from stale models. The Community Integrity team for example rely on frequently trained models to keep up with the ever changing ways adversaries try to bypass Facebook’s protections and show objectionable content to users. Here training iterations take on the order of days. Even more dependent on the incorporation of recent data into models is the news feed ranking. “Stale News Feed models have a measurable impact on quality.” And if we look at the very core of the business, the Ads Ranking models, “we learned that the impact of leveraging a stale ML model is measured in hours. In other words, using a one-day-old model is measurably worse than using a one-hour old model.” One of the conclusions in this section of the paper is that disaster recovery / high availability for training workloads is key importance. (Another place to practice your chaos engineering ;) ).

---

## Banana rotting time

---

### "If everyone's doing it, it's got to be the best way, right?"

---

## Kappa architecture

???

This looks great, but our favorite models such as LightGBM, can be updated incrementally.

---

background-color: #b5ddd1
class: middle

## Online machine learning

.bigbullets[
- Subdiscipline of machine learning
- Data is a stream, potentially infinite
- Models learn from one observation at a time
- Features and labels can be dynamic
- Can be supervised or unsupervised
]

---

background-color: #2ac380
class: middle, white

## Different names, same thing 🤔

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Continuous learning
- Out-of-core learning
]

---

background-color: #607bd4
class: middle, white

## Benefits of online machine learning

.bigbullets[
1. Models don't have to be retrained
2. Nothing is too big
3. Online models (usually) adapt to drift
4. Model development is closer to reality
5. Training can be monitored in real-time
]

---

background-color: #e69138
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

background-color: #FF7F50
class: middle, white

## Why is batch learning so popular?

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

.bullets[
- Online machine learning library for Python 🐍
- Easy to pick up API inspired by scikit-learn
- Written with production scenarios in mind
- First commit in January 2019
- Version `0.4.3` released this week (with wheels!)
]

---

### PyData Amsterdam, May 2019 🐍 🇳🇱 🧀

---

### Features

Representing a set of features using a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Web request payloads are dictionaries

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 'height': 29.7,
 'width': 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator**, it doesn't hold data in memory
- Source depends on your use case (CSV file, Kafka consumer, HTTP requests)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred = model.predict_one(x)
 model.fit_one(x, y)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Progressive validation 💯

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need for cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

counts = feature_extraction.CountVectorizer()
tdidf = feature_extraction.TFIDFVectorizer()

scale = preprocessing.StandardScaler()

log_reg = linear_model.LogisticRegression()

model = (counts + tdidf) | scale | log_reg

model.draw()
```
]

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

Using the mean and the variance, we can rescale incoming data.

```python
>>> scaler = creme.preprocessing.StandardScaler()

```

In practice, works better than normalized gradient descent 😲

---

### Linear models

Model is:

$$\hat{y}_t = f(w_t, x_t) + b_t$$

Update weights with gradients:

$$w_{t+1} = u(w_t, x_t, \partial L(y_t, \hat{y}_t))$$

Many models can be derived, for example:

- Use Hinge loss for SVM
- Add L1/L2 regularisation for LASSO/ridge
- Add interactions for factorization machines

---

### Many online optimizers to choose from

.bullets[
- Stochastic gradient descent (SGD)
- Passive-Aggressive (PA)
- ADAM
- RMSProp
- Follow the Regularized Leader (FTRL)
- Approximate Large Margin Algorithm (ALMA)
]

<div align="center">
Many variants of each, as you know
</div>

---

### Bayesian linear models

We want the posterior target distribution on the target:

$$\color{forestgreen} p(y\_t | x\_t) \color{black} \propto \color{crimson} p(y_t | w_t, x_t) \color{royalblue} p(w_t)$$

We first need to compute the posterior distribution of the weights:

$$\color{blueviolet} p(w\_{t} | w\_{t-1}, x\_t, y\_t) \color{black} \propto \color{crimson} p(y\_t | w\_{t-1}, x\_t) \color{royalblue} p(w\_{t-1})$$

This is old-school Bayesian learning, it is different from and predecesses the Monte-Carlo mumbo-jumbo.

---

### Online belief updating

Before any data comes in, the model parameters follow the initial distribution we picked, which is $\color{royalblue} p(w\_0)$. Next, once the first observation $(x\_0, y\_0)$ arrives, we can obtain the distribution of $w\_1$:

$$\color{blueviolet} p(w\_1 | w\_0, x\_0, y\_0) \color{black} \propto \color{crimson} p(y\_0 | w\_0, x\_0) \color{royalblue} p(w\_0)$$

Once the second observation $(x\_1, y\_1)$ is available, the distribution of the model parameters is obtained in the same way:

$$\color{blueviolet} p(w\_2 | w\_1, x\_1, y\_1) \color{black} \propto \color{crimson} p(y\_1 | w\_1, x\_1) \color{royalblue} p(w\_1) \propto \color{crimson} p(y\_1 | w\_1, x\_1) \color{blueviolet} p(w\_1 | w\_0, x\_0, y\_0)$$

The $\propto$ symbol means there is an analytical formula that can be derived.

---

### Nearest neighbors

.bullets[
- Three parameters:
 1. The distance function
 2. The number of neighbors
 3. The window size
- .green[Naturally adapts to drift]
- .red[Lazy]
]

---

### Decision trees 🌳

- A version of Hoeffding trees is being implemented
- Basic idea:
 - Start with a leaf 🍃
 - Find the leaf where an observation belongs 🔎
 - Update the leaf's "sufficient statistics" 📊
 - Measure information gain every so often 🔬
 - Split when the information gain is good enough 🍂
- Mondrian trees 👨‍🎨 are another possibility but they only work for continuous attributes

---

### Decision trees 🌳

Quality criterion of split $x < t$ can be evaluated with:

$$P(y \mid x < t) = \frac{P(x < t \mid y) \times P(y)}{P(x < t)}$$

and:

$$P(y \mid x \geq t) = \frac{(1 - P(x < t \mid y)) \times P(y)}{1 - P(x < t)}$$

- For classification, $P(x < t \mid y)$ is a set of online CDFs and $P(y)$ is a PMF.
- For regression, $P(x < t \mid y)$ is a 2D CDF and $P(y)$ is a PDF
- All these distributions can be updated online

---

### Bagging

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

This leads to a simple and efficient online algorithm:

```python
for model in models:
 for _ in range(random.poisson(λ=1)):
 model.fit_one(x, y)
```

---

### (S)(N)(AR)(I)(MA)(X)

ARMA model is defined as so:

$$\hat{y}\_t = \sum\_{i=1}^p \alpha\_i y\_{t-i} + \sum\_{i=1}^q \beta\_i (y\_{t-i} - \hat{y}\_{t-i}) $$

Classically, Kalman filters are used to find the weights $\alpha\_i$ and $\beta\_i$. But $y\_{t-i}$ and $\hat{y}\_{t-i}$ can also be [seen as features in an online setting](https://dl.acm.org/citation.cfm?id=3016160):

- Seasonality can be handled online
- Any online learning model can be used
- Detrending by differencing can be done online
- Heteroscedasticity can be handled online
- Exogenous variables can be added

---

### Online aggregated features

```python
>>> import creme

>>> X = [
... {'meal': 'tika masala', 'sales': 42},
... {'meal': 'kale salad', 'sales': 16},
... {'meal': 'kale salad', 'sales': 24},
... {'meal': 'tika masala', 'sales': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='sales',
... by='meal',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'sales_mean_by_meal': 42.0}
{'sales_mean_by_meal': 16.0}
{'sales_mean_by_meal': 20.0}
{'sales_mean_by_meal': 50.0}
```

---

background-color: #008080
class: middle, white

## There is much more

.bullets[
- Half-space trees for anomaly detection
- $k$-means clustering
- Latent Dirichlet allocation (LDA)
- Expert learning
- Stacking
- Recommendation systems
- See [creme-ml.github.io/api](https://creme-ml.github.io/api.html)
]

---

### Alternative frameworks

---

### Binary classification benchmark with default parameters

<div align="center">
.pure-table.pure-table-striped[
| Library | Method | Accuracy | Average fit time | Average predict time |
|---------|------------|----------|------------------|----------------------|
| creme | LogisticRegression | 0.61810 | .green[26μs] | .green[10μs] |
| creme | PAClassifier | 0.55009 | 35μs | 22μs |
| creme | DecisionTreeClassifier | 0.64663 | 356μs | 15μs |
| creme | RandomForestClassifier | .green[0.65915] | 3ms, 972μs | 208μs |
| Keras on TF (CPU) | Dense | 0.61840 | 463μs | 534μs |
| PyTorch (CPU) | Linear | 0.61840 | 926μs | 621μs |
| scikit-garden | MondrianTreeClassifier | .red[0.53875] | 864μs | 208μs |
| scikit-garden | MondrianForestClassifier | 0.60061 | .red[9ms, 773μs] | .red[1ms, 233μs] |
| scikit-learn | SGDClassifier | 0.56161 | 420μs | 116μs |
| scikit-learn | PassiveAggressiveClassifier | 0.55009 | 398μs | 114μs |
]
</div>

---

### Linear regression benchmark

<div align="center">
.pure-table.pure-table-striped[
| Library | Method | MSE | Average fit time | Average predict time |
|---------|------------|----------|------------------|----------------------|
| creme | LinearRegression | 23.035085 | 18μs | 4μs |
| Keras on TF (CPU) | Dense | 23.035086 | 1ms, 208μs | 722μs |
| PyTorch (CPU) | Linear | 23.035086 | 577μs | 187μs |
| scikit-learn | SGDRegressor | 25.295369 | 305μs | 108μs |
]
</div>

---

background-color: #1f282d
class: middle, white

## Current work (1)

.bullets[
- Boosting, many methods but no clear winner:
 - [Online Bagging and Boosting (Oza-Russell, 2005)](https://ti.arc.nasa.gov/m/profile/oza/files/ozru01a.pdf)
 - [Online Gradient Boosting (Beygelzimer, 2015)](https://arxiv.org/pdf/1506.04820.pdf)
 - [Optimal and Adaptive Algorithms for Online Boosting (Beygelzimer, 2015)](http://proceedings.mlr.press/v37/beygelzimer15.pdf)
- Mixture models through expectation-maximization:
 - [Recursive Parameter Estimation Using Incomplete Data (Titterington, 1982)](https://apps.dtic.mil/dtic/tr/fulltext/u2/a116190.pdf)
 - [A View of the EM Algorithm that Justifies Incremental, Sparse, and other Variants (Neal-Hinton, 1998)](http://www.cs.toronto.edu/~fritz/absps/emk.pdf)
 - [Online EM Algorithm for Latent Data Models (Cappé-Moulines 2009)](https://hal.archives-ouvertes.fr/hal-00201327/document)
]

---

background-color: #1f282d
class: middle, white

## Current work (2)

.bullets[
- Field-aware factorization machines (FFM):
 - [Factorization Machines (Rendle, 2010)](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)
 - [Field-aware Factorization Machines for CTR Prediction (Juan et al., 2016)](https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf)
 - [Field-aware Factorization Machines in a Real-world Online Advertising System (Juan-Lefortier-Chappelle, 2017)](https://arxiv.org/pdf/1701.04099.pdf)
- Metric learning:
 - [Online and Batch Learning of Pseudo-Metrics (Shwartz-Singer-Ng, 2004)](https://ai.stanford.edu/~ang/papers/icml04-onlinemetric.pdf)
 - [Information-Theoretic Metric Learning (Davis et al., 2007)](http://www.cs.utexas.edu/users/pjain/pubs/metriclearning_icml.pdf)
 - [Online Metric Learning and Fast Similarity Search (Jain et al., 2009)](http://people.bu.edu/bkulis/pubs/nips_online.pdf)
]

---

background-color: #85144b
class: middle, white

## You can help

.bigbullets[
- Use it and tell us about it
- Share it with others
- Take on issues on GitHub
- Become a core contributor
]

---

# Thanks for listening!

An introduction to symbolic regression

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

An introduction to symbolic regression

Max Halford - PhD student IRIT/IMT

Toulouse Data Science Meetup - December 2017

.center[ .left-column[![tds_logo](/assets/img/presentations/tds_logo.jpeg)] .right-column[![xgp_logo](/assets/img/presentations/xgp_logo.png)] ] --- layout: true # Symbolic regression --- ## Quick overview - The goal is to evolve "programs" with selection, mutation, and crossover - Selection keeps programs that perform well - Mutation changes a piece of the program - Crossover combines two programs --- --- ## Example programs --- ## Kaggle Titanic top 1% 🚢 2 years ago [scirpus](https://www.kaggle.com/scirpus) posted a [solution](https://www.kaggle.com/scirpus/genetic-programming-lb-0-88) to the [Kaggle Titanic competiton](https://www.kaggle.com/c/titanic) ```python y_raw = ((np.minimum(((((0.058823499828577 + X['Sex']) - np.cos((X['Pclass'] / 2.0))) * 2.0)), ((0.885868))) * 2.0) +np.maximum(((X['SibSp'] - 2.409090042114258)), (-(np.minimum((X['Sex']), (np.sin(X['Parch']))) * X['Pclass']))) +(0.138462007045746 * ((np.minimum((X['Sex']), (((X['Parch'] / 2.0) / 2.0))) * X['Age']) - X['Cabin'])) +np.minimum(((np.sin((X['Parch'] * ((X['Fare'] - 0.720430016517639) * 2.0))) * 2.0)), ((X['SibSp'] / 2.0))) +np.maximum((np.minimum((-np.cos(X['Embarked'])), (0.138462007045746))), (np.sin(((X['Cabin'] - X['Fare']) * 2.0)))) +-np.minimum(((((X['Age'] * X['Parch']) * X['Embarked']) + X['Parch'])), (np.sin(X['Pclass']))) +np.minimum((X['Sex']), ((np.sin(-(X['Fare'] * np.cos((X['Fare'] * 1.630429983139038)))) / 2.0))) +np.minimum(((0.230145)), (np.sin(np.minimum((((67.0 / 2.0) * np.sin(X['Fare']))), (0.31830988618379069))))) +np.sin((np.sin(X['Cabin']) * (np.sin((12.6275)) * np.maximum((X['Age']), (X['Fare']))))) +np.sin(((np.minimum((X['Fare']), ((X['Cabin'] * X['Embarked']))) / 2.0) * -X['Fare'])) +np.minimum((((2.675679922103882 * X['SibSp']) * np.sin(((96) * np.sin(X['Cabin']))))), (X['Parch'])) +np.sin(np.sin((np.maximum((np.minimum((X['Age']), (X['Cabin']))), ((X['Fare'] * 0.31830988618379069))) * X['Cabin']))) +np.maximum((np.sin(((12.4148) * (X['Age'] / 2.0)))), (np.sin((-3.0 * X['Cabin'])))) +(np.minimum((np.sin((((np.sin(((X['Fare'] * 2.0) * 2.0)) * 2.0) * 2.0) * 2.0))), (X['SibSp'])) / 2.0) +((X['Sex'] - X['SibSp']) * (np.cos(((X['Embarked'] - 0.730768978595734) + X['Age'])) / 2.0)) +((np.sin(X['Cabin']) / 2.0) - (np.cos(np.minimum((X['Age']), (X['Embarked']))) * np.sin(X['Embarked']))) +np.minimum((0.31830988618379069), ((X['Sex'] * (2.212120056152344 * (0.720430016517639 - np.sin((X['Age'] * 2.0))))))) +(np.minimum((np.cos(X['Fare'])), (np.maximum((np.sin(X['Age'])), (X['Parch'])))) * np.cos((X['Fare'] / 2.0))) +np.sin((X['Parch'] * np.minimum(((X['Age'] - 1.5707963267948966)), ((np.cos((X['Pclass'] * 2.0)) / 2.0))))) +(X['Parch'] * (np.sin(((X['Fare'] * (0.623655974864960 * X['Age'])) * 2.0)) / 2.0)) +(0.31830988618379069 * np.cos(np.maximum(((0.602940976619720 * X['Fare'])), ((np.sin(0.720430016517639) * X['Age']))))) +(np.minimum(((X['SibSp'] / 2.0)), (np.sin(((X['Pclass'] - X['Fare']) * X['SibSp'])))) * X['SibSp']) +np.tanh((X['Sex'] * np.sin((5.199999809265137 * np.sin((X['Cabin'] * np.cos(X['Fare']))))))) +(np.minimum((X['Parch']), (X['Sex'])) * np.cos(np.maximum(((np.cos(X['Parch']) + X['Age'])), (3.1415926535897931)))) +(np.minimum((np.tanh(((X['Cabin'] / 2.0) + X['Parch']))), ((X['Sex'] + np.cos(X['Age'])))) / 2.0) +(np.sin((np.sin(X['Sex']) * (np.sin((X['Age'] * X['Pclass'])) * X['Pclass']))) / 2.0) +(X['Sex'] * (np.cos(((X['Sex'] + X['Fare']) * ((8.48635) * (63)))) / 2.0)) +np.minimum((X['Sex']), ((np.cos((X['Age'] * np.tanh(np.sin(np.cos(X['Fare']))))) / 2.0))) +(np.tanh(np.tanh(-np.cos((np.maximum((np.cos(X['Fare'])), (0.094339601695538)) * X['Age'])))) / 2.0) +(np.tanh(np.cos((np.cos(X['Age']) + (X['Age'] + np.minimum((X['Fare']), (X['Age'])))))) / 2.0) +(np.tanh(np.cos((X['Age'] * ((-2.0 + np.sin(X['SibSp'])) + X['Fare'])))) / 2.0) +(np.minimum((((281) - X['Fare'])), (np.sin((np.maximum(((176)), (X['Fare'])) * X['SibSp'])))) * 2.0) +np.sin(((np.maximum((X['Embarked']), (X['Age'])) * 2.0) * (((785) * 3.1415926535897931) * X['Age']))) +np.minimum((X['Sex']), (np.sin(-(np.minimum(((X['Cabin'] / 2.0)), (X['SibSp'])) * (X['Fare'] / 2.0))))) +np.sin(np.sin((X['Cabin'] * (X['Embarked'] + (np.tanh(-X['Age']) + X['Fare']))))) +(np.cos(np.cos(X['Fare'])) * (np.sin((X['Embarked'] - ((734) * X['Fare']))) / 2.0)) +((np.minimum((X['SibSp']), (np.cos(X['Fare']))) * np.cos(X['SibSp'])) * np.sin((X['Age'] / 2.0))) +(np.sin((np.sin((X['SibSp'] * np.cos((X['Fare'] * 2.0)))) + (X['Cabin'] * 2.0))) / 2.0) +(((X['Sex'] * X['SibSp']) * np.sin(np.sin(-(X['Fare'] * X['Cabin'])))) * 2.0) +(np.sin((X['SibSp'] * ((((5.428569793701172 + 67.0) * 2.0) / 2.0) * X['Age']))) / 2.0) +(X['Pclass'] * (np.sin(((X['Embarked'] * X['Cabin']) * (X['Age'] - (1.07241)))) / 2.0)) +(np.cos(((((-X['SibSp'] + X['Age']) + X['Parch']) * X['Embarked']) / 2.0)) / 2.0) +(0.31830988618379069 * np.sin(((X['Age'] * ((X['Embarked'] * np.sin(X['Fare'])) * 2.0)) * 2.0))) +((np.minimum(((X['Age'] * 0.058823499828577)), (X['Sex'])) - 0.63661977236758138) * np.tanh(np.sin(X['Pclass']))) +-np.minimum(((np.cos(((727) * ((X['Fare'] + X['Parch']) * 2.0))) / 2.0)), (X['Fare'])) +(np.minimum((np.cos(X['Fare'])), (X['SibSp'])) * np.minimum((np.sin(X['Parch'])), (np.cos((X['Embarked'] * 2.0))))) +(np.minimum((((X['Fare'] / 2.0) - 2.675679922103882)), (0.138462007045746)) * np.sin((1.5707963267948966 * X['Age']))) +np.minimum(((0.0821533)), (((np.sin(X['Fare']) + X['Embarked']) - np.cos((X['Age'] * (9.89287))))))) y_pred = 1 / (1 + np.exp(-y_raw)) ``` 0.88516 (top 1% as of today) on the public leaderboard 😵 --- ## Types of nodes - Constants - Variables - Functions Huge search space because the shape of the model is included in the search space 🙊 --- ## Algorithm (high-level) ➰ ```python programs = make_random_programs() for i in range(generations): evaluate(programs) new_programs = select(programs) crossover(new_programs) mutate(new_programs) programs = new_programs ``` --- ## Mutation --- ## Crossover 💏 --- ## Evaluation 💯 --- ## Selection --- ## Initialization --- ## Pros 😺 - Flexible model - Built-in feature selection - Can optimise non-differentiable metrics - Useful for stacking - Makes you look cool 😎 --- ## Cons 😿 - Black box model - No mathematical foundation - Requires a lot of CPU power - Non-deterministic and volatile --- layout: true .center[![xgp_logo](/assets/img/presentations/xgp_logo.png)] --- ## Enter XGP 🎉 - Written in [Go](https://golang.org/) - Optimization done with [gago](https://github.com/MaxHalford/gago) - SIMD operations thanks to [gonum](https://www.gonum.org/) ⚡ - Readable code 📖 - Very customizable (with sensible defaults!) - Command line interface (CLI) 💻 - Python bindings (scikit-learn API) 🐍 --- ## Command line interface (CLI) 💻 ```sh $ xgp fit train.csv ``` ```sh $ xgp predict test.csv ``` --- ## Python package 🐍 ### Regression 📈 ```python model = xgp.XGPRegressor() model.fit(X_train, y_train) y_pred = model.predict(X_test) ``` --- ## Python package 🐍 ### Feature extraction 🔬 ```py model = xgp.XGPTransformer() model.fit(X_train, y_train) train_gp_features = model.predict(X_train) test_gp_features = model.predict(X_test) ``` --- ## Future work 🔮 - Binary and multi-class classification - Boosting (promising) - Caching to handle large datasets - More bindings (R, Ruby, ...) - Extensive testing - Documentation and examples --- layout: false class: center, middle # Thanks! ✌ [github.com/MaxHalford](https://github.com/MaxHalford/)