Logistic Regression Part 3

Alright, like I mentioned before — this time we’re looking at multi-class classification. Everything we’ve done so far? The data was always split into two buckets. 0 or 1. Binary. Done.

But what if we’ve got 3 or more classes to sort out?

Then the question is how we tweak this machine learning class to handle that.

So how do you classify 3+ things?

Right now we’ve got a classifier that can separate two things. The question is: how do we use it to sort three…? Let’s look at a picture.

OK so these guys are huddled up by color. With just the classifier we already have, we can do this much:

• A blue line separating blue from everything else
• A green line separating green from everything else
• A red line separating red from everything else

That much we can already handle with what we’ve got…. Wait a sec…?!

If we’ve got that much, can’t we pull off 3-class classification with some sneaky trick? What kind of dirty trick should we try here?!

Here’s what I’m thinking. First, the broad regions — the areas where each color clumps up — those are easy. Just use the “above or below the line” rule.

The tricky bit is the triangle in the middle where all three lines meet. For each point, compare the three classifiers’ signed decision scores and pick the largest. With logistic regression those scores are $w_k\cdot x+b_k$; the sigmoid preserves their order. It isn’t a nearest-line rule. So the calls sound like:

• “Oh, point 1 — blue scores highest → you’re blue.”
• “Oh, point 2 — red scores highest → you’re red.”
• “Oh, point 3 — blue wins → blue.”
• “Oh, point 4 — blue wins → blue.”
• “Oh, point 5 — green wins → green.”

Do that for every point, give each one its color, and you get something like:

Actually — the thing I just described is an algorithm called one-vs-rest (OvR).

But lol, there are only a few points here, so fine, we can do it like that…. What if the dataset gets huge? Calling out every point one by one like that would take forever……. At that point all you can do is type GG and back out.

Anyway — let’s eyeball it and move on. Because our goal right now isn’t to understand every single principle inside machine learning, right? Our goal is to use this stuff!!!

The parameter

In the original notebook, we selected OvR with multi_class='ovr'. Current scikit-learn uses OneVsRestClassifier around the binary model instead. Let’s use that below.

Let’s get it! First, load the data:

from sklearn.datasets import load_iris
iris = load_iris()
X = iris['data']
y = iris['target']

Shuffle & split:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1, stratify=y)

The iris dataset is 3-class and the class ratios are nicely balanced:

import pandas as pd
X = pd.DataFrame(X)
y = pd.Series(y)
print(y.value_counts()/len(y))

2    0.333333
1    0.333333
0    0.333333
dtype: float64

Now let’s fit the OvR model and see what we get. I’ll just throw in the other parameters at reasonable values:

from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier

model = OneVsRestClassifier(
    LogisticRegression(C=1, class_weight='balanced',
                       random_state=1, solver='lbfgs'),
    n_jobs=-1
).fit(X_train, y_train)

print('train score: ', model.score(X_train, y_train))
print('test score: ', model.score(X_test, y_test))

train score:  0.9285714285714286
test score:  0.9210526315789473

“But wait, is it really getting each of the 3 classes right????”

Fine, let’s ask it like this. X holds both the train set and the test set, so I’ll just throw it all into predict in one go:

print('predict: ', model.predict(X))

predict:  [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]

There it is — it answers 0, 1 or 2 for each one. All three classes. And we can look at how confident each prediction was like this:

print('P(class): ', model.predict_proba(X))

Hey — we learned precision and recall too, so let’s use those!!! Heart’s pounding a bit lol — will it actually give us precision and recall for each of the 3 classes separately…? heh heh heh

from sklearn.metrics import classification_report
pred_train = model.predict(X_train)
pred_test  = model.predict(X_test)
print('classification_report - training data')
print(classification_report(y_train, pred_train))

Ohhhhhh

print('classification_report - test data')
print(classification_report(y_test, pred_test))

Besides OvR — there’s also multinomial

model = LogisticRegression(C=1, class_weight='balanced', random_state=1,
                           solver='lbfgs').fit(X_train, y_train)
print('train score: ', model.score(X_train, y_train))
print('test score: ', model.score(X_test, y_test))

train score:  0.9642857142857143
test score:  0.9736842105263158

pred_train = model.predict(X_train)
pred_test  = model.predict(X_test)
print('classification_report - training data')
print(classification_report(y_train, pred_train))

print('classification_report - test data')
print(classification_report(y_test, pred_test))

The numbers came out a little nicer, heh heh heh.

The multinomial model uses a softmax-based approach under the hood, and it did a little better on this split. That doesn’t guarantee it’ll beat OvR on every dataset….

As for what softmax actually is — we’ll get to that when we briefly touch Deep Learning later. So I’m shelving that concept for next time, heh heh.

For now, let’s settle for “we did sort 3 things” and move on.

Up next…

SVM!

Heads-up. The figures here are the original notes, slips and all. A few things to watch for:

• The notebook screenshots are from 2019; the runnable code above is updated. multi_class was deprecated in scikit-learn 1.5 and removed in 1.8. For OvR use OneVsRestClassifier(LogisticRegression(...)); for the multinomial model with lbfgs, leave that argument out. The old screenshots’ nearest-line explanation is only a sketch: actual OvR prediction picks the highest signed classifier score.