On linear classifiers

Linear Classifier

This is a step up from the problem of image classification and acts as a precursor to neural networks.

Let's define:

• Score function: Maps raw data to class scores
• Loss function: Quantifies how well predicted scores match ground-truth labels

This is the core of supervised learning: we will later minimize the loss with respect to the parameters of the score function.

Score Function

If you remember from school, the linear function is:

$f(x)=ax+b$

where

• $a$ controls the slope
• $b$ controls the intercept

This simple linear equation is the building block. In image classification, instead of predicting one value, we extend it to predict one score per class simultaneously. Therefore, instead of $a$ being a scalar, it becomes a matrix $W$, and $b$ becomes a vector:

$f(x_i, W, b) = W x_i + b$

where:

• $x_i \in \mathbb{R}^D$ are flattened training images, i.e. $x_i$ is a floating point vector shape of [D,1]. D features per image (width × height × channels)
• $y_i \in {1 \dots K}$ are the labels.
• There are $N$ total images and $K$ total classes
• Each row of $W$ can be interpreted as a detector for one class.

The score function maps pixels to raw class scores: $f: \mathbb{R}^D \mapsto \mathbb{R}^K$.

The function score variable dimensions are:

Variable	Shape	Meaning
$x_i$	[D,1]	Flattened image
$W$	[K,D]	One row per class classifier
$b$	[K,1]	Class bias
$f(x_i)$	[K,1]	Class scores

Tracking dimensions is extremely useful when implementing models.

Class prediction is obtained by

$\hat{y} = \arg\max_k f_k(x_i)$

It means:

• $f(x_i)$ produces a vector of class scores [K,1].
• $f_k(x_i)$ is the score for class k (k-th element of the score vector)
• $\arg\max$ returns the argument (index) of maximum value.
• That index corresponds to the predicted class
• In this notation, $\hat{}$ on $\hat{y}$ means predicted

If you are uncomfortable with the math notation, I encourage you to go over it again and familiarise with the jargon here, this will become increasingly important. Rest assured I will make sure to introduce new notations as we go.