Classification Models and Learning Objectives

Statistical learning assumes that training examples provide evidence about the population on which a model will operate. Sampling bias, label error, and distribution shift can invalidate that assumption; a high validation score then does not establish production performance.

Linear Scores and Logistic Regression #

A linear model computes a score \(z = w^\mathsf{T}x + b\). Linear regression predicts a continuous target, so thresholding its output does not make it a probabilistic classifier.

Binary logistic regression maps the score through the sigmoid:

\[ P(y = 1 \mid x) = \sigma(z) = \frac{1}{1 + e^{-z}}. \]

Multinomial logistic regression uses softmax for mutually exclusive classes:

\[ P(y = k \mid x) = \frac{e^{z_k}}{\sum_j e^{z_j}}. \]

Cross-entropy compares predicted probabilities with labels. For a binary label \(y\), the per-example loss is \(-y\log p - (1-y)\log(1-p)\). Unlike a misclassification count, it is differentiable with respect to model scores and penalizes confident incorrect predictions.

Support Vector Machines #

A binary support vector machine learns a decision boundary \(w^\mathsf{T}x + b = 0\). The signed distance to that boundary is proportional to \((w^\mathsf{T}x+b)/\lVert w\rVert\). A soft-margin SVM minimizes regularization plus hinge loss:

\[ \frac{1}{2}\lVert w\rVert^2 + C\sum_i \max(0, 1-y_i(w^\mathsf{T}x_i+b)). \]

The margin objective rewards separation, while the penalty permits violations for data that are not perfectly separable. Kernel methods evaluate an implicit feature-space similarity without constructing every feature, but prediction and training costs depend on the number of support vectors.

Neural-Network Outputs #

Hidden units learn intermediate features; the final layer maps those features to task outputs. Use:

1. a sigmoid and independent binary cross-entropies for multi-label classification;
2. softmax and categorical cross-entropy for one-of-\(K\) classification; or
3. a linear output and regression loss for a continuous target.

Margin, probability calibration, ranking, and cost-sensitive decisions are distinct objectives. Select the loss and evaluation metric from the decision the model supports, then verify that held-out data represents that decision context.