Machine Learning Optimizers

An optimizer updates model parameters to reduce a training objective. In gradient-based learning, the objective measures the mismatch between predictions and targets, and automatic differentiation supplies gradients with respect to the parameters.

Gradient Descent #

For parameters \(\theta\), learning rate \(\eta\), and loss \(L\), gradient descent performs:

θ ← θ − η ∇θL

Batch gradient descent uses the entire training set for each update. Stochastic gradient descent uses one example, and mini-batch gradient descent uses a small batch. Mini-batches are the standard practical choice because they use accelerator hardware efficiently while adding gradient noise that can help exploration.

Momentum and Adaptive Methods #

Momentum accumulates a moving direction from past gradients, reducing oscillation in narrow valleys. Adam combines momentum with an adaptive per-parameter scale derived from squared gradients. AdamW decouples weight decay from Adam’s adaptive update and is a common default for Transformer training.

No optimizer is universally best. SGD with momentum can generalize well in some vision workloads; AdamW is often easier to tune for large language models. Learning-rate schedules, batch size, weight decay, gradient clipping, and data quality may have as much impact as the optimizer name.

Practical Tuning #

Choose a baseline optimizer and a conservative learning rate, then monitor training loss, validation quality, gradient norms, and instability. A falling training loss alone is not proof of a useful model. Use warm-up when large initial updates are unstable, reduce the learning rate as training approaches convergence, and keep optimizer state in checkpoints so resumed training behaves consistently.