Linear Algebra Essentials

Linear algebra describes vectors, matrices, and transformations between vector spaces. It is central to machine learning because data, parameters, embeddings, and many model operations are represented as arrays of numbers.

Vectors and Linear Combinations #

A vector is an ordered collection of scalars. Given vectors \(v_1, \ldots, v_n\) and scalars \(a_1, \ldots, a_n\), their linear combination is:

a₁v₁ + a₂v₂ + ... + aₙvₙ

The span of a set of vectors is the set of every linear combination of those vectors. A set spans a vector space when every vector in that space can be expressed using it.

Linear Independence and Bases #

Vectors are linearly independent when none can be written as a linear combination of the others. A basis is an independent set that spans the space. The number of vectors in every basis of a finite-dimensional space is its dimension.

These ideas matter because a basis provides coordinates without redundancy. In data analysis, redundant features can make a matrix rank-deficient and can complicate estimation.

Matrices and Column Space #

A matrix represents a linear transformation. For a matrix \(A\), the column space is the span of its columns: it contains every possible output \(Ax\) as \(x\) varies. The rank of \(A\) is the dimension of this space.

For a system \(Ax=b\), a solution exists exactly when \(b\) lies in the column space of \(A\). When it does not, least squares finds an \(x\) whose output \(Ax\) is as close to \(b\) as possible under a chosen norm.

Dot Products and Orthogonality #

The dot product measures alignment between vectors. Two nonzero vectors are orthogonal when their dot product is zero. Orthogonal bases simplify projections and numerical computations; an orthonormal basis additionally gives every basis vector unit length.

Matrix multiplication composes linear transformations. Its order matters: in general, \(AB\) is not equal to \(BA\). Dimension checks are a practical way to catch errors before computing.