Lagrange multiplier

Lagrange multipliers are an optimisation technique under a constraint particularly useful when it is impossible or difficult to reduce the function to be optimised to a single variable function. It forms the basis of the Lagrangian function.¹

Statement

Given a function to be optimised $𝑓 (⃗ 𝐯)$ and constraining function $𝑔 (⃗ 𝐯) = 𝑘$ , a maximising or minimising input $⃗ 𝐜$ will satisfy

\nabla 𝑓 (⃗ 𝐜) = 𝜆 \nabla 𝑔 (⃗ 𝐜)

where $𝜆 \neq 0$ is called the Lagrangian multiplier.

Multiple constraints

In the case of optimising $𝑓 (⃗ 𝐯)$ under multiple constraints

𝑔 1 (⃗ 𝐯) = 𝑘 1, 𝑔 2 (⃗ 𝐯) = 𝑘 2, \dots, 𝑔 𝑛 (⃗ 𝐯) = 𝑘 𝑛

the equation to be satisfied becomes

\nabla 𝑓 (⃗ 𝐜) = 𝑛 \sum 𝑖 = 1 𝜆 𝑖 \nabla 𝑔 𝑛 (𝑐) = 𝜆 1 \nabla 𝑔 1 (⃗ 𝐜) + 𝜆 2 \nabla 𝑔 2 (⃗ 𝐜) + \dots + 𝜆 𝑛 \nabla 𝑔 𝑛 (⃗ 𝐜)

Intuitive justification

The constraint $𝑔 (⃗ 𝐯) = 𝑘$ forms a Level set of $𝑔 (⃗ 𝐯)$ . Therefore any inputs to $𝑓$ which satisfy the constraint correspond to intersections of the level curve $𝑔 (⃗ 𝐯) = 𝑘$ and some level curve $𝑓 (⃗ 𝐯) = 𝑙$ . For any optimising (i.e. maximising or minimising) input of $⃗ 𝐜$ , the two level curves will be tangent. Since Gradient vectors are perpendicular to level curves, this necessarily implies the the gradients $\nabla 𝑓 (⃗ 𝐜)$ and $\nabla 𝑔 (⃗ 𝐜)$ are parallel, and hence there exists some nonzero $𝜆$ such that²

\nabla 𝑓 (⃗ 𝐜) = 𝜆 \nabla 𝑔 (⃗ 𝐜)

Usage

In order to solve an optimisation problem using Lagrangian multipliers with input of dimension $𝑛$ (i.e. $𝑓 : ℝ 𝑛 \to ℝ$ ), one must solve a system of $𝑛 + 1$ equations.

⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝜕 𝑓 𝜕 𝑥 1 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) 𝜕 𝑓 𝜕 𝑥 2 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) ⋮ 𝜕 𝑓 𝜕 𝑥 𝑛 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) 𝑔 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 𝜆 𝜕 𝑔 𝜕 𝑥 1 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) 𝜆 𝜕 𝑔 𝜕 𝑥 2 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) ⋮ 𝜆 𝜕 𝑔 𝜕 𝑥 𝑛 (𝑥 1, 𝑥 2 \dots 𝑥 𝑛) 𝑘 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

Significance of the multiplier

The lambda multiplier $𝜆$ is not an arbitrary, meaningless value. It is the derivative of the optimised value with respect to the constraining value $𝑘$ where $𝑔 (⃗ 𝐯) = 𝑘$ is the constraint.

Practice problems

2022. MATH1011: Multivariable calculus, pp. 65–66 (example 4.29; exercise 4.4.1)
2016. Calculus, pp. 1017–1018 (§14.8 exercises)
2023. Advanced Mathematical Methods, p. 77 (§4 problems)

#state/tidy | #SemBr | #lang/en | #review

2022. MATH1011: Multivariable calculus, pp. 65–66 ↩
2016. Calculus, pp. 1011–1012 ↩