Optimization

This chapter concerns optimization mathematics. Optimization is concerned with finding one or more best solution, called an optimal solution, to a problem. Optimal solutions are defined with respect to a given criterion, called an objective function. The objective function takes a set of input values and returns a single output value that represents the quality of the solution. The goal of optimization is to find one or more input values to an objective function that produce optimal output values, either by maximizing or minimizing the output value (i.e., resulting in an extremum). The set of argument values that produce the optimal output value are called the set of optimal solutions.

The notation for extrema is as follows:

• The minimum of an objective function \(f\) is denoted by \(\min f\). For argument \(x \in S\), the minimum value is \[\min_{x \in S} f(x).\]
• The maximum of an objective function \(f\) is denoted by \(\max f\). For argument \(x \in S\), the maximum value is \[\max_{x \in S} f(x).\]
• The set of solutions that minimize an objective function \(f\) is denoted by \(\argmin f\). For argument \(x \in S\), the set of solutions is \[\argmin_{x \in S} f(x).\]
• The set of solutions that maximize an objective function \(f\) is denoted by \(\argmax f\). For argument \(x \in S\), the set of solutions is \[\argmax_{x \in S} f(x).\]

Optimization problems can be classified into two categories: unconstrained optimization and constrained optimization. In unconstrained optimization, the objective function is optimized over the entire domain of the input space, whereas in constrained optimization, the objective function is optimized over a subset of the input space that satisfies a set of constraints. In this chapter, we will consider both unconstrained and constrained optimization problems.