To find the maxima and minima of f(x) we solve

f ’(x)=0

We can obviously apply the Newton method, like we did to find roots. By analogy, if a maximum or a minimum is close to x₀, then a better approximation is found by:

                     f ’(x₀)
x₁ = x₀ - ¾¾¾¾¾
                     f ’’(x₀)

The specified interval is explored for changes in the sign of f ’(x) to obtain an initial approximation, and the method is applied to obtain a sequence of approximations:  x₀, x₁, x₂, ... , x_n.  The x_n is assumed correct if it differs from the previous value in the sequence by less than the specified error.

The way in which the sign of f ’(x) changes indicates whether it is a maximum or a minimum:  If f ’(x) goes from - to + then the point is a minimum, and if it goes from + to - it is a maximum.

MAXIMA AND MINIMA OF MULTIVARIABLE FUNCTIONS

Suppose that, instead of a one-variable function we have a function of several variables and we wish to find its relative extrema. We can do this using a generalization of the Newton method. To illustrate, let's say we want to find the relative maxima and minima of F(x,y,z).  A necessary (though not sufficient) condition for point (a,b,c) to be an extremum is that all three equations

F_x = 0  ;  F_y = 0  ;  F_z = 0       (I)

be satisfied, where

      ¶F         ¶F                ¶F
F_x = ——— , F_y = ——— , F_z = ——— , all evaluated at (a,b,c).
      ¶x         ¶y                ¶z

Assume (x₀ ,y₀,z₀) is close to an extremum.  Now expand all three equations into their Taylor series.  Neglecting terms of order higher than the first, we get:

F_xx*(x-x₀) + F_xy*(y-y₀) + F_xz*(z-z₀) = - F_x(x₀,y₀,z₀)

F_yx*(x-x₀) + F_yy*(y-y₀) + F_yz*(z-z₀) = - F_y(x₀,y₀,z₀)

F_zx*(x-x₀) + F_zy*(y-y₀) + F_zz*(z-z₀) = - F_z(x₀,y₀,z₀)

where F_x, F_y and F_z are as defined above, but evaluated at (x₀,y₀,z₀), and

      ¶F_x          ¶F_x          ¶F_x
F_xx = ———— , F_xy = ———— , F_xz = ———— , all evaluated at (x₀,y₀,z₀).
      ¶x           ¶y           ¶z

Similarly for F_yx, F_yy, F_yz, F_zx, F_zy and F_zz.

If we solve this last set of linear equations for (x-x₀), (y-y₀) and (z-z₀), we obtain corrections on the initial approximation, that is, if the solutions to the system are D_x, D_y and D_z, where

x-x₀=D_x;  y-y₀=D_y ;  z-z₀=D_z

then a closer approximation to the extremum is

x₁=x₀+D_x ;  y₁=y₀+D_y ;  z₁=z₀+D_z

We now use x₁, y₁ and z₁ as initial approximations and again set a system of linear equations then solve it to obtain x₂, y₂, and z₂.  This process is continued until all the variables in the current approximation in the sequence differ from the values of the previous approximation by less than the specified error.

Points satisfying (I) above are called stationary, critical or singular points. They may be points of local maximum or minimum, or neither (saddle points). To identify a stationary point as a local minimum or a local maximum, we set up the matrix of second partial derivatives and establish whether it is positive or negative definite or neither.

This is a generalization of concavity to higher dimensions. If you recall, a one-variable function has a minimum at a certain point if it’s graph at that point is concave upward (positive second derivative), and a maximum if it is concave downward (negative second derivative).

Let's illustrate using a 4-variable case.  Suppose the function is F(x,y,z,t).  The matrix of second partials we need to consider is the symmetric matrix:

          éF_xx  F_xy  F_xz  F_xtù
[A] =     êF_yx  F_yy  F_yz  F_ytú
          êF_zx  F_zy  F_zz  F_ztú
          ëF_tx  F_ty  F_tz  F_ttû

which is called the HESSIAN MATRIX of F, where

       ¶²F          ¶²F           ¶²F           ¶²F
F_xx = ————— , F_xy = ————— , F_xz = ————— , F_xt = —————
       ¶x²          ¶x¶y          ¶x¶z          ¶x¶t

       ¶²F          ¶²F           ¶²F           ¶²F
F_yx = ————— , F_yy = ————— , F_yz = ————— , F_yt = —————
      ¶y¶x          ¶y²           ¶y¶z          ¶y¶t

       ¶²F          ¶²F           ¶²F           ¶²F
F_zx = ————— , F_zy = ————— , F_zz = ————— , F_zt = —————
            ¶z¶x                    ¶z¶y                    ¶z²                     ¶z¶t

       ¶²F          ¶²F           ¶²F           ¶²F
F_tx = ————— , F_ty = ————— , F_tz = ————— , F_tt = —————
       ¶t¶x         ¶t¶y          ¶t¶z          ¶t²

all evaluated at the stationary point.

Since we’re assuming the function and all of its 1st and 2nd order partial derivatives are continuous at the point we’re testing, the matrix is obviously symmetric since F_xy = F_yx, F_xz = F_zx, etc.  [A] is positive or negative definite if the product

[x y z t] éF_xx  F_xy  F_xz  F_xtù éxù
         êF_yx  F_yy  F_yz  F_ytú êyú
         êF_zx  F_zy  F_zz  F_ztú êzú
         ëF_tx  F_ty  F_tz  F_ttû ëtû

is positive or negative, respectively, for all nontrivial [x y z t].  Clearly, we can't test that product for all possible [x y z t], but there are other equivalent tests.  One is to check the signs of the eigenvalues:  if they are all positive or all negative, the matrix is positive or negative definite, respectively.  Another test is to compute the determinants A₁, A₂, A₃ and A₄ of these principal submatrices:

A₁ = det[F_xx]

A₂ = detéF_xx  F_xyù
        ëF_yx  F_yyû
                        
A₂ = detéF_xx  F_xy  F_xzù
        êF_yx  F_yy  F_yzú
        ëF_zx  F_zy  F_zzû

A₄ = detéF_xx  F_xy  F_xz  F_xtù
        êF_yx  F_yy  F_yz  F_ytú
        êF_zx  F_zy  F_zz  F_ztú
        ëF_tx  F_ty  F_tz  F_ttû

all of which are evaluated at the stationary point.

If the point is a minimum then [A] is positive definite:

A_i > 0    for  i = 1, 2, 3, 4   (all i)

If the point is a maximum then [A] is negative definite:

A_i < 0    for  i = 1, 3   (i odd)
A_i > 0    for  i = 2, 4   (i even)

The matrix may also be positive or negative semidefinite, or indefinite.