TODO

  • 相关系数矩阵
  • 协方差矩阵
  • Hermitan矩阵
  • 共轭矩阵
  • Hessian 矩阵

Hessian Matrix

  海森矩阵是一个多变量实值函数的二阶偏导数组成的方块矩阵,假设一个实数函数 ${\displaystyle \textstyle f(x_{1},x_{2},\dots ,x_{n})}$,如果 $f$ 所有的二阶偏导数都存在,那么 $f$ 的黑森矩阵为:

\[H(f)=\left[ \begin{array}{cccc}{\frac{\partial^{2} f}{\partial x_{1}^{2}}} & {\frac{\partial^{2} f}{\partial x_{1} \partial x_{2}}} & {\cdots} & {\frac{\partial^{2} f}{\partial x_{1} \partial x_{n}}} \\ {\frac{\partial^{2} f}{\partial x_{2} \partial x_{1}}} & {\frac{\partial^{2} f}{\partial x_{2}^{2}}} & {\dots} & {\frac{\partial^{2} f}{\partial x_{2} \partial x_{n}}} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ {\frac{\partial^{2} f}{\partial x_{n} \partial x_{1}}} & {\frac{\partial^{2} f}{\partial x_{n} \partial x_{2}}} & {\cdots} & {\frac{\partial^{2} f}{\partial x_{n}^{2}}}\end{array}\right]\]

  若一元函数 $f(x)$ 在 $x=x_0$ 点的某个邻域内具有任意阶导数,则 $f(x)$ 在 $x=x_0$ 点处的泰勒展开式:

\[f(x)=f\left(x_{0}\right)+f^{\prime}(x) \Delta x+\frac{f^{\prime \prime}(x)}{2 !} \Delta x^{2}+\cdots\]

  其中,${\displaystyle \Delta x=x-x_{0}}$。

  同理,二元函数 ${\displaystyle f(x_{1},x_{2})}$ 在 ${\displaystyle x_{0}(x_{10},x_{20})}$ 点处的泰勒展开式为

\[{\displaystyle f(x_{1},x_{2})=f(x_{10},x_{20})+f_{x_{1}}(x_{0})\Delta x_{1}+f_{x_{2}}(x_{0})\Delta x_{2}+{\frac {1}{2}}[f_{x_{1}x_{1}}(x_{0})\Delta x_{1}^{2}+2f_{x_{1}x_{2}}(x_{0})\Delta x_{1}\Delta x_{2}+f_{x_{2}x_{2}}(x_{0})\Delta x_{2}^{2}]+\cdots }\]

  其中, ${\displaystyle \Delta x_{1}=x_{1}-x_{10}}$, ${\displaystyle \Delta x_{2}=x_{2}-x_{20}}$,${\displaystyle f_{x_{1}x_{1}}={\frac {\partial ^{2}f}{\partial x_{1}^{2}}}}$, ${\displaystyle f_{x_{2}x_{2}}={\frac {\partial ^{2}f}{\partial x_{2}^{2}}}}$, ${\displaystyle f_{x_{1}x_{2}}={\frac {\partial ^{2}f}{\partial x_{1}\partial x_{2}}}={\frac {\partial ^{2}f}{\partial x_{2}\partial x_{1}}}}$。

  将上述展开式写成矩阵形式,则有

\[{\displaystyle f(x)=f(x_{0})+\nabla f(x_{0})^{T}\Delta x+{\frac {1}{2}}\Delta x^{T}H(x_{0})\Delta x+\cdots }\]

  其中, ${\displaystyle \Delta x=(\Delta x_{1},\Delta x_{2})}$,${\displaystyle \Delta x^{T}={\begin{bmatrix}\Delta x_{1}\\\Delta x_{2}\end{bmatrix}}}$ 是 ${\displaystyle \Delta x}$ 的转置(此处“转置”用上角标 ${\displaystyle T}$ 表示),${\displaystyle \nabla f(x_{0})={\begin{bmatrix}{\frac {\partial f}{\partial x_{1}}}\\{\frac {\partial f}{\partial x_{2}}}\end{bmatrix}}}$ 是函数 ${\displaystyle f(x_{1},x_{2})}$ 的梯度,矩阵

\[{\displaystyle H(x_{0})={\begin{bmatrix}{\frac {\partial ^{2}f}{\partial x_{1}^{2}}}&{\frac {\partial ^{2}f}{\partial x_{1}\,\partial x_{2}}}\\\\{\frac {\partial ^{2}f}{\partial x_{2}\,\partial x_{1}}}&{\frac {\partial ^{2}f}{\partial x_{2}^{2}}}\end{bmatrix}}_{x_{0}}}\]

  即函数 ${\displaystyle f(x_{1},x_{2})}$ 在 ${\displaystyle x_{0}(x_{10},x_{20})}$ 点处的二阶黑塞矩阵。它是由函数 ${\displaystyle f(x_{1},x_{2})}$ 在 ${\displaystyle x_{0}(x_{10},x_{20})}$ 点处的二阶偏导数所组成的方阵。由函数的二次连续性,有

\[{\displaystyle {\frac {\partial ^{2}f}{\partial x_{1}\partial x_{2}}}={\frac {\partial ^{2}f}{\partial x_{2}\partial x_{1}}}} {\displaystyle {\frac {\partial ^{2}f}{\partial x_{1}\partial x_{2}}}={\frac {\partial ^{2}f}{\partial x_{2}\partial x_{1}}}}\]

  所以,黑塞矩阵 ${\displaystyle H(x_{0})}$ 为对称矩阵。

  将二元函数的泰勒展开式推广到多元函数时,${\displaystyle f(x_{1},x_{2},\cdots ,x_{n})}$ 在 $x_{0}$ 点处的泰勒展开式为

\[{\displaystyle f(x)=f(x_{0})+\nabla f(x_{0})^{T}\Delta x+{\frac {1}{2}}\Delta x^{T}H(x_{0})\Delta x+\cdots }\]

  其中,

\[{\displaystyle \nabla f(x_{0})={\begin{bmatrix}{\frac {\partial f}{\partial x_{1}}}&{\frac {\partial f}{\partial x_{2}}}&\cdots &{\frac {\partial f}{\partial x_{n}}}\end{bmatrix}}_{x_{0}}^{T}}\]

  为函数 ${\displaystyle f(x)}$ 在 ${\displaystyle x_{0}}$ 点的梯度。

\[{\displaystyle H(x_{0})={\begin{bmatrix}{\frac {\partial ^{2}f}{\partial x_{1}^{2}}}&{\frac {\partial ^{2}f}{\partial x_{1}\,\partial x_{2}}}&\cdots &{\frac {\partial ^{2}f}{\partial x_{1}\,\partial x_{n}}}\\\\{\frac {\partial ^{2}f}{\partial x_{2}\,\partial x_{1}}}&{\frac {\partial ^{2}f}{\partial x_{2}^{2}}}&\cdots &{\frac {\partial ^{2}f}{\partial x_{2}\,\partial x_{n}}}\\\\\vdots &\vdots &\ddots &\vdots \\\\{\frac {\partial ^{2}f}{\partial x_{n}\,\partial x_{1}}}&{\frac {\partial ^{2}f}{\partial x_{n}\,\partial x_{2}}}&\cdots &{\frac {\partial ^{2}f}{\partial x_{n}^{2}}}\end{bmatrix}}_{x_{0}}}\]

  为函数 ${\displaystyle f(x)}$ 在 ${\displaystyle x_{0}}$ 点的 ${\displaystyle n}$ 阶黑塞矩阵。若函数有 ${\displaystyle n}$ 次连续性,则函数的 ${\displaystyle n}$ 阶黑塞矩阵也是对称矩阵。