This document provides formulas and definitions for concepts in multivariate calculus and machine learning. It defines the derivative and lists time-saving rules like the sum, product, and chain rules. It also defines the Jacobian and Hessian matrices, Taylor series, optimization techniques like Newton-Raphson and gradient descent, and concepts like Lagrange multipliers and least squares minimization. The document is a formula sheet for a course on mathematics for machine learning.

1. Mathematics for Machine Learning
Multivariate Calculus
Formula sheet
Dr Samuel J. Cooper
Prof. David Dye
Dr A. Freddie Page
Definition of a derivative
f (x) =
df(x)
dx
= lim
∆x→0
f(x + ∆x) − f(x)
∆x
Time saving rules
- Sum Rule:
d
dx
(f(x) + g(x)) =
d
dx
(f(x)) +
d
dx
(g(x))
- Power Rule:
f(x) = axb
f (x) = abx(b−1)
- Product Rule:
A(x) = f(x)g(x)
A (x) = f (x)g(x) + f(x)g (x)
- Chain Rule:
If h = h(p) and p = p(m)
then
dh
dm
=
dh
dp
×
dp
dm
- Total derivative:
For the function f(x, y, z, ...), where each variable
is a function of parameter t, the total derivative is
df
dt
=
∂f
∂x
dx
dt
+
∂f
∂y
dy
dt
+
∂f
∂z
dz
dt
+ . . .
Derivatives of named functions
∂
∂x
1
x
= −
1
x2
(1)
∂
∂x
sin x = cos x (2)
∂
∂x
cos x = − sin x (3)
∂
∂x
exp x = exp x (4)
Derivative structures
f = f(x, y, z)
- Jacobian:
Jf =
∂f
∂x
,
∂f
∂y
,
∂f
∂z
- Hessian:
Hf =








∂2f
∂x2
∂2f
∂x∂y
∂2f
∂x∂z
∂2f
∂y∂x
∂2f
∂y2
∂2f
∂y∂z
∂2f
∂z∂x
∂2f
∂z∂y
∂2f
∂z2








1

2. Taylor Series
- Univariate:
f(x) = f(c) + f (c)(x − c) +
1
2
f (c)(x − c)2
+ ...
=
∞
n=0
f(n)
(c)
n!
(x − c)n
- Multivariate:
f(x) = f(c) + Jf (c)(x − c) + ...
1
2
(x − c)t
Hf (c)(x − c) + ...
Optimization and Vector Calculus
- Newton-Raphson:
xi+1 = xi −
f(xi)
f (xi)
- Grad:
f =








∂f
∂x
∂f
∂z
∂f
∂z








- Directional Gradient:
f.ˆr
- Gradient Descent:
sn+1 = sn − γ f
- Lagrange Multipliers λ:
f = λ g
- Least Squares - χ2
minimization:
χ2
=
n
i
(yi − y(xi; ak))2
σi
criterion: χ2
= 0
anext = acur − γ χ2
= acur + γ
n
i
(yi − y(xi; ak))
σi
∂y
∂ak
2