Understanding the Adam Optimization Algorithm: A Review of Linear Algebra and Calculus

Introduction

I'm not great at mathematics.
However, as I have been exploring various machine learning techniques, I have keenly felt the necessity of linear algebra and calculus.
The book "Introduction to Calculus + Linear Algebra for No-Compromise Data Analysis" on Amazon was on sale,

and since it's the perfect time of the New Year to start something new, I decided to take on the challenge of linear algebra and calculus, which I have been running away from.

Writing out formulas in LaTeX because I can't read them

Since I couldn't understand the text just by skimming through it, I decided to deepen my understanding by:

• Writing out mathematical formulas in LaTeX to make them readable,
• Having an AI explain the formulas I converted to LaTeX text.

In particular, the problem of not being able to grasp the meaning just by staring at the formulas began to fade; when I had to look up the pronunciations of Greek letters and mathematical symbols while expressing them in LaTeX, the strings of symbols began to change into things that had some meaning.

This time, as an experiment in using LaTeX, I will express the update formulas for Adam (Adaptive Moment Estimation), one of the most widely used optimization algorithms in deep learning. I will describe the Adam update formulas in LaTeX and break down the meaning of each formula.

---

1. Adam Update Formulas (LaTeX)

Adam utilizes the "exponential moving average" of gradients to adjust the learning rate for each parameter. Below are its primary update formulas. I displayed the LaTeX using Google Colab's code blocks.

from IPython.core.display import Latex, display
# Fig 0.1.3: Examples of special applications of calculus in deep learning

display(Latex(r'$m^{(t+1)}_i=\beta_1m^{(t)}_i+(1-\beta_1)\frac{\partial L}{\partial\theta^{(t+1)}_i}$'))
display(Latex(r'$\hat{m}^{(t+1)}_i=\frac{1}{1-\beta_1^{t+1}}m^{(t+1)}_i$'))
display(Latex(r'$\upsilon^{(t+1)}_i=\beta_2\upsilon^{(t)}_i+(1-\beta_2)\left(\frac{\partial L}{\partial\theta^{(t)}_i}\right)^2$'))
display(Latex(r'$\hat{\upsilon}^{(t+1)}_i=\frac{1}{1-\beta_2^{t+1}}\upsilon^{(t+1)}_i '))
display(Latex(r'$\theta^{(t+1)}_i=\theta^{(t)}_i-\eta\frac{\hat{m}^{(t+1)}_i}{\sqrt{\hat{\upsilon}^{(t+1)}_i}+\epsilon}$'))

When displayed as mathematical formulas, they look like this:

$$m^{(t+1)}_i=\beta_1m^{(t)}_i+(1-\beta_1)\frac{\partial L}{\partial\theta^{(t+1)}_i}$$

$$\hat{m}^{(t+1)}_i=\frac{1}{1-\beta_1^{t+1}}m^{(t+1)}_i$$

$$\upsilon^{(t+1)}_i=\beta_2\upsilon^{(t)}_i+(1-\beta_2)\left(\frac{\partial L}{\partial\theta^{(t)}_i}\right)^2$$

$$\hat{\upsilon}^{(t+1)}_i=\frac{1}{1-\beta_2^{t+1}}\upsilon^{(t+1)}_i$$

$$\theta^{(t+1)}_i=\theta^{(t)}_i-\eta\frac{\hat{m}^{(t+1)}_i} {\sqrt{\hat{\upsilon}^{(t+1)}_i}+\epsilon}$$

---

2. Meaning and Role of Each Formula

① First Moment of Gradient (Inertia)

$$m^{(t+1)}_i=\beta_1m^{(t)}_i+(1-\beta_1)\frac{\partial L}{\partial\theta^{(t+1)}_i}$$

This represents how much of the "past gradient direction" to retain. It plays the role of inertia (Momentum) in physics, suppressing sudden changes and allowing optimization to proceed smoothly.

② Second Moment of Gradient (Magnitude of Fluctuation)

$$\upsilon^{(t+1)}_i=\beta_2\upsilon^{(t)}_i+(1-\beta_2)\left(\frac{\partial L}{\partial\theta^{(t)}_i}\right)^2$$

This records the square of the "magnitude" of the gradient. This allows the algorithm to judge whether the gradient for a specific parameter is consistently large or small.

③ Bias Correction

$$\hat{m}^{(t+1)}_i=\frac{1}{1-\beta_1^{t+1}}m^{(t+1)}_i$$

Since $m$ and $\upsilon$ are too close to 0 immediately after training starts, this process is used to correct that.

④ Parameter Update

$$\theta^{(t+1)}_i=\theta^{(t)}_i-\eta\frac{\hat{m}^{(t+1)}_i} {\sqrt{\hat{\upsilon}^{(t+1)}_i}+\epsilon}$$

This is the final update formula. By having $\sqrt{\hat{\upsilon}^{(t+1)}_i}$ in the denominator, the algorithm automatically adjusts: "cautiously for parameters that move frequently (smaller learning rate)" and "boldly for parameters that don't move much (larger learning rate)".

• $\epsilon$ (epsilon) is a small value to prevent division by zero.

---

3. Why are Linear Algebra and Calculus Necessary?

This formula contains condensed mathematics that form the foundation of data analysis.

• Calculus: Through (partial) differentiation, it identifies in which direction to move the parameters to reduce the error $L$.
• Linear Algebra: Although the formulas above are written for each element $i$, in actual implementation, these are treated as vectors and processed at high speed using matrix operations.

Summary

Even formulas that look complex at first glance reveal their rationality when broken down into individual elements (inertia, scaling, correction). By solidifying the basics of linear algebra and calculus, you can deepen your understanding of the inner workings of such state-of-the-art algorithms.

---