iTranslated by AI
Speaking at an Event [Slides Included]
Important Notes
I Gave a Presentation!
Someone from Tokyo AI Festival found my article, and I gave a lecture today!
I would like to express my sincere gratitude to Mr. Kanazawa, who invited me, as well as my co-speakers Nuko-nuko-san and Mr. Ohara, and everyone who took the time to come and listen today!
I would like to take this opportunity to express my gratitude.
At the networking event, I had the chance to talk with people I often see on social media and incredibly talented students (far beyond what I could imagine), which was very inspiring!
(Above all, I got to meet Nuko-nuko-san! I'm so glad I went! Thank you very much for presenting together!)
Today's Materials
The materials I used for today's presentation are published below. Please take a look.
Also, these materials were created based on the following article:
Furthermore, I created these materials using the template below. It is an excellent template that makes things look professional even when created somewhat roughly. Thank you very much.
Precautions Regarding the Materials
The following content is for my own protection.
First and foremost, please refrain from using these materials for profit, commercial purposes, or internal corporate use.
If you find usage within the above scope, please contact me (along with evidence).
On the other hand, for educational or research purposes (excluding internal corporate or commercial use), feel free to use them (limited to within that community).
Even in those cases, please keep in mind that rigor has been sacrificed and use them strictly as an aid to understanding.
Furthermore, there is no problem with "referring to" these materials to create similar materials of your own.
However, in such cases, please refrain from making statements or notes such as "asap's materials said this" or "quoted/adapted from asap's materials."
(Please create your materials under your own responsibility as the author.)
Examples of Parts Where Rigor Was Sacrificed
While not exhaustive, listed below are parts of the materials where rigor has been significantly sacrificed.
(I will describe them as brief notes without detailed explanation, so please research them if you are curious.)
Page 5
These are not the only methods proposed in DeepSeek-V2 and V3.
Page 24
PPO training requires a reference model (the LLM before reinforcement learning).
The KL divergence of the output between the pre-reinforcement learning model and the model currently undergoing reinforcement learning is introduced into the objective function as a regularization term.
Page 28
Regarding the explanation of the state-value model, I believe I haven't missed the essence, but I am not confident about its strict rigor.
As with other pages, please refer to the mathematical formulas (the loss function of the state-value model).
Page 29
Strictly speaking, Advantage is not the difference between the reward and the expected value.
Strictly, it is a weighted sum of TD errors from 1-step to n-steps.
However, as a rough discussion, considering it as the difference between cumulative reward and its expected value should not miss the essence.
Also, the contribution of PPO is not the introduction of Advantage. That appeared in even earlier methods.
The primary contribution of PPO lies in clipping the objective function value to prevent large changes before and after the update when updating the LLM.
(However, I intentionally did not mention it because I thought it would hinder understanding.)
Page 38
DPO is not a reinforcement learning method.
Final Thoughts
This article may have ended up sounding like a series of excuses, but my greatest fear is these materials taking on a life of their own and spreading misunderstandings.
Thank you for your understanding.

Discussion