Jiachen Yao(@jiacheny7):I found that Adam variants perform the best when beta_1=beta_2 in my work on optimizers for physics-informed neural networks (PINNs). And I'm glad to see that people have also found this secret sauce for Adam, on a wider domain. Setting beta_1 equal to beta_2 aligns the momentum tracking history length, stabilizing the scale. (especially when the loss surface is changing a lot) It also puts less penalization on the gradient norm, making the gradients larger, thus converging faster. ⏩Table 4 in this PINN literature: https://t.co/StKBtBdOqA ⏩Newer literatures: https://t.co/AO7rD3nzgh

Jiachen Yao

@jiacheny7

AI+Physics | Ph.D. student at @Caltech | Generative Modeling | Machine Learning | 🥂⛰️

加入 April 2023

76 正在關注 53 粉絲

Jiachen Yao@jiacheny7

2026.02.26 02:05

I found that Adam variants perform the best when beta_1=beta_2 in my work on optimizers for physics-informed neural networks (PINNs). And I'm glad to see that people have also found this secret sauce for Adam, on a wider domain. Setting beta_1 equal to beta_2 aligns the momentum tracking history length, stabilizing the scale. (especially when the loss surface is changing a lot) It also puts less penalization on the gradient norm, making the gradients larger, thus converging faster. ⏩Table 4 in this PINN literature: ⏩Newer literatures: