This content originally appeared on HackerNoon and was authored by Reinforcement Technology Advancements
Table of Links
3 Model and 3.1 Associative memories
6 Empirical Results and 6.1 Empirical evaluation of the radius
6.3 Training Vanilla Transformers
7 Conclusion and Acknowledgments
Appendix B. Some Properties of the Energy Functions
Appendix C. Deferred Proofs from Section 5
Appendix D. Transformer Details: Using GPT-2 as an Example
7 Conclusion
We model transformer-based networks with associative memory and study the cross-entropy loss with respect to model and data sizes. By proposing a new energy function in Eq. 5, which does not rely on additional regularization terms as is common in modern continuous Hopfield networks, we demonstrate that the proposed energy function corresponds to a nearest neighbor search across patterns memorized during training. We then construct a global energy function for the layered structure of the transformer models using the majorization-minimization technique.
\ In practice, we have observed that the majority of transformer models tend to achieve a cross-entropy loss of approximately 2.2. The optimal balance between model and data sizes, however, is often determined by the collective expertise of practitioners. Additionally, the performance of these models can be compromised by both early and delayed stopping.
\ We believe the current paper represents an important step towards understanding the convergence and generalization behaviors of large transformer models. It provides insights into the theoretically optimal cross-entropy loss, which can inform both budgetary planning and model termination strategies.
Acknowledgments
The author thanks Dr. Yongqi Xu for stimulating discussions and practical assistance with the experiments.
\
:::info Authors:
(1) Xueyan Niu, Theory Laboratory, Central Research Institute, 2012 Laboratories, Huawei Technologies Co., Ltd.;
(2) Bo Bai baibo (8@huawei.com);
(3) Lei Deng (deng.lei2@huawei.com);
(4) Wei Han (harvey.hanwei@huawei.com).
:::
:::info This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
:::
\
This content originally appeared on HackerNoon and was authored by Reinforcement Technology Advancements
Reinforcement Technology Advancements | Sciencx (2025-06-22T17:30:03+00:00) New Regularization-Free Energy Function for Transformer Analysis. Retrieved from https://www.scien.cx/2025/06/22/new-regularization-free-energy-function-for-transformer-analysis/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.