Academic Journal of Mathematical Sciences, 2024, 5(3); doi: 10.25236/AJMS.2024.050310.
Shuai Wang
School of Mathematics and Statistics, Hainan University, Haikou, China
In this paper, a new generalized memory mechanism is proposed, which makes it possible to partition the training set accurately without changing the equation constraints of the original least squares loss by using a new memory influence function that allows the model to avoid overfitting. We propose Least Squares Generalization-Memorization Machines (LSGMM) and give a memory influence function suitable for this model. Experimental results show that our LSGMM has better generalization performance and significant memory capability compared to the least squares support vector machine. Meanwhile, it has a significant advantage in terms of time cost compared with other memory models.
Least squares support vector machine, Generalization-memorization mechanism, Generalized memory, Memorization kernel
Shuai Wang. Least Squares Generalization-Memorization Machines. Academic Journal of Mathematical Sciences (2024) Vol. 5, Issue 3: 67-74. https://doi.org/10.25236/AJMS.2024.050310.
[1] Rosenfeld E, Ravikumar P, Risteski A. The risks of invariant risk minimization[J]. arXiv preprint arXiv: 2010.05761, 2020.
[2] Li T, Beirami A, Sanjabi M, et al. Tilted empirical risk minimization[J]. arXiv preprint arXiv: 2007.01162, 2020.
[3] Rice L, Wong E, Kolter Z. Overfitting in adversarially robust deep learning[C]//International Conference on Machine Learning, 2020. PMLR.
[4] Cortes C, Vapnik V. Support-vector networks[J]. Machine Learning, 1995, 20: 273-297.
[5] Vapnik V, Izmailov R. Reinforced SVM method and memorization mechanisms[J]. Pattern Recognition, 2021, 119: 108018.
[6] Ding X, Liu J, Yang F, et al. Random radial basis function kernel-based support vector machine[J]. Journal of the Franklin Institute, 2021, 358(18): 10121-10140.
[7] Wang Z, Shao Y. Generalization-Memorization Machines[J]. arXiv preprint arXiv: 2207.03976, 2022.
[8] Li Z, Hoiem D. Learning without forgetting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(12): 2935-2947.
[9] Li Z, Liu F, Yang W, et al. A survey of convolutional neural networks: analysis, applications, and prospects[J]. IEEE transactions on neural networks and learning systems, 2021, 33(12): 6999-7019.
[10] Kim D, Bae J, Jo Y, et al. Incremental learning with maximum entropy regularization: Rethinking forgetting and intransigence[J]. arXiv preprint arXiv: 1902.00829, 2019
[11] Lopez-Paz D, Ranzato M. Gradient episodic memory for continual learning[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, NY, USA, 2017.
[12] Arpit D, Jastrzębski S, Ballas N, et al. A closer look at memorization in deep networks[C]// International Conference on Machine Learning, 2017: 233-242.
[13] Feldman V. Does learning require memorization? A short tale about a long tail[C]//Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, 2020.
[14] Suykens J A K, Vandewalle J. Least squares support vector machine classifiers[J]. Neural processing letters, 1999, 9: 293-300.