Neural Network Regularization Techniques: A Scoping Review

Background. Regularization is a crucial technique used to prevent overfitting, improve generalization, and enhance the predictive accuracy of neural networks. Regularization allows you to find a balance between the complexity of the model and its performance on real data. Purpose. This review is aimed at studying the application of regularization techniques by scientists in studies. Materials and Methods. We searched the Scopus database. Eligible studies were chosen published in English after 2019 that include machine learning using neural networks and methods of regularization for different purposes. Titles and abstracts of obtained studies were screened. We extracted information on the field of neural networks, regularization methods, types of data sets and application area of study. These characteristics were then categorized and thematically grouped. Groups were distinguished regarding the regularization methods used, for example, L₁, L₂ and Dropout, used neural networks, for example, CNN, ANN, and datasets, for example, datasets containing pictures or text. Conclusion. The review covers various regularization techniques, including L₁ and L₂ regularization, dropout, batch normalization, data augmentation, early stopping, and others in conjunction with the types of neural networks, scientific fields of application, and types of data on which the networks were trained. We have also created statistics on publications regarding regularization methods by year and country, and identified the main journals and conferences where this topic is frequently discussed.