Gradient Optimization Methods in Machine Learning for the Identification of Dynamic Systems Parameters

The article considers one of the possible ways to solve the problem of estimating the unknown parameters of dynamic models described by differential-algebraic equations. Parameters are estimated based on the results of observations of the behavior of the mathematical model. Their values are found as a result of minimizing the criterion that describes the total quadratic deviation of the state vector coordinates from the exact values obtained at measurements at different points in time. The parallelepiped type constraints are imposed on the parameter values. To solve the optimization problem, it is proposed to use gradient optimization methods used in machine learning procedures: the stochastic gradient descent method, the classical moment method, the Nesterov accelerated gradient method, the adaptive gradient method, root mean square propagation method, the adaptive moment estimation method, the adaptive estimation method modification, Nesterov–accelerated adaptive moment estimation method. An example of identification of the parameters of a linear mathematical model describing a change in the characteristics of a chemical process is shown, which demonstrates the comparative effectiveness of the optimization methods of the selected group. The methods used to search for an extremum do not guarantee finding a result – a global extremum, but allow you to get a solution of good enough quality for an acceptable time. The results of calculations by all the listed optimization methods are presented. Recommendations on the selection of method parameters are given. The obtained numerical results demonstrated the effectiveness of the proposed approach. The found approximate values of the estimated parameters slightly differ from the best known solutions obtained by other methods.

1. Ivchenko G.I., Medvedev Yu. I. Vvedenie v matematicheskuyu statistiku [Introduction to Mathematical Statistics]. Moscow: Publ. Librocom, 2014.
2. Panteleev A.V., Letova Т.А. Metody optimizacii. Prakticheskij kurs [Optimization Methods. Practical Course]. Мoscow: Logos, 2011.
3. Panteleev A.V., Skavinskaya D.V. Metaehvristicheskie algoritmy globalnoj optimizacii [Global optimization metaheuristic algorithms]. Мoscow: Publ. Vuzovskaya kniga, 2019.
4. Panteleev A.V., Kryuchkov A. Yu. Metaehvristicheskie metody optimizacii v zadachah ocenki parametrov dinamicheskih sistem [Metaheuristic optimization methods for parameters estimation of dynamic systems] // Civil Aviation High Technologies. 2017; 20(2): 37–45.
5. Ruder S. An Overview of Gradient Descent Optimization Algorithms arXiv:1609.04747v2 [cs.LG] 15 Jun 2017.
6. Floudas C.A., Pardalos P.M., Adjimann C.S., Esposito W.R., Gumus Z.H., Harding S.T., Schweiger C.A. Handbook of test problems in local and global optimization, 1999. Vol. 67. Springer US. 442 p. https://titan.princeton.edu/TestProblems/
7. Tjoa I.–B., Biegler L.T. Simultaneous solution and optimization strategies for parameter estimation of differential–algebraic equation systems. Industrial & Engineering Chemistry Research, 1991, Vol. 30, No. 2, pp. 376–385. https://doi.org/10.1021/ie00050a015