Моделирование и анализ данных 2012. № 1. С. 70–81
2219-3758 / 2311-9454 (online)
Probabilistic artifact filtration in adaptive testing 733
Куравский Л.С., доктор технических наук, декан факультета информационных технологий, Московский государственный психолого-педагогический университет, Москва, Россия, firstname.lastname@example.org Юрьев Г.А., кандидат физико-математических наук, зам. декана, доцент, факультет информационных технологий, ФГБОУ ВО МГППУ, Москва, Россия, email@example.com
Computerized testing is widely used for diagnostics and estimation of professional skills including CM education quality control. Both quality of testing and reliability of its results depend on a selected test technology that was a scientific research object during last years. Numerous problems following applications of tra¬ditional testing techniques inspired creation of the adaptive testing technology under consideration, which is based on application of trained structures in the form of continuous-time Markov models. Its peculiarities, in particular, are revealing and using test solution capability changes in quantitative evaluation of their time-domain dynamics as well as taking into account timetable of testing process. The approach suggested has certain advantages over the testing techniques, which were used before, owing to its greater information capability and acceleration of a test procedure. The main subject under consideration is elimination of artifacts conditioned by certain forms of illegal purposeful interference in testing procedure. It is carried out on the basis of comparing observed and expected subject responses with the aid of the Kalman filter adapted to the peculiarities of the problem in question.
Estimation of probabilities for various skill levels is performed basing on
test results obtained with the aid of parametric mathematical models described
by Markov random processes with discretestates and continuous or
discrete time. Further discussion applies to the models with continuous
time only. Directly observable quantity is the difficulty of task being
executed, measured in logit. The valid range of this quantity is divided into
several intervals, each of them is considered as a separate state
xi, i=0,1,,n, in which a testee may be with
certain probability, transferring from one state to another according to
certain rules. The length of these intervals determines the discrimination of
estimates obtained in the testing process. In turn, the number of states is
determined by the desired discrimination of estimates and available sample
Avanesov V.S., “Pedagogical measurement of latent qualities”,
Pedagogical Diagnostics, 2003, No 4, pp 69-78 (in Russian).
Kardanova E.Yu., “Modeling and test parameterization: theory
foundations and applications”, Moscow: FGU «Federal Testing Center¬, 2008 (in
Cramer H., “Mathematical methods of statistics”, Moscow: Mir,
1976, pp 648 (in Russian).
Kuravsky L.S., Baranov S.N., “Synthesis of Markov networks for
forecasting of fatigue failures”, Neurocomputers: Development and Application,
2002, No 11, pp 29-40 (in Russian).
Kuravsky L.S., Baranov S.N., “Application of neural networks for
diagnostics and forecasting of fatigue failures of thin-walled structures”,
Neurocomputers: Development and Application, 2001, No 12, pp 47-63 (in
Kuravsky L.S., Baranov S.N., Kornienko P.A., “Trained multifactor
Markov networks and their application for studying psychological
characteristics”, Neurocomputers: Development and Application, 2005, No 12, pp
65-76 (in Russian).
Kuravsky L.S., Baranov S.N., Malykh S.B., “Neural networks for
forecasting, diagnostics and data analysis: Textbook”, Moscow: RUSAVIA, 2003,
100 pp (in Russian).
Kuravsky L.S., Baranov S.N., Yuryev G.A., “Synthesis and
identification of hidden Markov models for fatigue damage diagnostics”,
Neurocomputers: Development and Application, 2010, No 12, pp 20-36 (in
Kuravsky L.S., Marmalyuk P.A., Panfilova A.S., Ushakov D.V.,
“Studying factor influences on psychological characteristics development with
the aid of a new approach to estimation of goodness-of-fit measure”,
Information Technologies, 2011, No 11, pp 67-77 (in Russian).
Kuravsky L.S., Yuryev G.A., “Adaptive testing as a Markov process:
models and their identification”, Neurocomputers: Development and Application,
2011, No 2, pp 21-29 (in Russian).
Kuravsky L.S., Yuryev G.A., “Using Markov models in processing
testing results”, Voprosy Psychologii, 2011, No 2, pp 98-107 (in Russian).
Ovcharov L.A., “Applied problems of the queuing theory”, Moscow:
Mashinostroenie, 1969, 324 pp (in Russian).
Saaty Th.L., “Elements of queuing theory with applications”,
Moscow: LIBROCOM, 2010, 520 pp (in Russian).
Tikhonov V.I., Shakhtarin B.I., Sizykh V.V., “Random processes.
Examples and tasks. /V.5. Estimation of signals, their parameters and spectra.
Information theory foundations”, Moscow: Goryachaya liniya – Telecom, 2009, 400
pp (in Russian).
Tyumeneva Yu.A., “Psychological measurements”, Moscow:
Aspect-Press, 2007 (in Russian).
Shakhtarin B.I., “Random processes in radio frequency engineering.
/V.1. Linear transforms”, Moscow: Goryachaya liniya – Telecom, 2010, 520 pp (in
Baker F.B., “The basics of Item Response Theory”, ERIC
Clearinghouse on Assessment and Evaluation, University of Maryland, College
Park, MD, 2001.
Gregory R.J., “Psychological testing: History, principles, and
applications” (5th edition), New York: Pearson, 2007.
Gulliksen H., “Theory of mental tests”, John Wiley & Sons Inc,
Kuravsky L.S., Malykh S.B., “Application of Markov models for
analysis of development of psychological characteristics”, Australian Journal
of Educational & Developmental Psychology, 2004, Vol 2, pp 29-40.
Kuravsky L.S. and Baranov S.N., “Condition monitoring of the
structures suffered acoustic fatigue failure and forecasting their service
life”, Proc. Condition Monitoring 2003, Oxford, United Kingdom, pp 256-279,
Kuravsky L.S. and Baranov S.N., “Neural networks in fatigue damage
recognition: diagnostics and statistical analysis”, Proc. 11th International
Congress on Sound and Vibration, St.Petersburg, Russia, pp 2929-2944, July
Kuravsky L.S. and Baranov S.N., “The concept of multifactor Markov
networks and its application to forecasting and diagnostics of technical
systems”, Proc. Condition Monitoring 2005, Cambridge, United Kingdom, pp
111-117, July 2005.
Kuravsky L.S., Baranov S.N. and Yuryev G.A., “Synthesis and
identification of hidden Markov models based on a novel statistical technique
in condition monitoring”, Proc. 7th International Conference on Condition
Monitoring & Machinery Failure Prevention Technologies,
Stratford-upon-Avon, England, June 2010, 23 pp.
Kuravsky L.S., Marmalyuk P.A. and Panfilova A.S., “Estimation of
goodness-of-fit measures for identification of unrestricted factor models
employing arbitrarily distributed observed data”, Proc. 8th International
Conference on Condition Monitoring & Machinery Failure Prevention
Technologies, Cardiff, UK, June 2011, 19 pp.
Rasch G., “Probabilistic models for some intelligence and
attainment tests”, Copenhagen, Danish Institute for Educational Research,
expanded edition (1980) with foreword and afterword by
B.D. Wright, Chicago: The University of Chicago Press, 1960/1980.
Roweis S. and Ghahramani Z. “A unifying review of linear Gaussian
models”, Neural Computation, Vol 11, No 2, 1999, pp 305–345.