Probabilistic artifact filtration in adaptive testing

Computerized testing is widely used for diagnostics and estimation of professional skills including CM education quality control. Both quality of testing and reliability of its results depend on a selected test technology that was a scientific research object during last years. Numerous problems following applications of tra¬ditional testing techniques inspired creation of the adaptive testing technology under consideration, which is based on application of trained structures in the form of continuous-time Markov models. Its peculiarities, in particular, are revealing and using test solution capability changes in quantitative evaluation of their time-domain dynamics as well as taking into account timetable of testing process. The approach suggested has certain advantages over the testing techniques, which were used before, owing to its greater information capability and acceleration of a test procedure. The main subject under consideration is elimination of artifacts conditioned by certain forms of illegal purposeful interference in testing procedure. It is carried out on the basis of comparing observed and expected subject responses with the aid of the Kalman filter adapted to the peculiarities of the problem in question.

Estimation of probabilities for various skill levels is performed basing on test results obtained with the aid of parametric mathematical models described by Markov random processes with discrete states and continuous or discrete time. Further discussion applies to the models with continu­ous time only. Directly observable quantity is the difficulty of task being executed, measured in logit. The valid range of this quantity is divided into several intervals, each of them is considered as a separate state xi, i=0,1,­,n, in which a testee may be with certain probability, transferring from one state to another according to certain rules. The length of these intervals determines the discrim­ination of estimates obtained in the testing process. In turn, the number of states is determined by the desired discrimination of estimates and available sample size.

1. Avanesov V.S., “Pedagogical measurement of latent qualities”, Pedagogical Diagnostics, 2003, No 4, pp 69-78 (in Russian).
2. Kardanova E.Yu., “Modeling and test parameterization: theory foundations and applications”, Moscow: FGU «Federal Testing Center¬, 2008 (in Russian).
3. Cramer H., “Mathematical methods of statistics”, Moscow: Mir, 1976, pp 648 (in Russian).
4. Kuravsky L.S., Baranov S.N., “Synthesis of Markov networks for forecasting of fatigue fail­ures”, Neurocomputers: Development and Application, 2002, No 11, pp 29-40 (in Russian).
5. Kuravsky L.S., Baranov S.N., “Application of neural networks for diagnostics and forecasting of fatigue failures of thin-walled structures”, Neurocomputers: Development and Application, 2001, No 12, pp 47-63 (in Russian).
6. Kuravsky L.S., Baranov S.N., Kornienko P.A., “Trained multifactor Markov networks and their application for studying psychological characteristics”, Neurocomputers: Development and Ap­plication, 2005, No 12, pp 65-76 (in Russian).
7. Kuravsky L.S., Baranov S.N., Malykh S.B., “Neural networks for forecasting, diagnostics and data analysis: Textbook”, Moscow: RUSAVIA, 2003, 100 pp (in Russian).
8. Kuravsky L.S., Baranov S.N., Yuryev G.A., “Synthesis and identification of hidden Markov models for fatigue damage diagnostics”, Neurocomputers: Development and Application, 2010, No 12, pp 20-36 (in Russian).
9. Kuravsky L.S., Marmalyuk P.A., Panfilova A.S., Ushakov D.V., “Studying factor influences on psychological characteristics development with the aid of a new approach to estimation of goodness-of-fit measure”, Information Technologies, 2011, No 11, pp 67-77 (in Russian).
10. Kuravsky L.S., Yuryev G.A., “Adaptive testing as a Markov process: models and their identifi­cation”, Neurocomputers: Development and Application, 2011, No 2, pp 21-29 (in Russian).
11. Kuravsky L.S., Yuryev G.A., “Using Markov models in processing testing results”, Voprosy Psychologii, 2011, No 2, pp 98-107 (in Russian).
12. Ovcharov L.A., “Applied problems of the queuing theory”, Moscow: Mashinostroenie, 1969, 324 pp (in Russian).
13. Saaty Th.L., “Elements of queuing theory with applications”, Moscow: LIBROCOM, 2010, 520 pp (in Russian).
14. Tikhonov V.I., Shakhtarin B.I., Sizykh V.V., “Random processes. Examples and tasks. /V.5. Estimation of signals, their parameters and spectra. Information theory foundations”, Moscow: Goryachaya liniya – Telecom, 2009, 400 pp (in Russian).
15. Tyumeneva Yu.A., “Psychological measurements”, Moscow: Aspect-Press, 2007 (in Russian).
16. Shakhtarin B.I., “Random processes in radio frequency engineering. /V.1. Linear transforms”, Moscow: Goryachaya liniya – Telecom, 2010, 520 pp (in Russian).
17. Baker F.B., “The basics of Item Response Theory”, ERIC Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, MD, 2001.
18. Gregory R.J., “Psychological testing: History, principles, and applications” (5th edition), New York: Pearson, 2007.
19. Gulliksen H., “Theory of mental tests”, John Wiley & Sons Inc, 1950.
20. Kuravsky L.S., Malykh S.B., “Application of Markov models for analysis of development of psychological characteristics”, Australian Journal of Educational & Developmental Psychology, 2004, Vol 2, pp 29-40.
21. Kuravsky L.S. and Baranov S.N., “Condition monitoring of the structures suffered acoustic fa­tigue failure and forecasting their service life”, Proc. Condition Monitoring 2003, Oxford, Unit­ed Kingdom, pp 256-279, July 2003.
22. Kuravsky L.S. and Baranov S.N., “Neural networks in fatigue damage recognition: diagnostics and statistical analysis”, Proc. 11th International Congress on Sound and Vibration, St.­Petersburg, Russia, pp 2929-2944, July 2004.
23. Kuravsky L.S. and Baranov S.N., “The concept of multifactor Markov networks and its applica­tion to forecasting and diagnostics of technical systems”, Proc. Condition Monitoring 2005, Cambridge, United Kingdom, pp 111-117, July 2005.
24. Kuravsky L.S., Baranov S.N. and Yuryev G.A., “Synthesis and identification of hidden Markov models based on a novel statistical technique in condition monitoring”, Proc. 7th International Conference on Condition Monitoring & Machinery Failure Prevention Technologies, Stratford-upon-Avon, England, June 2010, 23 pp.
25. Kuravsky L.S., Marmalyuk P.A. and Panfilova A.S., “Estimation of goodness-of-fit measures for identification of unrestricted factor models employing arbitrarily distributed observed data”, Proc. 8th International Conference on Condition Monitoring & Machinery Failure Prevention Technologies, Cardiff, UK, June 2011, 19 pp.
26. Rasch G., “Probabilistic models for some intelligence and attainment tests”, Copenhagen, Dan­ish Institute for Educational Research, expanded edition (1980) with foreword and afterword by
27. B.D. Wright, Chicago: The University of Chicago Press, 1960/1980.
28. Roweis S. and Ghahramani Z. “A unifying review of linear Gaussian models”, Neural Computa­tion, Vol 11, No 2, 1999, pp 305–345.
29. Wright B.D., Masters G.N., “Rating scale analysis. Rasch measurements”, Chicago: MESA  Press, 1982.
30. Wright B.D., Stone M.N., “Best test design”, Chicago: MESA Press, 1979.