Validation of the Strategy for Determining the Numerical Rating of the Cognitive Complexity of Exam Items in the Field of Chemical Kinetics

  • Saša Horvat Faculty of Sciences at University of Novi Sad, Serbia
  • Dušica Rodic Faculty of Sciences at University of Novi Sad, Serbia
  • Nevena Jović Master student of chemistry education at Faculty of Sciences at University of Novi Sad, Serbia
  • Tamara Rončević Faculty of Sciences at University of Novi Sad, Serbia
  • Snežana Babić-Kekez Faculty of Sciences at University of Novi Sad, Serbia
Keywords: mental effort, performance, chemical equilibrium


The main goal of this study was to validate the strategy for the assessment of the cognitive complexity of chemical kinetics exam items. The strategy included three steps: 1) assessment of the difficulty of concepts, 2) assessment of distractor value. and 3) assessment of concepts’ interactivity. One of the tasks was to determine whether there were misconceptions by students that might have influenced their achievement. Eighty-seven students in the first year of secondary school participated in the study. A knowledge test was used as a research instrument to assess the performance, and a five-point Likert-type scale was used to evaluate the perceived mental effort. The strategy was validated using regression analysis from which significant correlation coefficients were obtained between selected variables: students’ achievement and invested mental effort (dependent variables) and a numerical rating of cognitive complexity (independent variable).


Download data is not yet available.


Bain, K., & Towns, M. (2016). A review of research on the teaching and learning of chemical kinetics. Chemistry Education Research and Practice, 17(2), 246–262.

Banerjee, A. (1991) Misconceptions of students and teachers in chemical equilibrium. International Journal of Science Education, 13(3), 355–362.

Banks, P. J. (1997). Students’ understanding of chemical equilibrium. Unpublished MA thesis. Department of Educational Studies, University of York.

Barczi, K. (2013). Applying cooperative techniques in teaching problem solving. Center for Educational Policy Studies Journal, 3(4), 61–78.

Bieri, J. (1955). Cognitive complexity–simplicity and predictive behaviour. Journal of Abnormal and Social Psychology, 51(2), 263–268.

BouJaoude, S., & Barakat, H. (2000). Secondary school students’ difficulties with stoichiometry, School Science Review, 81(296), 91–98.

Brace, N., Kemp, R., & Snelgar, R. (2006) SPSS for Psychologists: A guide to data analysis using SPSS for Windows (3rd ed.). Routledge.

Çakmakci, G. (2010). Identifying alternative conceptions of chemical kinetics among secondary school and undergraduate students in Turkey. Journal of Chemical Education, 87(4), 449–455.

Çam, A., Topçu, M. S., & Sülün, Y. (2015). Preservice science teachers’ attitudes towards chemistry and misconceptions about chemical kinetics. Asia-Pacific Forum on Science Learning and Teaching, 16(2), 1–6.

Cliff, W. H. (2009). Chemistry misconceptions associated with understanding calcium and phosphate homeostasis. Advance in Physiology Education, 33(4), 323–328.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates.

Daniel, R. C.. & Embretson, S.E. (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34(5), 348–364.

Embretson, S. E., & Daniel, R. C. (2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychological Science, 50, 328–344.

Evans, J. D. (1996). Straightforward statistics for the behavioral sciences. ‎Brooks/Cole.

Goldreich, O. (2008). Computational complexity: A conceptual perspective. Cambridge University Press.

Gorodetsky, M., & Gussarsky, E. (1986) Misconceptualisation of the chemical equilibrium concept as revealed by different evaluation methods. European Journal of Science Education, 8(4), 427–441.

Hackling, M.W., & Garnett, P. (1985). Misconceptions of chemical equilibria. European Journal of Science Education, 7(2), 205–214.

Horvat, S. A, Mihajlović, J., Rončević, T. N., & Rodić, D. D. (2021). Procedure for the assessment of cognitive complexity: Development and implementation in the topic "Hydrolysis of Salts". Macedonian Journal of Chemistry and Chemical Engineering, 40(1), 119–130.

Horvat, S. A. , Rončević, T. N., Arsenović, D. Z , Rodić, D. D., & Segedinac, M. D. (2020). Validation of the procedure for the assessment of cognitive complexity of chemical technology problem tasks. Journal of Baltic Science Education, 19(1), 64–75.

Horvat, S., Rodić, D. D., Segedinac, M. D., & Rončević, T. N. (2017). Evaluation of cognitive complexity of tasks for the topic hydrogen exponentin the solutions of acids and bases. Journal of Subject Didactics, 2(1), 33–45.

Horvat, S., Segedinac, M. D., Milenković, D. D., & Hrin, T.N. (2016). Development of procedure for the assessment of cognitive complexity of stoichiometric tasks. Macedonian Journal of Chemistry and Chemical Engineering, 35(2), 275–284.

Johnstone, A. H., Macdonald, J. J., & Webb, G. (1977). Chemical equilibria and its conceptual difficulties, Education in Chemistry, 14(6), 169– 171.

Jonsson, A., & Svingby, G. (2002). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research and Reviews, 2(2), 130–144.

Justi, R. (2002), Teaching and learning chemical kinetics. In J. K. Gilbert, De O. Jong, R. Justi, D. Treagust , & J. H. Van Driel (Eds.), Chemical Education: Towards Research based Practice (pp. 293–315). Kluwer.

Kalainoff, M., Lachance, R., Riegner, D., & Biaglow, A. A. (2012). Computer algebra approach to solving chemical equilibria in general chemistry. PRIMUS, 22(4), 284–302.

Kalyuga, S. (2008). Managing cognitive load in adaptive multimedia learning. information science reference. Hershey.

Kelly, G. A. (1955). The psychology of personal construct. A theory of personality. Taylor & Francis.

Kim, S. J., Aleven, V., & Dey, A. K. (2014). Understanding expert-novice differences in geometry problem-solving tasks. CHI '14 Extended Abstracts on Human Factors in Computing Systems.

Knaus, K., Murphy, K., Blecking, A., & Holme, T. (2011). A valid and reliable instrument for cognitive complexity rating assignment of chemistry exam items. Journal of Chemistry Education, 88(5), 554–560.

Kousathana, M., & Tsaparlis, G., (2002), Students’ errors in solving numerical chemical equilibrium problems. Chemistry Education Research and Practice, 3(1), 5–17

Loewenthal, K. M. & Lewis, C. A. (2001). An introduction to psychological tests and scales. Psychology Press.

Mayers, A. (2013). Introduction to statistics and SPSS in psychology. Pearson Education.

Maskill, R., & Cachapuz, A. F. C. (1989). Learning about the chemistry topic of equilibrium: The use of word association tests to detect developing conceptualisations. International Journal of Science Education, 11(1), 57–69.

Morgan, G. A., Leech, N. L., Gloackener, G. W., & Barret, K. C. (2011). BM SPSS for introductory statistics: Use and interpretation. Routledge.

Moss, S., Prosser, H., Costello, H., Simpson, N., Patel, P., Rowe, S.,Tuner, S., & Hatton, C. (1998). Reliability and validity of the PAS-ADD checklist for detecting psychiatric disorders in adults with intellectual disability. Journal of Intellectual Disability Research, 42(2), 173–183.

Šurjanović, M., & Nikolajević, R. (2011). Hemija - Zbirka zadataka iz hemije - za 1. i 2. razred gimnazije prirodno-matematičkog smera, medicinsku i poljoprivrednu školu [Chemistry - Collection of tasks in chemistry - for the 1st and 2nd grade of the gymnasium of natural mathematics, medical and agricultural school]. Zavod za udžbenike i nastavna sredstva.

Pande, S. S., Pande, R. P., Parate, V. P., Nikam, A. N., & Agrekar, S. H. (2013). Correlation between difficulty and dis-crimination indices of MCQs in formative exam in physiology. South-East Asian Journal of Medical Education, 7(1), 45–50.

Pippenger, N. (1978). Complexity theory. Scientific American, 238(6), 114–125.

Raker, J. R., Trate, J. M., Holme, T. A.. & Murphy, K. (2013). Adaptation of an instrument for measuring the cognitive complexity of organic chemistry exam Items. Journal of Chemical Education, 90(10), 1290–1295.

Segedinac, M., Segedinac, M., Konjović, Z., & Savić, G. (2011). A formal approach to organization of educational objectives. Psihologija, 44(4), 307–323.

Segedinac, M. T., Horvat, S., Rodić, D. D., Rončević, T. N., & Savić, G. (2018). Using knowledge space theory to compare expected and real knowledge spaces in learning stoichiometry. Chemistry Education Research and Practice, 19(3), 670–680.

Soeharto, S., Csapó, B., Sarimanah, E., Dewi, F.I., & Sabri, T. (2019). A review of students’ common misconceptions in science and their diagnostic assessment tools. Jurnal Pendidikan IPA Indonesia, 8(2), 247–266.

Soeharto, S., & Csapó, B. (2021). Evaluating item difficulty patterns for assessing student misconceptions in science across physics, chemistry, and biology concepts. Heliyon, 7(11), e08352.

Sözbilir M., Pinarbasi T., & Canpolar N. (2010). Prospective chemistry teachers’ conceptions of chemical thermodynamics and kinetics. Eurasian Journal of Mathematics, Science and Technology Education, 6(2), 111–120.

Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. Springer.

Taber, K. S., (2002). Chemical misconceptions – prevention, diagnosis and cure: Theoretical background. Royal Society of Chemistry.

Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296.

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55.

Turányi, T., & Tóth, Z., (2013). Hungarian university students’ misunderstandings in thermodynamics and chemical kinetics. Chemistry Education Research and Practice, 14(1), 105–116.

Tóth, Z. (1999), Egy kémiai tévképzet nyomában. Az egyensúlyi állandó bevezetésének lehetöségei és problémái [Tracing a chemical misconception. The challenges and problems of the introduction of the chemical equilibrium constant]. Iskolakultúra, 9, 108–112.

Towns, M. H. (2014). Guide to developing high-quality, reliable, and valid multiple-choice assessments. Journal of Chemical Education, 91(9), 1426−1431.

Van Driel, J. H. (2002) Students’ corpuscular conceptions in the context of chemical equilibrium and chemical kinetics. Chemistry Education: Research and Practice, 3(2), 201–213.

Van Gog, T., Kester., L., & Paas, F. (2011). Effects of concurrent monitoring on cognitive load and performance as a function of task complexity. Applied Cognitive Psychology, 25(4), 584–587.

Yan, Y. K., & Subramaniam, R. (2018). Using a multi-tier diagnostic test to explore the nature of students’ alternative conceptions on reaction kinetics. Chemistry Education Research and Practice, 19(1), 213–226

Yunus, W. M. D. Z. W., & Ali, Z .M. (2012). Urban Students’ Attitude towards Learning Chemistry. Procedia-Social and Behavioral Sciences, 68, 295–304. https://10.1016/j.sbspro.2012.12.228

Zubairi, A., Lide, N., & Abu Kassim, N. L. (2006). Classical And Rasch Analyses Of Dichotomously Scored Reading Comprehension Test Items. Malaysian Journal of ELT Research, 2, 1–20.