Student Engagement with a Science Simulation: Aspects that Matter

Northumbria University has developed Northumbria Research Link (NRL) to enable users to access the University’s research output. Copyright © and moral rights for items on NRL are retained by the individual author(s) and/or other copyright owners. Single copies of full items can be reproduced, displayed or performed, and given to third parties in any format or medium for personal research or study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata page. The content must not be changed in any way. Full items must not be sold commercially in any format or medium without formal permission of the copyright holder. The full policy is available online: http://nrl.northumbria.ac.uk/policies.html


Introduction
Information communication technology (ICT) has become ubiquitous as it has become more affordable and more powerful (Madden et al., 2005). By 2008, approximately 66% of British homes had Internet connection, (Office for National Statistics, 2008) and in more recent years, a change in connection to the Internet in the form of broadband has reduced the need for homes to have a computer-dedicated line, increased the speed of data transfer, and allowed for increased use of multimedia within web pages. Valentine, Marsh and Pattie (2005) found that the majority of children used their home computer for school work. Over ten years ago, when Rideout, Foehr and Roberts, (1999) asked a representative sample of American children aged 8-18 which medium they would take to a desert island, the preferred choice was a computer with Internet access. Thus it is not surprising that over recent decades, schools, researchers and policy makers have all shown growing interest in the use of ICT to support classroom teaching and learning.
As a consequence, we have seen increasing literature reporting on various forms of ICT for science education. This literature has included reporting on the use of audience response systems (Rodrigues, Taylor, Cameron, Syme-Smith, & Fortuna, 2010), dataloggers (Tortosa, Pinto, & Saez, 2008), email (Van derMeij & Boersma, 2002), the Internet (Mackenzie, 2010), modelling (Pallant & Tinker, 2004), simulations (Eilks, Witteck, & Pietzner, 2010), virtual character research (Rebolledo-Mendez, Burden, & de Freitas, 2008) and whiteboards (Redman, McDougal, & Rodrigues, 2010). Within this body of work, one can also find research linking the culture of informal computer games, student interest and the development and design of appropriate ICT for chemistry (see Prensky, 2004;Grimley et al., 2010), as well as work on attitudes (Tondeur, Van Keer, van Braak, & Valcke, 2008). In the present paper, we consider more than just the motivational aspect; we look at the process of engagement and the influence of the design element in terms of supporting cognitive and skill development in science education.
Designers' views of learners and their assumptions about learning theories, learning processes and learning practices ensure that content and pedagogy are intertwined before the technology reaches the classroom (Segall, 2004). Consequently, multimedia design for school purposes has been explored and continues to be explored, resulting in a debate about the influence of various factors in supporting or hindering learning. Mayer, Sobko and Mautone (2003) define multimedia learning as the use of at least two different types of media (graphics, audio, video and text) in presenting information. Clarke and Mayer (2003), Ginns (2005) and Moreno (2006) reported a modality principal and suggested that graphical information explained by onscreen text and audio narration led to cognitive overload and was therefore detrimental to learning. In more recent times, studies (see Dunsworth & Atkinson, 2007;Sanchez & Garcia-Rodicio, 2008) suggest that there is no difference in performance based on the presence or absence of audio narration. Eilks et al., (2010) suggested that technology that allows for a seamless interchange between tables, charts, graphs and model displays could support conceptual linking between these representations. Ploetzner, Bodemer and Neudert (2008) suggest that the required high transfer rate may, unfortunately, result in a limited attention span. Testa, Monroy and Sassi (2010) suggest that graphs depicted in textbooks are 'cleaned' of redundant details/irregularities, whereas technology-generated real-time graphs include 'noise' , resulting in some learners finding them challenging to interpret. Indeed, the argument pertaining to computer-based graphing exercises has had a long lifespan. For example, the Brasell (1987) study suggested that a delay in display, even if less than 30 seconds, resulted in subduing nearly all students, for they demonstrated less engagement and became preoccupied with procedural issues. However, Beichner (1990) suggested that student engagement could be lowered if the software constructed the graphs. Schnotz and Bannert (2003) suggested that picture use in multimedia learning processes may not be beneficial in every case, while Schwartz, Andersen, Hong, Howard and McGee (2004) and Azevedo (2004) suggest the use of non-linear learning environments may result in inadequate metacognitive competencies. Paivio's dual coding theory (2006) suggests that multiple references to information with connections between verbal and nonverbal (imagery) processing improves the learning process. Sweller's (1991, 1992) 'split attention' effect (with the learner addressing multiple information sources before trying to integrate the segments to make them intelligible) and their 'redundancy' concept suggest that disparate sources may generate cognitive overload. Paivio (2006), Sweller (1991, 1992) may appear to hold contradictory views, but both sets of ideas seem feasible and at present neither explanation has more currency than the other.
In light of these various arguments, and given the growing use and production of simulations and animations in school chemistry, we decided to explore the influence of chemistry simulation design facets on user progress through a simulation. It is argued that multimedia technology affords an opportunity to better visualise complex relationships. We were interested in the scope of this opportunity and hence developed the following research questions: • What are the differences in the nature of student interactions associated with an altered simulation design format?
• What are the effects of the changes in instruction formats on the process of students' engagement behaviour? • How effective are additional written instructions before the simulation? • How does altering the position of controls on the simulation screen affect students' engagement with the simulation?

Participants
The convenience sample included 57 volunteers from four schools and one tertiary institution. The data collected did not identify the volunteers on a personal level. They were anonymously allocated individual codes when they accessed the website and the different institutions were recognised by the log. The volunteers were asked to provide their age, gender, science subject (science, chemistry, physics, biology) and class/tertiary level, as well as to indicate their previous ICT experience and complete five multiple choice chemistry questions pre-simulation use and post-simulation use. Fifty-seven volunteers submitted required information and 36 of them interacted with the simulation. The data collected from the volunteers who submitted questionnaires and actually interacted with a simulation provided were used for the analysis presented in the present paper.
Among the 36 participants, there were 19 students aged 13-15 years (second year of secondary school) and 15 students aged 16 and over. Two participants did not indicate their age. There were roughly equal numbers of male and female participants (17 females and 16 males) using this simulation. Three participants did not supply details about gender. Research design Professor Thomas Greenbowe (2005) kindly provided access to the code for two of his flash-based simulations (a titration and reactivity of metals) available on the internet as learning resources aimed at introducing college chemistry (general chemistry). We modified the code to create three versions of each simulation and to add a facility for monitoring users' interactions with the simulations. A system was created that randomly allocated one version of the two simulations to each user as they accessed the website. A log of all mouse clicks and interactions with the simulation controls (buttons, sliders, text fields and selection boxes) was generated for each user. The computer tracked the time that the user spent on each stage and on each particular element of the simulations.
This behind-the-scenes recording of activity was chosen for three reasons. Firstly, we felt it would be less intrusive, and that it therefore had the potential to generate more reliable data. Secondly, collecting images of school children is increasingly discouraged by local authorities. Thirdly, the url was available for use outside the classroom, and filming its use in that milieu would be impractical.
Each user had to complete a pre-simulation questionnaire (specific to the chemistry topic for the simulation being viewed) before being randomly allocated one of three versions of the simulation. After the simulation, they were asked to complete a post-simulation test and a post-questionnaire. The pre-simulation and post-simulation chemistry questions were based on those found in standard textbooks. However, these questions are not discussed here, as the present paper focuses on patterns of interaction and engagement. Figure  1 provides an overview of the sequence.

The simulations
The simulations we used probably best fit within the Thomas and Hooper (1991) category of 'experiencing simulations' . Experiencing simulations model particular scenarios, allowing students to manipulate factors to see their impact or influence. The simulations we used were representative of many common types of simulations used in school science lessons. However, by selecting an acid-base titration simulation aimed at 'college level' we were able to explore the influence of age and, consequently, prior experience factors on user ability to follow instructions, as while the acid-base titration would be familiar to older students it would be completely novel to the younger students in our sample cohort. The 13-year-old students would have encountered the terms acid and base, but in our experience they would not have conducted a titration during practical or wet-lab work in schools. Our sample also included first-year university chemistry undergraduate students, who almost certainly would have conducted titrations during their senior years at school and during their first year at university.
The acid-base titration simulation had three versions: the original version (Version 1), a modified version (Version 2) that included a one paragraph pre-text advising students to pay attention to particular aspects (as can be seen in Figure 2), and another modified version (Version 3) that had altered positions for specific elements on the screen (as can be seen in Figure 3).
The following is the excerpt paragraph that appeared on the webpage before the Version 2 titration simulation loaded: "When you click on the button below you will see a simulation that represents a titration. To make the simulation work you must follow the numbered instructions in sequence. So start with instruction 1, then 2, then 3, etc. Some instructions have tabs. You must place the mouse on the tab and drag it open".
In Version 3, a menu tab, also identified with the number 3 on the simulation "Select the Acid and Base", was converted from a 'pull out tab' menu to a fixed position, visible menu. The position of other items on the screen was also modified so that the sequence of instructions was aligned with a common reading pattern (horizontal sequence of left to right) (Gvozdenko et al., 2010).

Data analysis
We used one-way unrelated analysis of variance (ANOVA) to determine whether the different versions of the acid-base titration simulation had an impact on how students followed the sequence of instructions. Each simulation version involved a separate and unrelated sample of subjects, so we assumed equal population variation and normal distribution of our random population within the different version cohort. The one-way ANOVA allowed us to deduce the mean for the three versions and then compare these means between the versions. Calculating the one-way ANOVA and the variation between scores meant we could compare the variation between sample means for each simulation version. In the null hypothesis, the assumption was that the mean for Version 1 was the same as the mean for Version 2 and Version 3. However, if the one-way ANOVA showed that the variation between the samples was bigger than the variation in the population, we would have to accept the alternative hypothesis, i.e., that the variation was due to an independent variable. If the variability was statistically significant, the findings would indicate that the independent variable was having an effect.
We used SPSS to separate the groups for analysis, creating a grouping variable called simulation, and represented each of the three Versions as 1, 2 or 3. As would be expected, the time required to complete each respective simulation version was entered under a variable named 'Time' . Means and standard deviations were determined for each version, and by using Levene's Test of Homogeneity of Variance we verified that the assumption of homogeneity of variance was met.
A modified grounded theory approach (see Strauss and Corbin, 1998) allowed us to group track patterns as they emerged from the logged data: a preliminary reading of the tracks allowed for familiarisation of the whole data set of 36 tracks. At this stage, we suggested explanations, which was followed by a closer reading of the tracks that led to interpreting and coding into themes. To ensure rigour, the data analysis was triangulated. As two independent researchers, we reviewed the data and then reflected on and compared the themes that emerged from our independent analysis. This process helped us to develop perspectives while reducing subjectivity bias. Themes that emerged from the tracks as common or typical, resulting in what van Manen (1990) called 'control and order' , allowed us to generate what Polkinghorne (1988) called 'plotlines' for the collated tracks. These plotlines inform the writing presented in this article. When reviewing the tracking data, we were particularly interested in the nature of actions and steps taken by the users, as we were interested in the nature of engagement with the different versions of the simulation. Tracking their engagement could also tell us about the influence of particular design elements, thus allowing for an evaluation of effectiveness and performance as gauged by the pre-simulation and post-simulation tests.

Findings
Our findings are based on the tracks generated by student engagement and actions when using a randomly assigned version of the titration simulation. As the simulations were allocated randomly to volunteers, six students completed a pre-survey and engaged with Version 1, while 14 students completed a pre-survey and engaged with Version 2 and 16 students completed a pre-survey and engaged with Version 3.
Our findings show that 62% of the participants thought the titration simulation was equally as interesting as a computer game, and 82% believed that science simulations were easy. Hence it could be argued that the students involved were not novices in using simulations, and perceived themselves to be efficient simulation users. Despite this, the tracks showed that, unfortunately, only one participant reached the correct response in the field CALC OK at STEP 6.

Positioning instructions and icons
An analysis of the tracks showed that if a button that controls the dropwise addition from the burette is at a more central location it increases the number of interactions with that particular control by approximately 25%. The analysis also showed that having control elements in a side position decreased the number of interactions.
Analysis showed that converting a tab menu (that slid out) into a fixed menu resulted in a decrease in the number of overall interactions, including non-productive interactions, by 30-40%.
The data collected also allows an analysis of the relationship between student responses (in terms of gender, age, computer game experience and simulation user experience) and two measures of their behaviour and activity when using the simulation: (a) the pattern of engagement with the simulation inputs/controls, (b) the total number of interactions between a student and the simulation.
A one-way unrelated analysis of variance (ANOVA) found that the simulation version had a significant effect on how students followed the order of the instructions (F2,29=3.69, p<0.05). The extent to which students followed a recommended sequence of controls was significantly higher among the students using simulation Version 3 (M=4.24, SD=1.43), with 16 students, than for students using simulation Version 2 (M=2.85, SD=1.46), with 14 students. This was independent of age or gender. The extent to which students followed the intended sequence of controls was also higher with students using Version 3 in comparison with students using Version 1 (M=3.20, SD=1.10). However, as indicated previously, the Version 3 and Version 1 comparative finding warrants a degree of care, as there was a smaller number of students (n=6) using Version 1.
Contrary to our expectations, prior experience in playing computer games had no significant effect on how students followed the order of the instructions (F1,29=0.132, p=0.719). However, prior experience in playing games had a significant effect on the number of interactions (F1,29=4.81, p=0.036), with those students who indicated that they did not play computer games (n=10, M=40, SD=33) having nearly three times fewer interactions than those who indicated that they played computer games (n=21, M=129, SD=23).
The students who had previous experience (n=11, M=68, SD=64) with simulations in a lesson were on average engaged in more interactions with the simulation than those who did not (n=20, M=118, SD=129). This effect was not statistically significant. A one-way unrelated analysis of variance (ANOVA) found that prior experience in using simulations in a lesson had a significant effect on how students followed the order of the instructions (F1, 29=4.21, p<0.05). Perhaps, as to be expected, the students with no experience in simulation use in classrooms (n=10, M=2.82, SD=1.47) on average followed the order of the controls less efficiently than those with prior experience (n=20, M=3.95, SD=1.54).
Student perception of 'easiness' in using a simulation was found to have a significant effect on the number of interactions (F2,24=5.31, p<0.05). The students who thought that it was "very easy" to use a simulation (n=2, M=336, SD=202) on average had twice as many interactions than those who thought that simulations are "easy" (n=20, M=90, SD= 97) or "not easy" (n=5, M=100, SD=85).
The analysis of data showed that age did not have any significant effect on student behaviour patterns (F3,27=0.274, p=0.843).
This would suggest that regardless of whether or not the students had previously encountered the chemistry (acid-base titrations) there was no significant effect on behaviour, which would imply prior knowledge of chemistry did not have a significant effect on either the number of interactions or the order in following instructions.

Patterns of behaviour
Two of the students using Version 1 and two of the students using Version 2 did not appear to pay attention to the 'number sequence' associated with the instructions. These numbered instructions were intended to steer them and guide the decisions they made with respect to their process order. What was noticeable was that the proportion of those wrongly following numbered instructions was less for the cohort using Version 2 (simulation with pre-direction) than the cohort using Version 1, but not less than the cohort using Version 3. The difference in behaviour between the three versions showed that 10 of the 14 participants using Version 2, which is over two thirds, had chaotic behaviour patterns. In contrast, only three of the 16 participants using Version 3 had chaotic behaviour patterns, while 11 of those using simulation Version 3 (fixed position openly displayed menu and modified reading pattern) followed the steps sequentially.
Interestingly, despite having directions to steer them towards the process sequence order, only one of the participants managed to follow the steps, and 10 of the 14 students who used Version 2 either showed chaotic behaviour or only managed to complete step/instruction 2 in sequence. In addition, three of the participants using Version 2 (which provided pre-direction before they commenced using the simulation) took between 2.5 and 3 minutes to find and operate the sliding tab menu (instruction 3). There appeared to be a similar age distribution across Version 2 and Version 3, so the chaotic patterns were not due solely to age and possible prior experience. In fact, there were five first-year undergraduates using Version 2, and only one of them reached step 4 in simulation Version 2.

Conclusion
Our findings involve a small sample size, and with this come the usual caveats regarding drawing generalisations. Nevertheless, our findings suggest that simply providing instructions for students to read prior to using a simulation does not necessarily result in the students following the sequence in the simulation as designed. However, if the design is less ambiguous, for example, a 'pull out tab' menu when converted to a fixed position visible menu, the result is better engagement. It seems that additional instructions before a simulation cannot compensate for ambiguity in simulation design: despite being given directions advising them of the process sequence order, most users of the original versions (1 and 2) showed chaotic engagement behaviour tracks. In contrast, the modified Version 3, with a left-to-right and top-to-bottom aligned sequence of menu controls and a fixed visible menu, saw only one fifth of the cohort displaying chaotic behaviour, with most users following the intended sequence.
The presence of an interactive component in a simulation needs to be justified by a learning goal. While visual demonstration involving chemical laboratory tools, such as a probe or a thermometer, or a depiction of atom movement in different chemical solutions could aid learning, the interactive sliding out menu tab was a hurdle for some students who clicked multiple times on the tab control.
Our findings also show that age did not have any significant effect on the student behaviour patterns. Given that some of the participants were undergraduate degree-level students, this would suggest that regardless of whether or not they had previously encountered acid-base titrations there was no significant effect on engagement behaviour. This implies that prior knowledge of chemistry did not have a significant effect on either the number of interactions or the order in following instructions. These findings suggest that simulation design is therefore crucial if, for example, a simulation is to be used for assessment purposes. For a student may have the requisite subject content knowledge to enable them to undertake a wet-lab practical, but when they encounter a simulated version of that wet-lab practical it may be their ability to engage effectively with the technology that hinders their ability to perform to capacity. Furthermore, given their apparent self-perception of their computer skills, they may underestimate the impact of the technological skill required to use a chemistry simulation, if they fail to pay attention to the instructions. Hence, while we would advocate that when creating simulations designers need to take care to ensure that what appears obvious to them is equally obvious to the user, we would also suggest that users, in this case students, need to start taking responsibility and understand that, while their chemistry may be sound, it may be their inability to follow instructions that affects their assessment if the assessment involves a simulation.

Biographical note
Susan Rodrigues is a Professor in Education at the Northumbria University, England. She is a science teacher educator. Her research is in the area of science education, with particular reference to the use of ICT and the role of context. Eugene Gvozdenko, Dr., is an academic researcher with Melbourne Graduate School of Education, The University of Melbourne, Australia. He has teaching experience in the area of foreign languages and IT in education in secondary and adult education. Research interests include on-line assessment systems. Currently, he works on the SMART project (smartvic.com) which provides diagnostics of students' mathematical thinking and conceptual development.