Designing a Teacher Dashboard to Analyze Student Interaction with Interactive Simulations

Concurrent Session 9

Session Materials

Brief Abstract

We present the design, development, and testing of a prototype teacher dashboard for highly interactive simulations. The dashboard aims to provide teachers with interpretable data to improve their activities, plan lessons, and identify students who may be having difficulty. Results from classroom testing and teacher surveys will be discussed.

Presenters

PhD student in Physics Education Research. Working with the group of PhET Interactive Simulation from the University of Colorado.

Additional Authors

Dr. Kathy Perkins directs PhET Interactive Simulations at University of Colorado Boulder, and is a faculty member in Physics Education Research. Her work focuses on advancing the design and classroom use of interactive simulations to increase engagement and learning in STEM, and on scaling impact with open educational resources. She previously directed CU’s Science Education Initiative. She has authored over 60 articles on STEM education. In recent years, her team launched PhET-iO sims – interoperable sims enabling customization, integration, and data – and has been a leading innovator in accessible simulations, creating new opportunities to advance learning and assessment for all students.

Extended Abstract

The use of interactive simulations (sims) is increasing in science and math education classrooms. How students interact with sims is connected to the level and type of guidance teachers choose in activities coupled with such simulations.  Research shows that appropriate guidance can help center the attention of students on sim elements that are important for achieving specific learning goals. However, excessive guidance can lead students to follow directions rather than achieving the style of deep exploration associated with high-quality engagement. Student engagement with the sims has proven crucial in reaching meta-goals such as self-questioning, exploring, making predictions, testing ideas, designing experiments, monitoring their own understanding, and authentic scientific inquiry.

Designing sim-centered activities that strike the optimal balance between generating engagement and focusing on specific elements can be challenging for teachers. For homework activities or online courses, for example, teachers appreciate knowing the extent to which the sim was used to answer questions, if the time assigned is appropriate for completing the activity, or if students interact with specific sim elements. With a dashboard for interactive sims, teachers can access information about how their students interact with a sim to answer these questions and more.

A dashboard displays important information in visual representations. The use of dashboards in educational services is not new, but currently all such dashboards are focused on Learning Management Systems (LMS), Massive Open Online Courses (MOOCs), and Intelligent Tutor Systems (ITS). These educational systems use well-defined metrics to generate their visualizations, such as logins, student products, and the instructional resources that students review. In contrast, interactive sims are open-ended environments where each student can manipulate them in many unique and complicated ways. The data produced by such interactions is quite different from other educational web services and a different approach to data sources, analysis, and visualizations should be considered.

The current research focuses on the design and use of a teacher dashboard for analyzing student interaction and engagement with interactive simulations. The research seeks to identify the kinds of interaction data that can generate useful information for teachers and how to present that information to teachers in an easily digestible and informative way through visualizations.   

In this research, student engagement is assumed to be correlated with the exploration of the sims, including the duration of exploration, the number and location of clicks in the sim, which interactive elements of the sims were used, and the evolution of clicks in time. It is important to note that if this dashboard design is intended to characterize student interaction and engagement, not student learning. Students may interact with certain sim elements or activate particular visual representations. By itself, this information does not provide insight into student understanding of those representations. However, the dashboard visualizations seeks to be useful in other ways - for instance, knowing what students may have seen or done in the sim could help generate class discussion or aid in improving activity design.

In this presentation, we will describe the research and design process and present dashboard prototypes, methods for testing, and results.

As an initial step in the design process of the dashboard, we used information gathered in a survey of teachers regarding student data collection for interactive sims answered by college, high school and middle school educators. The most popular data that teachers desire were: information about the states of the sim (e.g. did they light the light-bulb, progress to a certain level, create a saturated solution...), information about the controls used in the sim and the settings (e.g. range of values used in a slide bar), and information about the duration of use.

The sims used in designing and testing this dashboard are from the PhET Interactive Simulations project of the University of Colorado Boulder. The PhET project has over 150 sims for science and math, collectively used over 100 million times per year worldwide. Some PhET sims have been instrumented for interoperability and back end data, and are referred to as PhET-iO sims. The PhET-iO enabled sims allow all student-sim interaction to be captured in a structured data stream file, including input position and time of events (clicks, mousedown and mouseup), total duration of use, time between events, and the sim elements used.

Prototype dashboard mockups were created, adapting indicators and visualizations used in prior research studies about engagement with sims and for specific visualizations used in Google Analytics to understand website use. To refine the design, dashboard mockups were presented to expert designers and developers from the PhET team over a series of meetings and iteratively improved. These discussions included the behavior of the dashboard for the end-user. The primary dashboard visualizations show individual student interaction patterns and aggregated information of an entire group. These include a click map where red dots overlayed on a screenshot of the sim represent all the recorded clicks. This visualization helps to identify the settings of some elements, for example to identify the range values of a slider the students explored. A sim elements map shows the percentage of students that use each element of the sim. Time vs click graphs present each student’s total time of interaction and the moments of more and less interaction with the sim, as well as the median values of these values for the group.

Several sims were used in this first version of the dashboard prototype, including Capacitor Lab:Basics (a sim where students can explore the physics of a parallel plate capacitor, changing the plate area, the separation distance, connecting a battery, monitoring numeric data, and using a voltmeter),  Forces and Motion:Basics (a sim where students can push different objects and explore with the factors that affect movement, like friction, applied force, and mass), and Energy Skate Park (a sim where the user can put a skater on different tracks and observe changes of potential, kinetic and thermal energy using different graphic representations).

Pilot testing of this dashboard prototype was conducted in three college physics courses, collecting student interaction data from assigned homework activities with the sims. Results will be presented and discussed, and provide evidence that the dashboard can help to improve design activities, provide information to plan lessons following homework, and identify students that may have had difficulties with the activity or content.

As part of future work, teachers will be interviewed to investigate how teachers use and interpret the visualizations included in the prototype dashboard, and to learn more about their questions regarding the interaction of students with sims and the ways in which they want to use that information in their teaching practice. After further refinements, we will evaluate the dashboard’s effectiveness by inviting some teachers to test the dashboard with data from their classrooms. After this testing, we will interview the teachers about how they interpret the dashboard visualizations, additional information they would like to have and how they would plan to use that data, as well as any current information they feel is not useful.

This work advances our understanding of how students engage with online simulations, how to visualize student engagement in informative ways, and how instructional designers and teachers can be informed by and use a teacher dashboard for interactive simulations.