Examining Students' Cognitive, Affective, and Behavioral Components in Adaptive Learning

Concurrent Session 6

Brief Abstract

This session presents results of a pilot study investigating the implementation of adaptive learning with the Realizeit system for a range of courses at the University of Central Florida. The results address students’ cognitive, affective and behavioral components in this modality and frame the faculty perspective as it evolves.

Presenters

Dr. Colm Howlin is the Principal Researcher at Realizeit and leads the research and analytics team. He has been with the company since it was founded 8 years ago. He is responsible for the development of the Adaptive Learning Engine within Realizeit and the Learning and Academic Analytics derived from learner data. Colm has a background in Applied Mathematics earning his B.Sc. and Ph.D. in Applied Mathematics from the University of Limerick and was a Research Fellow at Loughborough University in the UK. Colm has over 10 years’ experience working on research, educational data, analytics and statistical analysis, including spending time as a Consultant Statistician before joining Realizeit.

Extended Abstract

Introduction

For the past several years, adaptive learning advocates have promoted its potential for transforming higher education. The concept, although considered by many a recent innovation, is anything but new. Even the most casual literature search will identify authors, professional organizations, industries, foundations and government agencies advocating the importance of adaptiveness for learning.

The adaptive learning approach emerges as a critical component of many disciplines ranging from information search to forecasting and cultural evolution. However, no matter how diverse the disciplines are, there is apparent consensus about the core elements of adaptive learning: incremental progress, continual feedback, regular assessment, benchmarking, indexing growth and the availability of many paths to a final destination—the interaction of which alters the educational environment from a fixed setting to a flexible (adaptive) context.

This study and its results represent the cooperative effort between the Research Initiative for Teaching Effectiveness at the University of Central Florida and Senior Researcher staff at CCKF- the developer of the Realizeit platform. The work sought to clarify student assessment of the adaptive learning environment, the degree to which students achieved success, and their behavior in this instructional modality. The research took place in four courses; college algebra, pathophysiology, general psychology and educational statistics involving approximately 800 students and four instructors.     

Adaptive Learning in the Realizeit System

Realizeit incorporates probabilistic reasoning using Bayesian estimation procedures within an instructor-created learning network.  The Bayesian procedure incorporates students’ initial baseline results to estimate their position in a curriculum framework. As learners progress through their adaptive courses, additional outcomes enable Realizeit to suggest alternative learning trajectories. This results in continuous updates of students’ ability estimates, the knowledge they have acquired, those objectives that still require mastery and recommendations for optimal paths through the course material.

Methods

The Data Collection Protocol and Analysis

Affective student reaction to their adaptive learning experience was obtained through a questionnaire that was developed and validated by students, the instructors, and investigators from RITE and CCKF researchers. The instrument was developed from a review of research on adaptive learning in higher education. From this review the investigators drafted a preliminary data collection protocol that captured student demographic information and their assessment of the adaptive learning experience that was analyzed via general factor analytic methods. The survey was pilot tested with students, evaluated by instructors, revised and subjected to a second   round of assessment for its psychometric properties.   

Achievement in each of the courses (via curriculum modules) was measured based on an assessment protocol developed by the instructor within Realizeit, producing a final score on each module for each student. These scores permitted comparison of the achievement levels of successful and unsuccessful students as the course progressed, documenting learning trends of those two groups.

In addition to outcome measures created by the instructor, the Realizeit system generates a large number of student performance analytics, internal to the system. Two of these indices were subjected to further analysis. The first outcome measure used in the study was an index created by Realizeit termed knowledge state. This denotes the average ability that students achieve in a module for which there is direct evidence (some assessment exercise) across all the course nodes.  The second was percentage of nodes covered by each student for the course modules-thereby representing a surrogate for the degree to which students engaged in this module (and throughout the course. Student profiles on these indices were derived by the CCKF research unit for the top and bottom twenty students as well as the overall class profile.     

Results

Attitudes toward adaptive learning in the Realizeit platform

On average students were positive about their experience with adaptive learning in the Realizeit platform. Survey questions were selected to assess the usability of the Realizeit system – whether students found the system and its various functions and features easy to use; how well students felt the system captured their knowledge through the difficulty of items and pathways presented to them; and the student perceptions of Realizeit’s ability to help them learn and remain engaged. In addition to demographics, students were asked their overall satisfaction with the system and its components—whether it became personalized to them, if they followed system recommendations, their perception of time spent in the system, and finally, whether they would take another class using Realizeit if given a choice.

In terms of system usability, the majority of students felt the system was easy to use, as were the learning path and guidance panel features, with clear instructions provided. While over half of the students indicated they felt the guidance panel was helpful and easy to use, 23% indicated they did not use this feature or did not know what it was. Seventy-seven percent of the class indicated that they felt Realizeit provided them with the feedback necessary to stay on track.

Students (74%) believed the system accurately assessed their ability levels in psychology and that the grades accurately reflected their ability. Three-fourths of the class felt that the system increased their engagement with the course content over traditional instructional methods.

In rating the difficulty levels, the majority of students felt the difficulty levels of the learning path sequence (45%), material (46%), and questions asked (47%) were neither easy nor hard. Few found each to be somewhat or too difficult (13%; 16%; 28%, respectively).

Factor Results

By evaluating the factors attributable to the Realizeit platform, students responded to adaptive learning by how well it facilitates their learning, engages them, assesses their progress and clearly specifies the rules of engagement. The first factor identified was named adaptive effectiveness indicated that students in this environment evaluate their course with a dimension that reflects their judgment about whether adaptive learning represents a viable course modality. The second factor represented a dimension named course noise that relates to the degree to which students encounter interference in the learning process emanating from the course model and underlying technology. In a sense this is the “did adaptive learning get in the way” factor. The final dimension indexes the core value of adaptive learning because students judged Realizeit by how effective the system’s guidance algorithms helped them learn the material and acquire the concepts of general psychology.  The factor correlations range from .39 to .15 indicating a general independence with the strongest relationship found between effectiveness and guidance.

Student Achievement Knowledge State and Nodes Complete Profiles

Student profiles on module achievements, knowledge state and nodes complete showed remarkably similar characteristics. The highest achieving and generally successful students demonstrated consistency in tier performance from beginning to end of the “courses.” Unsuccessful students, however, with the slope of their achievement for module one to module two gave early indication that they would benefit from an intervention. The presenters will explore this phenomenon by discussing the profiles with respect to shape level and distance. In addition, they will propose the concept of real-time adaptive analytics.      

Conclusion

Adaptive learning resonates with educational theories contending that students must be allocated the time needed to master facts, concepts or constructs. Additionally, it responds to the well understood principle that prior learning is necessary for the attainment of successively higher levels of achievement. Incorporating these two principles into learning design, although seemingly straightforward, represents a seismic shift in the traditional educational arrangement. Currently, courses are designed by incorporating certain constraints, even in the online environment. Semesters are sixteen weeks long. The final examination is administered during a specified time or time period. Assignments have due dates. The course ends. Grades are due by a specified date. These constraints are the means by which we manage the educational enterprise for students and instructors at the present time in higher education. However, most of these requirements can be loosened or eliminated in the adaptive learning environment. Theoretically, students can move at their own pace and are assessed for competence when they or the system deems they are ready.  They can finish a course early or they can continue working after the semester ends. The implications of this thinking are substantial.  Currently, if a student finishes a course early they might have to wait to take the exam at the end of the semester. If we are to embrace truly adaptive learning and gauge a true index of its effectiveness much of the current educational structure will require reconsideration. The presenters will argue that should we stay the course with adaptive learning the changes in our educational system will be profound but these changes, although potentially beneficial at many levels, may cause considerable angst and discomfort. In spite of that, adaptive learning can find its proper place in our transformed educational system. Presenters will offer the attendees the opportunity to interact with UCF and RealizeIT researchers via a shared Google Document during the session, documenting session Q&As and also allowing participants to exchange ideas and questions after the session is over. The presenters would like participants to be able to utilize lessons learned from the ongoing evaluation of adaptive learning at UCF to design and develop their evaluations of technology-enhanced learning at their respective institutions, possibly forming partnerships across institutions and with vendors.