Some Recent Developments in Adaptive Learning Research

Concurrent Session 5
Streamed Session

Watch This Session

Session Materials

Brief Abstract

As adaptive learning begins to mature in American higher education, the need for evolved research methods becomes increasingly evident. In this session the panelists present some new findings based on cooperative work between the University of Central Florida and Realizeit--specifically examining adaptive analytics and cognitive structures.     


Charles Dziuban is Director of the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where has been a faculty member since 1970 teaching research design and statistics and is the founding director of the university’s Faculty Center for Teaching and Learning. He received his Ph.D. from the University of Wisconsin. Since 1996, he has directed the impact evaluation of UCF’s distributed learning initiative examining student and faculty outcomes as well as gauging the impact of online, blended and lecture capture courses on the university. Chuck has published in numerous journals including Multivariate Behavioral Research, The Psychological Bulletin, Educational and Psychological Measurement, the American Education Research Journal, the Phi Delta Kappan, the Internet in Higher Education, the Journal of Asynchronous Learning Networks, and the Sloan-C View. His methods for determining psychometric adequacy have been featured in both the SPSS and the SAS packages. He has received funding from several government and industrial agencies including the Ford Foundation, Centers for Disease Control, National Science Foundation and the Alfred P. Sloan Foundation. In 2000, Chuck was named UCF’s first ever Pegasus Professor for extraordinary research, teaching, and service and in 2005 received the honor of Professor Emeritus. In 2005, he received the Sloan Consortium award for Most Outstanding Achievement in Online Learning by an Individual. In 2007 he was appointed to the National Information and Communication Technology (ICT) Literacy Policy Council. In 2010, Chuck was named an inaugural Sloan-C Fellow. In 2012 the University of Central Florida initiated the Chuck D. Dziuban Award for Excellence in Online Teaching for UCF faculty members in honor of Chuck’s impact on the field of online teaching and learning. In 2017 Chuck received UCF’s inaugural Collective Excellence award for his work strengthening the university’s impact with the Tangelo Park Program and assumed the position of University Representative to the Rosen Foundation Tangelo Park and Parramore programs.
Patsy Moskal is the Director of the Digital Learning Impact Evaluation in the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where she evaluates the impact of technology-enhanced learning and serves as the liaison for faculty scholarship of teaching and learning. In 2011 Dr. Moskal was named an OLC Fellow in recognition of her groundbreaking work in the assessment of the impact and efficacy of online and blended learning. She has written and co-authored numerous works on blended and online learning and is a frequent presenter on these topics. Patsy's co-authored book--Conducting Research in Online and Blended Learning: New Pedagogical Frontiers--with Dziuban, Picciano, and Graham, was published in August 2015. She currently serves on the OLC Board of Directors.
Dr. Colm Howlin is the Principal Researcher at Realizeit and leads the research and analytics team. He has been with the company since it was founded 8 years ago. He is responsible for the development of the Adaptive Learning Engine within Realizeit and the Learning and Academic Analytics derived from learner data. Colm has a background in Applied Mathematics earning his B.Sc. and Ph.D. in Applied Mathematics from the University of Limerick and was a Research Fellow at Loughborough University in the UK. Colm has over 10 years’ experience working on research, educational data, analytics and statistical analysis, including spending time as a Consultant Statistician before joining Realizeit.

Extended Abstract

Adaptive learning (AL) offers promise for improving the educational environment in higher education. As with any innovation, however, original expectations have had to be moderated to accommodate counterintuitive outcomes and unanticipated side effects, both positive and negative. Because of this complexity, the University of Central Florida (UCF) and its adaptive learning platform provider (Realizeit) have formed a research partnership that suspends the typical vender-customer relationship. The two organizations have agreed to concentrate on researching adaptive learning as a phenomenon rather research an individual platform. This arrangement has been productive because of the analyses and analytics capabilities of each – some unique to UCF’s capabilities and others more easily conducted by the Realizeit research unit.  This presentation will share some of these issues along with the research team’s most recent findings. 

Adaptive Learning as an Educational Construct and Some Issues

Why do adaptive learning? The answer may be because it:

  1. Personalizes the educational experience
  2. Customizes the content and
  3. Assesses students’ performance individually

Adaptive learning can provide a framework for understanding the cognitive, affective and behavioral components of students in the learning space. More specifically, adaptive teaching tells us what students know, how they behave, how they feel, and what competencies and how much time they are going to need to succeed.  Our findings suggest that those students who are at risk in a traditional system have a much greater opportunity for success.  A third issue addressed by adaptive learning relates to educational inequity in the United States. Students from underserved communities operate in an environment of scarcity - they simply have more life needs than resources. Time, money, transportation, health care, family responsibilities, crime, poverty and deficient early education all contribute to scarcity.  These things tend to reduce the metaphoric cognitive bandwidth of these students so that functioning in traditional education is simply overwhelming. There is just not enough time or energy to complete courses where a course depends on the one that proceeded it, especially when a student must wait for loans to buy his or her books. The tendency to miss class in this cohort is higher than average and that in of itself creates a cascading and increasing probability of failure.  These kinds of pressures are even devastating to the highest ability students.  Because adaptive learning features learn-at-your-own-pace it offers promise for responding to the needs of this learning cohort. Of course, AL places a good deal of pressure on the traditional academy in terms of such issues as the need for semesters, prerequisite course scheduling, financial aid, the need for academic probation, faculty roles and loads and the need for new assessment models. Much of this has yet to be resolved.

The Present Study and Current Results

In Fall 2014, UCF began an ongoing pilot study examining the use of adaptive learning in instruction. Currently, we have served almost 2,000 students in 39 unique sections utilizing the Realizeit platform. Both undergraduate and graduate courses have been taught in fully online (W), blended (M), and face-to-face (P) modalities.

Current results

The evaluation protocol at UCF has evolved as our study has progressed. This presentation will focus on the following research aspects.

1. Student Reactions to Adaptive Learning

Student reactions have been captured utilizing a standardized protocol to allow for comparison across various disciplines and courses. Faculty have the option of adding additional questions of interest to their specific field, and many do so. In general, student reactions to the adaptive learning instructional components have been positive.

2. Adaptive Analytics

This research also explored adaptive learning data using visualization techniques to gain an understanding of what insights could be generated. The results hint at the huge potential for the field of adaptive analytics.

Animations were developed that replayed how student metrics collected by the adaptive platform changed over time. These have the potential to generate many new insights including the identification of prototype patterns for students at risk of non-success. In exploring one semester of a Psychology course, several interesting prototypes emerged. Some students, labeled hares, complete their learning quickly at the start of the course, tortoises and rabbits take a slow and steady approach whereas kangaroos leave all their work to near the end of the course and complete it all in large jumps (short line).

In another UCF course, adaptive learning was used to allow students to begin with co-remediation (intermediate algebra) before progressing to the required college algebra course in the same semester. Animations developed with system analytics allowed these students’ success trajectories to be visualized and the clear benefit of this remediation becomes evident. Many of the students who were given this opportunity quickly completed the remediation material before starting the main algebra course. While completing 2 courses in one semester, these students went on to catch and in some cases bypass many of the students who began directly in College Algebra opening many interesting possibilities for the future of how algebra should be approached in an adaptive learning context.

3. Cognitive Structure of Adaptive Learning across Disciplines

The research attempted to both discover the cognitive structure captured by the adaptive learning metrics and compare these across several disciplines – intermediate algebra, college algebra, pathophysiology, and psychology. There were 13 separate indices included.

  • Knowledge State - A measure of student ability
  • Knowledge Covered (KC) - A measure of student progress.
  • Calculated - An institute defined combination of several metrics used to assign a grade to students.
  • Average Score - The mean result across all learning, revision, practice and assessment activities.
  • Determine Knowledge - The percentage of objectives on which the student completed the pre-test for prior knowledge.
  • Knowledge State Growth - The extent by which a student’s Knowledge State changed from the start of the course.
  • Knowledge Covered Growth - The extent by which a student’s KC changed from the start of the course.
  • Interactions - The engagement level of the instructor(s) with the student.
  • Messages Sent - The number of the interactions sent by the instructor that were simple messages.
  • Total Activities - The total number of non-assessment activities started by the student.
  • Total Time - The total time spent on non-assessment activities started by the student.
  • Number Revise - The total number of node level activities that are classified as revision.
  • Number Practice - The total number of objective level practice activities.

We chose principal components for this analysis because the Realizeit measures were not psychometrically derived but rather comprise markers of student achievement and behavior and, are to some degree, relatively independent of each other. New graphical techniques for gaining an understanding of factors and correlation matrices in adaptive learning work were also developed.

A visualization illustrated the correlation between the 13 indices across all courses. Indices are reordered so that highly correlated indices are close and those with no correlations are far apart. Hints and the underlying structure begin to emerge showing possibly 4 or 5 groups of indices. Indeed, when we perform the principal component analysis for each of the courses, factors reflecting this grouping emerge. They represent student knowledge, growth, effort, and feedback. This structure was found to be invariant across the courses.

4. Staying the course

A final point that has become evident from the pioneering work of UCF faculty on adaptive learning has been that you must persist. Adaptive learning is not an instant solution. Its strength lies in the feedback metrics that it provides to students, instructors, and instructional designers to allow them to make changes and drive improvement over time. This is evident when examining the change in the progress of College Algebra students over 4 semesters.  This shows the improvement in the interquartile range (IQR) of student progress (knowledge covered) over time for the 4 semesters.

The effect of the feedback and improvement cycle is evident. In the first semester, Spring 2015, we can observe a dispersion out of students over time. The IQR stretches from about 40% to 80%. However, in Fall 2015 and Spring 2016 we see a narrowing of the IQR before a final narrowing and increase in the top and bottom of the IQR in Fall 2016. The lower bound of the IQR has moved from 40% to about 75% and the upper bound has moved from 80% to over 95%. The instructor used what she learned in each semester to first bring students tighter together and then collectively increase their outcomes. The improvement in course design and its correlation to student outcomes is evident across time.

Audience participation

We propose to begin a discussion at this session, allowing for audience participation in Q&A form, then continue the discussion in the form of a Google Doc, providing ongoing discussion and interaction with those using adaptive learning. We are particularly interested in encouraging research into AL and collaborating with others using this technology.