Some Recent Developments in Adaptive Learning Research
Concurrent Session 5
Brief Abstract
As adaptive learning begins to mature in American higher education, the need for evolved research methods becomes increasingly evident. In this session the panelists present some new findings based on cooperative work between the University of Central Florida and Realizeit--specifically examining adaptive analytics and cognitive structures.
Presenters
Extended Abstract
Adaptive learning (AL) offers promise for improving the educational environment in higher education. As with any innovation, however, original expectations have had to be moderated to accommodate counterintuitive outcomes and unanticipated side effects, both positive and negative. Because of this complexity, the University of Central Florida (UCF) and its adaptive learning platform provider (Realizeit) have formed a research partnership that suspends the typical vender-customer relationship. The two organizations have agreed to concentrate on researching adaptive learning as a phenomenon rather research an individual platform. This arrangement has been productive because of the analyses and analytics capabilities of each – some unique to UCF’s capabilities and others more easily conducted by the Realizeit research unit. This presentation will share some of these issues along with the research team’s most recent findings.
Adaptive Learning as an Educational Construct and Some Issues
Why do adaptive learning? The answer may be because it:
- Personalizes the educational experience
- Customizes the content and
- Assesses students’ performance individually
Adaptive learning can provide a framework for understanding the cognitive, affective and behavioral components of students in the learning space. More specifically, adaptive teaching tells us what students know, how they behave, how they feel, and what competencies and how much time they are going to need to succeed. Our findings suggest that those students who are at risk in a traditional system have a much greater opportunity for success. A third issue addressed by adaptive learning relates to educational inequity in the United States. Students from underserved communities operate in an environment of scarcity - they simply have more life needs than resources. Time, money, transportation, health care, family responsibilities, crime, poverty and deficient early education all contribute to scarcity. These things tend to reduce the metaphoric cognitive bandwidth of these students so that functioning in traditional education is simply overwhelming. There is just not enough time or energy to complete courses where a course depends on the one that proceeded it, especially when a student must wait for loans to buy his or her books. The tendency to miss class in this cohort is higher than average and that in of itself creates a cascading and increasing probability of failure. These kinds of pressures are even devastating to the highest ability students. Because adaptive learning features learn-at-your-own-pace it offers promise for responding to the needs of this learning cohort. Of course, AL places a good deal of pressure on the traditional academy in terms of such issues as the need for semesters, prerequisite course scheduling, financial aid, the need for academic probation, faculty roles and loads and the need for new assessment models. Much of this has yet to be resolved.
The Present Study and Current Results
In Fall 2014, UCF began an ongoing pilot study examining the use of adaptive learning in instruction. Currently, we have served almost 2,000 students in 39 unique sections utilizing the Realizeit platform. Both undergraduate and graduate courses have been taught in fully online (W), blended (M), and face-to-face (P) modalities.
Current results
The evaluation protocol at UCF has evolved as our study has progressed. This presentation will focus on the following research aspects.
1. Student Reactions to Adaptive Learning
Student reactions have been captured utilizing a standardized protocol to allow for comparison across various disciplines and courses. Faculty have the option of adding additional questions of interest to their specific field, and many do so. In general, student reactions to the adaptive learning instructional components have been positive.
2. Adaptive Analytics
This research also explored adaptive learning data using visualization techniques to gain an understanding of what insights could be generated. The results hint at the huge potential for the field of adaptive analytics.
Animations were developed that replayed how student metrics collected by the adaptive platform changed over time. These have the potential to generate many new insights including the identification of prototype patterns for students at risk of non-success. In exploring one semester of a Psychology course, several interesting prototypes emerged. Some students, labeled hares, complete their learning quickly at the start of the course, tortoises and rabbits take a slow and steady approach whereas kangaroos leave all their work to near the end of the course and complete it all in large jumps (short line).
In another UCF course, adaptive learning was used to allow students to begin with co-remediation (intermediate algebra) before progressing to the required college algebra course in the same semester. Animations developed with system analytics allowed these students’ success trajectories to be visualized and the clear benefit of this remediation becomes evident. Many of the students who were given this opportunity quickly completed the remediation material before starting the main algebra course. While completing 2 courses in one semester, these students went on to catch and in some cases bypass many of the students who began directly in College Algebra opening many interesting possibilities for the future of how algebra should be approached in an adaptive learning context.
3. Cognitive Structure of Adaptive Learning across Disciplines
The research attempted to both discover the cognitive structure captured by the adaptive learning metrics and compare these across several disciplines – intermediate algebra, college algebra, pathophysiology, and psychology. There were 13 separate indices included.
- Knowledge State - A measure of student ability
- Knowledge Covered (KC) - A measure of student progress.
- Calculated - An institute defined combination of several metrics used to assign a grade to students.
- Average Score - The mean result across all learning, revision, practice and assessment activities.
- Determine Knowledge - The percentage of objectives on which the student completed the pre-test for prior knowledge.
- Knowledge State Growth - The extent by which a student’s Knowledge State changed from the start of the course.
- Knowledge Covered Growth - The extent by which a student’s KC changed from the start of the course.
- Interactions - The engagement level of the instructor(s) with the student.
- Messages Sent - The number of the interactions sent by the instructor that were simple messages.
- Total Activities - The total number of non-assessment activities started by the student.
- Total Time - The total time spent on non-assessment activities started by the student.
- Number Revise - The total number of node level activities that are classified as revision.
- Number Practice - The total number of objective level practice activities.
We chose principal components for this analysis because the Realizeit measures were not psychometrically derived but rather comprise markers of student achievement and behavior and, are to some degree, relatively independent of each other. New graphical techniques for gaining an understanding of factors and correlation matrices in adaptive learning work were also developed.
A visualization illustrated the correlation between the 13 indices across all courses. Indices are reordered so that highly correlated indices are close and those with no correlations are far apart. Hints and the underlying structure begin to emerge showing possibly 4 or 5 groups of indices. Indeed, when we perform the principal component analysis for each of the courses, factors reflecting this grouping emerge. They represent student knowledge, growth, effort, and feedback. This structure was found to be invariant across the courses.
4. Staying the course
A final point that has become evident from the pioneering work of UCF faculty on adaptive learning has been that you must persist. Adaptive learning is not an instant solution. Its strength lies in the feedback metrics that it provides to students, instructors, and instructional designers to allow them to make changes and drive improvement over time. This is evident when examining the change in the progress of College Algebra students over 4 semesters. This shows the improvement in the interquartile range (IQR) of student progress (knowledge covered) over time for the 4 semesters.
The effect of the feedback and improvement cycle is evident. In the first semester, Spring 2015, we can observe a dispersion out of students over time. The IQR stretches from about 40% to 80%. However, in Fall 2015 and Spring 2016 we see a narrowing of the IQR before a final narrowing and increase in the top and bottom of the IQR in Fall 2016. The lower bound of the IQR has moved from 40% to about 75% and the upper bound has moved from 80% to over 95%. The instructor used what she learned in each semester to first bring students tighter together and then collectively increase their outcomes. The improvement in course design and its correlation to student outcomes is evident across time.
Audience participation
We propose to begin a discussion at this session, allowing for audience participation in Q&A form, then continue the discussion in the form of a Google Doc, providing ongoing discussion and interaction with those using adaptive learning. We are particularly interested in encouraging research into AL and collaborating with others using this technology.