An Adaptive Learning Partnership

Concurrent Session 6
Streamed Session Best in Track Blended

Watch This Session

Brief Abstract

We present results from collaborative, adaptive learning research between two universities and a vendor. Results describe student attitudes across different college settings, students’ adaptive behavior patterns and gains in student outcomes. Suspending the vendor/university relationship allows for collaborative research partnerships--what should become the new normal in higher education.


Patsy Moskal is the Director of the Digital Learning Impact Evaluation in the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where she evaluates the impact of technology-enhanced learning and serves as the liaison for faculty scholarship of teaching and learning. In 2011 Dr. Moskal was named an OLC Fellow in recognition of her groundbreaking work in the assessment of the impact and efficacy of online and blended learning. She has written and co-authored numerous works on blended and online learning and is a frequent presenter on these topics. Patsy's co-authored book--Conducting Research in Online and Blended Learning: New Pedagogical Frontiers--with Dziuban, Picciano, and Graham, was published in August 2015. She currently serves on the OLC Board of Directors.
Charles Dziuban is Director of the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where has been a faculty member since 1970 teaching research design and statistics and is the founding director of the university’s Faculty Center for Teaching and Learning. He received his Ph.D. from the University of Wisconsin. Since 1996, he has directed the impact evaluation of UCF’s distributed learning initiative examining student and faculty outcomes as well as gauging the impact of online, blended and lecture capture courses on the university. Chuck has published in numerous journals including Multivariate Behavioral Research, The Psychological Bulletin, Educational and Psychological Measurement, the American Education Research Journal, the Phi Delta Kappan, the Internet in Higher Education, the Journal of Asynchronous Learning Networks, and the Sloan-C View. His methods for determining psychometric adequacy have been featured in both the SPSS and the SAS packages. He has received funding from several government and industrial agencies including the Ford Foundation, Centers for Disease Control, National Science Foundation and the Alfred P. Sloan Foundation. In 2000, Chuck was named UCF’s first ever Pegasus Professor for extraordinary research, teaching, and service and in 2005 received the honor of Professor Emeritus. In 2005, he received the Sloan Consortium award for Most Outstanding Achievement in Online Learning by an Individual. In 2007 he was appointed to the National Information and Communication Technology (ICT) Literacy Policy Council. In 2010, Chuck was named an inaugural Sloan-C Fellow. In 2012 the University of Central Florida initiated the Chuck D. Dziuban Award for Excellence in Online Teaching for UCF faculty members in honor of Chuck’s impact on the field of online teaching and learning. In 2017 Chuck received UCF’s inaugural Collective Excellence award for his work strengthening the university’s impact with the Tangelo Park Program and assumed the position of University Representative to the Rosen Foundation Tangelo Park and Parramore programs.
Dr. Connie Johnson is Colorado Technical University's (CTU) chief academic officer and provost, working with both online and ground degree programs. She has oversight of academic affairs, including faculty, curriculum, classroom experience, and accreditation. During her time at CTU, Connie has initiated adaptive learning technology implementation, effective leadership of academics, women's leadership, leading academics through change, and effective technology implementation in the online classroom including the promotion of academics, faculty and student engagement through social media. Connie has served in higher education for over 20 years with extensive experience in online and ground teaching, administration, and leadership. Additionally, Connie has extensive experience in regional accreditation, curriculum implementation, and faculty training and development. She is a trained peer evaluator for the Higher Learning Commission (HLC), has completed and served as a facilitator in the ACE Chief Academic Officer Institute, and is a member of the CTU Board of Trustees. Her educational background includes a Doctorate of Education, organizational leadership emphasis (2010), and a Master of Business Administration in management (1991) from Nova Southeastern University; and a Bachelor of Science with honors in criminal justice from Florida State University.
Dr. Colm Howlin is the Principal Researcher at Realizeit and leads the research and analytics team. He has been with the company since it was founded 8 years ago. He is responsible for the development of the Adaptive Learning Engine within Realizeit and the Learning and Academic Analytics derived from learner data. Colm has a background in Applied Mathematics earning his B.Sc. and Ph.D. in Applied Mathematics from the University of Limerick and was a Research Fellow at Loughborough University in the UK. Colm has over 10 years’ experience working on research, educational data, analytics and statistical analysis, including spending time as a Consultant Statistician before joining Realizeit.

Extended Abstract


The higher education innovation landscape involves universities who look for creative instructional technologies and the vendors who market their platforms. Historically, all parties perceive this relationship from their own perspectives. Universities develop strategies for initiatives they consider responsive and transformative, based on their research agendas and the communities they serve. Vendors may work closely with higher education professionals who may advise them on product development. But the vendor perspective of higher education is mediated by their business propositions.

This proposal describes a research relationship formed between two universities and a vendor. We describe our research and positive results achieved only with our cooperative relationship, working together to investigate adaptive learning. Suspending the vendor/university relationship allows for collaborative research partnerships, an essential bond of what should become the new normal in higher education.

Two University Partners: UCF and CTU  

The University of Central Florida (UCF) and Colorado Technical University (CTU) were early adopters of adaptive learning (AL) and began discussing their approaches at scientific meetings, describing their instructional and implementation strategies. Both UCF and CTU adopted Realizeit’s adaptive learning platform as their enterprise solution ( Realizeit has a robust ability to support the sharing of experiences and data, providing insight into successful adaptive learning practices. Both universities were experiencing initial positive results with the technology, but at considerably different scales.

After conversations and collaboration on writing and conference presentations, it became clear that a research partnership between UCF and CTU would be beneficial. A fundamental component for successful cross-institution collaboration was an openness to each other’s work, in spite of considerably different student populations, faculty composition and structure. UCF is a large public university (66,000 students) in the pilot phase of AL adoption while CTU is a private, for-profit institution (21,200 students) that is further along with scaling adaptive learning. The initiative at UCF is research intensive, while implementation centric at CTU.

Enter the Vendor -- The Realizeit Partnership

Vendors develop their products based on a view of how their approach and technology can empower the various instructional roles within an institution. The variety of institutional contexts means that vendors cannot treat a genuinely adaptive learning platform as a simple plug-and-play device. Each implementation must be responsive to the requirements of the courses, instructors, academic standards and goals of individual institutions.

The most successful integration occurs when an open, collaborative relationship is fostered between the vendor and its university partners. While any vendor will have intellectual property that cannot be disclosed, there needs to be transparency in core areas. These include how the platform makes decisions, which data points are available, how the platform provides feedback and guidance to the learners, how it influences the direction of learning, and how it supports the student. When working with Realizeit, all data points generated are owned by the participating institution and are made available to it for measuring the impact of adaptive learning. This information has been the primary source of insights that have influenced the evolution of the platform.

Student Adaptive Behavior

Working with UCF and CTU, Realizeit researchers conducted a study examining students’ behaviors as they engaged with the adaptive learning platform to better understand how they managed their learning in an adaptive environment. In more traditional settings, students progress through the learning cycle in a nearly uniform pace and may not be able to accelerate through new lessons or repeat prior lessons for remediation. The agency of adaptive learning enables a variety of effective behaviors that can inform us on improving student success.

Examining student progression metrics, we were able to identify their behavior patterns indexed by the proportion of concepts completed in courses over time. Common prototypes emerged across different content domains and university settings (a typical CTU course lasts around five weeks compared to fifteen weeks at UCF, except for summer twelve-week semesters). However, depending on how the course was structured and how instructors engaged with their students, not all prototypes appeared in every course. 

The Tortoise and Frog were used to represent the two most common behavior styles of students in higher education. These students make systematic, steady progress throughout the course--a behavior most likely encouraged by many instructors. On the other hand, the Hare and the Kangaroo are prototypes that emerge as a result of adaptive learning self-direction and self-pacing. These students accomplish the majority of their progress in altered time frames, jumping ahead in spurts or in a flash. Since all student prototypes tended to complete, the primary difference was the manner in which students engaged in the course.

Improving Outcomes for Students

One insight that has become evident from the collaborative work of UCF and CTU is that any adopter of adaptive learning must stay on course. Adaptive learning is not an instant solution. Its strength lies in the feedback cycle that it provides to students, instructors, and instructional designers, allowing them to formulate outcomes through an iterative improvement cycle over time. To highlight what is possible, we examined how improvements in both the courses and the adaptive platform have led to an increased attainment level of students taking College Algebra at UCF. For example, the instructor has taken what she has learned after each course, to implement changes in the materials and structure, which produce improvements in student outcomes that are measurable and significant.

The impact of the changes by the instructor becomes much more evident by splitting the students into three cohorts and examining what has happened for each.

  • Top 25% - These students have moved from covering at least 86% of concepts to covering 95% or more.
  • Middle 50% - These students moved from covering between 49% and 86% in Spring 2015 to covering between 61%and 95% in Fall 2016. Both the top and the bottom boundaries are achieving more.
  • Bottom 25% - In Fall 2016 these students covered less than 49% of the course. However, by Fall 2016 some of these lower achieving students are covering up to 60% of the course.

All cohorts have improved, the best have gotten better, and even the lower achieving students have improved. Adaptive learning has shifted the curve in this course, and as we can observe it, has been universally beneficial. The key is staying in the course and committing to using data from each iteration to learn what worked, what didn’t, and where improvements are needed. This same strategy is one that has been used at a much larger scale at CTU.

Student Reaction to Adaptive Learning

We were interested in students’ reactions to adaptive learning and how they might differ across CTU and UCF. UCF developed a survey as part of their pilot evaluation and CTU adopted the same survey allowing for comparisons. While the demographics and campuses vary, it would be beneficial to know if students at both institutions were equally receptive to adaptive learning. At UCF, 300 students were surveyed in General Psychology, an adaptive course delivered in Fall 2014 and Spring 2015 with 244 respondents (81% response rate). CTU sent their survey to all students enrolled in adaptive courses (14,400) and received 1,140 completed responses (10% response rate).

The survey gauged student reactions and experiences with their adaptive classes, capturing details on their interaction with the system, including ease of use, helpfulness of feedback, and guidance and accuracy of platform assessment metrics. In addition, the survey captured overall student attitudes about the nature of using adaptive learning in instruction, including what was most and least positive about this instructional method and how it impacted their interaction with and time spent on the course. Demographic questions allowed examination of differences across student cohorts. Overall, students at both institutions were positive. However, some reactions to adaptive learning illustrated significant differences between students’ reactions at CTU and UCF. We will present highlights in this session and more details can be found in Dziuban, Moskal, Johnson & Evans (2017).

Adaptive Leaning--It’s About Time

Over the past few years, higher education has been able to refocus on adaptive learning because emerging technologies have created more effective capabilities just as they have with artificial intelligence that is emerging from its years of doldrums. Platforms have different strengths and weaknesses but each one adheres to the adaptive paradigm. Our conversation describes a research partnership between two universities with considerably different missions and Realizeit that grew organically through the convenience of a common platform. However, effective partnerships can emerge across several different providers--a development that, in our opinion, would be highly beneficial avoiding the “which is the best platform” narrative, focusing on the notion of adaptive learning as a construct.

We will focus this session on the need for a thoughtful examination of adaptive learning as an instructional process and the value of research findings possible with a university/vendor collaborative relationship. We have presented a small glimpse at our findings here and will illustrate more in the session. In addition, we hope to open a dialog with other universities, encouraging and promoting future collaborative opportunities.