Adaptive Learning: Three Years of Collaborative Research

Concurrent Session 1

Brief Abstract

This session describes results from a maturing three-year research initiative among two organizationally different universities and their common adaptive learning platform provider. Current findings indicated that the underlying pattern of learning in adaptive courses remains comparable across disciplines and institutions. These findings have implications for predictive analytics and instructional design.    

Presenters

Patsy Moskal is the Director of the Digital Learning Impact Evaluation in the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where she evaluates the impact of technology-enhanced learning and serves as the liaison for faculty scholarship of teaching and learning. In 2011 Dr. Moskal was named an OLC Fellow in recognition of her groundbreaking work in the assessment of the impact and efficacy of online and blended learning. She has written and co-authored numerous works on blended and online learning and is a frequent presenter on these topics. Patsy's co-authored book--Conducting Research in Online and Blended Learning: New Pedagogical Frontiers--with Dziuban, Picciano, and Graham, was published in August 2015. She currently serves on the OLC Board of Directors.
Charles Dziuban is Director of the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF) where has been a faculty member since 1970 teaching research design and statistics and is the founding director of the university’s Faculty Center for Teaching and Learning. He received his Ph.D. from the University of Wisconsin. Since 1996, he has directed the impact evaluation of UCF’s distributed learning initiative examining student and faculty outcomes as well as gauging the impact of online, blended and lecture capture courses on the university. Chuck has published in numerous journals including Multivariate Behavioral Research, The Psychological Bulletin, Educational and Psychological Measurement, the American Education Research Journal, the Phi Delta Kappan, the Internet in Higher Education, the Journal of Asynchronous Learning Networks, and the Sloan-C View. His methods for determining psychometric adequacy have been featured in both the SPSS and the SAS packages. He has received funding from several government and industrial agencies including the Ford Foundation, Centers for Disease Control, National Science Foundation and the Alfred P. Sloan Foundation. In 2000, Chuck was named UCF’s first ever Pegasus Professor for extraordinary research, teaching, and service and in 2005 received the honor of Professor Emeritus. In 2005, he received the Sloan Consortium award for Most Outstanding Achievement in Online Learning by an Individual. In 2007 he was appointed to the National Information and Communication Technology (ICT) Literacy Policy Council. In 2010, Chuck was named an inaugural Sloan-C Fellow. In 2012 the University of Central Florida initiated the Chuck D. Dziuban Award for Excellence in Online Teaching for UCF faculty members in honor of Chuck’s impact on the field of online teaching and learning. In 2017 Chuck received UCF’s inaugural Collective Excellence award for his work strengthening the university’s impact with the Tangelo Park Program and assumed the position of University Representative to the Rosen Foundation Tangelo Park and Parramore programs.
Dr. Colm Howlin is the Principal Researcher at Realizeit and leads the research and analytics team. He has been with the company since it was founded 8 years ago. He is responsible for the development of the Adaptive Learning Engine within Realizeit and the Learning and Academic Analytics derived from learner data. Colm has a background in Applied Mathematics earning his B.Sc. and Ph.D. in Applied Mathematics from the University of Limerick and was a Research Fellow at Loughborough University in the UK. Colm has over 10 years’ experience working on research, educational data, analytics and statistical analysis, including spending time as a Consultant Statistician before joining Realizeit.
Dr. Connie Johnson is Colorado Technical University's (CTU) chief academic officer and provost, working with both online and ground degree programs. She has oversight of academic affairs, including faculty, curriculum, classroom experience, and accreditation. During her time at CTU, Connie has initiated adaptive learning technology implementation, effective leadership of academics, women's leadership, leading academics through change, and effective technology implementation in the online classroom including the promotion of academics, faculty and student engagement through social media. Connie has served in higher education for over 20 years with extensive experience in online and ground teaching, administration, and leadership. Additionally, Connie has extensive experience in regional accreditation, curriculum implementation, and faculty training and development. She is a trained peer evaluator for the Higher Learning Commission (HLC), has completed and served as a facilitator in the ACE Chief Academic Officer Institute, and is a member of the CTU Board of Trustees. Her educational background includes a Doctorate of Education, organizational leadership emphasis (2010), and a Master of Business Administration in management (1991) from Nova Southeastern University; and a Bachelor of Science with honors in criminal justice from Florida State University.

Extended Abstract

Background

Adaptive learning continues to grow in popularity as a promising instructional innovation used in the ongoing higher education battle to improve student success and retention. Adaptive learning acts like a GPS for students. As they progress through the course content, it allows for personalized instruction while altering their pathways -- continually assessing their knowledge to help them most efficiently and effectively progress through the course (Moskal, Carter, & Johnson, 2017). This ability to allow students to either advance or remediate is one of the reasons adaptive learning is being investigated for its potential to use mastery to improve instruction (Bienkowski, Feng, & Means, 2012; Dziuban, 2017). The hype is likely to continue in the near future with a number of national reports pointing to adaptive learning as one of the important developments or emerging technologies in education (Becker et al., 2017; Legon & Garrett, 2018; Office of Educational Technology, 2017).

Several national initiatives have provided funding for investigating adaptive courseware’s potential in higher education (Bill and Melinda Gates Foundation, 2014; Association of Public & Land-Grant Universities, 2016; Online Learning Consortium, 2016), and while some preliminary results have been positive, there is much more work to be done. Much of the research indicates the varied campus climates and adaptive courseware implementations can make comparisons and generalizability of findings difficult and the research to date has not been as prolific or as promising as hoped.

Our Cooperative Study

This presentation will describe a cooperative adaptive learning evaluation model between the research unit of the platform provider (Realizeit), the Research Initiative for Teaching Effectiveness at the University of Central Florida and Colorado Technical University. This ongoing collaboration has taken place since 2015 as researchers suspended the focus on the typical vendor/university relationship with the understanding that the three organizations have unique contributions that together strengthen the research possibilities. Our focus has been to investigate the phenomenon of adaptive learning, as opposed to examining individual platforms.

In this session, researchers will share some of the outcomes from the 3-year collaborative relationship including student reactions and student behaviors within these systems. Findings from joint survey research at both UCF and CTU indicate that students across both institutions react similarly to adaptive learning, revealing that they like the feedback and personalization this modality provides. They also feel it makes them more engaged in the learning process and would prefer more adaptive learning in their educational experiences (Dziuban, Howlin, Johnson, and Moskal, 2017; Dziuban, Moskal, Cassisi & Fawcett, 2016; Dziuban, Moskal, Johnson & Evans, 2017).

In addition, an examination of student behavior within adaptive learning identified latent dimensions underlying these courses across multiple disciplines and the two structurally different universities. The objective was to determine if differing disciplines and university contexts impacted adaptive learning patterns, again focusing on examining adaptive learning as a process rather than evaluating an adaptive learning platform.

A subset of key student performance indices generated as students progress through the adaptive content depicted students’ cognitive outcomes and behaviors for three courses from UCF, three courses from CTU, as well as combined samples for each institution. The indices were intercorrelated and subjected to the principal component procedure (Mulaik, 2009) to explain the variance and relationships (correlations) among the indices, and to reduce the dataset to a smaller dimensionality. Operationally, the study examined the question: is the cognitive organization of adaptive learning constant or do the patterns change by institution or course context?

Pattern matrices and similarity confidents indicated that the underlying dimensions of adaptive learning remain stable within disciplines, across disciplines, and across the two universities. With some minor variation across UCF and CTU, the component similarity and invariance was relatively stable and four components were identified:

Knowledge Attainment

Engagement Activities

Growth

Communication

Learning science suggests that there is a clear relationship between these traits -- engagement and communication are prerequisite for growth and achievement although in this study they are statistically independent of each other. This finding supports the notion that this underlying pattern is fundamental to effective teaching and learning using adaptive platforms.

Adaptive learning (AL) creates a fluid educational environment responding to the needs of many student cohorts. Features like initial student knowledge baselines, continuous assessment and feedback, redesigned learning paths, mastery certification and instructional format preference make a more flexible and responsive educational landscape. In the introduction we mentioned AL’s modification of learning time. Adam (2004), in her work with temporal culture, provides insights into what can happen with constant outcomes and variable learning time.

Learning transforms a student’s:

  1. Time frame:    The time boundaries of a course or program of study
  2. Timing:           When will learning will take place
  3. Tempo:           The pace of learning
  4. Duration:        How long will learning take place
  5. Sequence:       In what order will learning take pace

Faculty importance increases in the adaptive environment because they can identify learning objectives for students through course design, and analytics data provided by the system. Therefore, instructors can suggest effective interactions and interventions with students in areas that require support or additional instruction. Faculty members have a real-time view of student progress that is not available in other methods of teaching. For instance, adaptive systems can reliably identify skills or concepts with which the class on average is excelling or having difficulty. In addition, instructors can track individual student progression through course content. This provides faculty the opportunity to adapt their lecture, activities, or homework assignments to personalize instruction.

Conclusion

While we will discuss our research findings, this session will also focus on the importance of establishing collaborative research between universities and vendors. The growth of available learning analytics make this relationships particularly fruitful and also critical to moving educational research forward.

This study is a collaboration among two universities and their common adaptive leaning platform provider. Each organization brings different strengths to the partnership. CTU achieves scale with adaptive implementation. UCF integrates research and data into the decision-making and policy process. Realizeit brings advanced analysis skills and makes transparent analytic data available all its partners. Because of this small network each organization improved its adaptive learning process--the universities with pedagogy and Realizeit with its platform. This happens over time in a nonlinear process that encounters a good deal of productive failure. The technology does not drive the work, but rather the research helps improve the technology. The partners commit to pushing information and flexibility out as far as possible and believe that progress happens in small steps. Simple is more effective. Without the partnership and the sharing there would be no study. None of us could do it alone. Therefore, our major conclusion is that we need more extensive collaborative work. Each university can similarly contextualize adaptive learning and every platform provider can support an active research agenda to form similar, and increasingly productive, collaborative partnerships.

References

Adam, B. (2004). Time. Cambridge, IK: Polity.

Association of Public & Land-Grant Universities. (2016). Personalizing Learning with Adaptive Courseware. Retrieved from http://www.aplu.org/projects-and-initiatives/personalized- learning-consortium/plc-projects/plc-adaptive-courseware/

Becker, S. A., Cummins, M., Davis, A., Freeman, A., Hall, C. G., & Ananthanarayanan, V. (2017). NMC horizon report: 2017 higher education edition (pp. 1-60). The New Media Consortium.

Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. US Department of Education, Office of Educational Technology1, 1-57.

Bill & Melinda Gates Foundation. (2014, November). Early progress: Interim research on personalized learning. Retrieved from http://collegeready.gatesfoundation.org/wp-content/uploads/2015/06/Early-Progress-on-Personalized-Learning-Full-Report.pdf

Dziuban, C. (2017). The Technology of Adaptive Learning. Education Technology Insights.

Dziuban, C. D., Moskal, P. D., Cassisi, J., & Fawcett, A. (2016). Adaptive Learning in Psychology: Wayfinding in the Digital Age. Online Learning, 20(3), 74-96.

Dziuban, C., Howlin, C., Johnson, C., & Moskal, P. (2017, December 18). An Adaptive Learning Partnership. EDUCAUSE Review.

Dziuban, C., Moskal, P., Johnson, C., & Evans, D. (2017). Adaptive learning: A tale of two contexts. Current Issues in Emerging eLearning4(1), 3.

Legon, R. & Garrett, R. (2018). The Changing Landscape of Online Education (Chloe) 2: A Deeper Dive. CHLOE2.

Moskal, P., Carter, D., & Johnson, D. (2017). 7 Things You Should Know About Adaptive Learning. ELI.

Mulaik, S.A. (2009). The foundations of factor analysis, second edition. London, United Kingdom: Chapman and Hall.

Office of Educational Technology. (2017, January). Reimagining the Role of Technology in Education: 2017 National Education Technology Plan Update. US Department of Education. Retrieved from: https://tech.ed.gov/files/2017/01/NETP17.pdf

Online Learning Consortium. (2016). Digital Learning Innovation Award. Retrieved from https://onlinelearningconsortium.org/about/olc-awards/dlia/