Mining for Achievement Using Student Performance And End-Of Course Data: A Multicovariate Analysis of 60,000 Online Courses

Concurrent Session 3
Streamed Session Best in Track

Watch This Session

Session Materials

Brief Abstract

Our study on student achievement revealed that student grades, course progression, and end-of-course ratings are correlated and significantly higher for full-time rather than part-time faculty. After statistically controlling for grade bias, we found that students place a high value on faculty participation, fostering critical thinking, expertise, high expectations, and constructive feedback.  

Presenters

Dr. Barbara Zorn-Arnold has 20 years of teaching, research, and leadership experience in higher education. Prior to joining Ashford University in 2012, she served as a faculty lecturer and thesis advisor for the MS Degree in Plant Conservation and Biology at Northwestern University. She has received numerous grants, fellowships, and awards including a teaching fellowship from The National Science Foundation. From 2007 - 2012, Dr. Zorn-Arnold served on the Faculty Advisory Council at the University of Phoenix where she helped develop multiple online degree programs in math and science. Dr. Zorn-Arnold is active in ecological and educational research, with a specific focus on student achievement. Publications are available on Google Scholar. She earned her PhD in Biological Sciences from the University of Illinois- Chicago and her BA in Environmental Studies from Lake Forest College.
Dr. Adam L. Selhorst currently serves as the Executive Dean of the College of Liberal Arts at Ashford University. Previous academic experiences include positions as a Fellow at the National Science Foundation, Lecturer at the Ohio State University, Lecturer at Leeward Community College, Researcher at the University of Hawaii, and Student Research Scientist at Kenyon College. He holds a PhD and MS in Environmental Science from the Ohio State University, an MA in Public Policy and Management from the Ohio State University, an MBA in from Ashford University's Forbes School of Business, and a BA in Biology from Kenyon College. While past research has focused on mitigation techniques for slowing global climate change, he currently focuses on innovation in online education and strategies aimed at enhancing student success and satisfaction with in online classrooms.

Extended Abstract

Online universities are faced with the ongoing challenge of increasing student retention while maintaining a rigorous and satisfying learning experience.  Gaining a better understanding of correlations between student achievement and classroom learning experience can help universities address these challenges in the future.  End-of-course surveys (EoCS) are one way to measure student learning experience and are often considered in evaluating curriculum and instructional quality.  However, due to potential student grade bias, the reliability and validity of EOCS is called into question.

Despite these long-time concerns, there is no consensus regarding the correlation between EoCS ratings and student achievement (e.g. grades).  In addition, little research has been conducted in the eLearning environment where student and faculty population, student academic needs, delivery of course content can be different from traditional universities. To address these gaps, we use data from almost 60,000 online courses over a three year period to test whether any correlations exist between student grades, percentage of students passing a course, next course progression, and EOCS ratings of full-time and part-time faculty.  Specifically, we test whether:      (1) faculty and course related questions within the EOCS are correlated with student grades, % passing, % next course progression; and whether (2) faculty status (full-time versus part-time) is associated with student achievement and learning experience.  

We address our research tests in three major steps: (1) we use a multivariate analyses of variance (MANOVA) to determine the overall effects of faculty status on student achievement and learning experienc; (2) we use Pearson’s correlation coefficient to determine the strength of correlations between variables, especially the potential relationship between student grades and EOCS responses; and (3) we use a MANCOVA to statistically adjust for any correlations identified in step 2 in order to illuminate instructional practices that have the largest effect on student learning experience.  

In 2015 and 2017, the Pearson correlation coefficients indicated that there were significant positive associations between student grades and survey responses, and student recommendations for both faculty (r=0.57, and courses (r = 0.52).  The student grade correlation is still present in 2017 for faculty and course assessments, although the correlation becomes weaker over time.  The percentage of students passing a course and the percentage of students progressing to the next course was 9.17%  and 12.31% higher in courses taught by full-time, rather than by part-time faculty (F = 8.438, p < 0.01; F = 13.59, p < 0.0001, respectively). There was no significant influence of faculty status on student grades.   However, faculty status had a highly significant effect on the overall student learning experience after we statistically adjusted for grade bias (F = 11.390, p < 0.0001). Students consistently rated full-time faculty higher in areas of participation, fostering critical thinking, expertise, communicating high expectations, and quality feedback.  Feedback that aligns with expectations and timely feedback have the most influence on course evaluation ratings (F = 14.505, p < 0.0001 and F = 27.770, p < 0.0001, respectively).  In sum, our study of almost 60,000 online courses show that the percentage of students passing a course and progressing to the next course, as well as the student learning experience is significantly greater for full-time versus part-time faculty. 

To promote a vibrant community of educational research, we invite audience members to bring their experiences with designing and using EOCSs, as well as any questions related to analyzing large multivariate data sets with confounding variables.   One important audience outcome of this research is to view EOCSs as a stragetic pathway to support our students rather than primarily as a faculty evaluation tool.  Continued exploration can examine student feedback in the EOCS comments section to gain a deeper understanding of how students can be intellectually challenged and supported in the classroom.  Including faculty course feedback on student preparedness and student-student interactions is also an important avenue for research because faculty input can drive student support services, including library training, writing, reading comprehension, and career development.   Future collaborative and industry-wide research opportunities exist and include meta-analyses of existing publications or compilation of multi-university student achievement and learning experience data.  Our results on the factors impacting student achievement and experience can empower the eLearning community by improving the application of performance data, and strategically planning future university research and policy.