Mining for Achievement Using Student Performance And End-Of Course Data: A Multicovariate Analysis of 60,000 Online Courses
Concurrent Session 3
Our study on student achievement revealed that student grades, course progression, and end-of-course ratings are correlated and significantly higher for full-time rather than part-time faculty. After statistically controlling for grade bias, we found that students place a high value on faculty participation, fostering critical thinking, expertise, high expectations, and constructive feedback.
Online universities are faced with the ongoing challenge of increasing student retention while maintaining a rigorous and satisfying learning experience. Gaining a better understanding of correlations between student achievement and classroom learning experience can help universities address these challenges in the future. End-of-course surveys (EoCS) are one way to measure student learning experience and are often considered in evaluating curriculum and instructional quality. However, due to potential student grade bias, the reliability and validity of EOCS is called into question.
Despite these long-time concerns, there is no consensus regarding the correlation between EoCS ratings and student achievement (e.g. grades). In addition, little research has been conducted in the eLearning environment where student and faculty population, student academic needs, delivery of course content can be different from traditional universities. To address these gaps, we use data from almost 60,000 online courses over a three year period to test whether any correlations exist between student grades, percentage of students passing a course, next course progression, and EOCS ratings of full-time and part-time faculty. Specifically, we test whether: (1) faculty and course related questions within the EOCS are correlated with student grades, % passing, % next course progression; and whether (2) faculty status (full-time versus part-time) is associated with student achievement and learning experience.
We address our research tests in three major steps: (1) we use a multivariate analyses of variance (MANOVA) to determine the overall effects of faculty status on student achievement and learning experienc; (2) we use Pearson’s correlation coefficient to determine the strength of correlations between variables, especially the potential relationship between student grades and EOCS responses; and (3) we use a MANCOVA to statistically adjust for any correlations identified in step 2 in order to illuminate instructional practices that have the largest effect on student learning experience.
In 2015 and 2017, the Pearson correlation coefficients indicated that there were significant positive associations between student grades and survey responses, and student recommendations for both faculty (r=0.57, and courses (r = 0.52). The student grade correlation is still present in 2017 for faculty and course assessments, although the correlation becomes weaker over time. The percentage of students passing a course and the percentage of students progressing to the next course was 9.17% and 12.31% higher in courses taught by full-time, rather than by part-time faculty (F = 8.438, p < 0.01; F = 13.59, p < 0.0001, respectively). There was no significant influence of faculty status on student grades. However, faculty status had a highly significant effect on the overall student learning experience after we statistically adjusted for grade bias (F = 11.390, p < 0.0001). Students consistently rated full-time faculty higher in areas of participation, fostering critical thinking, expertise, communicating high expectations, and quality feedback. Feedback that aligns with expectations and timely feedback have the most influence on course evaluation ratings (F = 14.505, p < 0.0001 and F = 27.770, p < 0.0001, respectively). In sum, our study of almost 60,000 online courses show that the percentage of students passing a course and progressing to the next course, as well as the student learning experience is significantly greater for full-time versus part-time faculty.
To promote a vibrant community of educational research, we invite audience members to bring their experiences with designing and using EOCSs, as well as any questions related to analyzing large multivariate data sets with confounding variables. One important audience outcome of this research is to view EOCSs as a stragetic pathway to support our students rather than primarily as a faculty evaluation tool. Continued exploration can examine student feedback in the EOCS comments section to gain a deeper understanding of how students can be intellectually challenged and supported in the classroom. Including faculty course feedback on student preparedness and student-student interactions is also an important avenue for research because faculty input can drive student support services, including library training, writing, reading comprehension, and career development. Future collaborative and industry-wide research opportunities exist and include meta-analyses of existing publications or compilation of multi-university student achievement and learning experience data. Our results on the factors impacting student achievement and experience can empower the eLearning community by improving the application of performance data, and strategically planning future university research and policy.