The Quality of Student Learning--An inference approach for measuring quality of online education

Brief Abstract

The quality of online higher education is difficult to evaluate. Many quality standards and benchmarks have been defined, but there is little scientific evaluation in the literature.  This paper proposes a quantitative approach for evaluating the quality of online English composition courses by analyzing students’ learning outcomes, including their academic performance and self-reported measures of satisfaction and learning experiences. Four types of learners (Low Attainment, Positive Experience, Mixed Experience, and Negative Experience Learners) were identified, each with different perceptions of the quality of their online learning.  This study proposes the quality of online courses can be inferred by the distribution of these learner types and offers a baseline of course quality that can support educators as they improve their courses.

Presenters

Ying-Ying Kuo has worked as an instructional designer since 2003. In her work, she highlights the importance of integrating technology to support pedagogy and learning outcomes. Her designs promise efficient and effective online teaching and learning as well as course management. Her biggest reward comes from instructors and students who become proficient in teaching and learning in the digital world.

Extended Abstract

The quality of e-learning should be at least as good as face-to-face (f2f) teaching and learning on campus.  However, it is hard to evaluate the quality of online higher education because stakeholders, including administrators, designers, instructors, students, etc., have different perceptions of what constitutes quality in online education (Harvey & Green, 1993). Many standards, benchmarks, and rubrics, adopted by the higher education in the U.S. describe the criteria for quality, but they lack value judgements from students’ perspectives. Harvey and Green (1993, p. 24) stated “quality is seen in terms of the extent to which the education system transforms the conceptual ability and self-awareness of the student.”  To address the challenges of getting objective measures of e-learning quality, educators should focus on the quality of student learning outcomes since the student is the primary audience for online learning (Bates, 2015; June & Latchem, 2012; Stella & Gnanam, 2004). Measuring the quality of student learning should examine students’ learning outcomes, which include not only their academic success but also their self-reported perceived values about how satisfied they were with their learning experience.  

Conceptual Framework

Zeithaml (1988) emphasized that “objective quality” describes the actual superiority or excellence of the product from an objective standpoint, while “perceived quality” is measured from the consumer’s standpoint.  Objective quality in higher education is hard to estimate, compare to the quality of a product and service that is established via its features and specifications as well as its performance standards from the design to the production to the consumption. However, perceived quality is the individuals’ perspectives of quality, which can be influenced by their expectations, interactions with the environment and situational control, experiential value, etc. (Gotlieb, Grewal, & Brown, 1994; Oliver, 1981; Zeithaml, 1988).  Perceived quality of teaching and learning is a judgement usually is in relation to individuals’ perceived value that is an overall assessment of a tradeoff of what is given and received of learning experience (Zeithaml, 1988).  

Measuring quality is a comparison between individuals’ expectations and perceptions of the outcomes (Gotlieb et al., 1994; Zeithaml, 1988).  The academic achievement—grade—is a critical outcome that institutions have defined, which is based on instructors’ evaluation of students’ knowledge and skills.  In addition, students’ satisfaction with the learning context or physical surroundings has been viewed as a predecessor of perceived quality (Gotlieb et al., 1994; Oliver, 1981).  Students’ experiences of the quality benchmarks and standards, defined by IHEP (2000), QM (year), OLC (2019), California State University’s ROI (2019), etc., should be valued as a part of self-reported learning outcomes. 

Purpose

This study assumes that the quality of e-learning can be inferred by the measure of learning quality, as the student is the primary focus of e-learning.  It investigates how to measure the quality of student learning in a way that focuses on students’ learning outcomes, including their academic achievement and self-reported of their perceived values of online learning.  As the quality of online teaching and learning can be dynamic, this study establishes a baseline of learning quality in order to continue quality improvement for further instructional delivery and course maintenance.  

Methodology

This study examines the quality of online teaching and learning of English composition courses in a public university in the eastern U.S. Multivariate analysis and data mining techniques were applied, including an exploratory factor analysis (EFA), Analysis of Variance (ANOVA), and clustering analysis, to analyze students’ perceived quality of their online learning.   Descriptive and inference statistics are reported in the measurement of the learning outcomes.

Course Description & Data Collection

The English composition courses are required for all undergraduate students in the university.  All courses have the same uniform student learning outcomes across all sections, including goals in recursive and inquiry-based writing, analytical and critical reading and writing (rhetoric), and research and documentation. All English composition courses use Blackboard Learn 9.1. The data collection was conducted in Fall 2017, Spring and Fall 2018, and Spring 2019.  

The data was collected via a student course survey at the end of the semester. Two portions of the survey are included in this study: a section of learning experience and overall satisfaction. 

Learning experience. The learning experience section included 12 statements, which were created by considering the context of English composition courses and some quality benchmarks and standards in regard to teaching and learning, using, a 6-point Likert scale from Strongly Disagree (1) to Strongly Agree (6).  

Overall Satisfaction. Students evaluated their overall learning satisfaction with a 6-point Likert scale, from Very Dissatisfied (1) to Very Satisfied (6).  

In addition, the student participant’s final grade was collected. This academic achievement is evaluated by the instructor, based on course objectives and how students completed work on the requirements compared to the instructor’s expectations. 

Variables & Data Analysis

The data analysis in this report only included three semesters (Fall 2017, Spring and Fall 2018). Four instructors of the English first-year composition course and one English literature instructor supported this study. Three of them taught fully online while two instructors taught in hybrid settings. A total of 234 over 416 students voluntarily participated in this study, with an average of student response rate per instructor ranging from 41.8% to 64.4%.  Only 204 subjects were included in the data analysis, as 30 were excluded due to the missing data.        

Two new variables were created from the survey data of the learning experience section. Because there were medium correlations (ranged from 0.22 – 0.68, at p <0.01) within the 12 experience variables found and the Kaiser-Meyer-Olkin Measure and Bartlett’s Test of Sphericity (KMO= 0.868, p < 0.001) indicated suitability of the data structure and latent variables existed.  With the EFA and Oblimine rotation, a two factor structure was attained, which explained 61% of total variance of the student learning experiences. The two factors— entitled as “Course Design” and “Outcomes”—showed medium level of correlation (r = 0.46).  

Four variables in this study are defined below: 

Course Design: Seven items of learning experience on the use of Blackboard tools, navigation of Blackboard course site, content, lesson sequence, flexibility, and instructor feedback were included. Its internal consistency reliability was 0.87, as measured by Cronbach’s alpha. 

Outcomes: Five items of learning experience on group activities, interactions, writing skills and interest, and research skills were included. Its internal consistency reliability was 0.83, as measured by Cronbach’s alpha.  

Satisfaction: The self-report score of overall satisfaction, ranging from 1 to 6. 

Grade: Subjects’ final grade (in raw score out of 100 points) was provided by the instructor at the end of the semester. 

Then, a K-mean cluster analysis was conducted to group students’ perceived quality, based on the z-score of four variables—Course DesignOutcomesSatisfaction and Grade.  Four types of student groups were identified—Low Attainment LearnersPositive Experience LearnersMixed Experience Learners, and Negative Experience Learners, as shown on Table 2. The description of each student group is explained below:

  • Low Attainment Learners: These students received a failing final grade (C- and below). Their overall satisfaction ranged widely (= 4.64, SD = 1.12). Their scores in Course Design (z = -.03, SD= .52) and Outcomes (z = -.42, sd = 1.02) implied that they did not care about their online learning.
  • Positive Experience Learners: These students were highly satisfied with their learning (M = 5.53) and received passing grades (C or higher). Their learning experience was positive based on the score of Course Design (z = .65, SD = .59) and Outcomes (z = .72, SD = .55). 
  • Mixed Experience Learners: These students received passing grades but reported moderate satisfaction level (= 4.45).  Their score of Course Design (z = -.25, SD= .73) and Outcomes (z = -.40, SD= .76) indicated their learning experience could be mixed.
  • Negative Experience Learners: These students received passing grades but were very unsatisfied (= 2.79) with the online learning and their learning experiences were negative based on the score of Course Design (z = -1.54, SD= .75) and Outcomes 
    (z = -1.24, SD= .78).

An ANOVA result revealed that significant statistical differences (SSD) of their overall satisfaction, the learning experience of Course Design and Outcomes, and their final grades at the confidence level of 0.001. 

Discussion

This proposal discusses a few highlights of the findings.  The full paper will provide more detailed information.

Measure of Course Quality

This study advocates that course quality can be inferred from students’ perceived quality; it also suggests that the measure of learning quality should focus on whether or not students’ learning outcomes meet expected academic performance and how they perceived the value of their learning experience. Four types of students were identified in this study and each group has indicated that they perceived a different level of learning quality.  This study stresses that the quality of the English composition courses can be inferred from the distribution of student types. It means that course quality is better when there are more Positive Experience Learners; course quality is worse when there are fewer Positive Experience Learners and more Low Attainment Learners and Mixed/Negative Experience Learners.  

Continuous Quality Improvement 

The measure of course quality inferred by the quality of student learning does not intend to represent the true course quality; however, it can demonstrate  a baseline of course quality based on the student type distribution in one semester.  Educators can assess the course quality by referring the percentage of student types in different timeframes.   

According to the percentage of Positive Experience Learners and Mixed Experience Learners over the percentage of Low Attainment Learners and Negative Experience Learners in three semesters on Figure 1, educators can conclude the course quality has improved from Fall 2017 to Fall 2018.  However, no statistically significant differences were found in quality improvement within three semesters, according to the Pearson Chi-Square test.

Conclusion and future work

Literature reveals that it is challenging to have objective measures in online teaching and learning, as all stakeholders have their expectations and perspectives. This study has used the measures of learning quality to infer the status of course quality that is illustrated by the distribution of student type.  Online students perceive quality differently due to their own expectations, experiences, and perceived values.  This study advocates that quality of online teaching and learning can be assessed and managed when there is a baseline of course quality to be referred to and applied to support educators to continue  quality improvement in the long run.

Future work includes continuous data collection to enhance the statistical power in analysis and expand this approach to different subjects and disciplines.  A predictive model on measuring quality of student learning is also planned.  The goal is to apply this model in quality improvement and for the purpose of quality assurance.

Significance of the Study

This study has established a scientific approach for measuring the quality of student learning in online English Composition courses.  The quality of online education in current literature focuses largely on conceptual models or framework, and this study contributes important factors that also affect quality in the measure of learning quality and the statistical inference of course quality.  Online educators can adopt the process of quality evaluation in this study and work on continuous quality improvement, rather than estimating the true quality status of e-learning.