Evaluating and Supporting Student Engagement in Online Learning

Concurrent Session 1

Brief Abstract

This paper presentation reviews how results of a student survey measuring online student engagement were applied to creating evaluation tools for faculty, administrators, and instructional designers. The survey reflects the “Indicators of Engaged Learning Online” framework and evaluation tools that critique course design, teaching behaviors, and the use of technology. 


Paula Bigatel, PhD is an Instructional Designer for Penn State World Campus, Faculty Development Unit. She received her B.A. in Psychology and English at Concordia University in Montreal, Quebec. Subsequently, while teaching English as a Second Language, she earned her Teaching English as a Second Language certificate. Paula has also spent about 19 years in business serving various in financial management roles. She completed her Master in Education and PhD at Penn State in Instructional Systems (Educational Technology). Over the last twelve years, Dr. Bigatel has taught graduate level courses in the Masters in Education and Technology Certificate programs, and currently teaches various online courses for faculty professional development. Her research focus is in online pedagogy and student engagement.

Extended Abstract

Our research on student engagement is founded on a desire to improve the quality of online instruction in both course design and delivery. A high level of student engagement is associated with a wide range of educational practices including purposeful student-faculty contact, peer to peer contact, active and collaborative learning, and positive factors such as student satisfaction, persistence, achievement, and learning (Kuh, et al., 2006).  It is our belief that by encouraging student engagement, institutions of higher education can have a positive impact on student success that leads to retention and degree completion. Our paper presents how the results of our online student engagement survey were applied to creating evaluation tools (Toolkit) that can be used by faculty, administrators, and instructional designers as they create online courses and encourage/support faculty teaching practices to maximize the learning experience for online students.

The “Indicators of Engaged Learning Online” provide a framework that measures level of student engagement. Social constructivism and collaborative learning form the theoretical basis of the framework. Social constructivist pedagogy acknowledges the social nature of knowledge and its creation in the minds of individual learners. Some important themes that flow from social constructivism involve the importance of collaboration among instructors and students, active learning vs passive consumption of information, a learning environment that is learner and learning centered, which promotes multiple perspectives, and the use of social tools in the online environment to construct knowledge.

Furthermore, research provides evidence for the connection between higher student engagement and persistence and retention in online programs (Boston, et al., 2010; Rovai, 2003; Wyatt, 2011), which supports our rationale for encouraging student engagement especially in the online environment because attrition rates are higher than in the face-to-face environment (Allen & Seaman, 2015; Boston & Ice, 2011).

The goals for the audience of this session are:

  • Identify key elements of student engagement in online learning
  • Use a student survey, faculty checklist, and peer review tools based on indicators of engaged learning online
  • Implement the principles of engaged learning into their own teaching and learning environment


Indicators of Engaged Learning Online Framework

The framework is a comprehensive overview of elements to be considered in the design and delivery of an online course. The indicators are organized under three main categories: Instructional Approach, Teaching, and Learning. Furthermore, each dimension within each category represents three multi-dimensional aspects to the learning experience: cognitive investment (thoughtful/mindfulness), socio-emotional engagement (interactions/reactions), and behavioral engagement (participation). In our paper, we describe how the three dimensions can be used in course design and as a framework for best practices in instructional approach. 

We also describe how the framework can be used in course review to measure engaged learning.   High quality instruction contains specific elements that will improve the potential to engage students socially, emotionally and cognitively.  These elements have been researched primarily in the context of face to face, primary and secondary education.  Our approach stems from the belief that these elements are just as important to the quality of teaching and learning in an online environment for adult learners.  This framework was developed based largely on the 1995 publication Plugging in: Choosing and Using Educational Technology.

Certainly, other standards of quality like Quality Matters (QM) (www.qualitymatters.org) can be used to evaluate the quality of the design of online courses. QM is useful for the mechanics of how a quality course should be designed, but does not adequately address student engagement particularly in terms of how learners interact with the instructor, the course content, and each other.

Although there are many quality standards that contribute to the quality on online instruction, according to a study conducted by Southard and Mooney (2015), standards related to online course design (40.4%) far outnumbered the standards related to the delivery of online course (11%).  Thus, it is important to apply a framework that takes into account both types of standards among others such as faculty support, student support, and institutional infrastructure.

Student Engagement

When we asked students to describe what engaged learning meant to them, we saw themes emerge that emphasized the importance of:

  • Interaction among peers and faculty (collaboration)
  • Timely feedback on assignment/assessments
  • Faculty participation and visibility
  • Relevance and real-world application (in terms of assignments, activities, and assessments)
  • Motivation/interest (the need for faculty to motivate students especially those that may not have an interest in the coursework per se)

The qualitative aspect of this student engagement study by Bigatel & Williams (2015) revealed what instructors did well that made students feel engaged in the course, but also what instructors did not do well, which could inform our recommendations to faculty when expectations for teaching a course are reviewed.

The quantitative aspect of the survey drilled down to activities that students engaged in during their coursework and asked students to rate how often they engaged in such activities on a scale of 1 (not at all) to 4 (very often). The survey also asked students to rate how well instructors engaged them in their coursework.

In addition, the survey results informed our next iteration of a survey to be distributed in the summer and fall of 2016. Moreover, the results to date will become a part of the presentation as well.

During the presentation, we will review the revised survey and how that survey can be used as a reference or checklist when designing or re-designing an online course (See Evaluation Tools – Toolkit).

Evaluation Tools (Toolkit)

Based on the engagement study survey and the Engaged Learning Framework (Indicators of Engaged Learning), a toolkit was created for faculty, instructional designers, and administrators to use in order to assess the quality of online courses in terms of design and delivery. The toolkit includes:

  • Indicators used to assess course design – an excel spreadsheet pre-formatted
  • Check sheets for instructors and students
  • Survey
  • Review Cycle

In the presentation, this toolkit will be shared with the audience.


Allen, I., & Seaman, J. (2015). Grade level: Tracking online education in the United States. Babson Survey Research Group. Retrieved from http://www.onlinelearningsurvey.com/reports/gradelevel.pdf.

Bigatel, P., & Williams, V. (2015). Measuring Student Engagement in an Online Program. Online Journal of Distance Learning Administration18(2), n2.

Boston, W. E., & Ice, P. (2011). Assessing retention in online learning: An administrative perspective. Online Journal of Distance Learning Administration. Retrieved from http://www.westga.edu/~distance/ojdla/summer142/boston_ice142.html.

Stephanie Edel-Malizia and Kathleen Brautigam, “Gauging the Quality of Online Learning by Measuring 21st Century Engagement,” in Proceedings of the 13th European Conference on E-Learning, Copenhagen, 2014, 700–3. Copenhagen, Denmark: Aalborg University, 2014.

Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to student success: A review of the literature,
Commissioned Report for the National Symposium on Postsecondary Student Success: Spearheading a Dialog on Student Success. Retrieved from http://nces.ed.gov/npec/pdf/kuh_team_report.pdf.

Jones, B. F., Valdez, G., Nowakowski, J., & Rasmussen, C. (1995). Plugging in: Choosing and using educational technology. 

 Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. The Internet and Higher Education, 6, 1-16.

Southard, S., and Mooney, M. (2015. A comparative analysis of distance education quality assurance standards. The Quarterly Review of Distance Education, 16(1), 55-68.

Wyatt, L. G. (2011). Nontraditional student engagement: Increasing adult student success and retention. The Journal of Continuing Higher Education, 59(1), 10-20.