The VooDoo of Measuring Engaged Learning in Online and Blended Courses

Concurrent Session 7

Session Materials

Add to My Schedule

Brief Abstract

Designing instruction for engaged learning is critical to creating high quality blended learning courses.  We see similarities among critical elements necessary for creating highly engaging blended learning opportunities and highly engaging online courses.  Can the Indicators of Engaged Learning Online Framework be applied to Blended Learning? What do you think? 


Paula Bigatel, PhD is an Instructional Designer for Penn State World Campus, Faculty Development Unit. She received her B.A. in Psychology and English at Concordia University in Montreal, Quebec. Subsequently, while teaching English as a Second Language, she earned her Teaching English as a Second Language certificate. Paula has also spent about 19 years in business serving various in financial management roles. She completed her Master in Education and PhD at Penn State in Instructional Systems (Educational Technology). Over the last twelve years, Dr. Bigatel has taught graduate level courses in the Masters in Education and Technology Certificate programs, and currently teaches various online courses for faculty professional development. Her research focus is in online pedagogy and student engagement.
Dr. Stephanie Edel-Malizia is currently an Instructional Designer for Penn State University, with over 20 years of experience as a leader in instructional technology spanning the k-20 realm. She completed her Doctorate of Education at Delta State University with the dissertation Design and Implementation of Faculty Development for Student Required Internet Use. As a faculty member at Delta State, Dr. Malizia worked as an Instructional Designer and Instructor. Stephanie has ten years of experience as a Pennsylvania Department of Education certified Instructional Technology Specialist, working eight of those years as the Director of Instructional Media and Technology Services for a regional Educational Service Agency. She is also certified as a Superintendent of Schools and a 2013 graduate of the Institute for Educational Leadership Fellows program. Stephanie has taught graduate courses in instructional technology and pedagogy for Penn State DuBois, St. Bonaventure University, East Stroudsburg University, and Clarion University. Her conference presentations include the European Conference on E-Learning, Online Learning Consortium, Educause, The Teaching Professor Technology Conference and the International Society for Technology in Education.

Extended Abstract

This emerging idea explores the use of the Indicators of Engaged Learning Online Framework as a guide to measure the quality of blended learning courses. Student engagement has been an important topic of interest in higher education because of its positive impact on student persistence and retention (Boston & Ice, 2011; Robinson & Hullinger, 2008; Wyatt, 2011). In the online environment, retention rates are lower than in the face-to-face environment in higher education institutions (Allen & Seaman, 2015), thus strategies that engage students in online courses to a greater extent becomes an important goal to pursue. The Engaged Learning Framework lends itself well to measures of engagement particularly in terms of online course design.  We’d like to take what we’ve learned about measuring online course engagement and apply it to blended learning courses and we’d like to see what you think about using the Indicators of Engaged Learning Online Framework to measure engagement in blended learning courses.

In this emerging idea presentation, participants will:

  • Identify key elements of student engagement in online and blended learning

  • Discuss the use of the Indicators of Engaged Learning Online Framework for measuring student engagement in blended learning courses

  • Discuss the use of the Indicators of Engaged Learning Online Framework for improving blended learning course design by increasing planned opportunities for student engagement

  • Implement the principles of engaged learning into their own teaching and learning environment

To begin, we’ll share information from a previous study, where six online courses were selected to evaluate and discuss results in terms of what students and the research literature say engagement is.  Based on this previous study, a toolkit was created comprised of templates to be used to evaluate the degree of engagement designed within a course. Recommendations for practice include two aspects of engagement: how a course is designed and how it is delivered.

Our research on student engagement is founded on a desire to improve the quality of online instruction in both course design and delivery. A high level of student engagement is associated with a wide range of educational practices including purposeful student-faculty contact, peer to peer contact, active and collaborative learning, and positive factors such as student satisfaction, persistence, achievement, and learning (Kuh, et al., 2006).  It is our belief that by encouraging student engagement, institutions of higher education can have a positive impact on student success that leads to retention and degree completion. Our discussion explores how the Engaged Learning Online evaluation tools (Toolkit) can be used by faculty, administrators, and instructional designers as they create online and blended courses and encourage/support faculty teaching practices to maximize the learning experience for students in online and blended courses.

The Indicators of Engaged Learning Online Framework provide a framework that measures the level of student engagement. Social constructivism and collaborative learning form the theoretical basis of the framework. Social constructivist pedagogy acknowledges the social nature of knowledge and its creation in the minds of individual learners. Some important themes that flow from social constructivism involve the importance of collaboration among instructors and students, active learning vs passive consumption of information, a learning environment that is learner and learning centered, which promotes multiple perspectives, and the use of social tools in the online and blended environment to construct knowledge.

Indicators of Engaged Learning Online Framework

The framework is a comprehensive overview of elements to be considered in the design and delivery of an online course. The indicators are organized under three main categories: Instructional Approach, Teaching, and Learning. Furthermore, each dimension within each category represents three multi-dimensional aspects to the learning experience: cognitive investment (thoughtful/mindfulness), socio-emotional engagement (interactions/reactions), and behavioral engagement (participation). We will describe how the three dimensions can be used in course design and as a framework for best practices in instructional approach for blended learning.

We will also describe how the framework can be used in blended course review to measure engaged learning.   High quality instruction contains specific elements that will improve the potential to engage students socially, emotionally and cognitively.  These elements have been researched primarily in the context of face to face, primary and secondary education.  Our approach stems from the belief that these elements are just as important to the quality of teaching and learning in an online environment for adult learners.  This framework was developed based largely on the 1995 publication Plugging in: Choosing and Using Educational Technology.

Certainly, other standards of quality like Quality Matters (QM) ( can be used to evaluate the quality of the design of online courses. QM is useful for the mechanics of how a quality course should be designed, but does not adequately address student engagement particularly in terms of how learners interact with the instructor, the course content, and each other.  Additionally, QM is designed only to be used to evaluate online activities and does not transcend boundaries from online learning to blended learning.  When used to evaluate blended learning courses, QM reviewers are instructed to only review the online portion of the blended course.

Although there are many quality standards that contribute to the quality of online instruction, according to a study conducted by Southard and Mooney (2015), standards related to online course design (40.4%) far outnumbered the standards related to the delivery of online course (11%).  Thus, it is important to apply a framework that takes into account both types of standards among others such as faculty support, student support, and institutional infrastructure.

Evaluation Tools (Toolkit)

Based on an engagement study survey which measured qualitative aspect of this student engagement study by Bigatel & Williams (2015) and the Engaged Learning Framework (Indicators of Engaged Learning Online), a toolkit was created for faculty, instructional designers, and administrators to use in order to assess the quality of online courses in terms of design and delivery. The toolkit includes:

  • Indicators used to assess course design – an excel spreadsheet pre-formatted

  • Check sheets for instructors and students

  • Survey

  • Review Cycle

This toolkit will be shared with the audience and we will facilitate a discuss on how applicable these tools would be for designing and measuring engagement in blended learning courses.


Allen, I., & Seaman, J. (2015). Grade level: Tracking online education in the United States. Babson Survey Research Group. Retrieved from

Bigatel, P., & Williams, V. (2015). Measuring Student Engagement in an Online Program. Online Journal of Distance Learning Administration, 18(2), n2.

Boston, W. E., & Ice, P. (2011). Assessing retention in online learning: An administrative perspective. Online Journal of Distance Learning Administration. Retrieved from

Edel-Malizia, S., & Brautigam, K., “Gauging the Quality of Online Learning by Measuring 21st Century Engagement,” in Proceedings of the 13th European Conference on E-Learning, Copenhagen, 2014, 700–3. Copenhagen, Denmark: Aalborg University, 2014.

Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to student success: A review of the literature,

Commissioned Report for the National Symposium on Postsecondary Student Success: Spearheading a Dialog on Student Success. Retrieved from

Jones, B. F., Valdez, G., Nowakowski, J., & Rasmussen, C. (1995). Plugging in: Choosing and using educational technology.

Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. The Internet and Higher Education, 6, 1-16.

Southard, S., and Mooney, M. (2015. A comparative analysis of distance education quality assurance standards. The Quarterly Review of Distance Education, 16(1), 55-68.

Wyatt, L. G. (2011). Nontraditional student engagement: Increasing adult student success and retention. The Journal of Continuing Higher Education, 59(1), 10-20.