Establishing Quality Standards to Navigate Innovation: A Flexible Approach to Course Development

Concurrent Session 9

Brief Abstract

Changing technology and student demands have made the “one-size-fits-all” approach to course evaluation obsolete, yet quality standards are as important as ever. This session demonstrates a new approach to measuring course quality that divides the process into components based on accreditation standards, research, and best practices to provide an adaptable blueprint.

Presenters

Carrie Murphy is the Manager of Research and Analytics for The Learning House, Inc., overseeing course quality review and client reporting processes. Carrie investigates the relationships between course structure, and student success, satisfaction and retention. She also makes data-driven recommendations to enhance the effectiveness of online course delivery, and manages the research and analysis of Learning Houses course quality initiatives.

Extended Abstract

In a world where the traditional “one-size-fits-all” approach to learning is being broken down in favor of a more customized approach, why should we evaluate course quality against a similarly uniform scorecard? The next generation of learning management systems is expected to integrate a broad array of tools and modes of learning (http://er.educause.edu/articles/2015/6/whats-next-for-the-lms), only increasing the desire for customization. This leaves administrators and faculty with the difficult task of finding a systematic, consistent way to measure course quality while providing an innovative and unique learning experience to students.  

In 2014, The Learning House, Inc., began to re-evaluate its approach to course quality evaluation with this challenge in mind. This session will describe the intention and methods by which the company built, tested and implemented an improved approach to evaluating course quality.

During this session, attendees will have the opportunity to

  • review how this approach was developed, including the lessons learned from challenges and roadblocks;
  • walk through multiple scenarios to explore how this approach may be applied to the course development process; and
  • discuss how idea this approach can be adapted to different environments through an open Q&A session.

The new quality review method has three different and interconnected levels of factors. First, the core quality standards are a low-investment review to ensure all courses meet the minimum requirement for a functional course based on accreditation and legal requirements for all courses, including basic accessibility and copyright concerns. Second, the subsets build on the core standards and represent broadly applicable and empirically supported quality standards that have been grouped into factors that correlate with student outcomes. Our subsets were developed using a psychometric approach. We evaluated 88 courses (enrollment M = 12; SD = 7.2) and their associated student outcomes (M = 80.2%; SD = 15.7%) against the 132 proposed items based on a thorough review of available empirical literature. Based on the results of a factor analysis using principal component analysis to simplify the structure of our standards, we currently have nine factors (i.e., subsets) with 115 items. These nine subsets explain 57% of the total variance in student scores. Lastly, the focus areas are based on professional opinion and are used to fill specific gaps introduced by technology or the limited needs of particular departments or programs, including video production and accessibility, mathematics assessment, and eBook-format content delivery.

Attendees will walk away with an understanding of how this approach can be used in order to

  • create a nimble and iterative improvement plan for course creation as well as for the quality measure itself;
  • facilitate easy implementation of increasingly stringent quality standards over time as courses improve, allowing for continued growth and opportunity; and
  • provide targeted, actionable, and diagnostic feedback, allowing for an at-a-glance understanding of the broad strengths and weaknesses of a course, program, or institution.