Fixer Upper: Applying Evidence to Renovate an Online Course Development Process
Concurrent Session 3
This session will present the iterative evolution and lessons learned from design, to evaluation, to renovation of a faculty online and hybrid course redesign program. A solid foundation provides the necessary framework to support program modernization.
The goal of this panel session is to describe UMBC’s faculty online and hybrid course redesign program, discuss training and support successes, and highlight quantitative outcomes informing program redesign.
This presentation will focus on hybrid and online course redesign training, support and evaluation. When it is rigorously evaluated and replicated, our institution has found course redesign to be one of the most scalable ways to improve student academic success. We will share lessons learned in the iterative process of establishing, evaluating, and redesigning a faculty course design program. Specifically, we will describe UMBC’s original Alternate Delivery Program, discuss how analytics has informed effective practices, and detail how identifying, supporting, and promoting best practices within and among the faculty directed the redesign of this program.
History: Established in 2005 through a partnership between campus IT, Continuing Education and Faculty Development, the ADP is also a useful laboratory for exploring and applying learning analytics, which UMBC has pursued for several years. Since fall 2006, the Alternate Delivery Program (ADP) has provided financial, technical, and pedagogical support to 60 faculty members to redesign an existing face-to-face (F2F) course for online or hybrid (part online, part F2F) delivery during the compressed, intensive summer or winter session. The number of hybrid and online courses offered in the special sessions has grown steadily, and currently, 30 – 40% of the courses scheduled in summer or winter sessions are hybrid or online. In response to this increased demand and the need to assure design quality, program administrators have redesigned the training and support structures, beginning during the spring 2015 term.
Supporting: UMBC’s Instructional Technology Department is currently comprised four Specialists under the Director and a rotating contingency of student interns. Specialists provide administrative support for the learning management system (Blackboard), consult faculty on best practices for integrating technology into pedagogy, conduct analytics and coordinate campus-wide assessment and evaluation projects, and identify and train both faculty and staff on new and emerging technology tools and strategies.
Evaluation: The most recent program evaluation shows a statistically significant link in student academic outcomes associated with enrollment in courses instructed by teachers who participated in the ADP program. Using a large, student-level sample (250k records) with robust controls, this session presents the positive, statistically significant (p<.001) relationship between the faculty ADP training program and student academic outcomes. The data indicate that student LMS activity, a proxy for student engagement, is higher for courses that have undergone redesign than those that have not. Additionally, using a large, student-level sample with robust controls, this presentation describes a positive, statistically significant relationship between the ADP training program and student academic outcomes (i.e., Term GPA, Cumulative GPA, Term Credits Earned, and Cumulative Credits Earned), even when controlling for student LMS activity).
Redesign: Through evaluation and consultation of the current literature outlining best practices, we determined that ADP program needed adjustments to address the varying knowledge and skill needs of our program participants in order to assure quality of the redesigned courses. In turn, the program has been redesigned to include Skill Builder, Course Designer, and Quality Booster tracks.
The Skill Builder track offers modular training in core components of Blackboard and supporting technologies (e.g., Bb Collaborate, TechSmith Relay, Box) to help faculty better utilize the features available in Blackboard and these other tools and to prepare for the Course Designer track.
The Course Designer Track, which most closely resembles the original ADP program’s design, is available to faculty who have either completed the ADP Skill Builder track or demonstrated competency with the Skill Builder components. The Course Designer track typically requires at least a semester for course development before the redesigned course is offered in summer or winter session. It includes a two-day training workshop and an opportunity to work one-on-one with an Instructional Technology Specialist in the Division of Information Technology (DoIT). Faculty who participate in the Course Designer track should have prior experience teaching in summer or winter sessions and convert courses that have a proven enrollment track record in summer or winter sessions.
The Quality Booster track is designed for faculty who are already offering hybrid or online courses, but who have either not previously participated in formal course redesign training through the ADP or participated in ADP prior to the Quality Matters (QM) process integration. Quality Matters is an internationally recognized peer review process to assess the design quality of an online or hybrid course. The goal is to carefully evaluate the components of a hybrid or online course to ensure that it meets nationally identified quality standards.
We are developing mixed-methods strategies to identify design strategy differences between old and new course redesign programming to further refine best practices through this iterative process. Anyone involved in providing faculty development support that must address instructional technology and hybrid or fully online course delivery strategies should attend. Slides, handouts, and related website links will be provided during the presentation and posted on the conference web site.