Developing a MOOC Series: Pedagogical Considerations for Learning Designers

Watch This Session

Session Materials

Brief Abstract

This research examines 5 MOOC series to provide learning designers with guidelines for practice. Results suggest learning designers should attend to 1) objectives of series with respect to learners’ educational and professional goals, 2) course order within a series, including progressions that support learners in succeeding in higher order tasks.

Extended Abstract

Introduction 

As the landscape of Massive Open Online Courses (MOOC) continues to evolve, instructional forms are emerging that expand on the original MOOC form. For example, the MOOC series—a grouping or sequence of related courses—is rising in prominence (Hollands & Trithali, 2014) and has begun to eclipse the standalone MOOC in popularity and visibility (Shah, 2019). This development presents an important design challenge for instructional teams who must now consider the pedagogy of a series of courses while simultaneously attending to the coherence of individual courses. Instructional teams may find themselves confronted by new kinds of design questions:

  • How important is it to prioritize consistency of course structure (e.g., length, sequence of item types) so that learners become familiar with the learning experience and are able to anticipate workload over the duration of several courses? 
  • How important is it to consider pedagogical design from the perspective of course order, such as assessments in earlier courses that require learners to present declarative knowledge, and assessments in later courses that ask learners to apply knowledge to real world tasks? 

Objectives

Our overarching goal is to provide instructional teams with guidelines that will allow designers to be more intentional about how they approach the design of a series of MOOC courses. To achieve our goal, we set the following objectives: 

  • Define series requirements: Use Specializations—a particular type of MOOC series that are offered on the Coursera course delivery platform—as the object of our inquiry. Courses are generally designed to be taken in order, although this is not a requirement (Wong, 2016). 
  • Assess series characteristics: Use an expanded version Quintana et al., 2019) of the Assessing MOOC Pedagogies (AMP) tool (Swan et al., 2014) to characterize the pedagogies of courses within multiple Coursera Specializations. 

Given our overarching goal and the objectives we have set in order to meet that goal, we ask the following research questions:

  1. To what extent are courses that are designed as part of a series similar to other courses in that series pedagogically? Across which dimensions? 
  2. To what extent are courses that are designed as part of a series similar to courses in other series?
  3. What underlying factors may account for pedagogical similarities or differences?

Perspectives

Standalone MOOCs consist of a variety of elements including video-lectures, automated multiple-choice assessments, and discussion forum prompts (Conole, 2014; Margarayan et al., 2015). Although course delivery platforms have limited options that constrain design (Head, 2017), great pedagogical variation exists within MOOCs (Quintana et al., 2018; Major & Blackmon, 2016) and they are thus worth examining.

Typically, MOOCs include 6-10 weeks of content, but learner activity declines precipitously at week four (Hollands & Tirthali, 2014; Lacker et al., 2015). The cadence of these high drop-off rates led to the development of the MOOC series (Lackner et al., 2015). Researchers posited that a modular four-week course design would allow learners to “see the light at the end of the tunnel” and make greater progress in the course (Hollands & Tirthali, 2014; Lacker et al., 2015). A series could consist of multiple courses and each course in the series could focus on a specific aspect of a larger topic.

The AMP tool allows researchers to characterize the design of MOOCs across ten dimensions (Swan et al., 2014). Each dimension is anchored by two poles (e.g., artificial and authentic in the “Characteristics of Tasks” dimension), and has five score levels (see Table 1). Thus far, MOOC researchers have focused on characterizing the pedagogies of individual MOOCs using AMP (e.g., Admiraal et al., 2014; Bonk et al., 2015; Skrypnyk et al., 2015). In this research, we use an expanded version of the AMP tool (Quintana et al., 2019) that was designed to improve consistency of use through standardized language, detailed descriptions of score levels, and a list of course elements on which to focus. In our study, we will show that when combined with other research methods (i.e., hierarchical clustering methods), the AMP tool shows promise for providing researchers with a common language to describe potential pedagogical similarities and differences within and across MOOC series. 

Methods

Our study was conducted in three phases:

  • Phase 1: Characterize pedagogies of MOOCs within multiple Specializations using the expanded AMP tool. 
  • Phase 2: Plot hierarchical clusters (depicting pedagogical similarity) on a dendrogram (i.e., a tree style plot). 
  • Phase 3: Describe clusters, considering quantitative and qualitative aspects of the data 

Materials

  • We selected five Coursera Specializations across a range of subject domains, models of faculty involvement, and iteration stages. We selected Specializations a public research university (our home institution) which provided insight into unique aspects of their design and production processes. 
  • We generated nested clusters using hierarchical clustering in R using the hclust function from the stats package. 

Approach to analysis

Using the AMP tool, four researchers analyzed 23 individual MOOCs belonging to the five Specializations we selected. In order to ensure consistency and replicability of observations between independent observers, all four researchers analyzed the first course in each Specialization and the remaining courses in the front-end programming series. Over a series of four group meetings, they compared their scores across each dimension and reached consensus through discussion. Through this process, researchers developed a common understanding of how to apply score levels across each dimension and achieved an intra-class correlation coefficient of 0.86, which surpasses Koo and Li’s (2016) lower bound of “good” inter-rater reliability. Each researcher independently analyzed the remaining courses in one of the four remaining Specializations. We imported final scores into R and generated clusters. We built a dissimilarity matrix using the “daisy” function and fed the matrix into the “hclust” function, generating hierarchical clusters. Finally, we plotted the hierarchical clusters on a dendrogram, which allowed us to see which courses exhibit pedagogical similarities. 

Results

We identified three potential heights to cut the dendrogram and build clusters. This method allowed us to visualize three sets of clusters, which we have named according to each set’s most distinct feature: 

  • Pedagogy A: the highest cut set (2 clusters)
  • Pedagogy B: the middle cut set (4 clusters)
  • Series: the lowest cut set (7 clusters) 

Each set contained all of the courses we analyzed and emphasizes different aspects of the pedagogies underlying individual courses and Specializations as a whole.

  • The “Pedagogy A” set divides courses into two clusters that have distinct epistemologies: 1) learning through interaction with instructor-created content (objectivist) and 2) learning through interaction with a rich set of resources, including peers (constructivist). They also roughly divide according to the “left” and “right” poles of the AMP framework (refer to Table 1). 
  • The “Pedagogy B” set  separates courses into four clusters and reveals differences in approach to content presentation, task requirements, and structure: 1) concrete content and authentic tasks; 2) abstract content and artificial tasks;  3) self-directed tasks, with some structure; 4) self-directed tasks with little structure. 
  • The “Series” set displays seven clusters that roughly group courses by series (with some exceptions). 
    • Two series (financial and human anatomy) formed their own respective clusters, suggesting that the overall pedagogical design is similar across all courses in each series. 
    • Courses in two series (front-end development and data analysis) appeared in three clusters, showing pedagogical variation across courses in each respective series. Earlier courses in the series mapped to the “left” pole of the AMP framework and later courses mapped to the “right” pole. 
    • Clusters 6 and 7 contain one course each, showing that these courses are somewhat distinct from other courses. These courses are the capstone courses of the leadership and management series and the front-end development programming series, and they are distinguished by being far more learner-centered than courses in the other clusters. 

Scholarly Significance 

Our results show that instructional teams prioritized “fit for purpose” at the Specialization level over ensuring consistency from course to course for its own sake. 

  • In two Specializations (finance and anatomy), we observed that each course was pedagogically similar to all other courses in each respective series. In this example, these two Specializations were designed to support learners who intend to pursue advanced education within their respective domains (e.g., advanced business degrees and professional medical degrees). 
  • In two different Specializations (leadership and front-end development), we observed that courses appeared in multiple clusters and that course order determined if they fell in “left pole” clusters or “right pole” clusters. In other words, courses that were earlier in the series tended to be more instructor-centered in their epistemologies, and courses that were later in the Specialization tended to be more learner-centered and constructivist. These Specializations prioritized providing scaffolding in earlier courses that would allow learners to be more independent in later courses. 
  • In the final Specialization (data analysis), we observed that earlier courses used concrete approaches to content presentation with authentic tasks, and later courses used abstract approaches to content presentation with artificial tasks. 

This study reveals the pedagogies that underlie a specific set of MOOC Specializations and offers early insight into how instructional teams have approached the design of a series.