GAP ANALYSIS: AN INNOVATIVE LOOK AT GATEWAY COURSES & STUDENT RETENTION

Concurrent Session 4
Streamed Session

Brief Abstract

This session will demonstrate how analyses of gaps between expected and actual outcomes can identify differences in the efficacy of gateway courses, how such factors as student type and academic level affect success in such courses, and how that experience can differentially impact students’ likelihood of persisting in their studies.

Presenters

Premiere online scholar and James Stuckle professor, University of Illinois Southern; OLC Fellow and Outstanding Achievement Award in Online Learning; member of IACEHOF and significant role in development and dissemination of the Community of Inquiry (COI) framework. Karen Swan is the James J. Stukel Distinguished Professor of Educational Leadership and a Research Associate in the Center for Online Learning, Research, & Service (COLRS) at the University of Illinois Springfield. Karen’s research has been in the general area of electronic media and learning for the 25 years since she received her doctorate from Teachers College, Columbia University. For the past 20 years, she has been teaching online, researching online learning, and writing extensively about her experiences. She received the Online Learning Consortium (OLC) award for Outstanding Individual Achievement, National University Technology Network (NUTN) Distinguished Service Award, and the Burks Oakley II Distinguished Online Teaching Award for her work in this area. She is also an OLC Fellow and a member of the International Adult and Continuing Education Hall of Fame. In 2010 she also was given the Distinguished Alumni award by her alma mater.
Dr. Scott Day is Professor and Chair of the Department of Educational Leadership at the University of Illinois at Springfield. He holds an Ed. D. in Educational Organization and Leadership from the University of Illinois at Urbana-Champaign. Previously, Dr. Day served as Middle School Assistant Principal and Principal in Pennsylvania and Illinois before moving to a university teaching position. Dr. Day teaches courses on Instructional Leadership and Assessment for Learning online. In 2010, Dr. Day was awarded the Pearson Faculty Award for Outstanding Teaching at the University of Illinois at Springfield. His research includes publications and book chapters in a Design-Based Approach to Improving Online Courses, Using Peer Review and Analytics: From Incremental Course Changes, Developing Communities of Inquiry in Online Courses, and most recently, Pedagogical Approaches to Massive Open Online Courses (MOOCs).

Extended Abstract

INTRODUCTION

Improvement in the numbers of young Americans achieving a postsecondary degree has been a national priority for over a quarter of a century (Arnold, 1999; Shapiro, Dundar, Yuan, Harrell & Wakhungu, 2014), with little improvement seen. Indeed, a myriad of studies in the last decades of the 20th century tested the assumptions of theories concerning the reasons why students drop out of higher education institutions (Bean & Metzner, 1985; Mallette & Canrera, 1991; Munro, 1981; Tinto, 1987) to develop models of student progression.

Likewise, there is a substantial body of literature that has examined determinants of course non-completion (Juhong & Maloney, 2006; Ishtani, 2006; Jia, 2014; Montmarquette, Mahseredjian, & Houle, 2001; and Wetzel, O’Toole, & Peterson (1999), especially as regards online learning (Boston et al., 2009; Clay, Rowland, & Packard, 2008; Morris, Wu, & Finnegan, 2005; Rovai, 2003). Most recently, learning analytics are being applied to help online educators address undergraduate attrition (Baepler, & Murdoch, 2010; Barber, & Sharkey, 2012: Campbell & Oblinger, 2007).

An emerging strategy for enhancing postsecondary outcomes is to measure the patterns by which students reach and move through intermediate stages of degree completion. One of the issues identified as contributing to attrition is poor performance in gateway courses (Koch & Pistilli, 2015). The Gardner Institute (2015) has identified “gateway courses” as: foundational, credit-bearing, lower division courses, for which large numbers of students are at risk of failure and which accordingly stand as “gatekeepers” to further study and degree completion. Indeed, researchers have found that retention in these courses is strongly correlated with successful degree completion (Cabrera, Burkum & La Nasa, 2005; Herzog, 2005; Moore & Shulock, 2009; Offenstein & Shulock, 2010).  Koch and Pisitilli (2015) add that “courses with high rates of unsuccessful outcomes (DFWI rates) ‘kill’ a student’s grade point average (GPA), motivation, and academic progress” (p. 3). The problem of gateway courses is especially pernicious in online environments, and online educators are attempting to address the issue, primarily through course redesign (DePaul University, n.d.; Education Advisory Board, 2016).

The research reported in this paper looks at the issue of gateway courses from a slightly different perspective. It takes the position that redesign alone may not address problems with gateway courses.  It asks whether all gateway courses are equally detrimental to student success and/or whether the detrimental effects of poor performance in gateway courses apply equally to all students.  Answers to such questions are important so that institutional resources available for improving courses or advising and placement practices can focus on those areas in which the problems are most critical.

METHODS

Data was taken from student records for all degree-seeking undergraduate students enrolled in courses from the Fall 2009 semester through the Fall 2013 semester at a small, Midwestern university. Data sources were the grades posted at semesters’ end for these students who were grouped according to whether they began at the university as native freshman or they transferred into it.  These groups were in turn divided.  Native freshman were labeled as either enrolled in the honors program (Honors) or not (Native).  Transfer students were divided into those enrolled in regular, on-ground programs (Transfer) and those enrolled in fully online programs (Online Transfer).  Grades were also grouped by the stage in the students’ academic life cycle (first term, second term, second year, third year, or later) when they were assigned.  The outcome measure for this study was retention which was defined as persistence (re-enrollment or graduation) into the next term.

Persistence rates were calculated for all students by student types and the stage in their academic life cycle in which grades were assigned.  Binary logistic regression was used to explore the effects of differing combinations of student characteristics on persistence, and gap analyses (quantifying the difference between expected and actual outcomes) were used to explore the effects of particular courses and categories of courses on the same

RESULTS

There is no question that withdrawing from or a getting a D or an F in any course (D/W/F) had a negative effect on the persistence of students in the undergraduate population studied.  The data shows that 89.5% of all students on average persisted from one term to the next.  Among students without D/W/f’s, however, that rate rises to 94.7%.  For students with even one D/W/F, the persistence rate falls to 77.0%.  Overall persistence rates, moreover, vary by student type and by stage in the degree cycle

The persistence rates of Honors students in the population studied were greater than those of Native students (93.6% vs. 90.4%), greater among general Transfer students than among Online Transfer students (90.1% vs. 85.9%).  Persistence rates got larger for all categories of students as they progressed through their academic careers (1st term = 85.2%; 2nd term = 89.6%; 2nd year = 92.2%; 3rd year = 93.0%) until they start to decline at the fourth year or later (87.6%). While retaining pretty much the same patterns, however, the persistence rates are considerably lower for students who have any D/W/Fs, and considerably higher for those who don’t receive any

These results suggest that the impact of a getting a D/W/F in particular course may well be different for different kinds of students at different points in their academic life cycle.  Thus, instead of simply calculating the D/F/W rates for particular courses to identify “gateway courses”, it might make sense to additionally use binary logistic regression to develop a predictive model and to calculate the probability of the particular students in any course receiving a D/F/W based on student records as known at the start of the course. The difference between the actual D/F/W rate in the course and the D/F/W rate that would be expected for the particular group of students who are registered produces a gap analysis.  The gap analysis can reveal those courses in which students are performing worse than expected as well as those in which students are performing better than expected.  Indeed, the possibility exists that some courses with high D/F/W rates are actually performing better than expected given the students enrolled in them.

Moreover, while D/F/W grades are an obvious impediment to student progression, it is possible and illuminating to examine the impact of such grades on retention at the course level, broken down again by student type and point in the student life cycle.  Again, a gap analysis can be used to examine the difference between expected persistence and actual persistence of students receiving D/F/Ws in these courses

SIGNIFICANCE

There are at least three things one can and should take away from these findings. The first is that it is not possible to redesign gateway courses to serve all of the students all of the time

The second is that redesign efforts should be directed to where they can have the greatest positive impact. No matter what we do, there will be courses with high D/F/W rates simply because of the nature of their content and the preparation of the students who must take them. However, some gateway courses defy expectations while others increase them.  It is these latter courses on which we should focus redesign efforts

The third take-away is that placement matters, and it should be data-based.  The results of this study revealed many instances of courses that were clearly more effective for certain types of students at particular times in their academic careers, as well as courses that were especially damaging to particular students at specific times.  An obvious, inexpensive and effective remedy would be to place students, whenever possible, in course where and when they would have the greatest chance of success

Although the results reported in this paper are clearly limited to the undergraduate population at the university studied, they also plainly suggest that the effects of gateway courses are mediated by student types and academic stages.  D/F/W grades and gateway courses do impact student retention and progression, of course.  Thoughtful examination requires, however, adjusting observed course D/F/W rates for the characteristics of the students enrolled. It is essential to identify courses whose performance issues are real, not apparent, so that institutional resources available for improving courses or advising and placement practices can be focused on those areas in which problems actually exist.  And it is equally critical that students are placed in courses based on what we know about optimizing their chances for success.