Data vs. the Hunch: Letting Data Lead Innovation in Course Design

Concurrent Session 9
Streamed Session

Watch This Session

Session Materials

Brief Abstract

Is there a battle between innovation and best practices? In this session, we demonstrate how student feedback and other data helped confirm some assumptions about redesigning an engineering course, but also opened up new windows to innovation and viewing common problems like engagement and faculty time constraints from different angles.


Chris Willis, NC State - DELTA's Coordinator for Learning Analytics and Assessment, provides leadership on the evaluation and assessment of course redesign and educational technology projects to demonstrate ways in which learning technologies, pedagogy, and teaching best practices support and improve student success and engagement. He manages project data collection, analysis, and reporting, and provides training, guidance, and support for the effective integration and use of quantitative and qualitative data collection methods for evaluating and assessing DE and blended course redesigns. Chris holds Master's degrees in Survey Research, Education, and Public Administration, as well as a B.A. in Psychology. He is also a current PhD student in the Education Leadership, Policy, and Human Development program at NC State University.
Jakia Salam is an Instructional Designer at DELTA where she works as a project lead and instructional designer for DELTA Grant projects. She works closely with faculty, new media team and project coordinator team to design, develop, and evaluate on-line, hybrid and face-to-face courses. Jakia is a technology enthusiast and always looking for ways using innovative technologies to develop pedagogically sound courses and address instructional challenges. Jakia provides leadership in the area of instructional design, including the creation of engaging, effective and pedagogically sound instructional materials and activities, the development of assessment and evaluation techniques and process, and incorporate learning technologies based on sound pedagogy and effectiveness. Prior to joining DELTA, Jakia worked in various roles including as a Senior Instructional Designer at UNC Chapel Hill, an Instructional Technology Consultant at UNC Greensboro, a full-time faculty at Guilford Technical Community College, a Chemical Engineer and Application Developer. Jakia holds a bachelor’s degree in Chemical Engineering from Bangladesh University of Engineering and Technology and completed her Master of Science in Computer Science from UNC Greensboro in 2002. She also obtained a Graduate Certificate in e-learning from the College of Education at NC State in 2013.

Extended Abstract

This session focuses on using qualitative and quantitative data to create innovative redesign solutions for a traditional large lecture undergraduate-level environmental engineering course at NC State University. The project goals were to reduce instructional challenges, improve students’ learning experience, and increase engagement and excitement about environmental engineering. This session will demonstrate how we used baseline student attitudinal data and input from faculty nationwide to guide instructional design and decision making. We show how data can lead to innovation even when it might contradict assumptions grounded in course design best practices.

Learning Outcomes

  • Use student feedback to make pedagogically sound, data-driven design decisions
  • Analyze qualitative and quantitative data to verify or reject course design assumptions
  • ​Balance innovation with more traditional course design elements

The Redesign Project

NC State University’s Distance Education and Learning Technology Applications (DELTA) office “collaboratively applies expertise in innovative technologies and pedagogies to solve instructional challenges in an efficient, effective, and service-oriented environment, with the overarching goal of helping faculty build student success.” This course redesign project was funded by a DELTA grant, which helps faculty looking for new ideas to develop courses/materials using innovative strategies and technologies, incorporate best practices in course (re)design, and/or explore new ways to engage students with course material.

The project team redesigned a large course, Fundamentals of Environmental Engineering, from a face-to-face lecture to a partial blended learning model designed to improve students’ learning experience as well as increase student engagement and excitement about Environmental Engineering. Key instructional challenges included a diverse student population with low interest/engagement, a disconnect between homework and exams, a wide array of course topics, limited teaching assistant support, and rotating instructors. The project team, including faculty, started this redesign project with certain assumptions about how to redesign this course to accomplish these goals, mainly that a radical change to this very traditional lecture course was necessary to create a positive impact on students.

With the key challenges in mind, the team aimed to explore new technologies and learning practices to improve student engagement/support; increase interest in the field; improve faculty/TA time investments; develop students’ problem solving skills; and improve instructional consistency across semesters, among other goals. However, because DELTA aims to create a culture of inquiry, we created three primary data collection tools to gather data that could help make instructional design decisions by either verifying or rejecting our collective assumptions, and to foster creative solutions that kept in mind faculty time commitment, sustainability, and reusability. This baseline data will be compared to similar data gathered after the course redesign implementation in fall 2018, to assess its impact on student engagement and achievement.

Data-Driven Instructional Design

In addition to course evaluations, registration data, and some faculty anecdotal evidence, we utilized three tools prior to finalizing the redesign plan:

  • Classroom Observations: Conducted by the instructional designer and multimedia specialist to provide multiple perspectives on faculty teaching style, student engagement with the class/materials, use of multimedia, students’ learning styles, and course activities.
  • Baseline Student Survey: Administered to current students, it gathered feedback on their course experience, perceptions of the required homework and activities, and how much they think they have learned in the course.
  • Colleague Survey: Administered to faculty peers nationwide, it asked those teaching a similar course about its place in the curriculum, students’ perceptions of the course, effective teaching strategies, and course redesign.

Working closely with the faculty to reexamine how to present material to students, we used these tools to create a data-driven instructional design plan. In this session, we show how the data we gathered helped confirm some assumptions (e.g., increasing active learning), but also rejected some assumptions and opened up new windows to innovation and viewing common problems (e.g., engaging diverse students) from a different angle, including finding new ways to incorporate established instructional tools. This presentation will outline how to balance traditional teaching methodology (lectures, case studies) with innovation and cutting edge technology (active learning, animation/storytelling, mobile technology).

Presentation Activity

The presentation will include a combination of crowd polling and table teams, so attendees will be able to share their views and apply what they are learning to a sample redesign context.