Getting More Quickly to the Impact of Learning Technologies: The Rapid Cycle Evaluation Approach

Concurrent Session 1

Session Materials

Brief Abstract

Research on learning technology efficacy lags far behind the explosive growth of available products.  Rapid cycle technology evaluation is an emerging approach to generating timely evidence on the effectiveness of learning technologies for specific contexts and learners. Presenters will describe this approach and discuss possible applications in postsecondary education.

Presenters

Rebecca Griffiths is a principal education researcher at SRI Education, a non-profit research institute. She specializes in blending qualitative and quantitative methods to understand the efficacy and cost-effectiveness of technology-enhanced instruction in postsecondary education. Her work also examines organizational and systemic facilitators and obstacles to adoption of educational technologies. Griffiths has led large-scale research and evaluation projects involving numerous institutional partners and stakeholders. Her work spans multiple sectors of postsecondary education, including research universities, regional comprehensives, community colleges, and liberal arts colleges. Specific areas of focus are college readiness, open educational resources, and hybrid/blended learning.

Additional Authors

Dr. MJ Bishop directs the University System of Maryland’s William E. Kirwan Center for Academic Innovation, which was established in 2013 to enhance USM's position as a national leader in higher education transformation. The Kirwan Center conducts research on best practices, disseminates findings, and supports the system’s institutions as they develop innovative academic programs. Prior to USM, Bishop was Associate Professor of Instructional Design and Technology at Lehigh University and Director of the College of Education’s Teaching, Learning, and Technology Program. Author of numerous peer-reviewed publications, her research explores how instructional media might be designed and used more effectively to improve learning. She has received several awards for her research and innovative teaching, including Lehigh’s prestigious Stabler Award, conferred by students to faculty who have mentored them to excellence in their chosen field as well as excellence as human beings and leaders of society.

Extended Abstract

Faculty and administrators can choose from a growing number of educational technologies, ranging from comprehensive courseware to applications targeting specific student skills or content knowledge, to tools aimed at supporting faculty productivity and new teaching practices.  Many of these technologies – especially newer ones – lack robust evidence to support their claims of effectiveness. And when evidence exists, it may not be relevant for your institution or your students.  As a result, decisions often have to be made about technology adoption based on marketing materials, peer recommendations, and instinct.

Moreover, few institutions have supports in place to systematically evaluate the effectiveness of learning technologies.  It is typically left to individual faculty members to try to gauge whether a new resource “worked” as intended, or whether different approaches to implementation might have produced better results. Faculty members and departmental committees may compare aggregate course outcomes before and after the introduction of a new technology, but these evaluations rarely control for students’ backgrounds or analyze the impacts on subgroups of students, such as underrepresented minorities or students with lower (or higher) incoming GPAs.  As a result, decision makers may have a general sense of whether course outcomes rose or fell with the introduction of the technology, but not enough information to determine why or how best to move forward.

Rapid cycle technology evaluation is an emerging approach to generating timely evidence on the effectiveness of learning technologies -- along with strategies for implementing them -- for institution-specific context and students. This approach has been championed by researchers involved with the Brookings Institute’s Hamilton Project for a variety of social science research domains, including education. Rapid cycle evaluations have been embraced by the U.S. Department of Education as a way to address the dearth of rigorous evidence supporting learning technology products in K12 education.

The core concept of rapid cycle evaluation is to design tests that address narrow research questions and can be implemented over short time frames and at low cost. They aim to answer specific questions about the impact of a technology (or implementation approach) for a specific set of users and in a specific educational context. These evaluations are more likely to point to another set of questions than to provide conclusive answers, and can play a key role in continuous improvement processes.

Data for rapid cycle evaluations can come from administrative systems and course assessments. Usage logs from learning technologies themselves can also be a valuable source of data on outcomes, as well as for understanding how students and other users interact with the technology and what it reveals about growth, engagement, and learning behaviors.  

How might these methods be used by postsecondary faculty, departments, and institutional researchers to inform decisions about technology adoption and enable systematic improvement in implementation? MJ Bishop and Rebecca Griffiths will describe the tools and models being developed through the Department of Education initiative and an effort sponsored by the Bill & Melinda Gates Foundation to develop tools that can be used in postsecondary institutions to evaluate courseware adoptions.

The presenters bring two different perspectives: Griffiths serves as SRI’s principal investigator for the Department of Education initiative and has worked with colleges and universities to conduct numerous effectiveness studies of learning technology. Bishop leads a prominent university system center for academic innovation. In this capacity she works closely with faculty and staff across a diverse system of public universities and colleges and has a deep understanding of prevailing practices for evaluating effectiveness of learning technology.  

The presenters are eager to hear from audience members about the challenges various constituents face in evaluating technology products and how new tools and resources could help to close the gaps.  They will spend a significant portion of the session asking audience members to share experiences and discussing ways that rapid cycle evaluation might be used to enable more evidence-based decision making on their campuses.

Some specific topics include:

  • How are rapid cycle evaluations different from traditional efficacy studies?
  • What are some applications in postsecondary education?
  • Who within postsecondary education could take advantage of this approach?
  • What kinds of analytical tools would be most useful to these constituents?
  • How can system data from technology providers be used in rapid cycle evaluations?