Designing High-Quality Surveys for Online Education in The Digital Age

Concurrent Session 1

Session Materials

Brief Abstract

This hands-on workshop will cover the ins and outs of survey design for online education programs, including question types, response formats, layouts, pilot testing procedures, methods of delivery (social media/smartphones/other digital devices), and sampling methods with specific examples. The workshop will also introduce various survey development platforms and free resources.

Presenters

Lead Presenter: Kadriye Lewis, Childrens Mercy Hospital - Department of Pediatrics - UMKC School of Medicine

Kadriye O. Lewis, EdD, is the Director of Evaluation and Program Development in the Department of Graduate Medical Education at Children's Mercy Hospital CMH). She is also Professor of Pediatrics at the University of Missouri-Kansas City School of Medicine (UMKC SOM). Prior to coming to Children's Mercy, Dr. Lewis worked for Cincinnati Children's Hospital Medical Center (CCHMC) for more than 13 years. She played a major role in the development of the Online Master's Degree in Education Program for Healthcare Professionals. This program has developed a national and international reputation for excellence and played an important role in training future leaders in medical education. Dr. Lewis served as an education consultant to the medical center's faculty development program. She applied her educational background and academic skills to health literacy by establishing a Health Literacy Committee at CCHMC in 2007 and chaired this committee successfully for three years. She also received Medical and Academic Partnerships Pfizer Visiting Professorships in Health Literacy/Clear Health Communication grant in 2008. Along with her many accomplishments in the area of scholarly activities, she also established the e-Learning SIG in Medical Education for the Academic Pediatrics Association (APA) and chaired this group for six years. Dr. Lewis served as an education consultant for a national-level industry-sponsored project (Abbott Nutrition) on e-learning development in pediatric nutrition education for over six years. She also worked with the infectious disease team at CMH as a Co-PI for the Pfizer-funded CoVER project (Collaboration for Vaccination Education and Research for Residents). This project produced a unique training model in vaccine education for residency programs with its interactive modules that were implemented nationally at 26 institutions. Currently, she is involved in an NIH-funded grant project on genome, and various curriculum development projects for the graduate medical education programs at CMH. Dr. Lewis is active in medical education research and her scholarly interests are focused on e-learning design, implementation of innovative technologies for curriculum delivery at many levels in healthcare education, including performance-based assessment, the construction of new assessment tools as well as the improvement and validation of existing tools and methods. Dr. Lewis presents extensively at many professional meetings and conferences and has been a keynote/an invited speaker at many international and national universities. In addition, she is the Medical Education Section Editor of Annals of Medicine Journal (https://www.tandfonline.com/journals/iann20/sections/medical-education).

Extended Abstract

Surveys or questionnaires have been used for some time in all spectrums of online education from course development to program evaluation, including research purposes on many diverse topics. Recently, we have witnessed the rapid spread of surveys delivered through social media, smartphones, and numerous other digital devices that enabled us to collect/process data about human behaviors on visual scales or multi-channel formats that have never been imaginable before. Online surveys are usually structured questionnaires that students/target audience complete over a Web platform generally by a filling out a form. The main benefit of online surveys for online educators and researchers is that they increase productivity by saving time. Data is instantly available and can easily be transferred into specialized statistical software or spreadsheets when more detailed analysis is needed. While the length and format of online surveys can vary based on the purpose, developing questions that accurately assess respondents' opinions, experiences, and behaviors is a critical aspect of survey methodology.

Online survey tools have even increased their value further because online courses have shown significant growth with multiple applications to aid learning since the coronavirus outbreak. However, applying effective design principles in the survey development process is often overlooked by the online faculty although the key to a good survey is its design. Poorly designed surveys confuse respondents and yield inaccurate information, unreliable feedback, and low response rates. At this point, online educators/faculty should be cognizant of the multistage survey development process and effective design principles. Careful planning at the start of survey development will help them create accurate measures and improve data quality that will contribute to all levels of evaluation efforts of online education from teaching to research. This hands-on workshop will cover how to design and develop online surveys tailored to specific objectives for online education programs, including question types, response formats, layout, pilot testing, and sampling with specific examples. The workshop will cover the ins and outs of survey design and the best practices of survey question development with question types, response formats, layouts, pilot testing procedures, and sampling methods. In small groups, participants will be asked to critique sample surveys. They will also discuss the merits and flaws of questions and methods of delivery, e.g., social media, smartphones, and other digital devices. The workshop will also introduce various survey development platforms and free resources.

Level of Participation:

This workshop is structured to be a mutual learning experience with a combination of interactive lectures, and small group activities along with a thought-provoking discussion. The workshop will be a great refresher for an experienced faculty in survey development and for those who have little or no previous survey development experience will gain tremendous knowledge of the best approach to survey development with hands-on practice in small group activities.  

Session Goal:

The goal of this workshop is to introduce the best design principles for writing survey items that accurately represent what the online educators want to measure or capture data for their program needs.

Learning Objectives:

At the end of this session, participants will be able to:

  1. Identify the best design principles for the development of surveys tailored to specific objectives for online education programs, including question types, response formats, layout, pilot testing, and sampling

  2. Discuss techniques that optimize questionnaire design, the merits and flaws inherent in different types of questions, and additional considerations for surveys delivered through social media, smartphones, and numerous other digital devices

  3. Locate survey development platforms and free resources.

Workshop Agenda, Topics/Activities (90 Minutes)

Step 1 (5 minutes): Opening

Introduction (agenda review)

Step 2 (10 minutes): Icebreaker

  • Participants are invited to share three facts about surveys

Step 3 (20 minutes): Introduction to Survey Methods (Interactive lecture with questions and answers, and small group activity)

·What is an online survey? How are surveys used in online programs?

  • Surveys delivered through social media, smartphones, and numerous other digital devices

·Small Group Activity: Stage of development for survey questions

  • Question structure and wording

  • Sequencing of questions

  • Survey question design

  • Evaluation of survey questions

Step 3 (25 minutes): Survey question design

  • Best design principles with specific examples

  • Types of survey questions (Small Group Activity: Survey critique - Poor question vs. better question and discussion)

    • Multiple-choice questions
    • Rating scale questions
    • Likert scale questions
    • Matrix questions
    • Dropdown questions
    • Open-ended questions
    • Demographic questions
    • Ranking questions
    • Image choice questions
    • Click map questions
    • Slider questions
    • Closed-ended vs open-ended questions
  • Merits and flaws inherent in questions

Step 4 (15 minutes): How to test your survey questions to avoid common problems

  • Pilot testing

  • Sampling

  • Reliability of survey data

  • The survey response process

Step 4 (15 minutes): Summary & Key Takeaways

  • Take-home message and intended application after the workshop

  • Questions & Answers