Frameworks and Quality Indicators for assessing and evaluating e-Learning programs

Concurrent Session 5

Brief Abstract

This presentation provides an overview of different frameworks, benchmarks, guidelines and instruments that exist to assess the quality and effectiveness of e-Learning programs. Elements of multiple frameworks and instruments are combined to identify seven quality indicators that can support Elearning leaders to assess and evaluate quality in online offerings.

Presenters

Lead Presenter: Florence Martin, University of North Carolina Charlotte

I am an Associate Professor in the Instructional Systems Technology program at the University of North Carolina, Charlotte. I received my Doctorate and Master's in Educational Technology from Arizona State University. I have a bachelor's degree in Electronics and Communication Engineering from Bharathiyar University, India. Previous to my current position, I taught at University of North Carolina Wilmington for seven years. I also worked on instructional design projects for Shooolini University, Viridis Learning, Maricopa Community College, University of Phoenix, Intel, Cisco Learning Institute, and Arizona State University. I worked as a co-principal investigator on the Digital Visual Literacy NSF grant working with Maricopa Community College District in Arizona. My research focuses on designing and integrating online learning environments (OLE) to improve learner motivation and engagement to achieve effectiveness in learning.

Co-presenter: Swapna Kumar, University of Florida

Dr. Swapna Kumar is a Clinical Associate Professor of Educational Technology at the School of Teaching and Learning, College of Education, University of Florida. She directs the online doctorate in Educational Technology and studies online and blended learning in higher education. Her current research is focused on quality assurance in online programs and online mentoring in graduate education. Details of her publications can be found at http://www.swapnakumar.com

Tags

Institutional Strategies and Innovations Administration Institution Institutional change Institutional culture Institutional effectiveness Leadership Organization Program Quality assurance Quality matters Readiness Strategies Global Institutional strategies Leadership Online Quality

Extended Abstract

The tremendous growth in online learning, competition among institutions offering online courses or degrees, the need to offer the same level of quality in online learning in par with face to face learning, and continual scrutiny of quality in online courses has established a need for institutions to use quality standards to guide the design and facilitation of online learning. Quality assurance and accountability for higher education institutions are addressed by accreditation agencies in various countries. E-Learning leaders need to show effectiveness and quality of their online programs to these accrediting agencies, and simultaneously have to market their online programs to a large audience, maintain high quality in their courses, and demonstrate learning outcomes. Also, internally e-Learning leaders are expected to continuously improve programs and maintain cost-efficiency. For these various purposes, it is essential for them to be aware of the various frameworks that exist and can assist them in assessing and evaluating quality in their online offerings.

Frameworks, benchmarks and guidelines for quality assurance in E-learning.

The following frameworks, benchmarks and guidelines for quality assurance will be reviewed during the presentation.

1. Online Learning Consortium (Formerly Sloan-C) Pillars http://onlinelearningconsortium.org/about/quality-framework-five-pillars/

2. CHEA Accreditation and Assuring Quality in Distance Learning - http://www.chea.org/pdf/mono_1_accred_distance_02.pdf

3. Quality on the Line: Benchmarks for Success in Internet Based Distance Education http://www.ihep.org/sites/default/files/uploads/docs/pubs/qualityontheline.pdf

4. Open eQuality Learning Standards (Canada) www.futured.com/documents/OeQLsMay2004_000.pdf

5. Asian Association of Open Universities (AAOU). Quality Assurance Framework. http://www.aaou.org/ images/fi les/AAOU%20Quality%20Assurance%20Framework.pdf

6. Australasian Council on Open, Distance and e-Learning (2007). ACODE benchmarks for e-Learning in universities and guidelines for use.

7. European Association of Distance Teaching Universities (EADTU)

8. The National Association for Distance Education and Open Learning in South Africa - http://www.nadeosa.org.za/tags/quality-assurance

Instruments for quality in online courses and programs 

1. Online Learning Consortium Quality Scorecard http://onlinelearningconsortium.org/consult/quality-scorecard/

2. Blackboard Exemplary Rubric - http://www.blackboard.com/getdoc/7deaf501-4674-41b9-b2f2-554441ba099b/2012-blackboard-exemplary-course-rubric.aspx

3. Quality Matters (QM) - https://www.qualitymatters.org/rubric

4. iNACOL -  http://www.inacol.org/

On analyzing these frameworks and standards, some common indicators were identified and listed to provide guidance to e-Learning leaders on areas to focus while working on enhancing quality in online learning.

Quality Indicators derived from existing standards, benchmarks, frameworks and instruments

Figure 1. Indicators for Institutional Online Learning Quality will be shared at the presentation.

  • Institutional Support

E-Learning leaders are expected to get buy-in from the leadership and administrators to implement online learning. Without the support of the institutional leaders, online learning initiatives are not successful. Therefore, e-Learning leaders need to form strategic partnerships with key stakeholders in the implementation of online education. Some of the ways by which the e-Learning leaders can facilitate support to online programs are:

  • Advocating budget support in the university’s commitment to online programs.
  • Providing course development incentives for faculty who wish to develop quality online courses
  • Fostering understanding and implementation of quality frameworks to support online learning
  • Implementing awards for quality online courses
  • Updating policy statements to reflect university’s commitment to online programs.
  • Developing an institutional E-learning strategy
  • Communicating strategic directions of the institution
  • Collaborating with the institutional entities to receive all the appropriate accreditation and support for online programs.
  • Technology Infrastructure

Online learning heavily relies on technology infrastructure. E-Learning leaders have to take decisions on the purchase of technology (both hardware and software), implementation and services for the institutions online learning needs in the following manner:

  • Ensuring the availability of technology infrastructure and staff to host E-learning initiatives
  • Selecting and hosting the appropriate Learning Management System for the University
  • Aligning student information systems with online learning infrastructure and procedure
  • Providing infrastructure for the use of digital resources for both instructors and students (e.g. a video server)
  • Selecting and adopting synchronous tools for the university
  • Providing a Multimedia studio, technologies and support staff for faculty development of E-learning materials
  • Providing Multimedia software for faculty use
  • Course Design

E-Learning leaders need to be familiar with instruments that measure effective instructional design approaches and pedagogical elements essential for student satisfaction and learning outcomes in online courses. They also have to ensure that their organization or unit

  • Hires and trains instructional designers to assist with course design
  • Is familiar with frameworks that guide course design
  • Provides faculty with professional development opportunities focused on E-learning
  • Is familiar with quality criteria for online course design including, instructional alignment between objectives, assessment and instructional material, measurable goals and objectives and formative and summative assessments for grading
  • Implements quality control procedures before students access online courses
  • Provides accessible and usable instructional material
  • Learner and Instructor Support

Technical, pedagogic and facilitation support is needed for instructors during the design and delivery of online courses. Technical and administrative support is needed for online learners. Different aspects of course design support for instructors are addressed in the previous section, and faculty incentives for developing online courses is listed above under institutional support. Additionally, E-Learning leaders are responsible for ensuring such support structures are in place for instructors and students, in the form of:

  • Academic Technologies Support for instructors
  • Advising and administrative processes (e.g. registration) for students
  • Information literacy support for students
  • A Help desk system to support faculty and staff
  • Web resources with FAQs and tutorials to guide faculty and staff
  • Online writing and other academic resources for students
  • Course Assessment and Evaluation

Assessment and evaluation in online learning brings several challenges, including a need for new policies, guidelines and procedures. E-Learning leaders need to be familiar with various online assessment methodologies.

  • E-Learning leaders need to be familiar with the evaluation techniques to collect variety of data to assist with the evaluation for continual improvement.
  • Integration of Student Information System with Learning Management Systems to permit ease of enrollment and reporting grades
  • Providing online proctoring support for courses that need it
  • Providing plagiarism detecting applications that are integrated with Learning Management Systems
  • Establishing peer and student evaluation procedures for online courses
  • Ensuring online courses have the same instructional hours as contact hours in face to face courses
  • Learning effectiveness

In the end, the main indicator of measuring success is the effectiveness of online learning. E-Learning leaders have to:

  • Ensure collection of assessment and grade data that helps measure effectiveness at various levels at course, program, college and institution levels
  • Ensure collection of other learning analytics data in an ethical manner that informs of the online learner behavior and participation in the courses.
  • Ensure periodic collection of data which assists with student retention
  • Collect data on admissions and completion to identify patterns of growth in various programs
  • Faculty and Student Satisfaction

It is critical that both the online instructor and the online student are satisfied with the online learning experience. E-Learning leaders must be able to choose from different approaches to collect feedback from the instructors and the students and utilize this feedback for continual improvement and increase faculty and student satisfaction. They have to:

  • Collect student feedback on instruction, resources, feedback, interaction, support, and accessibility
  • Collect faculty feedback on resources, support, incentives, appreciation
  • Establish procedures for review and implementation of feedback.

Conclusion

This synthesis of frameworks and considerations will benefit e-Learning leaders who serve in leadership roles at various educational institutions in roles such as directors, coordinators, deans and associate deans, vice-presidents and provosts and have responsibility over e-learning/online learning/distance education in their organizations.

References

Allen, E., & Seaman, J. (2015). Grade Level: Tracking Online Education in the United States (Rep.). Babson Survey Research Group. Retrieved from http://www.onlinelearningsurvey.com/reports/gradelevel.pdf

Lorenzo, G. & Moore, J. (2002). The Sloan Consortium report to the nation: Five pillars of quality online education. Needham, MA: Sloan Center

Perraton, H. (2000). Open and distance learning in the developing world. London: Routledge.

Phipps, R., & Merisotis, J. (2000). Quality on the Line: Benchmarks for Success in Internet-Based Distance Education. Retrieved from http://eric.ed.gov/?id=ED444407

Sloan Consortium. (2002). The Sloan Consortium: The 5 pillars.  Retrieved from http://www.sloan-c.org/