The Online Student Learning Experience: A Comprehensive Assessment Approach

Concurrent Session 8

Session Materials

Brief Abstract

Explore a comprehensive assessment approach to the online learning experience and the models and theories by which it’s supported. Engage in discussions about effective assessment practices, challenges and opportunities throughout the assessment process, the practical application of quality assurance methods and tools, and additional assessment and evaluation activities and resources.

Presenters

Julie Hewitt, PhD, EdS, PMP, CSM is a learning and technology advocate. She is available for consulting and contract work related to higher education programming, curricular and operational assessment. Hewitt has almost 20 years of experience among higher education, K12 education, and training and development in industry. Hewitt's research interests include faculty development, teaching and learning, course design, and technology. Most recently Hewitt was with UW-Platteville where her diverse knowledge and skills with online faculty and course management, assessment, and research were leveraged. Hewitt was the lead on the Online Faculty Academy (OFA) and was collaborating with others on the exploration of Open Educational Resources (OER) and affordability. Prior to UW-Platteville, she served as a part of campus administration as a Dean of Student Services. Hewitt also has experiences in higher education as an Academic Program Director, an online graduate adjunct faculty member, and a face-to-face undergraduate faculty member. Her prior work experiences extend to the corporate world as a business and training systems analyst for a large aerospace company and K-12 public education as a secondary education teacher, coach, and district technology coordinator. Hewitt is a member of the Online Learning Consortium (OLC), the eLearning Guild, United States Distance Learning Association (USDLA), and Project Management Institute (PMI).
Education: Master of Organizational Leadership Master of Business Administration Work Experience: Academic Advising and Outreach Manager • Responsible for academic and outreach coordination by supervising academic advisors in the Distance Learning Center • Aids in the recruitment, hiring, and training of new academic advisors • Oversee academic success by monitoring student success and engagement • Oversee professional development budget and approval for all advising staff • Responsible for participation in enrollment management and effective targeted communication • Provides leadership and responsibility for policy development, interpretation, and execution • Triage student complaints and discipline-related processes and procedures • Facilitate collaborative discussions with on-campus offices such as Financial Aid, Registrar, and Veterans Office • Oversee and support external relationships with various institutions and organizations • Assist Distance Learning Center Director and Alternative Delivery System Executive Director with student communications and other special projects or assignments • Representative on the Student Advisory Board for distance students • Functional Analyst for Student Records and Academic Advisement for the Distance Learning Center Specialties: Outreach and advising, leadership, and assessments.

Extended Abstract

Background

The Distance Learning Center (DLC), which began to offer the University of Wisconsin-Platteville’s first online programs in 1998, will be celebrating 40 years in existence during the 2018-2019 academic year. As technology and assessment have evolved, our staff’s commitment to serving students who require flexibility while earning their degree remains steadfast. Through sound course design, passionate staff and faculty, rigorous content, and continual improvements; we have demonstrated success growing enrollments and delivering award-winning programs. We credit much of our success to our comprehensive assessment approach, which directly supports the unit’s mission and goals as well as a commitment to offering a quality educational experience for all stakeholders.

Practical Application and Audience Engagement

Participants of this session can expect to learn about the models and theories that support a comprehensive assessment approach. Participants can also expect to engage in sharing about challenges and opportunities throughout the assessment process and the application of well-vetted tools and resources such as the Sloan Quality Scorecard, online course design and facilitation rubrics, and other predominant online student services guidelines and standards. Technologies will be used to facilitate sharing and discussion throughout the interactive presentation.

In 2012 Bresciani noted, “the ability to incorporate assessment and evaluation into day-to-day practices is emerging as a necessary competency for student affairs practitioners” (p.321). With limited resources, changes in policy, and the overall environment of higher education; assessment and evaluation are crucial to our everyday practices to provide necessary services, support, and educational experiences in a fiscally and ethically responsible manner. The information and knowledge gathered as a component of assessment and evaluation is used as a driving force for the changes made to improve processes and procedures, which supports the student learning experience.

Effective Practice for Validation, Improvement, and Innovation

Student services theories and models have been an important aspect of the evidence-based practices implemented through DLC services and its collaborative efforts with other departments within the university. Through a review of current literature on changing educational environments, the DLC’s practices have been found to still align with many of the fundamental principles of student affairs practices, which have remained consistent over time. While we have some affirmation of our practices, we continue to assess our activities.

Continuous improvement is evident in the varied assessment and evaluation practices implemented in the DLC. Different data collection methods are used to collect quantitative and qualitative data. Survey tools are used in the areas of advising, student services, alumni engagement, course and instructor evaluation, and program completion. Focus group methodologies are used with program advisory boards, student advisory boards, and the alumni committee. Special focus groups or interviews may be completed from time to time for further specific information discovery or clarification purposes. Internal quality and validation processes also exist. Each of these data points are analyzed to allow for data-driven decisions to be made. When the qualitative data is triangulated and the results are integrated, the comprehensive approach provides knowledge upon which a strategic direction can be informed.  

The opportunity to innovate our services is important to us as we support learners through different methods. Without recurring evaluation and looking for opportunities to streamline, resources would not be available to hone our practices and services for the non-traditional student populations we serve.

A Sampling of Our Evidence

The following samples are only a brief glimpse of our efforts and results. In the presentation, more information will be provided on additional areas of assessment and the corresponding results, as well an active discussion of the challenges and opportunities along the way

In 2010, as a result of comprehensive assessment, one of the most impactful changes was the movement towards a one-stop-shop philosophy in the DLC. Previous models required at least three staff members to work with students during various points in the student life cycle; often times contacting them for similar needs. After reviewing the results of student services and advising surveys in conjunction with informal feedback received from students and staff, the student services team determined a model change was needed to improve the student experience, and increase student satisfaction. The efforts were validated by the DLC’s receipt of a Certificate of Merit from the National Academic Advising Association (NACADA) for a successful student advising department in 2013.

In the administration area, the DLC worked to implement student information system integration. The DLC once used a third party vendor for registration, student status tracking, and payments; but in 2011, the DLC moved to the full utilization of PeopleSoft, called the Pioneer Administrative Software System (PASS). This overall assessment lead to the entire campus going through a re-implementation of PASS. All areas from admission through graduation have been positively impacted. In 2014, as a result of the re-implementation, the DLC received outside recognition for its streamlined processes through the use of technology and other process improvement such as:

  • Reduction in staff hours of manually enrolling students, tracking financial aid, and updating contact information
  • Student access to grades, course registration, and financial accounts in one system
  • Leveraging functionality of PASS as a CRM, leading to a fiscal efficiency

Recent initiatives identified through the analysis of assessment results have also led to grant opportunities for the DLC. One of these new initiatives is the Advising Community of Support (CoS). Through assessment, it was found that students were not feeling as engaged or as connected to the university as expected. In 2015, the DLC was awarded $20,000 from the UW System Innovation Fund to create and support an online CoS. This CoS is designed to promote student engagement, be a high impact practice on student retention, and strengthen the student to student and student to advisor relationship. While the overall success of the CoS is yet to be determined, early assessments have shown a positive impact regarding student engagement outcomes.  

The DLC also recently received a grant of $630,410 from the National Science Foundation to increase access to online master’s degree programs in the STEM (science, technology, engineering, math) disciplines; this initiative was determined through the assessment of enrollment data. As a requirement of the scholarship, recipients actively participate in the STEM Scholar Master’s program., which provides increased engagement with support services.

OLC’s Five Pillars of Quality for Effective Practice

Learning Effectiveness: A partnership amongst the operational and student services areas for online learners and the academic areas is imperative to gauge learning effectiveness. DLC instructional designers work closely with faculty to develop quality teaching and learning experiences within the online environments. The DLC facilitates data collection from students and faculty on the learning outcomes and experience.

Scale: With the cost-recovery nature of our budget model, our commitment to streamlining processes while sustaining or increasing student services and satisfaction is a nonnegotiable aspect of what we do. Initiatives such as the re-implementation of PASS and the paperless movement are just a couple of examples of the scalable efforts which are have reduced costs, increased communication, improved response times and the accuracy of information collected.

Access: Our definition of access expands from technology to services. Each aspect is considered in our assessment practices. Collaboration with academic departments for course sequencing, accessibility of course design from a technology and assistive perspective, and access from diverse locations are evaluated on a recurring basis. Qualitative and quantitative results demonstrate progress such as the offering of new programs and courses and mobile design considerations.

Faculty Satisfaction: Online instructors require subject matter, technological, and pedagogical knowledge and competencies, each requiring significant effort and resources to employ in an effective manner in the classroom. The DLC has established processes and facilitates assessment to support instructors with the technological and pedagogical aspects of online course delivery. A DLC course resource team supports faculty from course authoring through course and instructional delivery. Online instructor retention demonstrates satisfaction in this area.

Student Satisfaction: The DLC examines the complete student life-cycle looking for opportunities to improve the online learning experience for all stakeholders, however, the primary focus is the student. From recruitment through graduation, each step and interaction is carefully assessed. Evidence exists through student enrollment, retention, and survey data of online learner satisfaction.  

Closing statement:  Assessment results provide evidence to support strategic direction and innovative initiatives. The scope is focused on the comprehensive assessment of online learner experience and success. The scalability and replicability of DLC practices continues to be essential as a result of growth and changes in technology and higher education, and the budget model under which we operate. The impact of our comprehensive assessment process has been wide ranging as demonstrated by the achievements and recognition received. The five pillars provide a framework on which we can anchor our assessment strategies.

Commitment

A commitment to the UW-Platteville DLC mission, goals, and students is demonstrated each day by dedicated DLC staff and faculty. The ongoing comprehensive assessment and evaluation practices have proven to be worthwhile efforts. The DLC intends to continue to be intentional and strategic with assessment in the delivery of quality educational experiences and services to online learners.

References

Bresciani, M. J. (2011). Assessment and Evaluation. In J.H. Schuh, S.R. Jones, & S.R. Harper S.R. (Eds.), Student services a handbook for the profession (5th Ed.) (pp. 321). San Francisco: Jossey-Bass.