Outcome Assessment of Learning: Content Analysis of Asynchronous Online Discussion

Concurrent Session 3

Session Materials

Brief Abstract

Designing assignments that optimize the educational value of asynchronous discussion tools requires careful consideration of strategies to evaluate learning outcomes. The current study utilized a post priori analysis of students’ responses to an online discussion board assignment to identify organically emerging indicators of learning outcomes.

Presenters

Orly Calderon, Psy.D. , is an associate professor at Long Island University, teaching courses in research design and data analysis, assessment and evaluation and psychopathology. Dr. Calderon earned her B.A. in psychology and education from Long Island University, her M.A. in developmental psychology from Teachers College, Columbia University, and her Psy.D. in school and community psychology from Hofstra University. For the last 19 years, she has been combining her teaching and scholarly work with clinical work, focusing on psychological evaluations. Dr. Calderon’s research reflects her clinical interest in at-risk populations. She has utilized her expertise in empirical approaches to study immigrant youths, youths with ADHD, individuals with disabilities, and people with chronic illness, as well as to evaluate best ways to teach future mental health professionals about the needs of these populations. Concurrently, Dr. Calderon specializes in program evaluation and has been involved in developing tools and strategies for pedagogical and clinical outcome assessment at LIU, as well as other organizations in the USA and abroad. For the last few years her work has focused on researching students’ and faculty responses to blended and online learning, as well as on developing pedagogical approaches for integration of information technology tools in and out of the classroom. Her research emphasizes the role of input from faculty and students in shaping the design, delivery, and assessment of distributed learning. Dr. Calderon’s work in this area has been presented and published in various peer-reviewed forums and journals, and has served as a model for evaluation in other institutions.

Extended Abstract

Use of learning management systems (LMS) has become prevalent across the continuum of education formats (Allen and Seaman, 2013). Specific asynchronous tools such as the discussion board have been shown to be effective tools for student-instructor as well as student- student communication (Calderon, Ginsberg and Ciabocchi, 2013; Ke, 2013); constructivist learning (Ke, 2013; Lane, 2014), and critical thinking skills (Williams & Lahman, 2011).  However, the literature suggests several challenges in designing, and especially in assessing online discussion assignments. There is a lack of systematic comprehensive analysis of online discussion assignments for the purpose of identifying direct learning dimensions that can be objectively evaluated. Current work in this area includes a priori parameters of critical thinking skills (Lai, 2012), focusing on  instrumental dimensions such as length of communication instances (Brooks & Bippus, 2012); and indirect measures of learning such as analysis of students’ reaction to the  assignment (Mathews & La Tronica-Herb, 2013) rather than an assessment of learning that has occurred as a result of the assignment, or analysis of instructors’ perceptions of students’ skills development in course  (Klisc, McGill & Hobbs, 2009).  Seeking to fill this gap, and to gain a better understanding of learning dimensions that emerge in response to an online discussion assignment, the current study utilized a post priori content analysis of students’ responses to an online discussion prompt in order to identify organically emerging indicators of learning for the purpose of creating an evaluation rubric for such assignments.  The analysis was guided by the following research question:  What are the organically emerging patterns in content and structure of discussion board posts that can indicate quality of learning?

This presentation describes the assignment, explains the data analysis methodology and presents a summary of the results with the hope that participants will Participants will become familiar with challenges associated with outcome assessment of online discussion assignments, learn  strategies for content analysis of students’ post on a discussion board learning assignment, and  explore the relevance of the method described here to their own teaching and outcome assessment of students learning in an online environment. 

 Participants

The participants included 49 graduate students in a large private university in New York, USA who were enrolled in a psychopathology class. The participants (mostly females) represent three cohorts (spring semesters of 2012, 2013 and 2014, respectively), all of which have been evaluated on the same assignment.  The students were required to a case study (adapted from Morrison, 1994 and identical across the three cohorts) and respond to a prompt by instructor regarding the proper diagnosis and treatment plan /or next steps in addressing the issues presented in the case study. Students were divided into groups and were instructed to respond to each other within their respective groups and to reach a consensus in the group in regards to the diagnosis.  Students were given 5 days to complete the assignment. A total of 83 responses were analyzed.  To maintain confidentiality, all responses were de-identified and numerically coded. 

The Assignment

Consistent with Ke (2013) and with Williams and Lahman (2011), who advocate a group approach in asynchronous online discussion assignments to promote peer-to- peer interaction, the assignment in this study was designed as a group discussion. Each class was randomly divided into 3 or 4 groups, typically consisting of three to four students.  Students had access to their own group’s asynchronous discussion thread, each of which also included the instructor, who had access to all the groups, respectively.   For all three cohorts, students in each group were required to read a weekly case study (this analysis is based on one case study, identical to all groups across the three cohorts) and respond to a prompt by the instructor regarding the proper diagnosis and treatment plan /or next steps in addressing the issues presented in the case study.  The case study focused on a topic to be discussed in class the following week and required students to complete the weekly assigned readings (as per the syllabus) in order to correctly respond to the prompt. The prompt to all the groups was identical and was posted by the instructor 6 days before the class was scheduled to meet face-to-face. The prompt directed the students to the location on the course LMS’ dashboard where the case study could be found, and included the question and the due date (24 hours before the class met face to face). It is important to note the prompt in 2014 also included explicit instructions for students to respond to each other's posts.  Such language was not included in the previous semesters, although the instructor orally encouraged the students to do so.  The instructor read all posts on the due date and provided feedback to each group, respectively.   

 

Data Analysis

All posts were de-identified and numerically coded by student, group and by cohort.  The content of students’ posts was analyzed using inductive constant comparison method, identifying units of meaning within each response, grouping units of meaning across responses into categories and generating themes. The researcher and her assistant conducted an initial joint analysis of sample random responses, followed by independent analyses for the purpose of testing inter-rater reliability. The researches then compared their results, discussed areas of disagreement and continued to engage in analysis until a point of saturation has been reached when no new categories or themes have emerged.  Additionally, responses were analyzed for promptness (determined by the relative sequential place of the response on the thread) and for frequency.

Results  

A total of 83 responses were analyzed.  There were 23 responses that involved disagreement between raters leading to 72% inter-rater agreement on the content analysis. These disagreements focused on variability in grouping identified units of meaning into categories. There was a complete agreement on most categories, with the exception of one and on all the generated themes.

Themes and Categories Emerging from the Data

Themes

Contextual Communication

Inter-personal Communication

Reflective Communication

Effort and Motivation

Categories

Diagnostic impressions

Positive feedback/encouragement

Reflection on the process of learning

Promptness of responses

 

Rationale for diagnostic impressions

Constructive criticism

Reflection on the quality of the material

Frequency of posts

 

Treatment plan

Engaging peers in conversation

 

Professional development

 

Discussion

The post priori themes that have emerged from the current analysis reflect the findings from previous a priori analyses (Ke, 2013). However, the current findings reveal that a comprehensive post priori analysis yields richer dimensions of online communication via a discussion board. For example, while the current analysis yielded many “agree” or “disagree” responses, as Lane (2014) cautions, these responses were often accompanied by rationale for the agreement or disagreement with peers, thus suggesting that the students engaged in constructivist learning.  The results also suggest that students engage in contextual learning of the material as evident in responses that addressed diagnostic impressions and rationale.  Of particular interest were results that suggested learning dimensions that are unique to the online format. For example, within the Inter-personal Communication theme, categories of responses included patterns of peer-to-peer communication that were characterized by positive  feedback and encouragement, providing constructive criticism and engaging peers  in   the conversation.  In a traditional face to face interaction, these patterns of communication tend to emerge in a didactic instructor-student interaction. However, in the online learning environment, students demonstrated learning of complex, and collaborative, inter-personal communication skills.  Similarly, the results suggest that students’ responses demonstrate a reflective process that extends beyond contextual learning.  For example, students reflected not only on the material in the prompt, but also on the process of learning, sharing what aspects of the assignment was enjoyable or challenging, thus suggesting a spontaneous and an independent development of meta learning skills that, unless directly prompted by the instructor, is typically missing in the time and space limited face to face environment.

Thus, the results indicate evidence of direct learning that encompass development of contextual learning as well as of communication and meta-learning skills. The identified themes serve as the dimensions of learning with the respective categories representing specific skills within each dimension, all of which can be ranked to yield indicators of successful learning. These dimensions and skills can be utilized deductively to evaluate students’ responses to an online discussion board assignments across curricular area for the purpose of direct learning outcome assessment.

References

Allen, I.E., & Seaman, J, (2013).  Changing Course: Ten Years of Tracking Online Education in

the United States.  Babson Survey Research Group.

Brooks, CF & Bippus, AM (2012). Underscoring the Social Nature of Classrooms by Examining the Amount of Virtual Talk across Online and Blended College Courses.  European Journal of Open, Distance and E-Learning.

Calderon, O., Ginsberg, P. A., & Ciabocchi, L. (2012). Multidimensional Assessment of Blended

Learning: Maximizing Program Effectiveness Based on Student and Faculty Feedback.

JALN- Journal of Asynchronous Learning Networks, 16 (4), 23-37.

doi: 10.1080/10437797.2013.796767

Ke, F (2013). Online Interaction Arrangements on Quality of Online Interactions Performed by Diverse Learners across Disciplines. Internet and Higher Education, 16, 14-22.

Klisc, C, McGill, T, & Hobbs, V (2009). The Effect of Assessment on the outcomes of asynchronous online discussion as perceived by instructors. Australian Journal of Educational Technology, 25(5), 666-682.

Lai, K (2012). Assessing participation skills: online discussions with peers.  Assessment &

 Evaluation in Higher Education, 37(8), 933-947.

Lane, LM (2014). Constructing the Past Online: Discussion Board as History Lab. History

Teacher, 47(2), 197-207.

Mathews, AL & La Tronica-Herb, A (2013). Using Blackboard to Increase Student Learning and Assessment Outcomes in a Congressional Simulation. Journal of Political Science Education,

9(2), 168-183.

Williams, L & Lahman, M (2011). Online Discussion, Student Engagement, and Critical Thinking. Journal of Political Science Education, 7(2), 143-162.