What do course analytics really indicate? Examining the relationship among course activity measurements, GRIT, and the Community of Inquiry.
Concurrent Session 6
This in-progress study seeks to evaluate the degree to which measures of activity in online courses are associated with measures of GRIT, and the Community of Inquiry framework. Using data from a 2-year study that will conclude data collection in July, 2017, results from a blended graduate teacher preparation program will be analyzed and discussed with session attendees.
Monitoring user interaction within modern online learning systems provide an opportunity for closely examining how students are consuming content and participating in course activities. The data that is generated from online learning systems can provide an immense amount of data. This data, however, cannot always be easily processed or clearly interpreted in a way that leads to action on the part of the instructor. In order to gain actionable intelligence from course activity data, it is necessary to better understand the relationship these measures may have and the degree to which this may vary from one course to another. As such, it is the aim of this exploratory study to examine the relationship between measures derived from online course activity logs and ratings of GRIT (Duckworth, et al, 2007), as well as Social, Cognitive, and Teaching Presence as defined in the Community of Inquiry model (Garrison, Anderson, & Archer, 2000).
The students and courses on which this study focuses is are part of a blended graduate teacher preparation program at a mid-sized public university in the midwestern United States. Typical students in this program are enrolled part-time and typically complete 2 courses per term over a 1-2 year period. The program includes three program tracks including a 1 year teacher licensure program or either a 2 or 2.5 year teacher licensure and Master of Education degree program. This program is largely cohort-based, thus students enroll in the same sequence of courses as others who started the program at the same time. The courses in this program feature 3 on-campus class sessions that are held at the beginning, middle, and near the end of the academic term.
During a two-year evaluation period that will conclude in July, 2017, data have been collected that includes the Community of Inquiry survey (Arbaugh et al, 2008) as well as data regarding frequency of online activity by students and instructors, number and type of course assignments, and course grades. This data will be used to address the following research question:
To what extent is there a relationship among measures of GRIT, Community of Inquiry, and online activity within a course?
Data collected were collected using three primary sources: (1) Community of Inquiry survey, (2) course activity records from the learning management system, and (3) Course syllabus.
The Community of Inquiry survey was administered via Qualtrics Survey Software. An invitation to complete a program survey was sent to all students enrolled in courses in the academic program at approximately the middle of the academic term. The survey included a separate 34-item Community of Inquiry survey for each course in which they were enrolled, thus each response could represent ratings of up to 3 courses. For Fall, Winter, and Spring academic terms, this occurred during the week following the 2nd on-campus class session. Summer courses in the program include a single on-campus class session at the beginning of the academic term, thus the survey was administered from the 4th-5th week of the summer courses. The survey was available during a period of 7 days and follow-up requests were sent every two days to students who had not completed the survey. Response rates ranged from 59.5% to 72.5%. During the 2-year time period, there were 614 responses to the survey.
An additional source of data for analysis was the Learning Management System. This data source included the student achievement data (grades) as well as records of student activity within the learning management system. Course grades were downloaded from the gradebook available in each course. Records of activity in the online courses were gathered using a procedure of downloading individual page-view records for each user within a course. Using the individual pageview records, a summary of course activity was generated for each student in the course that indicated the frequency of pageviews, pageviews with participation, pageviews in discussion forums, and pageviews with participation in discussion forums.
Data from the Community of Inquiry survey were analysed using SPSS 24. Data from the Learning Management system were downloaded using a custom-design processed using Python to access data available from the Application Programming Interface (API) provided by the LMS. Student course activity records were downloaded into a MySQL database. A frequency summary was then generated for each student in each course using Tableau 10.3 software. This summary was then imported into SPSS 24 in order to be matched and analyzed with the Community of Inquiry survey data.
Participation in the study was solicited from the students who were enrolled in program courses. Total course enrollment varied from a low of 75 students (Winter 2016-17) to a high of 96 students (Summer 2015-16). All participants were notified of the research student and invited to participate via an email invitation. The link that was provided by email directed the respondent to the Letter of Informed consent page of the survey in the Qualtrics Survey system. Participation was voluntary and no incentive was provided. Data were collected confidentially, including a unique numerical code that facilitated the matching of survey responses with LMS records. The data collection procedure was reviewed and approved by the appropriate Institutional Review Board.
The 2-year study period for this study will conclude in July, 2017. As such, the full results of the data analysis are not yet available until the study period is completed. Feasibility tests have been conducted to establish the process for downloading and analyzing this student activity data and ensuring data integrity.
Data will be analyzed first to identify bivariate correlations among the variables. Those variables for which a significant correlation is found with the course grades will be included in a follow-up analysis to identify a possible predictive model.
It is expected that this study will provide the audience with a relevant examination of the degree to which measures of student behaviour in an online course are associated with measures of GRIT and elements of the Community of Inquiry model. This can then serve as the basis for future projects focusing on the development of methods to assess student engagement from course activity logs.
Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the community of inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11(3-4), 133-136. doi:10.1016/j.iheduc.2008.06.003
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087-1101.
Garrison, D., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education model. The Internet and Higher Education, 2(2-3), 87-105