Using System Logs to Enhance Online Learning Research Projects

Concurrent Session 3

Brief Abstract

Researchers can access large volumes of data generated by hundreds of online courses each semester through the use of system logs. An analysis of these logs, paired with enrollment records and other data, can lead to rich insights. This method is less intrusive and does not contribute to survey fatigue.   


Dr. Robert Zotti heads the WebCampus Division at Stevens Institute of Technology. He leads over 150 faculty and staff in the delivery of 20 graduate programs in management, engineering, and computer science. Under Dr. Zotti's direction, the division's operations have been continuously improved, helping the school win the USDLA's 21st Century Award for Best Practices in Distance Learning seven times. Dr. Zotti has conducted research into synchronous online learning and the use of web conferencing systems. Before joining Stevens, Dr. Zotti worked at Prudential for 10 years as a programmer and trainer.

Extended Abstract

Getting survey data for online learning research projects is not getting any easier.  A proliferation of surveys on every conceivable aspect of the educational experience has led to a situation where students and faculty are simply tuning them out.  Institutional Review Boards, mindful of low response rates for even routine surveys, can be reluctant to approve new research activities that add to the “survey burn-out” of students and faculty.  What’s a researcher to do? Many have long used metrics gathered by learning management systems to analyze the effectiveness of teaching practices and patterns of online student participation.  However, many of these studies are limited in scope to a small number of classes that the researchers have direct access to.  Ironically, researchers may be able to access large volumes of LMS data generated by hundreds of online courses each semester.  An analysis of system logs that track user activity can be used to determine insights in a non-obtrusive fashion.  When combined with enrollment records, qualitative information from interviews, and even limited survey data, researchers can build a rich dataset to draw conclusions from.

This session will explore the use of system logs from learning management systems, web conferencing systems, and plagiarism-detection systems.  The different types of data within these systems will be reviewed, including:

  • Attendance and time on system
  • System features used
  • Devices and operating systems used to access systems

By matching the data from these user logs with enrollment data, a clear set of patterns can be determined. The information can be used to inform relatively limited and more targeted follow-up survey campaigns and/or a series of interviews.  Conversely, the results of previous surveys and interviews may inform the search of system logs.

Attendees of this session will learn potential ways to enhance their future online learning research projects. Strategies for gathering and reviewing system data will be presented. Basic procedures for downloading system logs will be presented. The role of collaboration with the Information Technology staff and Institutional Research Division will be covered. Privacy issues will also be reviewed.

Attendees will view the results of a research analysis of 10 years of web conferencing logs, as well as an operational analysis of TurnItIn logs.