In our May 2026 Snap Survey, we asked participants (n=52) to read a passage of text and identify it as either AI-generated or written by humans. The results may surprise you!
There are many invigilation products that promise to detect AI-written work. However, none are 100% reliable and researchers have found that “while AI-output detectors may serve as supplementary tools in peer review or abstract evaluation, they often misclassify texts and require improvement” (Erol et al., 2025, p. 10).
In our own mini-investigation into how detectable AI-generated content really is, we started by searching for academic material on the same subject matter. The two human-authored passages were drawn from openly licensed educational resources published in 2023, shortly after the public release of ChatGPT. Although no definitive claims can be made about the authors’ writing processes, the passages were selected because they represent authentic academic writing intended for educational use. Then an AI-generated passage was created on the same topic using Chat GPT. To reduce topic effects, all three passages addressed the same subject (qualitative research methods) and were similar in length and style. All three passages were submitted to Turnitin for baseline consistency. In this small demonstration, the tool classified the passages as expected; however, as mentioned above, broader research has shown that AI-detection tools are not consistently reliable.
What do we look for in AI-generated texts? Some of the hallmarks include both physical characteristics (em dashes, bullet points, line breaks, and short paragraphs that might be one sentence long) and linguistic (overly flowery, very definitive, or overuse of specific words like delve, tapestry, or pivotal). AI-generated text usually contains no typos characteristic of human writing and it completely avoids run-on sentences or other stylistic errors common to the average writer. There are other identifiable characteristics, but as AI becomes more sophisticated and so too the users, the writing has become less identifiable from a structural standpoint.
So how did our user community do when judging AI created text against human created text? Let’s find out!
Passage 1
At its simplest, qualitative research is research about the social world that does not use numbers in its analyses. All those who fear statistics can breathe a sigh of relief – there are no mathematical formulae or regression models in this book! But this definition is less about what qualitative research can be and more about what it is not. To be honest, any simple statement will fail to capture the power and depth of qualitative research. One way of contrasting qualitative research to quantitative research is to note that the focus of qualitative research is less about explaining and predicting relationships between variables and more about understanding the social world. To use our mother love example, the question about “what love looks like” is a good question for the qualitative researcher while all questions measuring love or comparing incidences of love (both of which require measurement) are good questions for quantitative researchers.
Is it AI?
No!
This passage was human-written (Hurst, 2023)! The results show that identifying this text was essentially a coin flip; slightly more people thought it was AI than human.
Passage 2
Qualitative research is often understood as an approach to studying the social world that does not depend on numerical analysis. Rather than starting with clearly defined variables or hypotheses, it tends to begin with broader questions about how people experience and interpret particular situations. This makes it especially useful in areas where there is limited prior research or where the boundaries of the phenomenon are still unclear. For example, instead of asking how frequently a behavior occurs or whether one factor predicts another, a qualitative researcher might explore how individuals make sense of that behavior in their everyday lives. In contrast to quantitative research, which focuses on measurement, comparison, and testing relationships, qualitative research emphasizes meaning, context, and the complexity of human experience, aiming to better understand how and why people see the world as they do.
Is it AI?
Yes!
This selection of text was generated by ChatGPT. This passage fooled most respondents. Nearly 65% classified the AI-generated passage as human.
Passage 3
Qualitative research designs are appropriate when we do not know much about the phenomenon (concept) under investigation, and when we may not even know what we are trying to find out.5,6 Qualitative research is suitable for seeking to understand a phenomenon from the perspectives of the persons experiencing it – for example, the phenomenon of presenting to a hospital’s emergency department can be explored from the perspectives of patients, caregivers and staff. The phenomenon of living in out-of-home care can be explored from the perspective of the person living in care as well as that of the care workers. Quantitative research is objective, hypothesis-testing and deductive; it seeks to assess, make associations, examine causation, relationships and effects; and seeks to answer questions of ‘what?’ (prevalence/incidence) and ‘do/does?’ (effectiveness). In contrast, qualitative research is subjective, hypothesis-generating and inductive; it seeks to describe, understand, explore, discover, generate and examine; and to answer questions of ‘why’, ‘how’ and what (experience).
Is it AI?
No!
This passage (Ayton et al., 2023) was identified correctly by about two-thirds of respondents.
So What Does This Mean?
The most interesting finding is that the AI-generated passage (Q2) was perceived as human far more often than not, while one of the human-written passages (Q1) was actually more likely to be labeled AI than human. The overall accuracy of our respondents was about 50%, essentially no better than chance.
The results suggest that many of the characteristics commonly associated with AI-generated writing may be less reliable indicators than we assume. When presented with relatively neutral academic prose on the same topic, respondents were unable to consistently distinguish AI-generated content from human-authored content. In fact, the AI-generated passage was more likely to be identified as human than one of the authentic human passages. These findings echo broader research showing that both humans and automated detection tools struggle to accurately identify AI-generated writing. As generative AI continues to improve, the question may be shifting from “Can we detect AI?” to “Does it matter who or what wrote the first draft if the final product is accurate, ethical, and fit for purpose?”
For educators, these findings present a challenge. Many faculty members believe they can recognize AI-generated work based on writing style alone, yet our respondents performed at approximately chance levels. This suggests that suspicion based solely on perceived writing characteristics may be misplaced. Rather than attempting to identify AI use after the fact, educators may be better served by designing learning activities that emphasize process, reflection, application, and demonstration of learning, elements that are difficult to outsource regardless of the tools being used. Perhaps the most important takeaway is that AI literacy requires more than learning a checklist of AI “tells.” As language models become more sophisticated, traditional indicators become less reliable. Developing critical evaluation skills such as focusing on the quality, accuracy, and appropriateness of content rather than assumptions about its origin may be a more useful competency for the future of teaching, learning, and knowledge work.
The real surprise isn’t that AI fooled people. It’s that the human-written passages fooled them too. If respondents could not reliably distinguish between human- and AI-generated content, perhaps the more important question is not whether we can detect AI, but whether detection is the right question in the first place.
About OLC Snap Surveys
OLC Snap Surveys provide quick insights into emerging trends, challenges, and perspectives in digital learning. Surveys are distributed to the OLC community and offer a rapid way to capture practitioner experiences across institutions.
OLC welcomes participation in future Snap Surveys to help inform ongoing conversations about the future of online and digital learning. Subscribe to OLC Today, our weekly newsletter, to get the latest Snap Survey delivered directly to your inbox.
References
Ayton, D., Tsindos, T., & Berkovic, D. (2023). Qualitative research: A practical guide for health and social care researchers and practitioners. Monash University Library. https://doi.org/10.60754/chqr-dn78
Erol, G., Ergen, A., Gülşen Erol, B., Kaya Ergen, Ş., Bora, T. S., Çölgeçen, A. D., Araz, B., Şahin, C., Bostancı, G., Kılıç, I., Macit, Z.B., Sevgi, U.T. & Güngör, A. (2025). Can we trust academic AI detective? Accuracy and limitations of AI-output detectors. Acta neurochirurgica, 167(1), 214. https://doi.org/10.1007/s00701-025-06622-4
Hurst, A. (2023). Introduction to qualitative research methods: A helpful guide for undergraduates and graduate students in the social sciences. Oregon State University. https://open.oregonstate.education/qualresearchmethods/
As senior researcher at OLC, Carrie designs, conducts and manages the portfolio of research projects that align with the mission, vision, and goals of the Online Learning Consortium. She brings with her over 15 years of experience as an online educator and instructional designer with a passion for research. She has peer-reviewed publications covering a variety of topics such as open educational resources, online course best practices, and game-based learning. In addition to a strong background in higher education teaching and instructional design, Carrie brings with her extensive experience in customer service and small business management. She holds a PhD in Educational Technology from Arizona State University, an MS in French from Minnesota State University, and BA in French from Arizona State University.
