Human researcher vs. AI-supported qualitative data analysis: Hybrid prompt design for constant comparison analysis
Authors
-
Nurşin AkmanComputer Education & Instructional Technology, Mersin University, Mersin, Türkiye
-
Erman UzunComputer Education & Instructional Technology, Mersin University, Mersin, Türkiye
-
Pınar ArslanMinistry of National Education, Niğde, Turkiye
DOI:
10.58583/EM.4.1.5Keywords:
AI-supported data analysis, Hybrid prompt design, Artificial intelligence, Qualitative data analysis, Constant comparison analysisAbstract
This study examines the potential integration of artificial intelligence (AI) tools, particularly ChatGPT, in qualitative data analysis educational research. The research tries to clarify AI-supported data analysis processes in qualitative data from teacher interviews that examine deficiencies in a "Fundamentals of Programming" course in vocational high schools. The methodology employs a hybrid (both heuristic and literature based) prompt engineering strategy, utilizing open, axial, and selective coding, to ensure a comprehensive analysis. The aim was to compare human and AI-supported analysis to evaluate the depth, efficiency, and reliability of AI tools in qualitative research. This study chose a comparative case study design because examining programming instruction with two different data analysis methods (human researcher and AI-supported) requires a comparative perspective. The findings indicate that hybrid prompt design for AI can significantly enhance the efficiency and accuracy of qualitative data analysis, providing deeper insights and more structured outputs. However, the study also highlights the importance of carefully designed prompts and human oversight to mitigate potential biases and errors inherent in AI-supported analysis. This research contributes to the growing field of AI in data analysis, offering a framework for future studies to leverage AI technologies for qualitative data analysis, thereby enhancing research quality and productivity.
Downloads
References
Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report, 13 (4), 544-559.
Bozkurt, A., Karadeniz, A., Baneres, D., Guerrero-Roldán, A.E. & Rodríguez, M.E. (2021). Artificial intelligence and reflections from educational landscape: A review of AI studies in half a century. Sustainability, 13 (800), 2-16. https://doi.org/10.3390/su13020800
Chmiliar, L. (2010). Multiple-case designs. In Encyclopedia of Case Study Research (pp. 582-584).
Christou, P. A. (2023). A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, 28 (7), 1981-1991. https://doi.org/10.46743/2160-3715/2023.6407
Christou, P. A. (2023a). The use of artificial intelligence (AI) in qualitative research for theory development. https://doi.org/10.46743/2160-3715/2023.6536
Christou, P. A. (2023b). How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research?. The Qualitative Report, 28 (7), 1968-1980.
Coffey, A., Beverley, H., & Paul, A. (1996). Qualitative data analysis: Technologies and representations. Sociological Research Online, 1 (1), 80-91. https://doi.org/10.5153/sro.
Cui, J., Wang, Z., Ho, S.-B., & Cambria, E. (2023). Survey on sentiment analysis: Evolution of research methods and topics. Artificial Intelligence Review, 56, 8469-8510. https://doi.org/10.1007/s10462-022-10386-z
Davenport, T. H., & Ronanki, R. (2018). Artificial intelligence for the real world. *Harvard Business Review, 96 (1), 108-116. https://doi.org/10.1109/ICEST58410.2023.10187269
Feuston, J. L., & Brubaker, J. R. (2021). Putting tools in their place: The role of time and perspective in human-AI collaboration for qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5 (CSCW2), 1-25. https://doi.org/10.1145/3479856
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J. (2023). PaTAT: Human-AI collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 362, 19 pages. https://doi.org/10.1145/3544548.3581352
Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51 (12), 2629-2633. https://doi.org/10.1007/s10439-023-03272-4
Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory. Aldine.
Göçmez, L., & Okur, M. R. (2023). Artificial intelligence applications in open and distance education: A systematic review of the articles (2007-2021). Asian Journal of Distance Education, 18 (1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/665
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
Gustafsson, J. (2017). Single case studies vs. multiple case studies: A comparative study.
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023). Evaluating large language models in generating synthetic HCI research data: A case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 433, 19 pages. https://doi.org/10.1145/3544548.3580688
Haque, M. U., Dharmadasa, I., Sworna, Z. T., Rajapakse, R. N., & Ahmad, H. (2022). "I think this is the most disruptive technology": Exploring sentiments of ChatGPT early adopters using Twitter data. arXiv preprint arXiv:2212.05856. https://arxiv.org/abs/2212.05856
Henrickson, L., & Meroño-Peñuela, A. (2023). Prompting meaning: A hermeneutic approach to optimising prompt engineering with ChatGPT. AI & SOCIETY, 1-16. https://doi.org/10.1007/s00146-023-01752-8
Hwang, G.-J. (2019). Definition, framework and research issues of smart learning environments-a context-aware ubiquitous learning perspective. Smart Learning Environments, 6 (1), 1-14. https://doi.org/10.1186/s40561-014-0004-5
Ifelebuegu, A. (2023). Rethinking online assessment strategies: Authenticity versus AI chatbot intervention. Journal of Applied Learning and Teaching, 6 (2). https://doi.org/10.37074/jalt.2023.6.2.2
Ifelebuegu, A. O., Kulume, P., & Cherukut, P. (2023). Chatbots and AI in education (AIEd) tools: The good, the bad, and the ugly. Journal of Applied Learning and Teaching, 6 (2). https://doi.org/10.37074/jalt.2023.6.2.29
İnik, Ö., & Ülker, E. (2017). Deep Learning and Deep Learning Models Used in Image Analysis. Gaziosmanpasa Journal of Scientific Research, 6 (3), 85-104. https://dergipark.org.tr/tr/pub/gbad/issue/31228/330663#article_cite
Joshi, S., Rambola, R. K., & Churi, P. (2021). Evaluating artificial intelligence in education for next generation. In Journal of Physics: Conference Series, 1714 (1), p. 012039. IOP Publishing. https://doi.org/10.1088/1742-6596/1714/1/012039
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82 (3), 3713-3744. https://doi.org/10.1007/s11042-022-13428-4
Krstić, L., Aleksić, V., & Krstić, M. (2022). Artificial Intelligence in Education: A Review. Technics and Informatics in Education – TIE 2022, 223-228. https://doi.org/10.46793/TIE22.223K
Lee, A. V. Y. (2020). Artificial Intelligence in Education (AIEd). Conference: 28th International Conference on Computers in Education.
Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson Education, London. https://discovery.ucl.ac.uk/id/eprint/1475756/
McCorduck, P., & Cfe, C. (2004). Machines who think: A personal inquiry into the history and prospects of artificial intelligence. AK Peters/CRC Press. https://doi.org/10.1201/9780429258985
Mesec, B. (2023). The language model of artificial intelligence ChatGPT: A tool of qualitative analysis of texts. Authorea Preprints.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., ... & Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435. https://ar5iv.labs.arxiv.org/html/2307.06435
Ojha, S., et al. (2023). From robots to books: An introduction to smart applications of AI in education (AIEd). arXiv preprint arXiv:2301.10026. https://arxiv.org/pdf/2301.10026.pdf
Panda, G., Upadhyay, A. K., & Khandelwal, K. (2019). Artificial intelligence: A strategic disruption in public relations. Journal of Creative Communications, 14 (3), 196-213. https://doi.org/10.1177/0973258619866585
Patton, M. Q. (2014). Qualitative research & evaluation methods: Integrating theory and practice. Sage publications.
Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., & Riedel, S. (2020). How context affects language models' factual predictions. arXiv preprint arXiv:2005.04611. https://arxiv.org/abs/2005.04611
Radford, A., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1*(8). https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf
Rietz, T., & Maedche, A. (2021). Cody: An AI-based system to semi-automate coding for qualitative research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 394, 14 pages. https://doi.org/10.1145/3411764.3445591
Serbest, A. B. (2024). Demand Forecasting And Inventory Management With Machine Learning. Unpublished master's thesis. Bursa Uludağ University Graduate School of Natural and Applied Sciences https://acikerisim.uludag.edu.tr/items/52611184-0b35-4c90-bd19-4dac173ffd80
Shaik, T., Tao, X., Li, Y., Dann, C., McDonald, J., Redmond, P., & Galligan, L. (2022). A review of the trends and challenges in adopting natural language processing methods for education feedback analysis. IEEE Access, 10, 56720-56739. https://doi.org/10.1109/ACCESS.2022.3177752
Shao, Z., Yu, Z., Wang, M., & Yu, J. (2023). Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14974-14983). https://doi.org/10.1109/CVPR52729.2023.01438
Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2023). An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. arXiv preprint arXiv:2309.08008.
Spasić, A. J., & Janković, D. S. (2023, June). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role, instructions and seed-word prompts. In 2023 58th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST) (pp. 47-50). IEEE. https://doi.org/10.1109/ICEST58410.2023.10187269
Spasić, A. J., & Janković, D. S. (2023, June). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role, instructions and seed-word prompts. In 2023 58th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST) (pp. 47-50). IEEE.
Stake, R.E. (2006). Multiple case study analysis. Guilford Press.
Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2024). An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods, 23, 16094069241231168. https://doi.org/10.1177/16094069241231168
Thunberg, S., & Arnell, L. (2022). Pioneering the use of technologies in qualitative research–A research review of the use of digital interviews. International Journal of Social Research Methodology, 25 (6), 757-768. https://doi.org/10.1080/13645579.2021.1935565
Wang, X., Attal, M. I., Rafiq, U., & Hubner-Benz, S. (2022, June). Turning large language models into AI assistants for startups using prompt patterns. In International Conference on Agile Software Development (pp. 192-200). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-48550-3_19
Weitzman, E., & Miles, M. B. (1995). Computer programs for qualitative data analysis. SAGE Publications. https://books.google.com/books?id=E4Y5DQAAQBAJ
Welsh, E., et al. (2002). Dealing with data: Using NVivo in the qualitative data analysis process. In Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 3 (2). https://doi.org/10.17169/fqs-3.2.865
White, J., et al. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. https://arxiv.org/abs/2302.11382
Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R., & Oudeyer, P.-Y. (2023). Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 75-78. https://doi.org/10.1145/3581754.3584136
Yin, R. K. (2003). Case study research: Design and methods. Sage Publications.
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., ... & Li, Y. (2021). A review of artificial intelligence (AI) in education from 2010 to 2020. Complexity, 2021, 1-18. https://doi.org/10.1155/2021/8812542
Zhang, H., Wu, C., Xie, J., Kim, C., & Carroll, J. M. (2023). QualiGPT: GPT as an easy-to-use tool for qualitative coding. arXiv preprint arXiv:2310.07061.
Downloads
How to Cite
Published
Issue
Section
Statistics
License
Copyright (c) 2025 Nurşin Akman, Erman Uzun, Pınar Arslan
This work is licensed under a Creative Commons Attribution 4.0 International License.
