Education Mind

Research Article Open Access

Human researcher vs. AI-supported qualitative data analysis: Hybrid prompt design for constant comparison analysis

Nurşin Akman , Erman Uzun , Pınar Arslan

Abstract

This study examines the potential integration of artificial intelligence (AI) tools, particularly ChatGPT, in qualitative data analysis educational research. The research tries to clarify AI-supported data analysis processes in qualitative data from teacher interviews that examine deficiencies in a "Fundamentals of Programming" course in vocational high schools. The methodology employs a hybrid (both heuristic and literature based) prompt engineering strategy, utilizing open, axial, and selective coding, to ensure a comprehensive analysis. The aim was to compare human and AI-supported analysis to evaluate the depth, efficiency, and reliability of AI tools in qualitative research. This study chose a comparative case study design because examining programming instruction with two different data analysis methods (human researcher and AI-supported) requires a comparative perspective. The findings indicate that hybrid prompt design for AI can significantly enhance the efficiency and accuracy of qualitative data analysis, providing deeper insights and more structured outputs. However, the study also highlights the importance of carefully designed prompts and human oversight to mitigate potential biases and errors inherent in AI-supported analysis. This research contributes to the growing field of AI in data analysis, offering a framework for future studies to leverage AI technologies for qualitative data analysis, thereby enhancing research quality and productivity.

Keywords: AI-supported data analysis Hybrid prompt design Artificial intelligence Qualitative data analysis Constant comparison analysis

References

Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report, 13 (4), 544-559.

Bozkurt, A., Karadeniz, A., Baneres, D., Guerrero-Roldán, A.E. & Rodríguez, M.E. (2021). Artificial intelligence and reflections from educational landscape: A review of AI studies in half a century. Sustainability, 13 (800), 2-16. https://doi.org/10.3390/su13020800

Chmiliar, L. (2010). Multiple-case designs. In Encyclopedia of Case Study Research (pp. 582-584).

Christou, P. A. (2023). A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, 28 (7), 1981-1991. https://doi.org/10.46743/2160-3715/2023.6407

Christou, P. A. (2023a). The use of artificial intelligence (AI) in qualitative research for theory development. https://doi.org/10.46743/2160-3715/2023.6536

Christou, P. A. (2023b). How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research?. The Qualitative Report, 28 (7), 1968-1980.

Coffey, A., Beverley, H., & Paul, A. (1996). Qualitative data analysis: Technologies and representations. Sociological Research Online, 1 (1), 80-91. https://doi.org/10.5153/sro.

Cui, J., Wang, Z., Ho, S.-B., & Cambria, E. (2023). Survey on sentiment analysis: Evolution of research methods and topics. Artificial Intelligence Review, 56, 8469-8510. https://doi.org/10.1007/s10462-022-10386-z

Davenport, T. H., & Ronanki, R. (2018). Artificial intelligence for the real world. *Harvard Business Review, 96 (1), 108-116. https://doi.org/10.1109/ICEST58410.2023.10187269

Feuston, J. L., & Brubaker, J. R. (2021). Putting tools in their place: The role of time and perspective in human-AI collaboration for qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5 (CSCW2), 1-25. https://doi.org/10.1145/3479856

Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J. (2023). PaTAT: Human-AI collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 362, 19 pages. https://doi.org/10.1145/3544548.3581352

Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51 (12), 2629-2633. https://doi.org/10.1007/s10439-023-03272-4

Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory. Aldine.

Göçmez, L., & Okur, M. R. (2023). Artificial intelligence applications in open and distance education: A systematic review of the articles (2007-2021). Asian Journal of Distance Education, 18 (1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/665

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.

Gustafsson, J. (2017). Single case studies vs. multiple case studies: A comparative study.

Hämäläinen, P., Tavast, M., & Kunnari, A. (2023). Evaluating large language models in generating synthetic HCI research data: A case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 433, 19 pages. https://doi.org/10.1145/3544548.3580688

Haque, M. U., Dharmadasa, I., Sworna, Z. T., Rajapakse, R. N., & Ahmad, H. (2022). "I think this is the most disruptive technology": Exploring sentiments of ChatGPT early adopters using Twitter data. arXiv preprint arXiv:2212.05856. https://arxiv.org/abs/2212.05856

Henrickson, L., & Meroño-Peñuela, A. (2023). Prompting meaning: A hermeneutic approach to optimising prompt engineering with ChatGPT. AI & SOCIETY, 1-16. https://doi.org/10.1007/s00146-023-01752-8

Hwang, G.-J. (2019). Definition, framework and research issues of smart learning environments-a context-aware ubiquitous learning perspective. Smart Learning Environments, 6 (1), 1-14. https://doi.org/10.1186/s40561-014-0004-5

Ifelebuegu, A. (2023). Rethinking online assessment strategies: Authenticity versus AI chatbot intervention. Journal of Applied Learning and Teaching, 6 (2). https://doi.org/10.37074/jalt.2023.6.2.2

Ifelebuegu, A. O., Kulume, P., & Cherukut, P. (2023). Chatbots and AI in education (AIEd) tools: The good, the bad, and the ugly. Journal of Applied Learning and Teaching, 6 (2). https://doi.org/10.37074/jalt.2023.6.2.29

İnik, Ö., & Ülker, E. (2017). Deep Learning and Deep Learning Models Used in Image Analysis. Gaziosmanpasa Journal of Scientific Research, 6 (3), 85-104. https://dergipark.org.tr/tr/pub/gbad/issue/31228/330663#article_cite

Joshi, S., Rambola, R. K., & Churi, P. (2021). Evaluating artificial intelligence in education for next generation. In Journal of Physics: Conference Series, 1714 (1), p. 012039. IOP Publishing. https://doi.org/10.1088/1742-6596/1714/1/012039

Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82 (3), 3713-3744. https://doi.org/10.1007/s11042-022-13428-4

Krstić, L., Aleksić, V., & Krstić, M. (2022). Artificial Intelligence in Education: A Review. Technics and Informatics in Education – TIE 2022, 223-228. https://doi.org/10.46793/TIE22.223K

Lee, A. V. Y. (2020). Artificial Intelligence in Education (AIEd). Conference: 28th International Conference on Computers in Education.

Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson Education, London. https://discovery.ucl.ac.uk/id/eprint/1475756/

McCorduck, P., & Cfe, C. (2004). Machines who think: A personal inquiry into the history and prospects of artificial intelligence. AK Peters/CRC Press. https://doi.org/10.1201/9780429258985

Mesec, B. (2023). The language model of artificial intelligence ChatGPT: A tool of qualitative analysis of texts. Authorea Preprints.

Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., ... & Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435. https://ar5iv.labs.arxiv.org/html/2307.06435

Ojha, S., et al. (2023). From robots to books: An introduction to smart applications of AI in education (AIEd). arXiv preprint arXiv:2301.10026. https://arxiv.org/pdf/2301.10026.pdf

Panda, G., Upadhyay, A. K., & Khandelwal, K. (2019). Artificial intelligence: A strategic disruption in public relations. Journal of Creative Communications, 14 (3), 196-213. https://doi.org/10.1177/0973258619866585

Patton, M. Q. (2014). Qualitative research & evaluation methods: Integrating theory and practice. Sage publications.

Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., & Riedel, S. (2020). How context affects language models' factual predictions. arXiv preprint arXiv:2005.04611. https://arxiv.org/abs/2005.04611

Radford, A., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1*(8). https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf

Rietz, T., & Maedche, A. (2021). Cody: An AI-based system to semi-automate coding for qualitative research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 394, 14 pages. https://doi.org/10.1145/3411764.3445591

Serbest, A. B. (2024). Demand Forecasting And Inventory Management With Machine Learning. Unpublished master's thesis. Bursa Uludağ University Graduate School of Natural and Applied Sciences https://acikerisim.uludag.edu.tr/items/52611184-0b35-4c90-bd19-4dac173ffd80

Shaik, T., Tao, X., Li, Y., Dann, C., McDonald, J., Redmond, P., & Galligan, L. (2022). A review of the trends and challenges in adopting natural language processing methods for education feedback analysis. IEEE Access, 10, 56720-56739. https://doi.org/10.1109/ACCESS.2022.3177752

Shao, Z., Yu, Z., Wang, M., & Yu, J. (2023). Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14974-14983). https://doi.org/10.1109/CVPR52729.2023.01438

Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2023). An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. arXiv preprint arXiv:2309.08008.

Spasić, A. J., & Janković, D. S. (2023, June). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role, instructions and seed-word prompts. In 2023 58th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST) (pp. 47-50). IEEE. https://doi.org/10.1109/ICEST58410.2023.10187269

Stake, R.E. (2006). Multiple case study analysis. Guilford Press.

Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2024). An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods, 23, 16094069241231168. https://doi.org/10.1177/16094069241231168

Thunberg, S., & Arnell, L. (2022). Pioneering the use of technologies in qualitative research–A research review of the use of digital interviews. International Journal of Social Research Methodology, 25 (6), 757-768. https://doi.org/10.1080/13645579.2021.1935565

Wang, X., Attal, M. I., Rafiq, U., & Hubner-Benz, S. (2022, June). Turning large language models into AI assistants for startups using prompt patterns. In International Conference on Agile Software Development (pp. 192-200). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-48550-3_19

Weitzman, E., & Miles, M. B. (1995). Computer programs for qualitative data analysis. SAGE Publications. https://books.google.com/books?id=E4Y5DQAAQBAJ

Welsh, E., et al. (2002). Dealing with data: Using NVivo in the qualitative data analysis process. In Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 3 (2). https://doi.org/10.17169/fqs-3.2.865

White, J., et al. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. https://arxiv.org/abs/2302.11382

Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R., & Oudeyer, P.-Y. (2023). Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 75-78. https://doi.org/10.1145/3581754.3584136

Yin, R. K. (2003). Case study research: Design and methods. Sage Publications.

Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., ... & Li, Y. (2021). A review of artificial intelligence (AI) in education from 2010 to 2020. Complexity, 2021, 1-18. https://doi.org/10.1155/2021/8812542

Zhang, H., Wu, C., Xie, J., Kim, C., & Carroll, J. M. (2023). QualiGPT: GPT as an easy-to-use tool for qualitative coding. arXiv preprint arXiv:2310.07061.

Human researcher vs. AI-supported qualitative data analysis: Hybrid prompt design for constant comparison analysis

Abstract

References

Suggested Article

Cookie Notice