Evaluating Generative AI's Applicability in Software Testing: User study and quality evaluation of unit test generation
2025 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesis
Abstract [en]
This thesis explores the current capabilities and perceptions of generative AI tools for software testing in an industrial environment, focusing on unit test generation. Despite the potential for generative AI to automate test generation and improve efficiency, the study reveals challenges related to output quality, compilability, and practical integration barriers. Using a commercial coding assistant, we generate unit tests for an industrial code base. We then evaluate the generated tests, revealing a compilation rate of 36.09% and emphasizing compilability as the primary obstacle in generating high-quality tests. To reveal the perceived benefits and challenges of using generative AI in software testing, we conduct interviews with engineers to gather qualitative data and perform a thematic analysis. The analysis reveals perceived benefits such as time savings, efficiency improvements, and organizational support and encouragement. It also highlights challenges related to security and data confidentiality concerns, validation requirements, and output quality and reliability. While engineers recognize the theoretical potential of generative AI tools in unit test generation, the adoption of such tools in industry remains limited, with some viewing it as a misapplication of the technology. This research contributes to more informed decision-making for companies considering integrating generative AI tools into their software testing workflows.
Place, publisher, year, edition, pages
2025. , p. 59
Keywords [en]
Artificial Intelligence, AI, Large Language Models, LLMs, Coding Assistant, User Study, Software Testing, Unit Test, Unit Test Generation, Generative AI, Benefits, Challenges, Thematic Analysis
National Category
Artificial Intelligence
Identifiers
URN: urn:nbn:se:liu:diva-219190ISRN: LIU-IDA/LITH-EX-A--25/021--SEOAI: oai:DiVA.org:liu-219190DiVA, id: diva2:2010682
External cooperation
Undisclosed company
Subject / course
Computer Engineering
Presentation
2025-06-16, Donald Knuth, Linköping, 13:00 (English)
Supervisors
Examiners
2025-11-282025-11-022025-11-28Bibliographically approved