liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Failed attempt at creating Guidelines for Visual GUI Testing: An industrial case study
Blekinge Inst Technol, Sweden.
Linköping University, Department of Clinical and Experimental Medicine, Division of Speech language pathology, Audiology and Otorhinolaryngology. Linköping University, Faculty of Medicine and Health Sciences.
Linköping University, Department of Computer and Information Science, Software and Systems. Linköping University, Faculty of Science & Engineering.
2021 (English)In: 2021 14TH IEEE CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION (ICST 2021), IEEE COMPUTER SOC , 2021, p. 340-350Conference paper, Published paper (Refereed)
Abstract [en]

Software development is governed by guidelines that aim to improve the codes qualities, such as maintainability. However, whilst coding guidelines are commonplace for software, guidelines for testware are much less common. In particular, for GUI-based tests driven with image recognition, also referred to as Visual GUI Testing (VGT), explicit coding guidelines are missing. In this industrial case study, performed at the Swedish defence contractor Saab AB, we propose a set of coding guidelines for VGT and evaluate their impact on test scripts for an industrial, safety-critical system. To study the guidelines effect on maintenance costs, five representative manual test cases are each translated with and without the proposed guidelines in the two VGT tools SikuliX and EyeAutomate. As such, 20 test scripts were developed, with a combined development cost of more than 100 man-hours. Three of the tests are then maintained by one researcher and two practitioners for another version of the system and costs measured to evaluate return on investment. This analysis is complemented with observations and interviews to elicit practitioners perceptions and experiences with VGT. Results show that scripts developed with the guidelines had higher maintenance costs than scripts developed without guide-lines. This is supported by qualitative results that many of the guidelines are considered inappropriate, superfluous or unnecessary due to the inherent properties of the scripts, e.g. their natural small size, linear flows, natural separation of concerns, and more. We conclude that there are differences between VGT scripts and software that prohibit direct translation of guidelines between the two. As such, we consider our study as a failure but argue that several lessons can be drawn from our results to guide future research into guidelines for VGT and GUI-based test automation.

Place, publisher, year, edition, pages
IEEE COMPUTER SOC , 2021. p. 340-350
Series
IEEE International Conference on Software Testing Verification and Validation, ISSN 2381-2834
Keywords [en]
Automated Testing; Visual GUI testing; Guidelines for testing; Case study; Industrial study
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:liu:diva-178565DOI: 10.1109/ICST49551.2021.00046ISI: 000680831800034ISBN: 978-1-7281-6836-4 (print)OAI: oai:DiVA.org:liu-178565DiVA, id: diva2:1588473
Conference
14th IEEE Conference on Software Testing, Verification and Validation (ICST), ELECTR NETWORK, apr 12-16, 2021
Note

Funding Agencies|KKS foundation through the S.E.R.T. Research Profile project; KKS foundation through M.E.T.A. project at Blekinge Institute of Technology

Available from: 2021-08-27 Created: 2021-08-27 Last updated: 2021-08-27

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full text

Search in DiVA

By author/editor
Borch Petersen, ElineTinnerholm, John
By organisation
Division of Speech language pathology, Audiology and OtorhinolaryngologyFaculty of Medicine and Health SciencesSoftware and SystemsFaculty of Science & Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 307 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf