Using Run-Time Information to Enhance Static Analysis of Machine Learning Code in Notebooks
2024 (English)In: COMPANION PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, FSE COMPANION 2024, ASSOC COMPUTING MACHINERY , 2024, p. 497-501Conference paper, Published paper (Refereed)
Abstract [en]
A prevalent method for developing machine learning (ML) prototypes involves the use of notebooks. Notebooks are sequences of cells containing both code and natural language documentation. When executed during development, these code cells provide valuable run-time information. Nevertheless, current static analyzers for notebooks do not leverage this run-time information to detect ML bugs. Consequently, our primary proposition in this paper is that harvesting this run-time information in notebooks can significantly improve the effectiveness of static analysis in detecting ML bugs. To substantiate our claim, we focus on bugs related to tensor shapes and conduct experiments using two static analyzers: 1) PYTHIA, a traditional rule-based static analyzer, and 2) GPT-4, a large language model that can also be used as a static analyzer. The results demonstrate that using run-time information in static analyzers enhances their bug detection performance and it also helped reveal a hidden bug in a public dataset.
Place, publisher, year, edition, pages
ASSOC COMPUTING MACHINERY , 2024. p. 497-501
Keywords [en]
static analysis; run-time information; notebook; machine learning bugs; large language models
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:liu:diva-207654DOI: 10.1145/3663529.3663785ISI: 001273423600054ISBN: 9798400706585 (print)OAI: oai:DiVA.org:liu-207654DiVA, id: diva2:1898430
Conference
32nd ACM International Conference on the Foundations of Software Engineering (FSE), Porto de Galinhas, BRAZIL, jul 15-19, 2024
Note
Funding Agencies|Wallenberg AI, Autonomous Systems and Software Program (WASP) - Knut and Alice Wallenberg Foundation; Software Center Project [30]
2024-09-172024-09-172024-11-22