liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
LLM-based Satisfiability Checking of String Requirements by Consistent Data and Checker Generation
ECE, McGill University, Montreal, Canada.
Dep’t of Computer Science, University of Toronto, Toronto, Canada.
ECE, McGill University, Montreal, Canada.
Linköping University, Department of Computer and Information Science, Software and Systems. Linköping University, Faculty of Science & Engineering.ORCID iD: 0000-0002-8790-252X
Show others and affiliations
2025 (English)In: 2025 IEEE 33rd International Requirements Engineering Conference (RE), IEEE Computer Society, 2025, p. 231-243Conference paper, Published paper (Refereed)
Abstract [en]

Requirements over strings, commonly represented using natural language (NL), are particularly relevant for software systems due to their heavy reliance on string data manipulation. While individual requirements can usually be analyzed manually, verifying properties (e.g., satisfiability) over sets of NL requirements is particularly challenging. Formal approaches (e.g., SMT solvers) may efficiently verify such properties, but are known to have theoretical limitations. Additionally, the translation of NL requirements into formal constraints typically requires significant manual effort. Recently, large language models (LLMs) have emerged as an alternative approach for formal reasoning tasks, but their effectiveness in verifying requirements over strings is less studied. In this paper, we introduce a hybrid approach that verifies the satisfiability of NL requirements over strings by using LLMs (1) to derive a satisfiability outcome (and a consistent string, if possible), and (2) to generate declarative (i.e., SMT) and imperative (i.e., Python) checkers, used to validate the correctness of (1). In our experiments, we assess the performance of four LLMs. Results show that LLMs effectively translate natural language into checkers, even achieving perfect testing accuracy for Python-based checkers. These checkers substantially help LLMs in generating a consistent string and accurately identifying unsatisfiable requirements, leading to more than doubled generation success rate and F1-score in certain cases compared to baselines without generated checkers.

Place, publisher, year, edition, pages
IEEE Computer Society, 2025. p. 231-243
National Category
Computer Sciences Formal Methods Computer Systems
Identifiers
URN: urn:nbn:se:liu:diva-218877DOI: 10.1109/RE63999.2025.00030ISI: 001733909100022Scopus ID: 2-s2.0-105020011394ISBN: 9798331524135 (electronic)ISBN: 9798331524142 (print)OAI: oai:DiVA.org:liu-218877DiVA, id: diva2:2007011
Conference
IEEE 33rd International Requirements Engineering Conference (RE), Valencia, Spain, 01-05 September, 2025
Note

Funding: FRQNT-B2X project [319955, IT30340]; Wallenberg AI, Autonomous Systems and Software Program (WASP), Sweden

Available from: 2025-10-16 Created: 2025-10-16 Last updated: 2026-08-12

Open Access in DiVA

fulltext(697 kB)6 downloads
File information
File name FULLTEXT02.pdfFile size 697 kBChecksum SHA-512
dbf8e46c34eba52c9b7ae1db4c614af41925d805479a108e10b92e2dfb693077619e6f42488417e6dcf64793268ddeda58aebb78eae43d158d0033b36b1da454
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Varro, Daniel

Search in DiVA

By author/editor
Varro, Daniel
By organisation
Software and SystemsFaculty of Science & Engineering
Computer SciencesFormal MethodsComputer Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 6 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 82 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf