Modern software development relies on automated testing through CI/CD pipelines that generate large volumes of test data daily. This data is often spread across multiple independent sources, each capturing a different aspect of the testing process, such as raw execution records, failure classifications, commit history, and rerun outcomes. These data sources share no enforced schema and must be connected through shared identifiers, making it difficult to extract insights without manually correlating records across sources, a process that is time consuming and requires deep knowledge of how the sources relate to each other.

Large language models have shown strong capabilities in understanding natural language and reasoning about complex information. However, they lack access to data stored in enterprise systems. Retrieval-augmented generation (RAG) addresses this by grounding the model in retrieved context, but the design choices around how retrieval is done significantly affect whether the system can answer the full range of questions that arise in practice.

Text-to-SQL has been the dominant paradigm for structured data retrieval in existing research, while semantic RAG has been applied primarily to unstructured text or single source structured data. This thesis evaluates semantic RAG on structured CI data distributed across multiple independent sources, where answering a question may require evidence from several sources that were not designed to be queried together.

Evaluating such a system requires a benchmark that reflects the actual structure of the data. Current public CI/CD test datasets tend to cover one source type in isolation, such as execution logs or flaky test records, and do not include questions that require evidence from multiple independent sources jointly, which is the setting targeted by this thesis. A custom benchmark of 500 questions across 11 types was therefore constructed. The types range from simple point lookups and aggregate counts to multi-hop questions where the second query can only be formed after the first result is known.

Four RAG configurations were evaluated in a stepwise ablation across metrics covering correctness, retrieval quality, reliability, and latency. The Baseline is a naive RAG system that embeds the raw question and retrieves the top matching chunks. C1 adds hypothetical document generation to close the vocabulary gap between natural language questions and structured records. C2 adds scope constraint extraction to constrain retrieval to the correct scope before retrieval. C3 adds iterative retrieval for questions that require more than one retrieval pass. Baseline reached 10.3\% accuracy. C1 reached 33.9\%, C2 67.8\%, and C3 72.0\%. Scope constraint extraction produced the largest single gain at 33.9 percentage points and is the most cost-effective addition. Overall accuracy however masks important differences across types, some types score near 100\% while others remain largely unsolved.