Retrieval-Augmented Generation with Charts: Improving retrieval for images of charts
2025 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesis
Abstract [en]
Recent advancements in Retrieval-Augmented Generation (RAG) systems have enabled large language models (LLMs) to better access and utilize external knowledge, yet the integration of non-textual data, particularly charts, remains underexplored. This thesis investigates strategies for preprocessing and embedding chart images in multimodal RAG systems to improve retrieval and generation performance. Three embedding methods, text summary, dual storage, and page screenshot, were developed and evaluated on two datasets: the chart-only CHARTQA and the newly created real-world financial report based COINQA dataset. Results show that while all strategies perform similarly on chart-only data, the page screenshot embedding significantly outperforms other methods in retrieval tasks for mixed text-and-chart documents, achieving more than twice the retrieval performance on COINQA. These findings highlight the effectiveness of page screenshot embeddings for multimodal retrieval, particularly in real-world financial contexts. Additionally, we reveal a need for further research on evaluation metrics in multimodal RAG systems.
Place, publisher, year, edition, pages
2025. , p. 46
Keywords [en]
Retrieval-Augmented Generation, Multimodal RAG, Large Language Models, Multimodal Embedding, Page Screenshot Embedding, Multimodal Retrieval, Financial Document QA, ChartQA
National Category
Natural Language Processing
Identifiers
URN: urn:nbn:se:liu:diva-214635ISRN: LIU-IDA/LITH-EX-A--25/032--SEOAI: oai:DiVA.org:liu-214635DiVA, id: diva2:1967837
External cooperation
Skandinaviska Enskilda Banken (SEB)
Subject / course
Computer science
Presentation
2025-05-28, John von Neumann, B-Huset, Linköping, 09:15 (English)
Supervisors
Examiners
2025-06-122025-06-122025-06-12Bibliographically approved