liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Retrieval-Augmented Generation with Charts: Improving retrieval for images of charts
Linköping University, Department of Computer and Information Science, The Division of Statistics and Machine Learning.
Linköping University, Department of Computer and Information Science, The Division of Statistics and Machine Learning.
2025 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Recent advancements in Retrieval-Augmented Generation (RAG) systems have enabled large language models (LLMs) to better access and utilize external knowledge, yet the integration of non-textual data, particularly charts, remains underexplored. This thesis investigates strategies for preprocessing and embedding chart images in multimodal RAG systems to improve retrieval and generation performance. Three embedding methods, text summary, dual storage, and page screenshot, were developed and evaluated on two datasets: the chart-only CHARTQA and the newly created real-world financial report based COINQA dataset. Results show that while all strategies perform similarly on chart-only data, the page screenshot embedding significantly outperforms other methods in retrieval tasks for mixed text-and-chart documents, achieving more than twice the retrieval performance on COINQA. These findings highlight the effectiveness of page screenshot embeddings for multimodal retrieval, particularly in real-world financial contexts. Additionally, we reveal a need for further research on evaluation metrics in multimodal RAG systems.

Place, publisher, year, edition, pages
2025. , p. 46
Keywords [en]
Retrieval-Augmented Generation, Multimodal RAG, Large Language Models, Multimodal Embedding, Page Screenshot Embedding, Multimodal Retrieval, Financial Document QA, ChartQA
National Category
Natural Language Processing
Identifiers
URN: urn:nbn:se:liu:diva-214635ISRN: LIU-IDA/LITH-EX-A--25/032--SEOAI: oai:DiVA.org:liu-214635DiVA, id: diva2:1967837
External cooperation
Skandinaviska Enskilda Banken (SEB)
Subject / course
Computer science
Presentation
2025-05-28, John von Neumann, B-Huset, Linköping, 09:15 (English)
Supervisors
Examiners
Available from: 2025-06-12 Created: 2025-06-12 Last updated: 2025-06-12Bibliographically approved

Open Access in DiVA

Retrieval-Augmented Generation with Charts(18486 kB)1120 downloads
File information
File name FULLTEXT01.pdfFile size 18486 kBChecksum SHA-512
3b5d1e0662fd5505dfa21fd91cbe93e65c0c09182dca322f3c775ad51725f74f52518dac6f678a3d4b0fb73a534d3d39ec7c48292e04b39168c534a78b3d680d
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Swedberg, OlofÖhman, Hjalmar
By organisation
The Division of Statistics and Machine Learning
Natural Language Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 1121 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 698 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf