liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
From Privacy Chains to ChainShield: Structured Privacy Risks and Defense in Vision-Language Models
Linköping University, Department of Computer and Information Science, Database and information techniques. Linköping University, Faculty of Science & Engineering.ORCID iD: 0009-0001-0554-3720
Linköping University, Department of Computer and Information Science, Database and information techniques. Linköping University, Faculty of Science & Engineering.ORCID iD: 0000-0003-2391-5951
Linköping University, Department of Computer and Information Science, Database and information techniques. Linköping University, Faculty of Science & Engineering.ORCID iD: 0000-0003-1367-1594
2025 (English)In: PROCEEDINGS OF THE 24TH WORKSHOP ON PRIVACY IN THE ELECTRONIC SOCIETY, WPES 2025, Association for Computing Machinery (ACM), 2025, p. 116-133Conference paper, Published paper (Refereed)
Abstract [en]

Vision-Language Models (VLMs) are increasingly deployed in applications that interpret and generate information from visual and textual inputs. While powerful, these models pose emerging privacy risks. In this paper, we introduce the concept of privacy chains: structured narratives that emerge when adversaries aggregate outputs from VLMs across multiple images, often exposing sensitive information even when the individual outputs are seemingly innocuous. Using LangChain, an open-source orchestration framework, we show how identity-linked data extracted via both benign and targeted prompts can be compiled into detailed timelines of private behavior, significantly amplifying privacy threats. To systematically assess this risk, we develop a privacy leakage pipeline within the Visual Question Answering (VQA) framework and evaluate six open-source VLMs across three tailored datasets: Celebrity, Car, and Tattoo. Our analysis reveals substantial and model-dependent privacy leakage, even from general-purpose queries. To mitigate this threat, we propose ChainShield, a white-box adversarial defense that applies targeted, imperceptible perturbations to images. ChainShield reduces privacy-relevant outputs by redirecting VLM responses toward benign alternatives, while preserving image realism. Our experiments show that ChainShield substantially lowers privacy leakage across models and datasets, effectively disrupting the formation of privacy chains.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2025. p. 116-133
Keywords [en]
Privacy; Vision-Language Models; LangChain; Adversarial Attacks
National Category
Bioinformatics (Computational Biology)
Identifiers
URN: urn:nbn:se:liu:diva-220968DOI: 10.1145/3733802.3764048ISI: 001656324800010Scopus ID: 2-s2.0-105023662249ISBN: 9798400718984 (print)OAI: oai:DiVA.org:liu-220968DiVA, id: diva2:2034558
Conference
24th Workshop on Privacy in the Electronic Society-WPES, Taipei, TAIWAN, oct 13, 2025
Note

Funding Agencies|Swedish Research Council (VR); Graduate School in Computer Science (CUGS) at Linkoping University

Available from: 2026-02-02 Created: 2026-02-02 Last updated: 2026-06-17

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Le, Minh-Ha

Search in DiVA

By author/editor
Liu, MinxingLe, Minh-HaCarlsson, Niklas
By organisation
Database and information techniquesFaculty of Science & Engineering
Bioinformatics (Computational Biology)

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 26 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf