Vision-Language Models (VLMs) are increasingly deployed in applications that interpret and generate information from visual and textual inputs. While powerful, these models pose emerging privacy risks. In this paper, we introduce the concept of privacy chains: structured narratives that emerge when adversaries aggregate outputs from VLMs across multiple images, often exposing sensitive information even when the individual outputs are seemingly innocuous. Using LangChain, an open-source orchestration framework, we show how identity-linked data extracted via both benign and targeted prompts can be compiled into detailed timelines of private behavior, significantly amplifying privacy threats. To systematically assess this risk, we develop a privacy leakage pipeline within the Visual Question Answering (VQA) framework and evaluate six open-source VLMs across three tailored datasets: Celebrity, Car, and Tattoo. Our analysis reveals substantial and model-dependent privacy leakage, even from general-purpose queries. To mitigate this threat, we propose ChainShield, a white-box adversarial defense that applies targeted, imperceptible perturbations to images. ChainShield reduces privacy-relevant outputs by redirecting VLM responses toward benign alternatives, while preserving image realism. Our experiments show that ChainShield substantially lowers privacy leakage across models and datasets, effectively disrupting the formation of privacy chains.

This thesis addresses the need to balance the use of facial recognition systems with the need to protect personal privacy in machine learning and biometric identification. As advances in deep learning accelerate their evolution, facial recognition systems enhance security capabilities, but also risk invading personal privacy. Our research identifies and addresses critical vulnerabilities inherent in facial recognition systems, and proposes innovative privacy-enhancing technologies that anonymize facial data while maintaining its utility for legitimate applications.

Our investigation centers on the development of methodologies and frameworks that achieve k-anonymity in facial datasets; leverage identity disentanglement to facilitate anonymization; exploit the vulnerabilities of facial recognition systems to underscore their limitations; and implement practical defenses against unauthorized recognition systems. We introduce novel contributions such as AnonFACES, StyleID, IdDecoder, StyleAdv, and DiffPrivate, each designed to protect facial privacy through advanced adversarial machine learning techniques and generative models. These solutions not only demonstrate the feasibility of protecting facial privacy in an increasingly surveilled world, but also highlight the ongoing need for robust countermeasures against the ever-evolving capabilities of facial recognition technology.

Continuous innovation in privacy-enhancing technologies is required to safeguard individuals from the pervasive reach of digital surveillance and protect their fundamental right to privacy. By providing open-source, publicly available tools, and frameworks, this thesis contributes to the collective effort to ensure that advancements in facial recognition serve the public good without compromising individual rights. Our multi-disciplinary approach bridges the gap between biometric systems, adversarial machine learning, and generative modeling to pave the way for future research in the domain and support AI innovation where technological advancement and privacy are balanced.  

In this era of ubiquitous surveillance and online presence, protecting facial privacy has become a critical concern for individuals and society as a whole. Adversarial attacks have emerged as a promising solution to this problem, but current methods are limited in quality or are impractical for sensitive domains such as facial editing. This paper presents a novel adversarial image editing framework called StyleAdv, which leverages StyleGAN's latent spaces to generate powerful adversarial images, providing an effective tool against facial recognition systems. StyleAdv achieves high success rates by employing meaningful facial editing with StyleGAN while maintaining image quality, addressing a challenge faced by existing methods. To do so, the comprehensive framework integrates semantic editing, adversarial attacks, and face recognition systems, providing a cohesive and robust tool for privacy protection. We also introduce the ``residual attack`` strategy, using residual information to enhance attack success rates. Our evaluation offers insights into effective editing, discussing tradeoffs in latent spaces, optimal edits for our optimizer, and the impact of utilizing residual information. Our approach is transferable to state-of-the-art facial recognition systems, making it a versatile tool for privacy protection. In addition, we provide a user-friendly interface with multiple editing options to help users create effective adversarial images. Extensive experiments are used to provide insights and demonstrate that StyleAdv outperforms state-of-the-art methods in terms of both attack success rate and image quality. By providing a versatile tool for generating high-quality adversarial samples, StyleAdv can be used both to enhance individual users' privacy and to stimulate advances in adversarial attack and defense research.

Most state-of-the-art facial recognition systems (FRS:s) use face embeddings. In this paper, we present the IdDecoder framework, capable of effectively synthesizing realistic-neutralized face images from face embeddings, and two effective attacks on state-of-the-art facial recognition models using embeddings. The first attack is a black-box version of a model inversion attack that allows the attacker to reconstruct a realistic face image that is both visually and numerically (as determined by the FRS:s) recognized as the same identity as the original face used to create a given face embedding. This attack raises significant privacy concerns regarding the membership of the gallery dataset of these systems and highlights the importance of both the people designing and deploying FRS:s paying greater attention to the protection of the face embeddings than currently done. The second attack is a novel attack that performs the model inversion, so to instead create the face of an alternative identity that is visually different from the original identity but has close identity distance (ensuring that it is recognized as being of the same identity). This attack increases the attacked system's false acceptance rate and raises significant security concerns. Finally, we use IdDecoder to visualize, evaluate, and provide insights into differences between three state-of-the-art facial embedding models.

Privacy of machine learning models is one of the remaining challenges that hinder the broad adoption of Artificial Intelligent (AI). This paper considers this problem in the context of image datasets containing faces. Anonymization of such datasets is becoming increasingly important due to their central role in the training of autonomous cars, for example, and the vast amount of data generated by surveillance systems. While most prior work de-identifies facial images by modifying identity features in pixel space, we instead project the image onto the latent space of a Generative Adversarial Network (GAN) model, find the features that provide the biggest identity disentanglement, and then manipulate these features in latent space, pixel space, or both. The main contribution of the paper is the design of a feature-preserving anonymization framework, StyleID, which protects the individuals’ identity, while preserving as many characteristics of the original faces in the image dataset as possible. As part of the contribution, we present a novel disentanglement metric, three complementing disentanglement methods, and new insights into identity disentanglement. StyleID provides tunable privacy, has low computational complexity, and is shown to outperform current state-of-the-art solutions.

There is limited prior work studying how the ad personalization experienced by different users is impacted by the use of adblockers, geographic location, the user's persona, or what browser they use. To address this void, this paper presents a novel profile-based evaluation of the personalization experienced by carefully crafted user profiles. Our evaluation framework impersonates different users and captures how the personalization changes over time, how it changes when adding or removing an extension, and perhaps most importantly how the results differ depending on the profile's persona (e.g., interest, occupation, age, gender), geographic location (US East, US West, UK), what browser extension they use (none, AdBlock, AdBlock Plus, Ghostery, CatBlock), what browser they use (Chrome, Firefox), and whether they are logged in to their Google account. By comparing and contrasting observed differences we provide insights that help explain why some user groups may feel more targeted than others and why some people may feel even more targeted after having turned on their adblocker.  

Telco systems usually run large-scale, centralized key management systems. However, centralized approaches based on conventional public key encryption like RSA raise problems such as key escrow, secure channel to delivery key, and third-party query as well as single point of failure. To address these problems, we propose both certificate-based encryption (CBE) and hierarchical certificate-based encryption (HCBE) schemes proved secure in the standard model. Compared with other schemes, our schemes are proved IND-CCA2 (Indistinguishability under Adaptive Chosen Ciphertext Attack) secure in full model, where the number of group elements is independent of the value of security parameter. As far as we know, the proposed HCBE is the first fully IND-CCA2 secure scheme with ciphetexts of constant size.

To achieve confidentiality, integrity, authentication, and non-repudiation simultaneously, the concept of signcryption was introduced by combining encryption and a signature in a single scheme. Certificate-based encryption schemes are designed to resolve the key escrow problem of identity-based encryption, as well as to simplify the certificate management problem in traditional public key cryptosystems. In this paper, we propose a new certificate-based signcryption scheme that has been proved to be secure against adaptive chosen ciphertext attacks and existentially unforgeable against chosen-message attacks in the random oracle model. Our scheme is not based on pairing and thus is efficient and practical. Furthermore, it allows a signcrypted message to be immediately verified by the public key of the sender. This means that verification and decryption of the signcrypted message are decoupled. To the best of our knowledge, this is the first signcryption scheme without pairing to have this feature.

Recently, a lot of researches focused on identity-based encryption (IBE). The advantage of this scheme is that it can reduce the cost of the public key infrastructure by simplifying certificate management. Although IBE has its own innovations, one of its weaknesses is the key escrow problem. That is, the private key generator in IBE knows decryption keys for all identities and consequently can decrypt any ciphertexts. The certificate-based encryption (CBE) scheme proposed in EUROCRYPT 2003 provides a solution for the key escrow problem by allowing the certification authority to possess a partial decryption key that comprises the full decryption key together with the user-generated private key. In this paper, we propose new CBE schemes without pairing and prove them to be Indistinguishability under Chosen Ciphertext Attack secure in the random oracle model based on the hardness of the computational Diffie-Hellman problem. When compared with other CBE schemes, our schemes are significantly efficient in terms of performance, which makes our schemes suitable for computation-limited node (e.g., sensor, wearable device) networks. Copyright (c) 2016 John Wiley & Sons, Ltd.