From Hallucination Detection to Adversarial Defense:
A Unified Framework for LVLM and LLM Safety

Outline

Introduction: Why trustworthy multimodal language models?

We begin by describing where LLMs and LVLMs are deployed, why their failures differ from those of conventional classifiers, and how safety depends on the interaction between model knowledge, prompts, multimodal grounding, decoding, external tools, and users. This motivates our central framework: detect, quantify, and defend.

Part I: Hallucinations, bias, grounding, and verification.

This part introduces a taxonomy of hallucinations, including intrinsic errors that contradict the provided source, extrinsic errors that conflict with external knowledge, object and attribute hallucinations in LVLMs, and reasoning failures. We discuss the role of pre-training data, fine-tuning, user misconceptions, decoding, and insufficient visual grounding. We then review fact-checking loops, retrieval, self-correction, consistency tests, and external verification.

Part II: Uncertainty Quantification for LLMs and LVLMs.

Modern generative models can assign high probability to incorrect outputs. We explain why raw token probabilities and token entropy do not provide a complete measure of confidence. The tutorial then covers calibration, predictive entropy, sequence likelihood, sampling-based uncertainty, semantic clustering, semantic entropy, consistency across generations, ensembles, Bayesian approximations, and uncertainty-aware abstention. Particular attention is given to the distinction between local token-level uncertainty and global sentence- or meaning-level uncertainty.

Part III: Adversarial attacks on language and multimodal models.

We map the attack surface across user prompts, system prompts, visual inputs, demonstrations, retrieval databases, context windows, fine-tuning datasets, and tool-using agents. We introduce prompt injection, jailbreaks, adversarial images, data poisoning, backdoors, model extraction, gradient-based discrete suffix optimization such as GCG, and black-box social-engineering attacks. We emphasize the special difficulties created by discrete language inputs and multimodal interactions.

Part IV: Mitigation and robust defense.

This part compares defenses at different stages of the pipeline: input filtering and purification, trustworthy retrieval and context enhancement, model alignment, adversarial training, uncertainty-aware rejection, safety classifiers, robust system prompts, constrained or verified decoding, output monitoring, post-inference correction, and human oversight. We discuss how to evaluate defenses without relying only on average benchmark performance and why adaptive attacks must be considered.

Interactive case study and practical evaluation.

Participants will audit a small LLM or LVLM pipeline. They will generate multiple answers, measure lexical and semantic disagreement, inspect calibration and failure prediction, construct adversarial prompts, and compare simple mitigation strategies. The session will demonstrate how hallucination detection, uncertainty quantification, and adversarial robustness complement one another in a single safety workflow.

Open-source tools and supplementary material.

The website and repository will provide slides, annotated references, code, and Google Colab notebooks. The tutorial will also introduce the Torch-Uncertainty library and complementary tools from the Hugging Face ecosystem for model inference, evaluation, and safety auditing.

Intended audience and prerequisites

The tutorial targets academic researchers, graduate students, machine learning engineers, and industry practitioners interested in AI safety, security, trustworthy machine learning, and multimodal foundation models. It is particularly relevant to researchers and engineers deploying generative models in high-stakes environments.

Recommended prerequisites:

• Graduate-level knowledge of deep learning and optimization.
• Basic familiarity with Transformers, attention, and autoregressive generation.
• Basic Python knowledge and familiarity with the Hugging Face ecosystem for the practical session.

Relation to prior tutorials and short courses

Previous tutorials have studied individual dimensions of trustworthy generative AI. Some have focused on uncertainty quantification and reliability for deep learning, while others have addressed hallucinations, factuality, jailbreaks, or agentic attacks. Examples include The Many Faces of Reliability of Deep Learning for Real-World Deployment, the AAAI tutorial on uncertainty quantification for large language models, and the Jailbreaking LLMs and Agentic Systems tutorial.

Our tutorial differs by placing hallucination analysis, semantic uncertainty, adversarial threat modeling, and robust defense within one unified framework. It also emphasizes practical evaluation: participants will learn not only how failures arise, but also how to measure predictive risk and test whether a defense remains effective against adaptive attacks.

Selected References

1. Laurent, O., Lafage, A., Tartaglione, E., Daniel, G., Martinez, J. M., Bursuc, A., & Franchi, G. Packed-Ensembles for Efficient Uncertainty Estimation. ICLR, 2023.
2. Lafage, A., Laurent, O., Gabtni, F., & Franchi, G. Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification. NeurIPS, 2025.
3. Franchi, G., Laurent, O., Leguéry, M., Bursuc, A., Pilzer, A., & Yao, A. Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models. CVPR, 2024.
4. Hoche, J., Bursuc, A., Brellmann, D., Louppe, G., Izmailov, P., Yao, A., & Franchi, G. Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes. arXiv, 2025.
5. Xiao, J., Yao, A., Li, Y., & Chua, T. S. Can I Trust Your Answer? Visually Grounded Video Question Answering. CVPR, 2024.
6. Zou, A., Wang, Z., Kolter, J. Z., & Fredrikson, M. Universal and Transferable Adversarial Attacks on Aligned Language Models. 2023.
7. Ji, Z., Lee, N., Frieske, R., et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023.
8. Kuhn, L., Gal, Y., & Farquhar, S. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. ICLR, 2023.

Additional references and an annotated safety roadmap will be released with the tutorial materials.

A tutorial connecting academia and industry

Organized by researchers from AMIAD, ENSTA Paris, Safran Tech, Inria, valeo.ai, and the National University of Singapore.