Large vision-language models (LVLMs) still hallucinate. But why? We find that the main culprit is not the vision backbone: it's the model trusting what the prompt says over what the image shows.
We diagnose this with a new benchmark, HalluScope, and fix it with a preference-optimization method, HalluVL-DPO, which reduces instruction-driven hallucinations while keeping performance strong on other benchmarks.
🔬 HalluScope
A benchmark that separates where hallucinations come from: perception, co-occurrence priors, or the prompt itself.
🛠️ HalluVL-DPO
A weighted DPO fine-tuning recipe and dataset that teach the model to prefer grounded answers over hallucinated ones.
Hallucinations in LVLMs increasingly arise from conflicts between language priors and visual information, rather than from perceptual limitations alone. Yet existing benchmarks (POPE, CHAIR, SHR, MMHal-Bench) do not tell apart hallucinations caused by perception failures, learned object co-occurrence priors, or presuppositions introduced by the instruction itself.
HalluScope 
Where do LVLM hallucinations actually come from?
What we do. HalluScope asks the same model about the same image in different ways, so that each question isolates one possible cause of hallucination:
Perception failures
Can the model correctly see what is in the image?
Real & random object recognition.
Co-occurrence priors
Does it claim objects that usually appear in such scenes but are absent here?
Adversarial recognition.
Instruction presuppositions
Does it accept a false assumption stated in the prompt?
Adversarial presupposition.
🔍 Examples from HalluScope
Each image comes with four kinds of questions. Use the arrows to browse.
✨ What we found
Vision is not the bottleneck
Recognition accuracy on present and random absent objects stays high across all models.
Co-occurrence priors hurt
Accuracy drops on objects that are absent but likely to appear in such scenes.
The prompt hurts the most
When the question presupposes an absent object, performance drops the most: at least 15% more than co-occurrence alone.
🤗 HalluScope benchmark is available on Hugging Face.
Get the dataset coming soonHalluVL-DPO 🛠️
Can we teach the model to trust the image over the prompt?
What we do. We fine-tune off-the-shelf LVLMs with a variant of Direct Preference Optimization (DPO) that targets prompt-induced hallucinations:
- Build preference pairs. For each image and question, we create a preferred (visually grounded) answer and a rejected (hallucinated) one.
- Weight each sample. Pairs are weighted by their semantic gap, so that the most informative pairs count more during training.
- Fine-tune with DPO. The model learns to prefer grounded answers, even when the prompt pushes it the other way.
🤗 HalluVL-DPO training dataset is available on Hugging Face.
Get the dataset coming soon📈 Results
The improvement doesn't come at the expense of other abilities: the fine-tuned model stays competitive or improves on other hallucination benchmarks and general multimodal evaluations.
