What are VLMs, anyway?
Vision-Language Models enable AI to watch videos and then tell you what they saw. They are designed to act as interpreters between visual inputs and human language, allowing them to both understand and explain what they observe. Unlike traditional AI that either looks at pictures or reads text, VLMs do both at once.
Zero-shot learning is a machine learning technique where an AI model can recognize and categorize concepts, objects or tasks it has never been specifically trained on. Instead, the model uses its pre-existing knowledge and a text description to come up with the correct answer. This saves security companies massive amounts of money.
How VLMs transform video analytics
Want to find a man with a blue backpack walking out of a corner shop at a certain address? Just type it in. VLMs match your query to visual patterns without needing pre-defined markups. VLMs watch surveillance videos and generate plain-language summaries, as well as detect safety hazards like fires, anomalies or unauthorized access in real-time.
The benefits:
- Instant flexibility: Discovered something new? Just update the text prompt and deploy instantly.
- No retraining needed: One model detects everything and anything, you just need to change prompts.
- Privacy built-in: Faces automatically blurred in real-time.
- Natural language search: Any authorized person can request video materials in plain language without special training.
- Zero-shot capability: VLMs work on new tasks without subject-specific retraining.
- Video understanding: Tracks actions across time, not just in individual frames.
- Structured output: Generates JSON files for database integration.
- Minimal annotation: Needs less labeled training data than traditional AI.
Now, let’s look at the challenges
Hallucinations
When AI watches security footage and confidently lies about events or objects that are not actually present, it's called a hallucination. In a public safety context, such false reports carry serious risks, potentially causing wrongful accusations, wasted investigative resources, or the failure to identify genuine threats.
Why AI makes stuff up:
- AI predicts what words usually go together, but doesn't actually "see" what's in the video.
- AI might memorize patterns instead of understanding concepts. If trained mostly on red safety helmets, it gets confused by a yellow one.
- Some models struggle to pinpoint the time when an event occurred. They might tell that something happened, but won’t provide the timestamp.
Hallucinations aren’t rare, they happen all the time. The key is: AI must serve strictly as an assistant whose decisions require mandatory human supervision. When humans stay in the loop, we enable AI to be highly effective without the danger of making up fakes.
Why human supervision is mandatory
VLMs should never operate without constant human supervision. In Europe, this is actually a legal requirement under the EU Artificial Intelligence Act for high-risk systems used in law enforcement and public safety. The EU AI Act explicitly states: "No action or decision shall be taken based on AI identification, unless separately verified and confirmed by natural persons".
“High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use”.
- EU AI Act, Article 14: Human Oversight
- EU AI Act, Article 14: Human Oversight
Because VLMs frequently hallucinate, misjudge context and lack common sense, human supervision is mandatory to catch false positives and false negatives. In public safety, a mistake costs far more than a few extra minutes of review time.
To implement this level of control effectively, humans in charge should adopt clear operational protocols:
- Never trust, always verify every VLM output
- Two-person verification is necessary in specific sectors
- Human-in-the-loop approval for any action when modifying systems or accessing sensitive data
- Confidence scores, if below 80%, must be automatically sent to a human reviewer
- AI training: "If not sure, say “unable to confirm”.
The bottom line
VLMs are powerful tools that make video analytics accessible, fast and intelligent. They turn hours of footage into searchable, understandable information.
But they're assistants, not replacements. The ideal configuration pairs VLMs with human reviewers, allowing the AI to manage bulk scanning while humans provide the necessary judgment and context for final decisions.
Unlike traditional models that demand fine-tuning for each case, VLMs rely on natural language prompts, making them highly adaptable. At IREX.ai, this solution is ideal for irregular or slow-unfolding events, like fall detections, flooding, or monitoring video quality control, where per-frame real-time analysis is unnecessary. Key benefits include quick deployment for new tasks without the need for neural network training, the ability to collect data for future optimization, and GPU-based performance. This module will be integrated into a central rules management system, offering a uniform interface for managing prompts. The feature is currently under development and will be available in the near future.
Used responsibly, with constant human supervision, VLMs can make public safety more effective while protecting against the risks of full automation. The technology is exciting, but human oversight is what keeps it trustworthy.
