New use-cases for Vision Language Models: An update on our latest projects

# Ai

# Artificial intelligence

# SCEWC

# Hafnia

# VLM

# Training-aaS

# Analytics

An update on our latest projects

Taylor Seidel

Fulgencio Navarro

Last December, we released our first vision-language model (VLM), a traffic model built on NVIDIA Cosmos and fine-tuned on 75,000 hours of real-world traffic video from Europe and the United States. It powers Video Summarization for XProtect , which turns video segments into structured text summaries, and VLM as a Service, which gives developers and partners API access to the same underlying model. In March, we added an EU-optimized version, now running with selected EU cities.

The Traffic VLM is built for breadth within a smart-city vertical. It describes a wide range of road scenes and conditions and follows prompts for many traffic-related tasks. But a summary of the whole scene is not always what an operator needs. On a busy road network, the moments that demand the fastest action are often traffic accidents. When one happens, a Department of Transportation monitoring hundreds of cameras needs to know whether a collision actually happened, which vehicles were involved, and what happened, when, and where. A missed or misinterpreted accident delays a response. And too many false alarms create unnecessary work for operators.

For our latest projects, we’re going narrower and deeper. Our Accident VLM reuses much of what we built for the Traffic VLM, including Hafnia’s video library and our data pipelines built using NVIDIA Cosmos Curator . It builds on NVIDIA Cosmos 3 VLM and focuses on just one event type: traffic accidents. This post covers how we are building it, what the early numbers show, and what we still need to prove.

Why accidents need a use-case VLM

A traffic camera captures a continuous scene, but an accident is a brief event within it. To recognize one, a model needs to interpret how the scene changes over time: what vehicles were doing before the event, what happened at the moment of impact, and what followed.

That makes accident detection a different task from the scene summarization the Traffic VLM was built for. Getting there is not a matter of changing a prompt. It takes training data of collisions and careful labels on the video about what is visible, what is happening, and which vehicles are involved.

The model’s main job is to help find out whether an accident has occurred. Beyond that, we are training it to identify the vehicles involved, describe their movements and directions, recognize the collision type, and note lanes, positions, and the sequence of events. It can also produce structured descriptions to support incident reporting, without drawing conclusions about who was responsible.

Figure 1: The Accident VLM locating an accident in time and in the frame, with descriptions of the vehicles involved and the sequence of events.

Finding a focused dataset

Hafnia’s video library includes approximately 13,000 recorded traffic accidents, contributed by customers who want better AI for their cities. This library covers different road environments, camera angles, and conditions. Before any video is used for training, faces and license plates are anonymized using brighter AI technology. The data stays secure within our controlled infrastructure and is not shared with external foundation-model providers. Read more about the public part of our library here.

We began by testing general-purpose VLMs against real-world scenarios to see where they struggled. From there, the team designed a workflow with four stages:

Select candidate crashes

Trim the footage to the moments around the crash

Generate draft labels automatically

Have people review and complete the labels used for training

The selection process distinguishes between accidents that are visible, partly obscured, or not directly visible but inferable from the scene. That distinction matters: a useful model must learn the limits of what a camera can actually show, not just describe the clearest examples.

To keep the human labels consistent, annotators work from carefully defined guidelines, and several of them label the same clips to check that they agree. For our first checkpoints, we have carefully curated around 3,940 30-second clips covering 2,381 unique crashes. That is a checkpoint, not the final production model.

Early results

The current pre-release model starts from Cosmos 3 Nano (reasoner), an 8-billion-parameter (8B) model. To get a first read on where it stands, we compared it with three open video-language models (the original Cosmos3 Nano without post-training, Qwen3 VL 8B, and Gemma 4 31B) on a small test set: 43 accident events and 36 non-accident events, from cameras that were not used in training. All models received adapted prompts. Gemma 4 31B saw 32 frames per clip because of its video limit; the 8B models saw 100-128 frames per clip.

Figure 2: Early results for the pre-release Accident VLM compared with open video-language models, with reasoning. Test set: 43 accident clips and 36 normal clips from cameras unseen in training.

With reasoning turned on (the model works through the clip step by step before answering), our Accident VLM caught 86% of the accidents, compared with 16% to 28% for the other models. It also scored higher on analyzing and describing the crash (50%) and on identifying the involved parties (59%). But that came at a cost: it correctly cleared 72% of the normal clips, while the other models cleared 92% to 100%. In other words, it flagged crashes that did not actually happen.

These are early numbers from a small test set, not a final benchmark. Before we make firmer claims about accuracy, we need results from a larger test set of accident footage the model has never seen. The model also will not be right in every situation. Accidents can be hard to interpret, camera views are often limited, and unusual scenarios will keep challenging it.

However, initial results point in the direction of our model closing the existing gap on physical AI VLMs from the state of the art on true detections of accidents, with a minor cost of false positives.

What’s next

The early results point to a clear priority: bringing false alarms down without missing real accidents. We are also exploring synthetic and modified video to cover scenarios that are hard to capture in real-world footage, and we will keep releasing updated versions of the model as the work progresses.

Now, we are looking for partners to help us transform this unique AI technology into a product that improves their Traffic Management Centers. If you monitor traffic cameras or work on a smart-city traffic solution and want to see how the Accident VLM handles your video, it is available for controlled pilots.

Traffic accidents are only the beginning. Our specialization capabilities can apply to other challenges across public safety, transportation hubs, ports, logistics, manufacturing, retail, and other industries that rely on video. We think that is where AI delivers real value: solving specific problems and making people’s work easier, faster, and more reliable.

We want our customers to help decide where we go next. If you have a video AI use case that existing solutions cannot handle reliably, tell us about it. We can explore the technology together, evaluate the opportunity, and identify the potential business value. If it’s a good match, it might become our next use-case VLM. Do you have a proposition for the next VLM use case? Let us know here

Comments (0)

Popular