ViBR: Automated Bug Replay from Video-Based Reports using Vision-Language Models
Abstract
Bug reports play a critical role in software maintenance by helping users convey encountered issues to developers. Recently, GUI screen capture videos have gained popularity as a bug reporting artifact due to their ease of use and ability to retain rich contextual information. However, automatically reproducing bugs from such recordings remains a significant challenge. Existing methods often rely on fragile image-processing heuristics, explicit touch indicators, or pre-constructed UI transition graphs, which require non-trivial instrumentation and app-specific setup. This paper presents ViBR, a lightweight and fully automated approach that reproduces bugs directly from GUI recordings. Specifically, ViBR combines CLIP-based embedding similarity for action boundary segmentation with Vision-Language Models (VLMs) for region-aware GUI state comparison and guided bug replay. Experimental results show that ViBR successfully reproduces 72% of bug recordings, significantly outperforming state-of-the-art baselines and ablation variants.
BibTeX
@article{Feng-al:FSE26,
author = {Sidong Feng and
Dingbang Wang and
Nikola Tomic and
Tingting Yu and
Aldeida Aleti and
Chunyang Chen},
title = {{ViBR:} Automated Bug Replay from {Video-Based} Reports using {Vision-Language} Models},
journal = {{PACMSE}},
volume = {3},
number = {{FSE}},
pages = {3275--3297},
year = {2026},
}