Open Collections

Chen, Xuezheng

University of British Columbia

Construction safety inspections typically involve a human inspector iden tifying safety concerns on-site. With the rise of powerful large pre-trained Vision Language Models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, the question “whether these large models trained on data scraped from the internet understand concepts related to construction safety” is yet answered. Evaluating this capability is critical before deploying pre-trained models on real construction sites, especially in zero-shot settings (i.e., directly deploying models without training). In this thesis, I attempt to evaluate artificial intelligence (AI)’s capability of understanding safety concepts in zero-shot settings by deploying a state-of-the-art (SOTA) pre-trained VLM to match 443 construction safety posters collected from the internet (i.e., visual inputs) with detailed descriptions of the poster written manually by the researchers (i.e., natural language inputs). Moreover, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this thesis, I propose the ConstructionSite 10k, featuring 10,013 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. My subsequent evaluation of current SOTA large pre-trained generative VLMs shows strong generalization abilities in zero-shot and few shot settings. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection. The aforementioned dataset and testing framework pave the way for large-scale evaluation and deployment of these models onsite.

Text

2024-12-19

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/89945

Civil Engineering

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/