Applied Science, Faculty of
Civil Engineering, Department of
DSpace
UBCV
Suen, Wun Ki (Christine)
Master of Applied Science - MASc
University of British Columbia
The hazardous working environment of construction workers leads to fatal accidents, whereas it is challenging to balance worker safety and productivity. Egocentric (i.e., first-person view) pose estimation is effective in identifying, localizing, and tracking worker poses under construction scenarios. It can be used as a human-centric approach to reduce fatalities and increase productivity through various downstream applications within Intelligent Construction, such as safety monitoring and human-robot collaboration. However, most of the egocentric pose estimation models are trained with daily life poses and have limited generalization ability to construction poses. The scarcity of publicly available datasets designated for egocentric worker pose estimation further impedes the training and evaluation of existing and newly developed egocentric worker pose estimation models. In this regard, this thesis presents a dual-modules egocentric worker pose estimation approach, which effectively addresses the major issues of self-occlusion and depth ambiguity in worker pose estimation. The proposed approach employs a supervised 2D module to address self-occlusion in construction poses and an unsupervised 3D module to handle depth ambiguity using only monocular-view input. It not only achieves a PCKh@0.5 of 98.7% in 2D pose estimation and a N-MPJPE of 96.43mm in 3D pose estimation, but also demonstrates strong generalization ability in the construction domain. To address the significant gap in egocentric pose estimation dataset for construction environments, this thesis also introduces ICON-Pose, the first open dataset includes thousands of egocentric images and corresponding 2D and 3D worker poses data across 63 action types. ICON-Pose accurately depicts complex construction worker poses with the proposed approach and shows a high level of diversity in action types, data forms, participants, and construction site settings. This dataset has the potential to serve as a general benchmark dataset not only for subsequent analyses in Intelligent Construction but also in the broader field of egocentric vision, particularly for capturing and analyzing out-of-distribution poses. It is expected to support artificial intelligence research in the construction domain along with the egocentric worker pose estimation approach.
Egocentric Monocular ConstructionWorker Pose Estimation for IntelligentConstructionbyWun Ki (Christine) SuenBSc., The Hong Kong Polytechnic University, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Civil Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)April 2024© Wun Ki (Christine) Suen 2024The following individuals certify that they have read, and recommend to theFaculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Egocentric Monocular Construction Worker Pose Estimation for Intel-ligent Constructionsubmitted by Wun Ki (Christine) Suen in partial fulfillment of the require-ments for the degree of Master of Applied Science in Civil Engineering.Examining Committee:Zhengbo Zou, Assistant Professor, Department of Civil Engineering, UBCSupervisorHelge Rhodin, Professor, Faculty of Technology, Bielefeld University;Adjunct Professor, Department of Computer Science, UBCSupervisory Committee MemberiiAbstractThe hazardous working environment of construction workers leads to fatal acci-dents, whereas it is challenging to balance worker safety and productivity. Ego-centric (i.e., first-person view) pose estimation is effective in identifying, localiz-ing, and tracking worker poses under construction scenarios. It can be used asa human-centric approach to reduce fatalities and increase productivity throughvarious downstream applications within Intelligent Construction, such as safetymonitoring and human-robot collaboration. However, most of the egocentric poseestimation models are trained with daily life poses and have limited generaliza-tion ability to construction poses. The scarcity of publicly available datasetsdesignated for egocentric worker pose estimation further impedes the trainingand evaluation of existing and newly developed egocentric worker pose estimationmodels.In this regard, this thesis presents a dual-modules egocentric worker pose esti-mation approach, which effectively addresses the major issues of self-occlusionand depth ambiguity in worker pose estimation. The proposed approach employsa supervised 2D module to address self-occlusion in construction poses and anunsupervised 3D module to handle depth ambiguity using only monocular-viewinput. It not only achieves a PCKh@0.5 of 98.7% in 2D pose estimation and aN-MPJPE of 96.43mm in 3D pose estimation, but also demonstrates strong gen-eralization ability in the construction domain. To address the significant gap inegocentric pose estimation dataset for construction environments, this thesis alsointroduces ICON-Pose, the first open dataset includes thousands of egocentricimages and corresponding 2D and 3D worker poses data across 63 action types.ICON-Pose accurately depicts complex construction worker poses with the pro-posed approach and shows a high level of diversity in action types, data forms,participants, and construction site settings. This dataset has the potential toiiiAbstractserve as a general benchmark dataset not only for subsequent analyses in Intelli-gent Construction but also in the broader field of egocentric vision, particularlyfor capturing and analyzing out-of-distribution poses. It is expected to supportartificial intelligence research in the construction domain along with the egocentricworker pose estimation approach.ivLay SummaryThis paper introduces a novel approach and dataset to improve safety and produc-tivity in the construction industry through Egocentric Human Pose Estimation,which tracks and analyzes construction worker body positions in the first per-son perspective collected through a camera mounted on the worker’s hardhat.Traditional machine learning models in pose estimation often fail in constructionsettings due to the complexity of construction poses and the difficulty in obtain-ing 3D ground-truth data for training. To fill this gap, the paper proposes anapproach that accurately identifies construction worker poses without the needof 3D data collection and annotation. Moreover, it presents ICON-Pose, the firstdataset with thousands of egocentric images showing construction workers in adiverse range of actions. The proposed approach and dataset are expected to en-hance the safety and operational efficiency in the field of construction by assistingin applications such as worker safety monitoring and human-robot collaboration.vPrefaceThe research in this thesis was conducted by the author under the direct su-pervision of Dr.Zhengbo Zou, an assistant professor in the Department of CivilEngineering at The University of British Columbia. The author was responsiblefor the background study, literature review, design and implementation of themodel presented in this study, as well as the analysis and interpretation of themodel results.This thesis consists of an introductory chapter (Chapter 1); a chapter (Chap-ter 2) introduces related works in my study; a main chapter (Chapter 3) thatdescribes both the proposed model developed and the proposed dataset; a chap-ter (Chapter 4) evaluates our proposed model and dataset; and a final concludingchapter (Chapter 5) that synthesizes this composition’s findings and opportunitiesfor future work. A version of section 3.1.1 in Chapter 3 has been presented in-Computing in Civil Engineering (Suen, Christine Wun Ki, Ziming Liu, YangmingShi, and Zhengbo Zou. ”ICON-Pose: Toward Egocentric Action Recognition forIntelligent Construction.” In Computing in Civil Engineering 2023, pp. 682-689.).viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiList of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Human Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . 72.1.1 2D Human Pose Estimation . . . . . . . . . . . . . . . . . 72.1.1.1 Regression-based Pose Estimation . . . . . . . . . 82.1.1.2 Heatmap-based Pose Estimation . . . . . . . . . . 82.1.2 3D Human Pose Estimation . . . . . . . . . . . . . . . . . 92.1.2.1 Egocentric 3D Human Pose Estimation . . . . . . 9viiTable of Contents2.1.2.2 Unsupervised 3D Monocular Human Pose Estima-tion . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.3 Construction Worker Pose Estimation . . . . . . . . . . . . 122.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2 Related Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.1 General Egocentric Datasets . . . . . . . . . . . . . . . . . 142.2.2 Construction Worker Pose Estimation Datasets . . . . . . 152.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1 Egocentric Monocular Worker Pose Estimation . . . . . . . . . . . 193.1.1 2D Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.1.1.1 Model Design . . . . . . . . . . . . . . . . . . . . 203.1.1.2 Training details . . . . . . . . . . . . . . . . . . . 223.1.2 3D Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.1.2.1 Model Design . . . . . . . . . . . . . . . . . . . . 233.1.2.2 Training details: . . . . . . . . . . . . . . . . . . 243.2 ICON-Pose Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2.1 Motion Capture Setup . . . . . . . . . . . . . . . . . . . . 263.2.2 Construction Actions . . . . . . . . . . . . . . . . . . . . . 263.2.3 Dataset Collection and Augmentation . . . . . . . . . . . . 273.2.4 Data Annotation . . . . . . . . . . . . . . . . . . . . . . . 283.2.5 Dataset Design . . . . . . . . . . . . . . . . . . . . . . . . . 293.2.6 Downstream Applications . . . . . . . . . . . . . . . . . . . 313.2.6.1 Worker Activity Recognition . . . . . . . . . . . . 324 Result and evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 354.1 2D Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.1.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.1.2 Ablation for Integral Worker Pose Estimation Model . . . 374.2 3D Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.2.1 Value of ICON-Pose . . . . . . . . . . . . . . . . . . . . . . 404.3 Limitations and Future Works . . . . . . . . . . . . . . . . . . . . 425 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43viiiTable of ContentsBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44ixList of Tables3.1 Construction Actions in ICON-Pose . . . . . . . . . . . . . . . . . 283.2 Configuration of pre-defined skeleton . . . . . . . . . . . . . . . . . 324.1 Ablation of integral, heatmap, and regression methods in HPE . . 38xList of Figures1.1 Comparison of 3rd and 1st captures. . . . . . . . . . . . . . . . . . 32.1 Comparison between existing worker pose datasets and ICON-Pose. 172.2 Comparison between existing egocentric datasets and ICON-Pose. 183.1 Overview of the egocentric worker pose estimation approach. . . . 203.2 Architecture of the 2D module. . . . . . . . . . . . . . . . . . . . . 203.3 Architecture of the 2D module. . . . . . . . . . . . . . . . . . . . . 213.4 Architecture of the 2D module. . . . . . . . . . . . . . . . . . . . . 223.5 Architecture of 3D module. . . . . . . . . . . . . . . . . . . . . . . 243.6 Illustration of our dataset background augmentation. . . . . . . . . 293.7 Configuration of pre-defined skeleton. . . . . . . . . . . . . . . . . 313.8 Sample data modalities of ICON-Pose. . . . . . . . . . . . . . . . . 333.9 Temporal interpolation for action classification. . . . . . . . . . . . 344.1 2D Worker pose estimation results in PCK. . . . . . . . . . . . . . 364.2 2D Worker pose estimation results. . . . . . . . . . . . . . . . . . . 384.3 3D Worker pose estimation results. . . . . . . . . . . . . . . . . . . 404.4 Worker pose estimation result fine-tuning by ICON-Pose. . . . . . 41xiList of Abbreviationsfps frames per secondGAN Generative Adversarial NetsHoG Histograms of Oriented GradientsHoF Histograms of Optical FlowHPE Human pose estimationIEF Iterative Error FeedbackIndustry 5.0 Fifth Industrial RevolutionMBH Motion Boundary Histogramsmocap Motion CaptureMPJPE Mean Per Joint Position ErrorMSE Mean Squared ErrorN-MPJPE Normalized Mean Per Joint Position ErrorPA-MPJPE Procrustes Alignment Mean Per Joint Position ErrorPCK Percentage of Correct KeypointsPCP Percentage of Correct Partsrbf radial basis functionSVM Support Vector MachineVY Virtual RealityXR Extended RealityxiiList of SymbolsH HeatmapsJ JointP PoseH˜i Normalized Heatmapp Joint LocationD DepthY 3D Posex 2D PoseL3D 3D LossL2D 2D LossLdepth Depth LossT Time SpanxiiiAcknowledgementsFirst and foremost, I would like to express my wholehearted gratitude to mysupervisor, Dr. Zhengbo Zou. His guidance and support throughout my master’sdegree are precious. Dr. Zou not only gave me room to explore my researchinterests freely but also showed me the directions whenever I faced moments ofuncertainty or confusion. I would also like to extend my gratitude to Dr. HelgeRhodin for his valuable opinions and constructive feedback. His insights haveenriched my understanding and contributed significantly to the quality of mywork. Both Dr. Zou and Dr. Rhodin have made significant influence on myacademic development and I am truly grateful for their mentorship and support.I am also thankful to my friends and lab mates. They assisted my troublesomedata collection in their busy time. Special thanks to Lei and Weijia, they providedme lots of inspiration whenever I met obstacles in my research.I would like to express my highest gratitude to my family, especially my mumand dad. They love me the most as always. My dad supports and respects everydecision I made. He gives me the courage to be here and go beyond. Last but notleast, I deeply appreciate my partner, Vincent, for his patience and consideration.He knows me the best and have been the greatest source of strength during mymaster’s journey. I feel blessed with all these kindnesses received.xivDedicationTo my family.xvChapter 1Introduction1.1 Problem StatementThe recently released Fifth Industrial Revolution (Industry 5.0) marks an emerg-ing phase of industrialisation that emphasizes the integration of humans and ad-vanced technology to enhance the performance and effectiveness of the productioncycle with human-centric approaches. This evolution is particularly relevant tothe construction sector, where nearly 50% of the 4,764 fatal occupational injuriesin the U.S. occurred in construction-related jobs. These numbers indicate thehazards of the construction industry and the critical need for a human-centricapproach for improving worker safety and construction productivity.Human pose estimation is a key topic in computer vision that focuses on capturinghuman motion and understanding human behavior [1]. It has a board spectrum ofapplications across various domains and have had a profound impact particularlyin the field of Intelligent Construction [2]. The construction industry has seenthe adoption of human pose estimation for its potential to reduce constructionfatalities and increase operational efficiency [2, 3]. It can serve as a tool for safetymonitoring by interpreting worker poses and identifying dangerous actions suchas jumping off a ladder or taking off construction hardhat during work. On theother hand, it facilitates interactions between construction workers and robots inthe workplace. Human pose estimation for construction workers allows robotsto understand the intention of workers by detecting their poses. Robots can notonly synchronize their tasks with human workers but also avoid collisions andaccidents, which leads to a more efficient workflows and increased productivity.Human pose estimation can be cast into two broad categories, third-person viewand first-person view pose estimation (as known as egocentric pose estimation) [4].11.1. Problem StatementIn the construction sector, implementing third-person view worker pose estima-tion is particularly challenging due to the complex and dynamic environment ofconstruction sites. This difficulty arises from three main factors: (1) the com-plexity of choosing an optimal number of vision sensors and their placement toachieve the best viewing angles; (2) frequent occlusions caused by the backgroundscene structure, including human workers, construction materials and equipment;(3) the lack of comprehensive datasets with sufficient range of actions [5]. In con-trast, egocentric view pose estimation is an alternative with distinct advantages.Unlike third-person view pose estimation, which sets up the vision sensor (e.g.,camera) from a fixed and distant position, egocentric pose estimation captureshuman motions from the first person’s perspective through the sensor with a dy-namic viewpoint, meaning its view is continuously aligned with the worker’s fieldof view [6]. By attaching the sensor to a worker’s hardhat for an egocentric viewsetting, the limitations from the fixed camera field-of-view and environmentalocclusions can be overcome.On the other hand, an egocentric view setting offers rich visual cues for down-stream applications like worker activity recognition. To elaborate, worker ac-tions can be classified to coarse-grained actions such as walking and climbing, orconstruction-specific fine-grained actions [7], which refer to actions with subtlemovements [8] such as drilling, screwing, and nailing. While these fine-graineddetails (e.g., handheld construction tools) are mostly invisible or ambiguous inthird-person capturing [9] because of the distant third-person view setup or occlu-sions, an egocentric view setting can capture both coarse and fine-grained actionswith precision and detail. For example, the distinct characteristics of construc-tion tools held by workers, such as their color and geometry, are clearly shownin egocentric images. Hence, egocentric pose estimation offers a more effectivesolution for worker pose estimation within the construction site scenarios. It canprovide a comprehensive assessment of worker behaviour with captures containinga wealth of information on worker actions. Figure 1.1 demonstrates a compari-son between third-person view images and egocentric view images capturing theworker actions.Furthermore, egocentric pose estimation has greater practical benefits than third-person view pose estimation. Its applications are not limited to activity recog-21.1. Problem Statement(a) Third-person view captures with occlusion and low visibility of construction tools.(b) Egocentric view captures without occlusion and relatively high visibility ofconstruction tools.Figure 1.1: Comparison between captures of construction tasks from thethird-person view (a) and egocentric view (b) cameras.nition, safety monitoring, and human-robot interaction but are also beneficial toExtended Reality (XR) technologies. There has been a growing focus on usingVirtual Reality (VR) for construction worker safety training [10–13], where work-ers can learn and assess safety risks without being exposed to real-world dangersin the virtual setting. In VR training, worker pose estimation is particularly im-portant as it helps capture and transfer the workers’ physical movements intothe virtual environment. The worker pose estimation adopts an egocentric ap-proach that workers’ movements are analysed through the cameras attached toVR glasses, where an egocentric pose estimation model is well-suited for this taskbut a third-person view pose estimation model will struggle to interpret their posesdue to the first person perspective captures. Hence, egocentric pose estimation is31.1. Problem Statementmore advantageous than third-person pose estimation in XR applications.Despite the importance and effectiveness of egocentric pose estimation, it is underexplored in the construction industry, where research in worker pose estimationhas mainly focused on the third-person view [7, 9, 14]. The main cause lies in theinsufficiency of egocentric data for the construction domain. Existing datasets andapproaches involving egocentric poses are either designed for general actions ortrained on synthetic data [15–17]. Due to the uniqueness of construction poses,it is expected that current egocentric pose estimation models used for generalhuman pose estimation will experience a performance drop when attempting toinfer construction worker poses. The lack of construction-specific and publiclyaccessible egocentric worker pose datasets challenges the training and evaluation ofdeep learning models in egocentric pose estimation for the construction industry.In addition to the insufficiency of egocentric datasets for construction workerpose estimation, the approach for egocentric pose estimation presents significantchallenges. A primary issue is the depth ambiguity due to the use of monocularcameras. While multi-view geometry from multiple egocentric cameras [16, 18]can help infer depth, setting up a stereo camera system is complex and impracticalfor on-site construction environments with the need for calibration, rectification,and synchronization. A monocular camera, on the other hand, is feasible forworker pose estimation but struggles to estimate the depth because of depthambiguity, which means position of the 3D points projected to a 2D plane isuncertain along the optical axis. The second challenge is the occurrence of self-occlusion in construction poses, where worker’s movements can be obscured bytheir body parts (e.g., squatting). These challenges obstruct the development ofegocentric worker pose estimation in the construction industry.To address the challenges outlined above, this paper proposes a two-stage ap-proach for worker pose estimation, which consists of a 2d module followed by a3d module. Firstly, the 2D module utilizes a heatmap-regression-integral modelto predict the 2D poses of workers from egocentric images. The goal of the 2Dmodule is to localize a set of human anatomical keypoints from the egocentric im-ages and solve the self-occlusion caused by egocentric setting with 2D labelling.Subsequently, the predicted 2D poses pass through the 3D module to generate 3Drepresentations. The proposed architecture of 3D worker pose estimation involves41.1. Problem Statementa pseudo multiple view to solve the depth ambiguity and adopts a self-supervisedlearning strategy to resolve the challenge associated with acquiring 3D data. Theentire egocentric pose estimation approach can be generalized in different use casesor domains where 2D pose data is available.Due to the lack of egocentric worker pose estimation dataset, this paper alsoproposes ICON-Pose, the first egocentric dataset specifically designed for con-struction worker pose estimation. ICON-Pose captures 63 construction workeractions in total with both 2D and 3D human body joints among 10 categories ofbasic construction tasks, including “Connect”, “Cover”, “Cut”, “Dig”, “Finish”,“Place”, “Position”, “Work from height”, “Spread” and “Others”. ICON-Poseconsiders the temporal dynamics of action, presenting it as a series of poses alongthe time axis rather than a single frame. Compared with the majority of publiclyavailable datasets such as MPII [19] and Human3.6M [20], ICON-Pose focuses onrecognizing diverse construction worker poses from the egocentric perspective andcontains both coarse and fine-grained construction actions. The egocentric framesin ICON-Pose are captured by a motion capture setup tailored to the constructionenvironment, which is a 360-degrees fisheye camera mounted on a constructionhardhat. This tailored setup is practical and easy-to-adapt for construction tasks.The proposed dataset ICON-Pose can be used for downstream applications suchas human-robot interaction, safety monitoring and XR technologies with the con-struction sectors.In summary, the contributions of this paper are as follows:1. An egocentric worker pose estimation approach that only take monocularegocentric images as input and output both 2D and 3D worker poses is proposed.The proposed method effectively tackles the challenges of depth ambiguity andocclusion without the need for 3D ground-truth data. Furthermore, it offersflexibility and generalization ability on different domains with independent 2Dand 3D modules.2. An egocentric worker pose estimation dataset ICON-Pose that presents avariety of construction worker poses in the egocentric frame is introduced. Thisdataset aims to serve as a resource for the research community to enhance thestate-of-the-art egocentric-based construction worker pose estimation, as well asto offer benchmarking data for evaluating the performance and shortcomings of51.1. Problem Statementexisting and newly developed algorithms.3. The proposed egocentric worker pose estimation approach and ICON-Pose,including the motion capture setup and data collection protocol, are throughlydescribed. This comprehensive documentation enables easy replication for furtherinvestigations and applications. This paper aims to drive progress in worker poseestimation within the construction industry, promoting the implementation ofIntelligent Construction practices.The rest of the paper is structured as follows: The section Related Work reviewsthe existing literature on construction worker pose estimation and general humanpose estimation. We also examine the datasets available for construction workerposes and egocentric datasets for everyday poses. In the Methodology section, theproposed approach for egocentric worker pose estimation is introduced in detail,including the design of 2D and 3D modules. Following this, the design and compo-nents of the ICON-Pose dataset with a demonstration of its practical applicationare introduced. The effectiveness of the approach is then evaluated in the Resultsand Analysis section, where the 2D and 3D modules are assessed independently.Finally, the Conclusion summarizes the findings and outlines directions for futureresearch.6Chapter 2Related Work2.1 Human Pose EstimationHuman pose estimation is a fundamental computer vision task for predictinghuman pose information such as the spatial locations of body joints based on theinputs. Research works in 3D human pose estimation fall into two categories,an end-to-end framework that directly predicts 3D poses from inputs or a dual-module approach that independently trains with 2D and 3D poses. In this paper,we focus on the latter and explore the substantial studies in both 2D and 3Dpose estimation. Moreover, We further narrow our discussion of human poseestimation to the domain of construction and review recent studies that haveadvanced construction worker pose estimation.2.1.1 2D Human Pose Estimation2D Human pose estimation (HPE) aims at localizing 2D human body keypointsfrom the input data. Early works in HPE rely on hand-crafted features suchas HOG features [21], shape, color and contour-based features [22], and imagesilhouettes [23]. Recent HPE solutions employ deep-learning methods that areprovably able to learn complex functional mappings and obtain feature repre-sentations directly from the data have achieved significant breakthroughs. Sev-eral surveys [24–26] have covered different categories of deep learning-based HPEmodels based on the characteristics of pose estimation models (e.g., top-downand bottom-up). In this section, we limit the scope to the deep-learning basedHPE and focus on two commonly used approaches in HPE, Heatmap-based andregression-based approaches.72.1. Human Pose Estimation2.1.1.1 Regression-based Pose EstimationThe main idea of regression-based pose estimation is to map the input image tothe kinematic body joints (e.g., shoulder, elbow) via an end-to-end network anddirectly regress the keypoint coordinates. One of the early works in regression-based HPE with deep learning was DeepPose [27], which formulated body jointsas a problem of a CNN-based regression. DeepPose adopted the CNN-basedAlexNet [28] as the backbone and regressed the joint coordinates from input im-ages through a three-stage cascade of refining regressors. With the success ofDeepPose, research in HPE began to transform from conventional methods todeep networks. Carreira et al. [29] proposed an Iterative Error Feedback (IEF)network based on GoogLeNet [30] which is a self-correcting model that trainsthe initial pose by feeding back error predictions. Sun et al. [31] introduced astructure-aware regression approach based on ResNet-50 [32] and produced sig-nificant improvement over regression-based methods. It adopts a more stable andprimitive reparametrized and bone-based pose representation instead of a joint-based representation. Li et al. [33] suggested a heterogeneous multi-task learningframework that simultaneously learns a human pose regressor and body-part andjoint-point detectors with sliding-window classifiers. It shows the possibility ofjoint training of pose regression with the detection tasks.The regression-based methods efficiently learn the joint coordinates in an end-to-end manner. However, it has two clear drawbacks. First, direct regression ofbody joints requires fully connected layers with heavy parameters. This leadsto a model with high capacity, which can cause overfitting on the training dataand fail to generalize on unseen data. Second, individual regression of bodyjoints does not consider the spatial relationship between joints. Consequently, themodel demonstrates limited robustness in handling variations of joint positionsand orientations2.1.1.2 Heatmap-based Pose EstimationIn response to the above limitations, heatmap-based methods emerged as an alter-native. The heatmap-based method aims at predicting N heatmaps, representedas {H1, H2, . . . ,HN}, corresponding to N different joints. A heatmap Hi is aone-channel image with a 2D Gaussian centered at the joint Ji, implying the82.1. Human Pose Estimationprobability of the target joint location (xi, yi). The prediction of joint Ji can beexpressed by 2.1:Ji = argmaxx,yHi(x, y) . (2.1)A heatmap prediction network is designed with a simple architecture that doesn’tinvolve fully connected layers. For example, Rafi et al. [34] proposed a GoogLeNet-based deep network architecture with convolutional and deconvolutional layers forpose estimation. It takes multi-resolution input and adds a skip connection layerto combine feature maps. Xiao et al. [35] introduced a pose estimation architecturewith a few deconvolutional layers added on ResNet. It combines the upsamplingand convolutional parameters into deconvolutional layers without the use of skipconnections. Newell et al. [36] presented a novel stacked hourglass architecture forthe task of HPE. The presented network employs convolutional and max poolinglayers, as well as skip connections to preserve spatial information.In general, heatmap-based methods have impressive performance in HPE, whereasit is associated with a few limitations, including the demanding computationalcost of N heatmaps and the quantization error derived from the transformationbetween heatmap representation and joint coordinates. Considering both thestrength and weaknesses of the existing regression or heatmap-based HPE, thispaper explores the integral operations for construction workers pose estimationto retain the advantages of both representations and avoid their downsides.2.1.2 3D Human Pose EstimationIn this section, we will introduce existing works in 3D human pose estimationmethods from a first-person viewpoint and 3D human pose estimation throughunsupervised training.2.1.2.1 Egocentric 3D Human Pose EstimationMotivated by the potential applications in various domains, considerable progresshas been achieved in the field of egocentric 3D human pose estimation in recentyears. Rhodin et al. [37] proposed the first egocentric human motion capturesystem by two-head-mounted fisheye camera to estimate full-body through dis-criminative joint-location detector suggested by Insafutdinov et al. [38]. Following92.1. Human Pose Estimationworks [16, 18] also performed 3D pose estimation based on stereo camera. Akadaet al. [16] introduced a multi-branch autoencoder to estimate 3D joints from 2Dheatmaps predicted from stereo images, while one branch estimates 3D joints lo-cations and one branch reconstructs the corresponding 2d heatmaps. Akada etal. [18] suggested a transformer-based 3D pose estimation approach with differ-ent modules that take egocentric stereo fisheye video as input and predict depthmaps, human body masks and 2d joint heatmap for 3D joint estimation.Despite their success, stereo camera configuration tends to be bulky and oftenrequires heavy computational costs. Xu et al. [15] thus proposed single head-mounted fisheye camera motion capture setup with a CNN-based monocular 3Dhuman pose estimation approach, which estimates 2D joint location heatmapsfrom the original fisheye images and zoomed-in images. It then recovers the 3Djoint positions using predicted camera-to-joints distances and intrinsic informationof the fisheye camera. Tome et al. [39] suggested a two-step approach for egocen-tric 3D human pose estimation, which estimated 2D heatmaps from single-viewegocentric images using a ResNet-based [32] heatmap estimator. The encodedheatmaps are then regressed as 3D joints and reconstructed as 2D heatmaps by adual-branches decoder. To further tackle the challenge of depth ambiguity causedby single-view input and self-occlusion, Wang et al. [40] employed a pose priorlearned from sequential variational auto-encoder to optimise 3D pose predictedfrom a sequence of monocular-view egocentric image.2.1.2.2 Unsupervised 3D Monocular Human Pose EstimationAlthough the previously mentioned studies in egocentric human pose estimationhave demonstrated promising performance, they train the model with ground-truth data such as 3D joints under strong supervision. The collection of 3Dground truth data is not only challenging but also leads to a model with limitedgeneralization capabilities when applied to downstream applications across vari-ous domains. To address this challenge, unsupervised learning for 3D human poseestimation has attracted increasing attention. In this section, we explore unsuper-vised monocular 3D human pose estimation in both egocentric and third-personperspectives.A primary obstacle in unsupervised egocentric human pose estimation is to resolve102.1. Human Pose Estimationdepth ambiguity by single-view egocentric input without relying on ground-truth3D poses. Liu et al. [41] overcome this problem by proposing an unsupervisedapproach which takes both third-person view and egocentric view images as in-put and predicts their corresponding 3D poses as well as the rotation differencesbetween them. The egocentric 3D pose estimation model is supervised by thepredicted rotation difference without the use of ground-truth 3D data and onlyegocentric image is required to estimate the 3D egocentric pose during the infer-ence stage. However, the estimation performance of the proposed approach highlyrelies on the use of extrinsic parameters generated from the Aruco markers whenlearning the rotation difference. On the other hand, Wang et al. [42] introduceda weakly supervised approach in which an egocentric pose estimation network istrained using pseudo labels of egocentric 3D poses. The pseudo labels are pre-dicted from large-scale egocentric pose estimation datasets with pseudo-groundtruth, where the egocentric view and third-person view images are collected fromsynchronized egocentric camera and external cameras. Compared to supervised3D egocentric pose estimation, unsupervised egocentric pose estimation general-izes better to new domains without the need of 3D labelling. Yet, present studieshighly depend on external view images and additional ground-truth informationduring training.In recent studies of third-person view human pose estimation, unsupervised learn-ing is employed with single-view input solely. Drover et al. [43] proposed an un-supervised approach in 3D pose estimation solely based on 2D pose input withthe hypopaper that only plausible 3D pose can project to realistic 2D pose. Inthis approach, 2D pose is lifted to 3D pose by a depth predictor and randomlyprojected back to 2D. A discriminator is then used to supervised the plausibilityof the projected 2d poses. Based on [43], Chen et al. [44] extends the architectureof [43] with a pseudo view by randomly sampled virtual rotation to bypass themultiple-view need of generating the 3D poses. The network is supervised by acycle consistency loss without 3D labelling. [45, 46] then built upon [44]. Yuet al. [45] introduced a learnable scaling parameter to optimize the scale of 3Dposes. Wandt et al. [46] learnt the distribution of elevation angle in camera ori-entation and applied to the poses before the virtual rotation to ensure a realisticrotation. Furthermore, normalizing flow is used instead of a discriminator to inferthe probability of the reconstructed 3D pose.112.1. Human Pose Estimation2.1.3 Construction Worker Pose EstimationIn this section, we narrow our focus to construction worker pose estimation. In theconstruction domain, worker pose estimation are limited to the third-person per-spective. A range of methodologies for third-person view worker pose estimationhas been explored. The bag-of-feature framework is one of the commonly usedmethods, which represents the high-dimensional features by feature descriptorand presents their clustering by a histogram. For example, Gong et al. [47] intro-duced a Bag-of-Video-Feature-Words-based automated workers action classifica-tion. Worker actions are represented as video feature words by HoG (Histogramsof Oriented Gradients) descriptor and formed as a codebook. A similar studyadopts the Bag-of-Feature pipeline. Yang et al. [7] described the recorded videoby dense trajectories obtained from dense sampling and feature points tracking.The motion pattern is depicted by three types of descriptors HoG, HoF (His-togram of Optical Flow), and MBH (Motion Boundary Histograms). Althoughthe Bag-of-Feature approaches have shown robustness to noise on input imagesof construction sites, representing worker actions by Bag-of-Feature ignores thespatial information and is hard to represent complex actions.The skeletal structure is an alternative representation of human actions contain-ing rich information about human pose. Escorcia et al. [48] tracked human poseinformation in skeleton form with a Microsoft Kinect Sensor providing RGB-Dinformation. With the rapid development of artificial intelligence, recent studieshave adopted deep learning models for better performance in worker pose estima-tion. Li & Li [14] obtained the skeleton information of workers by OpenPose andencoded them as a feature matrix, where occlusions are imputed by GAN (Gen-erative Adversarial Nets)[49]. Ishioka et al. [50] detected workers with boundingboxes by FasterRCNN and tracked the boundary boxes by a Kalman filter-basedtracking method. Pose estimation is performed with a 3D-ConvNet architecturewith an RGB appearance stream and an optical flow dynamics stream.2.1.4 DiscussionApproach for egocentric worker pose estimation is expected to have two key char-acteristics so that it is applicable to the construction domain: (1) Accurate poseestimation with rich information of poses to effectively represent complex con-122.2. Related Datasetstruction actions; (2) lightweight motion capture setup with high flexibility forconstruction workers. Our paper introduces a monocular worker pose estimationapproach with dual-modules to ensure the accurate detection with lightweightsetup.For the 2D module, we adopt an integral worker pose regression model for 2Dworker pose estimation. The implementation of the integral worker pose regres-sion model not only utilizes the advantages of both regression and heatmap-basedpose estimation models, but also ensures the predicted poses can be comprehen-sively represented complex construction poses with 2D annotations. This pro-posed setting facilitates the learning of the accurate and complex spatial configu-ration of worker poses with the consideration of possible challenges in egocentricconstruction worker poses such as self-occlusion, spatial interference, and imagedeficiencies encountered in egocentric captures. For the 3D module, we presentsan unsupervised architecture that only require single-view of 2D eccentric posesfor 3D pose estimation. Existing 3D pose estimation models are either heavily re-liant on large 3D datasets under strong supervision or require multiple-view inputto mitigate depth ambiguity. Consequently, they are limited in generalizing to thedownstream domains like construction sector and are not feasible to constructionworker pose estimation because of the heavy motion capture setup required. Inlight of these limitations, this paper introduces a novel approach for unsupervised3D egocentric monocular worker pose estimation. Drawing inspiration from thework by Chen et al. [44], our approach uses a pseudo multiple-view, so only 2Dinputs is required for egocentric 3D worker pose estimation. Our proposed ap-proach not only overcomes the difficulties of acquiring 3D construction pose databut also bridges the gap between generic human pose estimation models and thespecific demands of the construction industry. This paper marks a significant stepforward in the field of construction worker pose estimation with the first unsuper-vised monocular egocentric pose estimation framework designed specifically forconstruction scenarios.2.2 Related DatasetTo the best of the author’s knowledge, there is no existing dataset for egocentric-based construction worker pose estimation. Therefore, this section will focus on132.2. Related Dataset(1) egocentric datasets from domains other than construction and (2) datasets forconstruction worker pose estimation from a third-person view.2.2.1 General Egocentric DatasetsA number of egocentric datasets have been established over the past decade. Thissection aims to present a selection of representative egocentric datasets and assessthem based on their action types, device setup and training environment.In terms of the action types, representative egocentric datasets, such as xR-EgoPose [39], UnrealEgo [16], EgoBody [51], EgoCap [37], Mo2Cap2 [15], andHPS [52] contain only everyday human motions like “walking”, “exercising”, “sit-ting” and “standing”. Although these datasets have covered a variety of dailyactions, without the necessary focus on construction worker poses, their gener-alization capability diminishes significantly when applied to construction tasks,considering the construction tasks usually involve more complex and distinctiveposes compared to the daily poses. Therefore, existing egocentric datasets cannotbe directly applied to construction worker pose estimation.Regarding the motion capture setup, existing egocentric datasets attempts tocapture full-body poses through different mocap (motion-capture) setup. Ego-Cap [37] is the first to demonstrate a successful full-body mocap setup using astereo pair of bike helmet-mounted fisheye cameras that is approximately 25cmaway from the head. Mo2Cap2 [15] then proposed a single monocular baseballcap-mounted fisheye camera mocap setup, where the camera is 8cm away fromthe head. HPS [52] later adopted and improved the single head-mounted cameramocap setup with body-mounted IMUs. Various alternative mocap setups havealso been implemented for different purposes. EgoBody [51] captured egocentricframes by a Microsoft Hololens2 headset with a camera mounted in order to repli-cate VR scenarios. UnrealEgo [16] designed an eye-glasses-based mocap setup thatoffers convenience for everyday use. This setup incorporates two fisheye camerasequipped on the frames of the glasses. While the aforementioned mocap setupswork well in egocentric pose estimation, they are designed for various daily scenar-ios. These setups typically involve wire-connected cameras with cables hangingfrom the mounting equipment on the head, presenting compatibility challenges inthe congested construction environment. There is a gap between existing mocap142.2. Related Datasetsetups and their applicability for construction applications, so alternative designof mocap setup is required for egocentric tasks in the construction industry.For the training environment, synthetic datasets are gaining increasing popularity.xR-EgoPose [39], UnrealEgo [16] and Mo2Cap2 [15] adopted a synthetic trainingcorpus and contain synthetic rendering egocentric captures. Although syntheticdatasets require minimal human effort in data generation and annotation, theymay not fully capture the complexity, variability, and nuances in real-world envi-ronments. Applying synthetic egocentric data to real-world scenarios, especiallyto a complex construction environment, requires sufficient validation to ensurethe reliability and generalizability of synthetic datasets.2.2.2 Construction Worker Pose Estimation DatasetsDatasets in construction worker pose estimation are mainly focused on the third-person perspective. Gong et al. [47] introduced two video-based worker posedatasets that consist of 300 labeled video segments with a total of 3,000 seconds.The first video data set includes three action categories, and the second data setincludes five worker action categories. They are “Swing”, “Excavating”. “Relo-cating”, “Traveling”, “Transporting”, “Bending down”, “Nailing with hammer”and “Aligning formwork”. Yang et al. [7] presented a publicly available datasetof 1,176 real-world video clips (an average of 6.7 seconds each) with 11 workeraction types annotated, including ”Lay Brick”, ”Transporting”, ”Cut plate”, ”Tierebar”, ”Nailing”, ”Plastering”, ”Shoveling”, ”Bolting”, ”Welding” and ”Sawing”.Escorcia et al. [48] introduced a video dataset with 5 construction poses, including“Fire caulking”, “Hammering”, “Idle”, “Painting” and “Walking”. The durationof each action varied from 5 to 15 seconds and assembled 80 videos. Li & Li [14]proposed a collected dataset containing 254 minutes of video clips in 8 actiontypes. Ishioka et al. [50] presented a dataset designed for Japanese constructionsites, which contains 52 videos, around 151 minutes in total, and 17 scenes from 6sites in Japan to classify 9 worker actions, including “walk”, “crouch”, “pick-up”,“carry”, “place”, “stand-up”, “crouching”, “standing” and “other”.152.2. Related Dataset2.2.3 DiscussionEgocentric datasets currently available for everyday activities have several limi-tations. Firstly, they are incompatible for construction worker pose estimationbecause they do not include poses specific to construction workers and are of-ten built on motion capture setups that are not suitable to be repurposed forthe construction industry. Furthermore, these datasets often consist of syntheticdata that does not accurately replicate the construction site environment. On theother hand, existing datasets for third-person view construction worker poses faceseveral key limitations, such as a lack of egocentric captures, insufficient varietyof action types, and limited public accessibility. The insufficiency of egocentricdataset for construction worker poses influence the implementation of egocen-tric worker pose estimation, which is critical for downstream applications suchas human-robot interaction and hazard monitoring in creating a sustainable andworker-centric construction environment.To address these gaps, we introduces a publicly accessible dataset named ICON-Pose. ICON-Pose tackles the previously mentioned limitations by including first-person view captures of construction worker poses with a large variety of construction-related actions. A comparison between existing worker poses datasets and ICON-Pose is presented in Figure 2.2. ICON-Pose adopts a tailored full-body mocapsetup that is well-suited for the construction environment. We employ a wire-less and lightweight camera mounted on a hardhat to capture a diverse range ofposes performed by construction workers with the considerations of various fac-tors such as task diversity, working conditions, and equipment usage. ICON-Posecan be adopted for different scenes of construction worker pose estimation easily.The comparison between existing egocentric datasets and ICON-Pose is shown inFigure 2.1.162.2.RelatedDatasetFigure 2.1: Comparison between existing worker pose datasets and ICON-Pose.172.2.RelatedDatasetFigure 2.2: Comparison between existing egocentric datasets and ICON-Pose.18Chapter 3MethodologyIn this chapter, we will discuss our proposed worker pose estimation approach andthe design of ICON-Pose. We will present the worker pose estimation methodologyby its modules with the details of architectural design. Following this, we willintroduce each component of the ICON-Pose dataset. To show the applicability ofICON-Pose, we will demonstrate a practical application of the dataset to illustrateits potential impact in the field of Intelligent Construction.3.1 Egocentric Monocular Worker Pose EstimationOur proposed worker pose estimation architecture is a two-step approach with twomodules as shown in Figure 3.1. The first module employs a heatmap-regressionintegral model to estimate the 2D locations of body joints. Followed by is thesecond module that takes the 2D poses as input and elevate them to 3D throughunsupervised learning. Its design offers two major benefits. Firstly, it effectivelyaddresses the challenges of self-occlusion and the requirement for 3D data. As 2Dlabels are relatively straightforward and easy to acquire, we apply a fully super-vised approach to tackle the self-occlusion issue within the 2D module to ensurethe final 3D poses are free of occlusion. Moreover, given the challenges associ-ated with obtaining 3D labels, we elevate the 2D poses to 3D in an unsupervisedmanner, which bypass the need for direct 3D annotations. Secondly, our modelis designed with modular independence, meaning the 2D and 3D modules canbe independently trained and deployed. This flexibility not only facilitates easyadaptation of the proposed worker pose estimation model to different use cases,but also extends its applicability to diverse domains where 2D training data isavailable.193.1. Egocentric Monocular Worker Pose EstimationFigure 3.1: Overview of the egocentric worker pose estimation approach withdual modules.3.1.1 2D Module3.1.1.1 Model DesignTo leverage the advantages of heatmap-based and regression-based approaches inpose estimation while simultaneously mitigating the downsides, this paper pro-poses an integral worker pose regression model [31] that unifies the heatmap rep-resentation and joint regression [29]. Our model consists of a heatmap predictionnetwork and a regression-based integral layer. The input is the egocentric imageswith construction workers’ poses and the output is the 2D construction workerposes in the form of a skeleton as shown in Figure 3.2.Figure 3.2: Architecture of the 2D module.Specifically, a heatmap H of pose P is first generated through a heatmap pre-diction network. We adopt the architecture of DeepLabV3 [53] and embed thedeep convolutional network ResNet-50 as the backbone network for image featureextraction, which consists of an initial convolutional block and four layers of resid-ual blocks. Each layer of the residual block is composed of a convolutional block203.1. Egocentric Monocular Worker Pose Estimation(Conv Block) and a varying number of identity blocks (ID Block), from three tosix and then back to three in the successive layers. The convolutional layers ineach identity block are 1x1, 3x3, and 1x1 bottleneck convolutions. The structureof the backbone ResNet-50 is illustrated in Figure 3.3. As the input image passesthrough the backbone network, it is down-sampled by a factor of 32. The finalfeature map produced is fed to a classifier to refine the heatmap. The feature mapfirst passes through an astrous spatial pyramid pooling (ASPP) module, whichis made up of five branches, including a 1x1 convolutional layer, three dilatedconvolution layers with dilation rates of 12, 24, and 36, and a pooling layer. Thedilated convolution layers can increase the receptive field of the network withoutintroducing additional parameters [54]. Using dilated convolutions with a differ-ent dilation rate in the ASPP module can thereby incorporate a larger contextin heatmap prediction while minimizing the computational cost. The output ofASPP is concatenated to the final block and up-sampled to produce the finaloutput heatmap Hi. The overall structure of the heatmap prediction network isdemonstrated in Figure 3.4.Figure 3.3: Architecture of the 2D module.To transform the predicted heatmap Hi into joint coordinates, a regression-basedintegration layer is constructed to regress the ith joints Ji. It estimates the joints Jiby taking the expectation of joint location p in (x, y) on the normalized heatmapH˜i instead of performing the argmax operation (Eq. 1). Softmax is first per-formed to compute the normalized heatmap H˜i, also known as the probabilitymap. During the process of softmax, all pixels in the predicted heatmap H˜i arenormalized so all elements in H˜i become non-negative and sum to one.The normalized heatmap H˜i(p) is computed using the softmax function, as shown213.1. Egocentric Monocular Worker Pose EstimationFigure 3.4: Architecture of the 2D module.in 3.1:H˜i(p) = softmax(Hi(p)) =eHi(p)∑Heightu=1∑Widthv=1 eHi(u,v)(3.1)The joint Ji is regressed by integrating all expected joint locations p weighted byits probability H˜i(p) in the integration layer. The integral function is shown in3.2:Ji =Height∑py=1Width∑px=1p · H˜i(p) (3.2)3.1.1.2 Training detailsWe adopt a fully-supervised training approach to address the expected challengesof our egocentric worker pose estimation model, such as self-occlusion arisingfrom complex worker poses, occlusion caused by large construction equipment(e.g., a shovel in hand), and barrel distortions resulting from the fisheye cameraand perspective distortion from a top-down viewing angle of the egocentric frame.We extensively annotate the egocentric images with worker poses and feed bothegocentric image and the ground truth skeleton from annotation for model train-ing. By training the model in a fully-supervised manner, we provide it with thenecessary information to handle worker poses even in scenarios with the afore-mentioned challenges. To ensure the learned representation can be transferredwell to construction egocentric images, the pose estimation model is pretrainedon 5,646 egocentric images depicting daily poses in the sizes of 160*128 (image223.1. Egocentric Monocular Worker Pose Estimationwidth*height) and poses randomly selected from Egocap (Rhodin et al. 2016).The selected Egocap images are captured in front of a green screen by two head-mounted fisheye cameras. The pre-training was performed for 100 epochs with alearning rate of 0.0001 and batch size of 8. The network is then fine-tuned for130 additional epochs using egocentric images with construction poses from theICON-Pose dataset. The ICON-Pose dataset contains 1,294 egocentric imageswith construction poses. It is split into a training set of 1036 images and a testingset of 258 images. The raw images collected are in the size of 1920*1080, whichare resized to 1350*1080 and rescaled to 160*128. The fine-tuning adopts a batchsize of 8 and an optimal learning rate of 0.0001, taking about 2 hours of trainingtime. Mean squared error (MSE), also called L2 loss as shown in 3.3, is adoptedas the loss function for heatmap prediction, which refers to the squared differ-ences between the predicted heatmap and ground truth heatmap. Our networkconverges at the point where MSE achieves 1.80× 10−5 pixels2 (px2).L2 =1nn∑i=1(yi − f(xi))2 (3.3)3.1.2 3D Module3.1.2.1 Model DesignConsidering the complexity in obtaining 3D construction poses, our objective isto lift a 2D construction pose x ∈ R2J to 3D pose y ∈ R3J using an unsupervisedapproach, where no dataset of 3D poses is available for training. The overallarchitecture of the 3D module is illustrated in Figure 3.5. We adopt the baselinepose estimation model [55] as the lifting network. It takes 2D construction posesconsist of joints Ji in [ui, vi] as input and output the corresponding depth Di. Thelifting network consists of two residual blocks. Each residual blocks included twobuilding blocks, which is a linear layer, followed by batch normalization, a RELUactivation and dropout. Two extra linear layers applied to the input to increasethe dimensionality of input to 1024, and one before the final prediction. Giventhe predicted depth Di and a fixed translation vector t, the zi value of the 3Djoints is zi = Di + t and the 3D joint positions [xi, yi, zi] is calculated as below:Yi = [ui · zi, vi · zi, zi] (3.4)233.1. Egocentric Monocular Worker Pose EstimationTo create a virtual second view for estimating the depth in an unsupervised way,the constructed 3D pose Y1 is then rotated to Y2 at its root joint by a randomrotation matrix R, sampled from a uniform distribution with an azimuth anglebetween [−π, π] and an elevation angle between [−π9 , π9 ]. The rotated 3D posesY2 is then projected to x2 by perspective projection with each 2D joints location[xi\zi, yi\zi].The symmetrical lifting and projection step is performed on the synthetic 2D posex2 for self-supervision. A synthetic 3D pose Y˜2 is constructed with the predicteddepth of x2 from the same lifting network used in the upper part of the cycle.Y˜2is then rotated inversely to Y˜1 by R and Y˜1 is projected to 2D pose x˜1.Figure 3.5: Architecture of 3D module.3.1.2.2 Training details:We employ 3D loss L3D and 2D loss L2D similar to [45, 46]. As 2D pose x2 isprojected from 3D pose Y2 and lifted to Y˜2 again by the same lifting network, L2243.2. ICON-Pose Datasetloss can be defined with Y2 and Y˜2:L3D = ∥Y˜2 − Y2∥2 (3.5)With the symmetrical lifting and projection, Y˜1 is projected to the 2D pose x˜1and it should be matched with x1 which is the 2D pose of Y1. Therefore, 2D lossis defined as:L2D = ∥x˜1 − x1∥1 (3.6)Unlike [45, 46] that are designed for general pose estimation model, our approachis customized to egocentric pose estimation. We utilized a characteristics of ego-centric setting that the camera is usually head-mounted so the viewing directionof the egocentric camera is closely matches with the vertical orientation (y-axis)of the 2D images, which extends from the head down to the lower body joints. Asdepth Di is the distance along the optical ray from the center of egocentric camera(attached to head) to joints Ji, it has proportional properties among the jointssimilar to those along the y-axis, which can be represented by the y-coordinatesof 2D joints inputs xv. Hence, L1 loss is employed on predicted depth D andy-coordinates of the 2D joints inputs xv as depth loss Ldepth:Ldepth = ∥D − xv∥1 (3.7)The lifting network is updated to minimize the total loss function L:L = w3DL3D + wdepthLdepth + w2DL2D (3.8)3.2 ICON-Pose DatasetThis section provides an in-depth overview of ICON-Pose, particularly emphasiz-ing our motion capture setup, the construction actions performed, data collectionprocedures, data annotation process, and dataset design.253.2. ICON-Pose Dataset3.2.1 Motion Capture SetupOur motion capture setup is designed to cater to the working environment ofconstruction workers. We mounted a fisheye camera, Insta360 ONE X2, on ahardhat using a 17.7-inch adjustable gooseneck mount. The fisheye camera itselfis compact with a size of 4.62 x 11.30 x 2.98 cm (width x height x depth). Thiswide-angle perspective enables the capture of not only the full body pose of theconstruction worker but also the surrounding environments and the constructiontools they are holding without requiring additional sensors. One notable feature ofour motion capture setup is that unlike existing mocap setups, our fisheye camerais operated remotely by Bluetooth or Wi-Fi without the need for cumbersomewires. The mocap setup is thereby lightweight, easy-to-employ, and practical thatcan be implemented in the construction environment so that ICON-Pose can berepurposed easily or utilized for further applications in Intelligent Construction.In summary, our motion capture setup with the mounted fisheye camera offersa solution customized to the needs of construction workers, providing improvedadaptability, mobility, and convenience in their daily construction tasks.3.2.2 Construction ActionsOne of the primary objectives of ICON-Pose is to include a wide range of con-struction actions to ensure a comprehensive coverage. We drew inspiration fromEverett [56] previous study and identified 10 fundamental construction cate-gories: “connect,” “cover,” “cut,” “dig,” “finish,” “place,” “position,” “workfrom height,” “spread,” and “others.” Within these categories, we selected 63construction actions that span across the 10 basic construction types. Accordingto Yang et al. [7], an action is a series of spatial movements across time. Thus,a construction action in ICON-Pose is decomposed into a sequence of consecutiveposes, denoted as A = {pt, pt+1, pt+2 . . . pt+n} over time T = {t, t1, t2 . . . tn+1},where p is a pose, ti is the ith time stamp, and n+1 refers to the duration of T aswell as the total number of poses constructed for the corresponding action. Forinstance, the construction action “climb up ladder” consists of a sequence of con-secutive poses that represent the sequential steps taken to climb from the groundto the third tier of the ladder. The total number of poses for each action is deter-mined by the duration of each action at a frame rate of approximately 2 frames263.2. ICON-Pose Datasetper second (fps). The duration of each action varies based on the behavioral char-acteristics of different actors. The consideration of the temporal structure of anaction can facilitate the performance of subsequent downstream applications likeworker action recognition, especially for the understanding of fine-grained action.Our goal is to construct actions spanning a broad spectrum of construction tasks,containing both coarse and fine-grained actions. This ensures the dataset’s appli-cability to subsequent analyses and applications within the construction domain,such as construction safety monitoring, construction project management, andWorker-Robot Collaboration. To the best of our knowledge, ICON-Pose includesthe largest collection of construction action types compared to existing datasetsfocusing on worker pose estimation. Table 3.1 provides detailed information onthe worker actions.3.2.3 Dataset Collection and AugmentationThe construction poses are collected in a laboratory environment with an ap-proximately 3m×2.5m effective capture space and a green-screen studio with2.5m*2.5m capture space. A total of 63 construction actions were performed.Each action is performed multiple times and is sustained for a minimum dura-tion of 10 seconds in order to ensure an adequate amount of data for analysis, aswell as capture the essential pose variations and movements associated with eachspecific action.To increase the variability of the datasets, the green-screen background is replacedby random real backgrounds, as illustrated in Figure 3.6, to represent differenttypes of construction sites including indoor and outdoor construction sites. Taskswere carried out by 5 actors with construction hardhat and safety vest put on toensure the variations of height, body shape, and physique, and behavioral char-acteristics in ICON-Pose. Furthermore, to replicate varying lighting conditionsthat construction workers encounter in their working environments, constructiontasks were performed at different times with variations in lighting setup. Theinclusion of background and actor diversity in our dataset contributes to a morecomprehensive and realistic representation of construction scenarios. This en-sures the robustness of ICON-Pose in model training and real-world applicationsin construction-related domains. ICON-Pose will be updated and enhanced on an273.2. ICON-Pose DatasetTable 3.1: Construction Actions in ICON-PoseType of Action Construction ActionsConnect Screw on floor; Screw on wall; Nail on floor; Nail on wall;Assembly of two conduits(stand); Assembly of two con-duits(squat); Install hole cover by driller; Mount screw todriller; Wrench(squat); WrenchCover Overlay sheet on material; Cover hole with boardCut Cut by plier(squat); Cut by plier(stand); Drill the wallwith electric driller; Drill with electric driller(squat); SawDig Excavate; Backfill; Move soil; Move soil by cart; LoosensoilFinish Grind floor; Grind wall; Scrape with trowelPlace Drop off the box with one hand (bend); Drop off thebox(squat); Stack conduit; Transport objects by cartPosition Carry box with one hand; Pick up a board; Pick up a longboard; Pick up a box; Drop off a box under table; Pick upa box under table; Lift a boxWork from height Climb down ladder; Climb up ladder; Set up ladder; Workfrom heightSpread Pour concrete; Paint by brush(stand); Paint bybrush(squat); Paint by roller; Brush floor; Brush wallOthers Put on gloves; Take off gloves; Put on/take off reflectivejacket; Put on/take off construction hardhat; Clean up thesite; Talk over phone; Measure and draw; Jet particles;Scroll phone; Sit on chair; Sit on floor; Stand up; Standstill; Walk; Walk while holding big boxongoing basis to meet the needs of the community, such as including egocentricdata and poses with a larger variety of construction environments.3.2.4 Data AnnotationThe egocentric images collected with the corresponding pose annotations are fedinto the training of the 2D worker pose estimation model. The annotation workwas conducted by annotators who were instructed to label the keypoints represent-ing the joints of the construction workers. These keypoints were placed preciselyat the middle of the centerline of the respective body parts. For worker poseswith occlusion or self-occlusion, workers’ keypoints are annotated with approxi-283.2. ICON-Pose Dataset(a) Egocentric images in a green-screen studio.(b) Egocentric images with background augmentation using different construction sitebackgrounds.Figure 3.6: Illustration of our dataset background augmentation.mate joint location. To create the skeletal structure, the edges were automaticallygenerated by connecting the keypoints. Additionally, bounding boxes that enclosethe labeled keypoints were computed by selecting the largest and smallest valueof the x-coordinate and y-coordinate among the labeled keypoints. By employ-ing this annotation approach and our dedicated annotation platform, we ensurethe accurate labeling of keypoints, which served as training data for the 2D poseestimation model with the egocentric images.3.2.5 Dataset DesignIn ICON-Pose, each construction action is constructed by a sequence of consecu-tive poses and each pose contains four components: RGB egocentric images, 2Dand 3D joints, worker localization (bounding box), and heatmap. An example of293.2. ICON-Pose Datasetdata modalities contained in our dataset is illustrated in Figure 3.8 (Figure 3.8demonstrates 2D joints and bounding box information with RGB images over-lapped for better visualization).RGB Egocentric Images. The fisheye camera used in data collection for ICON-Pose records raw data in the form of 360-degree RGB video footage. The rawfootage is subsequently processed to generate egocentric images with a resolutionof 1080*1350 pixels. The egocentric images adopt a crystal ball perspective, repli-cating the viewpoint of the construction worker wearing the camera. The highresolution of the images ensures that fine details are preserved, and the crystalball perspective allows for the comprehensive inclusion of the surroundings. Thedetailed visual information derived from the combination of the high resolutionand the crystal ball perspective in the egocentric images guarantees the abilityto conduct precise analyses of the construction poses, including the interaction ofobjects (e.g., construction tools).2D and 3D Joints. The pose of the worker is parameterized by a set of 2Dand 3D keypoints representing the locations of body joints, denoted as P ={J1, J2, J3, . . . , J18}, where Ji is the ith joint of pose P on an egocentric im-age, with Ji ∈ R2. To learn representations that can describe the complex andunique construction poses, a body configuration with a high degree of freedom isemployed. This is achieved by visualizing the worker’s pose as a human skeletonformed by 18 keypoints and the corresponding 19 edges. The 18 keypoints are rep-resented in (x, y) coordinate vectors, zero-based starting from the top-left pixel.They appear in the order of ’head’, ’neck’, ’left-shoulder’, ’left-elbow’, ’left-wrist’,’left-finger’, ’right-shoulder’, ’right-elbow’, ’right-wrist’, ’right-finger’, ’left-hip’,’left-knee’, ’left-ankle’, ’left-toe’, ’right-hip’, ’right-knee’, ’right-ankle’, ’right-toe’.The skeleton defined is illustrated in Figure 3.7 and Table 3.2.Bounding Box. The bounding box information is for localizing the worker on theegocentric images. It is presented in the form of [xmin, ymin,widthbbox, heightbbox].xmin represents the top-left x-coordinate of the bounding box, and ymin representsthe top-left y-coordinate of the bounding box. The inclusion of bounding boxinformation in ICON-Pose is intended to aid downstream tasks such as humandetection or segmentation.Heatmap. The normalized heatmap H˜ is generated from the worker pose es-303.2. ICON-Pose DatasetFigure 3.7: Configuration of pre-defined skeleton.timation model implemented. It is a composite of 18 heatmaps of each joint Jiin a pose, {H˜1, H˜2, . . . , H˜18}, indicating the level of confidence of each workerjoint occurs at specific locations within the image. The inclusion of heatmap inICON-Pose is intended to aid downstream tasks such as human reconstruction.3.2.6 Downstream ApplicationsThe proposed egocentric worker pose estimation dataset, ICON-Pose can be con-ducive to various facets of Intelligent Construction with the proposed egocen-tric worker pose estimation approach. The impact of worker pose estimation isparticularly significant in construction activity recognition, construction safetymonitoring, Human-Robot Interaction and Virtual Reality (VR) applications. Inthis regard, we will demonstrate the implementation of ICON-Pose in one of thepotential downstream applications, worker activity recognition.313.2. ICON-Pose DatasetTable 3.2: Configuration of pre-defined skeletonKeypoints EdgesHead - 0 Head - NeckNeck - 1 Neck - Left-shoulderLeft-shoulder - 2 Left-shoulder - Left-elbowLeft-elbow - 3 Left-elbow - Left-wristLeft-wrist - 4 Left-wrist - Left-fingerLeft-finger - 5 Neck - Right-shoulderRight-shoulder - 6 Right-shoulder - Right-elbowRight-elbow - 7 Right-elbow - Right-wristRight-wrist - 8 Right-wrist - Right-fingerRight-finger - 9 Left-shoulder - Left-hipLeft-hip - 10 Left-hip - Left-kneeLeft-knee - 11 Left-knee - Left-ankleLeft-ankle - 12 Left-ankle - Left-toeLeft-toe - 13 Right-shoulder - Right-hipRight-hip - 14 Right-hip - Right-kneeRight-knee - 15 Right-knee - Right-ankleRight-ankle - 16 Right-ankle - Right-toeRight-toe - 17 Right-shoulder - Left-shoulderRight-hip - Left-hip3.2.6.1 Worker Activity RecognitionWe utilize the 2D worker poses data in our dataset with a simple classifier, Sup-port Vector Machine (SVM), for the classification of 63 construction action typesbecause of its efficient use of computational resources and high level of perfor-mance [57]. To incorporate the temporal information of actions, we consider eachaction as a sequence of poses and treat each action as a single input for theclassifier. Data augmentation is performed before model training.Temporal interpolation is applied to create intermediate poses between the ex-isting ones, enhancing the smoothness of transitions and capturing more detailedtemporal variations in the pose sequences. For this, linear interpolation is usedacross 18 classes of keypoints, where each class corresponds to a specific keypoint(e.g., head, left finger) over the time span T of an action. This process yields18 interpolated keypoints, which collectively form a complete skeleton of a pose,as demonstrated in Figure 3.9. To standardize the number of poses for each ac-323.2. ICON-Pose DatasetFigure 3.8: Example of data modalities contained in ICON-Pose: Brushing.Construction action “Brushing” is constructed by a set of consecutive posesalong time axis T, 3 of them are selected for demonstration (shown in row).From left to right are Egocentric image, 2D joints and bounding box on image,3D joints, and Heatmap.tion type during model training, we calculate the required number of interpolatedposes by subtracting the current count of poses from the highest pose count acrossall 63 action types. Subsequent to temporal interpolation, translation augmenta-tion is applied, altering the worker’s pose location within the image frame. Thisaugmentation affects both original and interpolated poses, generating a new setof augmented data. A random translation factor within the range [−0.2, 0.2] inpixel units is used to shift the poses along both the x-axis and y-axis, aiding therecognition model in learning the spatial invariance of actions. It is crucial toapply uniform data augmentation transformations across all poses within eachaction sequence.After data augmentation, there are 963 inputs in total. A testing set is created333.2. ICON-Pose DatasetFigure 3.9: Temporal interpolation: green skeleton represents the existing posesand orange skeleton represents the interpolated poses.by randomly selecting 20% of the actions while the remaining 80% are allocatedfor training. The SVM classifier is implemented with radial basis function (RBF)kernel and has successfully classify 63 construction actions in ICON-Pose with anaccuracy rate of 81.87% and an inference time of 0.13 seconds.This is comparableto the recent state-of-the-art action recognition models. For instance, Yang et al.[7] has a 77.2% accuracy in classifying 21 daily actions from JHMDB dataset [58]using a CNN-based network and Avola et al. [59] achieves 72.1% accuracy in classi-fying 51 actions from HMDB51 [60]. In summary, ICON-Pose is highly adaptablefor various downstream applications within the field of Intelligent Construction,such as worker activity recognition. With its comprehensive and detailed posedata, ICON-Pose provides a rich foundation for developing sophisticated machinelearning models that can accurately classify and monitor construction worker ac-tivities. This capability is crucial for automating the process of worker activitytracking on construction sites, consequently leads to more a efficient and safeworking environment for construction workers.34Chapter 4Result and evaluationIn this chapter, we will evaluate the proposed method for egocentric worker poseestimation using both quantitative and qualitative measures. The assessmentcovers both 2D and 3D modules to provide a comprehensive view of the capabilitiesof our approach. This evaluation will not only demonstrate the effectiveness of ourmethodology but also highlight the value of the ICON-Pose dataset in advancingthe development of worker pose estimation. We will then discuss the limitationsof the study and outline potential future research directions.4.1 2D Module4.1.1 EvaluationThe 2D module is evaluated on 258 images from the ICON-Pose testing datasetthat is independent of the training set. We employ PCK (Percentage of Cor-rect Keypoints) as indicators of pose estimation performance, which measures theaccuracy of body joint localization. In most studies, the threshold for match-ing the joint position to the ground truth is defined as a fraction of the personbounding box size [61] or as 50% of the head segment length (PCKh@0.5) [19].We evaluate the 2D module with multiple thresholds to thoroughly evaluate theaccuracy of joints in poses and the plots are shown in Figure 4.1. Overall, our pro-posed 2D module for 2D worker pose estimation has achieved high accuracy witha PCKh@0.5 of 98.7% on ICON-Pose, indicating precise localization of workerbody joints in egocentric images with construction poses. Further evaluation ofthe PCK for upper and lower body joints in ICON-Pose reveals consistently highaccuracy, with a slightly diminished detection precision for endpoints in upperand lower body parts, such as ’right-finger’ and ’left-finger’. This minor discrep-ancy can be caused by fisheye distortion at the image edges and the frequent354.1. 2D Modulehand movements occurring near the edges. This minor discrepancy can be causedby the frequent hand movements occurring near the edges while fisheye distor-tion intensifies as objects approach the camera center and occur at the edges ofegocentric images captured from the head-mounted camera. In addition, PCP(Percentage of Correct Parts) is adopted to determine if a body part is localizedcorrectly by assessing if the estimated endpoints of the body segment are within50% of the ground-truth segment length from their actual locations. The pre-diction result shows a PCP of 92.2% which demonstrates precise localization ofbody part edges. These scores validate the reliability of worker poses containedin ICON-Pose and prove the robustness and capability of our proposed pose esti-mation model in accurately capturing and identifying construction worker posesfrom egocentric frames.(a) PCK for all poses. (b) PCK for all upperbody joints.(c) PCK for all lowerbody joints.Figure 4.1: 2D Worker pose estimation results in percentage of correct keypoints(PCK) for multiple thresholds on ICON-Pose.ICON-Pose offers a comprehensive collection of construction worker poses with adiverse set of construction actions, including both fine-grained and coarse tasks.A set of construction poses are selected from the testing dataset and demon-strated in Figure 4.2 for qualitative evaluation, from top left to bottom right arefine-grained actions “nail”, “paint”, “drill” and “grind”, as well as coarse actions“climb up a ladder”, “take off the vest”, “take off the hardhat” and “pick up abox with squatting pose”. Figure 4.2 shows that ICON-Pose successfully tack-les the challenges associated with egocentric worker poses by adopting a fully-supervised training approach: (1) Self-Occlusion/Occlusion. In egocentric framesand complex construction poses (e.g., squat), self-occlusion often occurs, whileocclusion by construction tools and equipment is a common issue during con-364.1. 2D Modulestruction tasks. ICON-Pose includes construction poses that are self-occluded oroccluded, as demonstrated in Figure 12 with examples like ”pick up a box withsquatting pose” that is occluded by a big box and ”drill” showing a high de-gree of self-occlusion. Despite these challenges, their poses are clearly outlined.Therefore, we conclude that our proposed 2D module is effective in handling con-struction poses with occlusion and self-occlusion so that a realistic representationof worker poses can be provided. (2) Distortion. Egocentric images captured fromhead-mounted camera naturally exhibit perspective distortion that upper humanbody parts appear larger than the lower body parts, which can be noticed in everyimages included in ICON-Pose. In addition, fisheye images include barrel distor-tions that straight lines appear to be curvy. For example, perspective distortion isparticularly noticeable in the upper portion of the image during the ”Brush floor”task, resulting in a distorted representation of the worker’s head. Yet, ICON-Poseaccurately detects the joints of the worker despite the distortions. (3) ComplexSpatial Interference. To improve the comprehensiveness of the dataset, ICON-Pose includes data captured under conditions that introduce spatial interference,such as variations in lighting and clothing. Despite these challenging factors, theposes involve spatial interference (e.g., “Work from height” and “Saw” in dif-ferent clothing) are accurately predicted. This demonstrates the robustness ofICON-Pose in handling diverse construction working environments and reinforcesits reliability as benchmarking data.4.1.2 Ablation for Integral Worker Pose Estimation ModelThe proposed integral pose estimation model used for construction workers poseprediction is the integration of heatmap-based and regression-based approach.We compare their pose prediction performance based on the testing set. Table4.1 below shows their comparison. From Table 4.1, the integral model achievesthe highest scores in both metrics, with a PCK of 98.7% compared to the othermethods of 97.8% and 92.0%, and a PCP of 92.2% compared to scores of 79.8%and 82.7%. These results indicate that the integral worker pose regression modelemployed in our work performs better than the traditional pose estimation modelin localizing the joints and edges of the worker’s body parts within the egocentricframes of ICON-Pose. These results also support our motivation to implement anintegral approach in the workers pose estimation model374.2. 3D Module(a) Nailing. (b) Painting. (c) Drilling. (d) Grinding.(e) Climbing upa ladder.(f) Taking off thevest.(g) Taking off thehardhat.(h) Picking up abox withsquatting pose.Figure 4.2: Selected 2D Worker pose estimation results by the proposed 2Dmodule.Table 4.1: Ablation of integral, heatmap, and regression methods in HPEMethod Integral Heatmap RegressionPCKh@0.5 98.7 97.8 92.0PCP 92.2 79.8 82.74.2 3D ModuleFor the quantitative performance of the proposed 3D module, PA-MPJPE(MeanPer Joint Position Error) and N-MPJPE are used to evaluate the performance of3D pose estimation. MPJPE calculates the mean of the Euclidean distances (mm)between the predicted and ground truth coordinates over each joint. PA-MPJPEcalculates the MPJPE after performing Procrustes alignment [62] to eliminatethe effects of translation, rotation, and scale on the estimated and ground-truthjoints, while N-MPJPE is the normalized MPJPE. The primary result of ourproposed 3D module is evaluated on a sub-set of a large-scale egocentric datasetXR-EgoPose [39] with 4404 2D daily poses. The model is trained with 3000epoch with a learning rate of 0.0001 and a batch size of 64. It has achieved384.2. 3D Modulea PA-MPJPE of 95.89mm and a N-MPJPE of 96.43mm, which show promisingperformance since it has outperformed the baseline model of 122.1mm from [55]and has comparable performance of [39] which has a PA-MPJPE of 97.48mm.While our 3D module incorporates the baseline model [55] as the lifting network,our module surpasses it primarily due to its tailored design for egocentric settings.This allows our model to utilize the unique characteristics of egocentric captures,which in turn significantly enhances the accuracy and effectiveness of our poseestimation performance. For example, we integrate the mechanisms that camerais often head-mounted in an egocentric framework into our 3D module by opti-mizing the lifting network with L1 loss on the y-coordinate of the 2D poses andthe predicted depth. This tailored approach for egocentric pose estimation notonly facilitates easier and faster convergence but also enhances overall accuracy.A more comprehensive evaluation will be conducted using the full XR-EgoPosedataset with ablation studies in the future.We further evaluate the module performance on construction worker poses. Thetraining utilized 1294 2D poses from the ICON-Pose dataset, which were predictedusing our 2D module. To generate the 3D construction worker poses, the modeltrained from scratch without pre-trained weights. This training involved 300epochs, with a learning rate of 0.0001 and a batch size of 8. The qualitativeassessment of the model’s output is shown in Figure 4.3, which is a selection of3D construction worker poses generated by our proposed 3D module. The resultshows that our module can accurately estimate 3D construction worker poses andeffectively handles complex construction poses such as squatting. It successfullymitigates the depth ambiguity without the need of ground-truth 3D poses dataor multiple-view input. This performance demonstrates the model’s capabilityin handling diverse pose scenarios and high degree of generalizability to differentdomains.In summary, our egocentric monocular worker pose estimation method with adual-module architecture, delivers notable performance and offers four significantadvantages: (1) It effectively addresses the issues of self-occlusion and occlusionderived from the complex construction poses performed in the dynamic construc-tion working environments through a fully-supervised 2D module; (2) Our ap-proach overcomes depth ambiguities using only monocular input, which bypasses394.2. 3D Modulethe need for multiple-view inputs in generating 3D poses and thus maintain alightweight setup for construction worker pose estimation; (3) It outperforms thebaseline pose estimation models without relying on hard-to-obtain ground-truth3D pose data, which proves its practicality across various domains. This alsodemonstrates its strong generalization capability with the modular independentdesign that can be adapted wherever 2D training data is available, which is com-paratively easier to obtain; (4) Our proposed approach is highly customized tohuman pose estimation with egocentric setting. It shows an improved performanceover general pose estimation models. To the best of the author’s knowledge, this isalso the first unsupervised monocular egocentric pose estimation model proposed.(a) Nailing.(b) Scraping. (c) Wrenching.Figure 4.3: Selected 3D worker pose estimation result by proposed 3D module.4.2.1 Value of ICON-PoseIn addition to the evaluation of the proposed worker pose estimation approach,we provide insights into the value of ICON-Pose in studying construction workerposes. In particular, we compare the pose estimation result of the pose esti-mation model initially pre-trained on a general egocentric dataset, EgoCap [37],with the results obtained after fine-tuning the model with ICON-Pose. Figure 4.4visually depicts the notable improvement in estimated worker poses, which ini-tially deviated significantly from the actual joint locations before transfer learn-404.2. 3D Moduleing. Considering the primary difference in data distribution of these two datasetslies in the types of action, the findings underscore the distinct characteristicsof construction poses compared to everyday poses even within a similar egocen-tric setting. This validates the need to develop construction-specific egocentricdatasets for domain-specific model training while ICON-Pose is effective in study-ing worker poses from an egocentric perspective in construction-related scenarios.We further compare the quantitative performance of ICON-Pose with the existingrepresentative benchmark 2D pose estimation dataset: MPII [19] achieves 92.5%in PCKh@0.5 with a GNN-based model [63] using 28,821 training data samplesand 11,701 testing data samples; EgoCap [37] with a Conv-Net-based body-partdetector achieves a PCK with a threshold of 20 pixels of 92.5%. It is trained byapproximately 75,000 images and validated by 1,000 images. With the PCKh@0.5of 98.7%, ICON-Pose demonstrates a comparable performance to the state-of-the-art models and benchmark datasets without requiring a large amount of trainingdata. ICON-Pose can serve as a benchmarking dataset in worker pose estimationwith impressive performance and accuracy in construction scenarios.(a) Worker pose estimation resultbefore fine-tuning.(b) Worker pose estimation resultafter fine-tuning using ICON-Pose.Figure 4.4: Predicted worker pose estimation result (solid line) and ground-truth(dotted line) overlapped with the egocentric image.414.3. Limitations and Future Works4.3 Limitations and Future WorksThe 3D module has showed promising results in primary evaluations, but the ac-curacy of the predicted 3D poses is lowered by two significant types of distortions.Firstly, egocentric images captured using head-mounted cameras results in per-spective distortion, where the upper parts of the human body appear larger thanthe lower parts, which can be observed in Figure 4.3. Secondly, the use of fisheyelenses introduces barrel distortion that straight lines appears to be curvy in theimages. Although the 2D pose estimation model is robust to these distortions inthe input images, it does not eliminate them, and thus leads to distorted 3D posereconstructions.To mitigate these distortions, two interesting research directions have been identi-fied. The first approach involves training for barrel distortion in parallel with thedepth estimation in the lifting network. Barrel distortion can be mathematicallyrepresented by a fisheye distortion model θ′ = θ(1 + k1θ2 + k2θ4 + k3θ6 + k4θ8),where ki are the coefficients representing different degrees of fisheye distortion.These distortion coefficients ki can be integrated into the network’s architectureto facilitate learning and correction of this distortion. The second approach ad-dresses perspective distortion, which is traditionally corrected through calibra-tion. Unlike barrel distortion that can be quantified and modeled, perspectivedistortion does not have a numerical representation. This could be alleviated byincorporating a human pose prior into the module, such as human bone lengthconstraints or learned distribution of human pose. They could possibly correctfor perspective distortion and enhance the accuracy of 3D pose estimations.42Chapter 5ConclusionsThe construction industry is actively exploring new ways to enhance its pro-ductivity, efficiency, and safety. Worker pose estimation is a human-centric ap-proach that prioritizes the needs and safety of construction workers by under-standing their poses and motions. In the egocentric setting, worker movementsand behaviors are closely analyzed from a first person perspective with richer vi-sual information than the traditional third-person view setting. Considering thelack of egocentric worker pose estimation model and dataset, this paper presentsan innovative egocentric worker pose estimation approach and an open datasetICON-Pose. The approach not only successfully overcomes the major challengesof self-occlusions and depth ambiguity without the need of 3D ground truth an-notation, but also maintains the lightweight setup for construction workers byusing only monocular view in the model design. The ICON-Pose dataset coversa wide range of construction actions and includes diverse data types. It has thepotential to drive the implementation of AI in various construction tasks, rangingfrom Human-Robot Interaction to VR safety training. We hope that ICON-Poseand the proposed approach will inspire and drive further exploration of egocentricpose estimation in the field of Intelligent Construction.43Bibliography[1] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen,Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human poseestimation: A survey. ACM Computing Surveys, 56(1):1–37, 2023.[2] Meiyin Liu, SangUk Han, and SangHyun Lee. Potential of convolutionalneural network-based 2d human pose estimation for on-site activity analysisof construction workers. In Computing in Civil Engineering 2017, pages 141–149. 2017.[3] Aanuoluwapo Ojelade and Frederick Paige. Construction worker postureestimation using openpose. In Construction Research Congress 2020, pages556–564. American Society of Civil Engineers Reston, VA, 2020.[4] Dongxu Zhao, Zhen Wei, Jisan Mahmud, and Jan-Michael Frahm. Egoglass:Egocentric-view human pose estimation from an eyeglass frame. In 2021International Conference on 3D Vision (3DV), pages 32–41. IEEE, 2021.[5] JoonOh Seo, SangUk Han, SangHyun Lee, and Hyoungkwan Kim. Computervision techniques for construction safety and health monitoring. AdvancedEngineering Informatics, 29(2):239–251, 2015.[6] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. InProceedings of the IEEE conference on computer vision and pattern recogni-tion, pages 287–295, 2015.[7] Jun Yang, Zhongke Shi, and Ziyan Wu. Vision-based action recognition ofconstruction workers using dense trajectories. Advanced Engineering Infor-matics, 30(3):327–336, 2016.44Bibliography[8] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchicalvideo dataset for fine-grained action understanding. In Proceedings of theIEEE/CVF conference on computer vision and pattern recognition, pages2616–2625, 2020.[9] Jun Yang. Enhancing action recognition of construction workers usingdata-driven scene parsing. Journal of Civil Engineering and Management,24(7):568–580, 2018.[10] Mehdi Hafsia, Eric Monacelli, and Hugo Martin. Virtual reality simulatorfor construction workers. In Proceedings of the Virtual Reality InternationalConference-Laval Virtual, pages 1–7, 2018.[11] Haiyan Xie, E Tudoreanu, and Wei Shi. Development of a virtual realitysafety-training system for construction workers. Digital library of construc-tion informatics and information technology in civil engineering and con-struction, pages 1–9, 2006.[12] Aanuoluwapo Ojelade and Frederick Paige. Virtual reality postural trainingfor construction. In Construction Research Congress 2020, pages 565–573.American Society of Civil Engineers Reston, VA, 2020.[13] Karan R Patil, Siddharth Bhandari, Steven K Ayer, and Matthew R Hal-lowell. Potential for virtual reality and haptic feedback to enhance learningoutcomes among construction workers. In Proceedings of the 18th Interna-tional Conference on Construction Applications of Virtual Reality Nov, pages22–23, 2018.[14] Ziqi Li and Dongsheng Li. Action recognition of construction workers underocclusion. Journal of Building Engineering, 45:103352, 2022.[15] Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer, Helge Rhodin, PascalFua, Hans-Peter Seidel, and Christian Theobalt. Mo 2 cap 2: Real-timemobile 3d motion capture with a cap-mounted fisheye camera. IEEE trans-actions on visualization and computer graphics, 25(5):2093–2101, 2019.[16] Hiroyasu Akada, Jian Wang, Soshi Shimada, Masaki Takahashi, ChristianTheobalt, and Vladislav Golyanik. Unrealego: A new dataset for robust45Bibliographyegocentric 3d human motion capture. In European Conference on ComputerVision, pages 1–17. Springer, 2022.[17] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, andKarteek Alahari. Charades-ego: A large-scale dataset of paired third andfirst person videos. arXiv preprint arXiv:1804.09626, 2018.[18] Hiroyasu Akada, Jian Wang, Vladislav Golyanik, and Christian Theobalt.3d human pose perception from egocentric stereo videos. arXiv preprintarXiv:2401.00889, 2023.[19] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2dhuman pose estimation: New benchmark and state of the art analysis. InProceedings of the IEEE Conference on computer Vision and Pattern Recog-nition, pages 3686–3693, 2014.[20] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.Human3. 6m: Large scale datasets and predictive methods for 3d humansensing in natural environments. IEEE transactions on pattern analysis andmachine intelligence, 36(7):1325–1339, 2013.[21] Fang Wang and Yi Li. Beyond physical connections: Tree models in humanpose estimation. In Proceedings of the IEEE conference on computer visionand pattern recognition, pages 596–603, 2013.[22] Benjamin Sapp, Alexander Toshev, and Ben Taskar. Cascaded models forarticulated pose estimation. In Computer Vision–ECCV 2010: 11th Euro-pean Conference on Computer Vision, Heraklion, Crete, Greece, September5-11, 2010, Proceedings, Part II 11, pages 406–420. Springer, 2010.[23] Ankur Agarwal and Bill Triggs. Recovering 3d human pose from monocularimages. IEEE transactions on pattern analysis and machine intelligence,28(1):44–58, 2005.[24] Haoming Chen, Runyang Feng, Sifan Wu, Hao Xu, Fengcheng Zhou, andZhenguang Liu. 2d human pose estimation: A survey. Multimedia Systems,29(5):3115–3138, 2023.46Bibliography[25] Tewodros Legesse Munea, Yalew Zelalem Jembre, Halefom Tekle Weldege-briel, Longbiao Chen, Chenxi Huang, and Chenhui Yang. The progress ofhuman pose estimation: A survey and taxonomy of models applied in 2dhuman pose estimation. IEEE Access, 8:133330–133348, 2020.[26] Qi Dang, Jianqin Yin, Bin Wang, and Wenqing Zheng. Deep learning based2d human pose estimation: A survey. Tsinghua Science and Technology,24(6):663–676, 2019.[27] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimationvia deep neural networks. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1653–1660, 2014.[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classifica-tion with deep convolutional neural networks. Advances in neural informationprocessing systems, 25, 2012.[29] Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik.Human pose estimation with iterative error feedback. In Proceedings of theIEEE conference on computer vision and pattern recognition, pages 4733–4742, 2016.[30] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed,Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Ra-binovich. Going deeper with convolutions. In Proceedings of the IEEE con-ference on computer vision and pattern recognition, pages 1–9, 2015.[31] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Inte-gral human pose regression. In Proceedings of the European conference oncomputer vision (ECCV), pages 529–545, 2018.[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residuallearning for image recognition. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 770–778, 2016.[33] Sijin Li, Zhi-Qiang Liu, and Antoni B Chan. Heterogeneous multi-task learn-ing for human pose estimation with deep convolutional neural network. InProceedings of the IEEE conference on computer vision and pattern recogni-tion workshops, pages 482–489, 2014.47Bibliography[34] Umer Rafi, Bastian Leibe, Juergen Gall, and Ilya Kostrikov. An efficientconvolutional network for human pose estimation. In BMVC, volume 1,page 2, 2016.[35] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human poseestimation and tracking. In Proceedings of the European conference on com-puter vision (ECCV), pages 466–481, 2018.[36] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks forhuman pose estimation. In Computer Vision–ECCV 2016: 14th EuropeanConference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings,Part VIII 14, pages 483–499. Springer, 2016.[37] Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinov, Moham-mad Shafiei, Hans-Peter Seidel, Bernt Schiele, and Christian Theobalt. Ego-cap: egocentric marker-less motion capture with two fisheye cameras. ACMTransactions on Graphics (TOG), 35(6):1–11, 2016.[38] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka,and Bernt Schiele. Deepercut: A deeper, stronger, and faster multi-personpose estimation model. In European Conference on Computer Vision, 2016.[39] Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino. xr-egopose: Egocentric 3d human pose from an hmd camera. In Proceedings ofthe IEEE/CVF International Conference on Computer Vision, pages 7728–7738, 2019.[40] Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, and ChristianTheobalt. Estimating egocentric 3d human pose in global space. In Proceed-ings of the IEEE/CVF International Conference on Computer Vision, pages11500–11509, 2021.[41] Yuxuan Liu, Jianxin Yang, Xiao Gu, Yijun Chen, Yao Guo, and Guang-Zhong Yang. Egofish3d: Egocentric 3d pose estimation from a fisheye cameravia self-supervised learning. IEEE Transactions on Multimedia, 2023.[42] Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Diogo Luvizon,and Christian Theobalt. Estimating egocentric 3d human pose in the wild48Bibliographywith external weak supervision. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 13157–13166, 2022.[43] Dylan Drover, Rohith MV, Ching-Hang Chen, Amit Agrawal, AmbrishTyagi, and Cong Phuoc Huynh. Can 3d pose be learned from 2d projec-tions alone? In Proceedings of the European Conference on Computer Vision(ECCV) Workshops, pages 0–0, 2018.[44] Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Rohith Mv,Stefan Stojanov, and James M Rehg. Unsupervised 3d pose estimation withgeometric self-supervision. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 5714–5724, 2019.[45] Zhenbo Yu, Bingbing Ni, Jingwei Xu, Junjie Wang, Chenglong Zhao, andWenjun Zhang. Towards alleviating the modeling ambiguity of unsupervisedmonocular 3d human pose estimation. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision, pages 8651–8660, 2021.[46] Bastian Wandt, James J Little, and Helge Rhodin. Elepose: Unsupervised3d human pose estimation by predicting camera elevation and learning nor-malizing flows on 2d poses. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 6635–6645, 2022.[47] Jie Gong, Carlos H Caldas, and Chris Gordon. Learning and classifying ac-tions of construction workers and equipment using bag-of-video-feature-wordsand bayesian network models. Advanced Engineering Informatics, 25(4):771–782, 2011.[48] Vı´ctor Escorcia, Mar´ıa A Da´vila, Mani Golparvar-Fard, and Juan CarlosNiebles. Automated vision-based recognition of construction worker actionsfor building interior construction operations using rgbd cameras. In Con-struction Research Congress 2012: Construction Challenges in a Flat World,pages 879–888, 2012.[49] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative ad-versarial networks. Communications of the ACM, 63(11):139–144, 2020.49Bibliography[50] Hiroaki Ishioka, Xinshuo Weng, Yunze Man, and Kris Kitani. Single cam-era worker detection, tracking and action recognition in construction site.In ISARC. Proceedings of the International Symposium on Automation andRobotics in Construction, volume 37, pages 653–660. IAARC Publications,2020.[51] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Polle-feys, Federica Bogo, and Siyu Tang. Egobody: Human body shape andmotion of interacting people from head-mounted devices. In European con-ference on computer vision, pages 180–200. Springer, 2022.[52] Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. Humanposeitioning system (hps): 3d human pose estimation and self-localization inlarge scenes from body-mounted sensors. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages 4318–4329,2021.[53] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.Rethinking atrous convolution for semantic image segmentation. arXivpreprint arXiv:1706.05587, 2017.[54] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilatedconvolutions. arXiv preprint arXiv:1511.07122, 2015.[55] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. Asimple yet effective baseline for 3d human pose estimation. In Proceedingsof the IEEE international conference on computer vision, pages 2640–2649,2017.[56] John G Everett. Construction automation–basic task selection and develop-ment of the CRANIUM. PhD thesis, Massachusetts Institute of Technology,1991.[57] X Wu, V Kumar, J Ross Quinlan, J Ghosh, Q Yang, H Motoda, GJ McLach-lan, A Ng, B Liu, PS Yu, et al. Top 10 algorithms in data mining knowledgeand information systems, vol. 14, no. 1. Dec, 2007.50Bibliography[58] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael JBlack. Towards understanding action recognition. In Proceedings of the IEEEinternational conference on computer vision, pages 3192–3199, 2013.[59] Danilo Avola, Marco Cascio, Luigi Cinque, Gian Luca Foresti, Cristiano Mas-saroni, and Emanuele Rodola`. 2-d skeleton-based action recognition via two-branch stacked lstm-rnns. IEEE Transactions on Multimedia, 22(10):2481–2496, 2019.[60] Hildegard Kuehne, Hueihan Jhuang, Est´ıbaliz Garrote, Tomaso Poggio, andThomas Serre. Hmdb: a large video database for human motion recognition.In 2011 International conference on computer vision, pages 2556–2563. IEEE,2011.[61] Yi Yang and Deva Ramanan. Articulated human detection with flexiblemixtures of parts. IEEE transactions on pattern analysis and machine intel-ligence, 35(12):2878–2890, 2012.[62] John C Gower. Generalized procrustes analysis. Psychometrika, 40:33–51,1975.[63] Hong Zhang, Hao Ouyang, Shu Liu, Xiaojuan Qi, Xiaoyong Shen, RuigangYang, and Jiaya Jia. Human pose estimation with spatial contextual infor-mation. arXiv preprint arXiv:1901.01760, 2019.51
Thesis/Dissertation
eng
Civil Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Zou, Zhengbo
Egocentric monocular construction worker pose estimation for intelligent construction
Text