Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Chen, Guanxiong
Master of Science - MSc
University of British Columbia
Accurate estimation of mass properties of the human musculoskeletal system is of great interest to many tasks, from gait analysis in biomechanics to motion tracking and control in computer animation. Previous work typically simplified the human musculoskeletal structure as a chain of rigid capsules, with muscle mass lumped with body segments. Such simplifications lead to errors in the system’s inertia matrix, and the error propagates to torque and pose estimates. In this study, we show that we can estimate the generalized joint-space inertia matrix of a human in motion, using a deep neural network or with a simple statistical model. The models do not make any assumptions other than that effective inertia matrices must be symmetric and positive definite. The models are trained and tested with real-world human data that includes synchronized motion and ground reaction forces. We show that a joint-space inertia matrix estimated from data can be physically plausible by revealing inertial coupling which a rigid, lumped inertia matrix fails to entail, and that effective inertia estimates are motion-type dependent. Moreover, we show that our neural inertia model SPDNet can predict inertia matrices parametrized by pose, body mass and height, that its predicted matrices are physically plausible, and that it generalizes well to unseen poses and mass distributions when used to reconstruct motion.
Data-driven Models of Human Body InertiabyGuanxiong ChenB.A.Sc. in Computer Engineering, The University of British Columbia, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)April 2024© Guanxiong Chen, 2024The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Data-driven Models of Human Body Inertiasubmitted by Guanxiong Chen in partial fulfillment of the requirements for thedegree of Master of Science in Computer Science.Examining Committee:Dinesh K. Pai, Professor, Department of Computer Science, The University ofBritish ColumbiaSupervisorHelge Rhodin, Professor, Faculty of Technology, Bielefeld UniversitySupervisory Committee MemberiiAbstractAccurate estimation of mass properties of the human musculoskeletal system is ofgreat interest to many tasks, from gait analysis in biomechanics to motion trackingand control in computer animation. Previous work typically simplified the humanmusculoskeletal structure as a chain of rigid capsules, with muscle mass lumpedwith body segments. Such simplifications lead to errors in the system’s inertia ma-trix, and the error propagates to torque and pose estimates. In this study, we showthat we can estimate the generalized joint-space inertia matrix of a human in mo-tion, using a deep neural network or with a simple statistical model. The models donot make any assumptions other than that effective inertia matrices must be sym-metric and positive definite. The models are trained and tested with real-world hu-man data that includes synchronized motion and ground reaction forces. We showthat a joint-space inertia matrix estimated from data can be physically plausible byrevealing inertial coupling which a rigid, lumped inertia matrix fails to entail, andthat effective inertia estimates are motion-type dependent. Moreover, we show thatour neural inertia model SPDNet can predict inertia matrices parametrized by pose,body mass and height, that its predicted matrices are physically plausible, and thatit generalizes well to unseen poses and mass distributions when used to reconstructmotion.iiiLay SummaryAccurate estimation of inertial properties of a human body is an important taskthat many biomechanists and computer animators have sought to solve. In thiswork, we propose methods of estimating the joint-space inertia matrix of humanbodies from motion capture and force plate data. The estimated inertia matrix canbe useful for many downstream tasks: letting biomechanists solve joint moments,or letting vision researchers generate new motions for a humanoid character. Weshow that our data-driven methods work better than prior methods that analyticallyconstruct inertia matrices, and that our method works well when injected into asimulator for reconstructing motions.ivPrefaceThe project was initially conceived by Dr. Bastian Wandt and Prof. Helge Rhodinas a follow-up work to the ECCV 2020 paper by Zell et. al, “Weakly-supervisedLearning of Human Motion Dynamics”. Prof. Dinesh Pai then formulated theresearch question as an inertia estimation problem, and developed several high-level objectives for the goal of identifying inertia matrix from publicly availablemotion capture and force plate data. Following the objectives put forth by Prof.Pai, I designed experiments, implemented the code for running the experimentsand for visualizing experimental results, executed the experiments, analyzed theirresults and wrote up most of the thesis. I also made use of the publicly availablemocap and force plate dataset, as well as the data preprocessing code from Prof.Bodo Rosenhahn’s group at Leibniz University Hannover for preprocessing ourdata. Throughout the course of the project, Prof. Pai has helped me with designingexperiments and understanding technical details related to the publicly availablehuman motion dataset by Zell et. al. Also, he offered me suggestions regard-ing implementation of the simulation framework, as well as his analysis of resultsfrom biomechanical perspectives. Prof. Rhodin and Prof. Wandt have offered mesuggestions on training and testing of neural networks. Prof. Pai, Prof. Rhodin andProf. Wandt have all been involved in refining the thesis write-up.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1 Inertia Parameter Estimation . . . . . . . . . . . . . . . . . . . . 42.2 Body Segment Models for Downstream Tasks . . . . . . . . . . . 62.3 Human Motion Datasets . . . . . . . . . . . . . . . . . . . . . . 93 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1 Effective Inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3 Constant Inertia Models . . . . . . . . . . . . . . . . . . . . . . . 13vi3.4 Neural Inertia Models . . . . . . . . . . . . . . . . . . . . . . . . 153.5 The Articulated Rigid Body (ARB) Inertia Model . . . . . . . . . 173.6 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.7 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.1 Inertia Matrices Estimated with Constant Inertia Models . . . . . 204.2 Inertia Matrices Estimated with Neural Inertia Models . . . . . . 215 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . 35Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37A Preliminary Experiments on the Synthetic Dataset . . . . . . . . . . 45A.1 Synthetic Dataset Construction . . . . . . . . . . . . . . . . . . . 45A.2 Synthetic Data Fitting . . . . . . . . . . . . . . . . . . . . . . . . 46viiList of TablesTable 4.1 Test set MAE and MPJPE of different models for inertia repre-sentation. We evaluated each model on six folds; for each foldwe use gait cycles from one subject for testing. (a) Walking Setresults. (b) Running Set results. . . . . . . . . . . . . . . . . 29Table 4.2 Mean test set MAE (↓) and MPJPE (↓) of walking gait inertiaestimation from different SPDLayer configurations. We alsolist the MAE and MPJPE from SymmNet, UnconNet, ConstantInertia Model and the ARB Inertia Model for comparison. . . 30viiiList of FiguresFigure 3.1 Method overview. (a) Curation of mocap and force plate dataand the dataset partitioning scheme. (b) Selection of data to usefor inertia estimation. An inertia model, be it a neural networkor a constant inertia model, takes data from the first 100 msof the initial single support phase of a gait cycle, and predictsa joint-space inertia matrix corresponding to the initial poseof the cycle. (c) Kinematic model of the dataset. We focuson analyzing inertial properties and reconstructing motion oflower body segments and the torso. . . . . . . . . . . . . . . 11Figure 3.2 Architecture of SPDNet. First, we feed body mass and heightinto the MLP-based feature extractor to predict an 8-D personvector p. Next, we feed p along with joint pose q into the MLP-based inertia predictor to produce d(d+1)2 entries. Finally, wefeed these entries to the SPD Layer [Jekel et al., 2022] to makethe final prediction SPD. . . . . . . . . . . . . . . . . . . . . 15ixFigure 4.1 Distribution of effective inertia estimated by the constant iner-tia model vs computed by the ARB inertia model over all sub-jects. Each subject is (a) walking with left foot on ground; (b)walking with right foot on ground; (c) running with left footon ground; (d) running with right foot on ground. The mag-nitudes are computed by diagonally lumping inertia matricesfirst, then aggregating over angles (displacements) associatedwith a joint (direction). We plot the distribution of magnitudeson log scale (left) and linear scale (right). . . . . . . . . . . . 25Figure 4.2 Distribution of walking vs running effective inertia with theconstant inertia model and the ARB inertia model. (a) Con-stant inertia model’s predictions for left-foot contact gaits and(b) right-foot contact gaits. (c) ARB inertia model’s construc-tions for left-foot gaits and (d) right-foot gaits. The magnitudesare also computed by diagonal lumping then aggregated. Weplot the distribution across all subjects on log scale (left) andlinear scale (right). . . . . . . . . . . . . . . . . . . . . . . . 26Figure 4.3 Distribution of effective inertia estimated by SPDNet vs com-puted by the ARB inertia model over all subjects. Each sub-ject is (a) walking with left foot on ground; (b) walking withright foot on ground; (c) running with left foot on ground; (d)running with right foot on ground. The magnitudes are alsocomputed by diagonal lumping then aggregating, then plottedon log scale (left) and linear scale (right). . . . . . . . . . . . 27Figure 4.4 Distribution of walking vs running effective inertia with SPDNetand the ARB inertia model. (a) SPDNet’s predictions for left-foot contact gaits and (b) right-foot contact gaits. (c) The ARBinertia model’s constructions for left-foot gaits and (d) right-foot gaits. The magnitudes are also computed by diagonallumping then aggregated. We plot the distribution across allsubjects on log scale (left) and linear scale (right). . . . . . . . 28xFigure 4.5 Box-and-whisker plots of metrics from reconstructing test setmotion clips of the six folds by different inertia models: (a)Walking Set MAE and (b) MPJPE; (c) Running Set MAE and(d) MPJPE. Left: log scale, right: linear scale. . . . . . . . . . 30Figure 4.6 Joint angles’ mean ± one standard deviation vs time from mo-tion clips reconstructed by different models. The joint anglesare grouped by anatomical planes they are situated in: e.g. thehip Sagittal angle is the hip-x axis angle shown in Fig. 3.1 (c),and the knee Horizontal angle is the knee-y axis angle shownin Fig. 3.1 (c). An empty plot in the grid means that angle wasnot measured in the dataset provided by [Zell et al., 2020]. . . 31Figure 4.7 Qualitative comparison of reconstructed motion between neu-ral networks, the constant inertia model and the ARB inertiamodel. The ground truth motion is measured on Subject No.17 while running, during a stance phase with right foot stay-ing in contact with the ground. We compare at three time in-stances: 10 ms, 50 ms, 90 ms. . . . . . . . . . . . . . . . . . 32Figure 4.8 Eigenvalues of inertia matrices, either constructed or predictedon Subject No. 17 from a running clip with right foot con-tacting ground. We can tell SPDNet guarantees eigenvalues ofpredicted matrix being positive. Note that we set the minimumacceptable eigenvalue to be 1−10 but that value is too small tobe printed out in the figure. . . . . . . . . . . . . . . . . . . . 33Figure 4.9 An inertia matrix with negative eigenvalues can make the sim-ulated human produce physically implausible behaviors. Hereto the simulated human we apply an eigenforce, i.e. an impulseforce in the same direction as an eigenvector corresponding toa negative eigenvalue of the system’s predicted inertia matrix.(a) The generalized unitary eigenforce applied to the system.(a) The system’s responding acceleration q¨. We see the appliedforce and the induced acceleration are in opposite directions,and that is not physically plausible. . . . . . . . . . . . . . . 34xiFigure A.1 Synthetic dataset construction and partitioning. (a) We cre-ate two leg models to generate synthetic data from: an unactu-ated double pendulum and a tendon-driven double pendulum.With the unactuated pendulum, we generate data at differentinitial poses to obtain different effective inertia matrices. Forthe tendon-actuated model, we apply different muscle activa-tions so the initial pose would be different. We also set the pen-dulum’s plane of motion at different tilt angles w.r.t. ground inorder to diversify our data. (b) Musculotendious actuators thatmap to real-world leg muscles. . . . . . . . . . . . . . . . . . 47Figure A.2 R2 scores of effective inertia fits, on synthetic data from thetendon-actuated double pendulum. (a) - (b): Fitting constantinertia models on the Pose, Mass Set. (c) - (d): Fitting NeuralNetwork on the Pose, Mass Set. . . . . . . . . . . . . . . . . 48xiiAcknowledgmentsFirst and foremost, I would like to offer my sincerest appreciation to Prof. Pai,who guided me throughout the entire project, offering his invaluable suggestionson nearly every single aspect of it, from the design of experiments to analysis of re-sults. Without his guidance I may never have pulled off this project. But in fact, tome Prof. Pai’s role is more than my thesis supervisor who oversees just a project;he is a true mentor who taught me how to conduct research and specifically, re-search in physics-based simulation. During our meetings he presented to me manytextbooks, papers (especially those that were authored by himself and his formerstudents) that are highly relevant to the project, and it was just mind blowing tome seeing how he was able to draw connections between trendy, current researchtopics to studies he did decades ago with his deep insights into physics-based sim-ulation. Moreover, from his reading group discussions I was able to pick up manyinvaluable lessons, from literature review strategies to his and Dr. Shu’s invalu-able opinions related to most cutting edge research work in the field. On top ofthat, Prof. Pai is an amazing, supportive supervisor who is willing to offer his en-couragement and help when I was either stuck and lacking progress in research, orwhen I bumped into issues in other aspects of my studies. So I believe Prof. Paiis not only an expert in the field and a true scholar, but also a supervisor of goodcharacter. I truely appreciate all the amazing opportunities he has offered to me,and I consider him as a role model, a moral compass, a Jedi Master in the sensethat if I have students to mentor in the future I should be guiding the same way thathe guides me.Next, I want to thank Prof. Rhodin and Prof. Wandt for their genuine supportover the course of the project. They are real experts in machine learning and vision,xiiiand only with their help in designing and tuning of neural architectures I was ableto build a neural network that outperforms baseline. They are always willing tolisten to my newbie questions and offer their opinions with great patience. Also, Iwant to thank Prof. Wandt for connecting me to Prof. Bodo Rosenhahn’s group so Icould ask them questions related to how their data was collected and preprocessed.And I hereby extend my appreciation to Natural Sciences and Engineering Re-search Council of Canada (NSERC) for its funding of the project.Moreover, I want to thank Felix Winkler for helping me with organizing thedata preprocessing scripts used for preparing the publicly human motion datasetfrom Prof. Rosenhahn’s group, and answering many of my questions related to thedataset. He is an expert in his project, a wonderful human being, and I wish himthe best in his studies.Other than people directly involved in the project, I would like to thank myschoolmates and friends including Pearson, Megha, Egor, Jennie, Yibo, Haomiao,Bole, Ming for their help and company; especially to Yuhao, Tianxin, Dingqing,Soheil for their valuable advice on my studies. Hearing what they think aboutresearch and telling them what I think about my experiences make me feel that Iam not alone in my academic endeavor, that my experiences are more shared thanindividual, and that I am connected to my peers in many ways. I also want to thankmy friend Chun for offering me a cozy space to write up the thesis during my tripto China. The place was quiet, lovely, with amazing food options, and it reallyhelped me focus on the writing. Without your help my time during the trip wouldhave been a lot less productive.Last but not least, I want to thank my mother, my grandmother and my fatherfor offering me unconditional love and every kind of support I can ever think of.Knowing that they will always love me, protect me, believe in me and support meon pursuing graduate studies is an essential part of what drives my forward in therealm of academia. Furthermore, I want to thank grandfather Tailin and uncle Ruifor their great advice on how to work with others in academia and on conductingresearch. Their wise words offered me new perspectives on many subjects that mat-ter to graduate students, and these words played rather crucial roles in my makingof several important decisions. I cannot thank them more for the generous supportthey are willing to offer me.xivDedicationTo my grandmother Jingwen and my late grandfather Chenglin.xvChapter 1IntroductionIdentifying inertial parameters of human subjects has been a pivotal and long-standing problem in biomechanics. The identified inertia can be useful for a mul-titude of applications: from solving inverse dynamics [Dao, 2019, Lam and Vu-jaklija, 2021, Shourijeh et al., 2017, Xiang et al., 2011, Zell et al., 2020] andsimulating human motions [Clancy et al., 2023, Ren et al., 2007, Sharif Razavianet al., 2019, Song et al., 2021] within the domain of biomechanics, to simulatinganatomically realistic animated characters [Jiang et al., 2019, Lee et al., 2019a,Nakada et al., 2018, Park et al., 2022] or improving accuracy and physical realismof motion tracking in computer vision [Andrews et al., 2016, Ga¨rtner et al., 2022,Shimada et al., 2020].Pioneering work that tackled the problem estimated body segment inertial pa-rameters (BSIPs) of live subjects from photometric data and anthropometric mea-surements [Dumas et al., 2007, McConville et al., 1980]; the data though gener-alizes to the average human population, is not specific to individuals outside thegroup of measured subjects. Recent work measured BSIPs through in-vivo stud-ies on live subjects using motion capture and force plate data [Chen et al., 2011,Damavandi et al., 2009, Hansen et al., 2014, Jovic et al., 2016, Pataky et al., 2003,Venture et al., 2008]. Regardless of their data sampling approach or estimationmethod, these approaches were all built upon the articulated rigid body (ARB)assumptions: body segments were approximated with rigid capsules or cylinders,chained up by joints; the mass of a muscle is lumped to the skeletal mass of its near-1est segment and uniformly distributed along the segment [Pai, 2010]. In fact, theseassumptions were employed not only by prior work of estimating inertia from real-world data, but also by software that simulate biomechanical systems [Rajagopalet al., 2016, Scholz et al., 2016, Seth et al., 2018]. But the two assumptions can failto fully characterize the complexity of human musculoskeletal systems: (i) bodysegments are non-rigid with complex geometries; (ii) a muscle can be attached tomultiple bones, can elongate or contract in motion. Since muscles contribute a sig-nificant portion of human body mass and thus significantly to the inertia of the fullbody, overlooking these aspects in musculoskeletal modelling can lead to 7.6% oferror in estimates of the system’s effective ankle inertia [Pai, 2010] and propagatethe error to downstream applications, e.g. gait prediction or joint torque estima-tion [Verheul et al., 2023]. To mitigate errors due to the ARB assumptions, Wanget al. [2022b] built a differentiable simulator that unlumps muscle mass from skele-tal mass, by approximating muscles as threads of “mass points” attached to bones,then derived the joint space inertia matrix of a leg system analytically from theapproximated model. Although this approach does account for the non-rigidnessof body segments and inertia can scale to individual subjects, the mass points arenevertheless an approximation, so are the computed inertia.In this study, we aim to develop data-driven representations of the joint-spaceinertia matrix of human subjects in motion. To this end, first we developed the con-stant inertia model, an inertia matrix representation defined per subject, per posefrom kinematic and dynamic data collected via a motion capture system and forceplates [Zell et al., 2020]. The model assumes nothing other than that the matrixshould be symmetric and positive definite (SPD). We show that after fitting themodel to data with constrained least squares, the yielded matrix is biomechani-cally plausible, and it reveals inertial effects that the articulated rigid body (ARB)inertia model fail to capture, including larger inertia on distal joints and the mo-tion type dependence of inertia. On the other hand, we recognize that least-squaresfitting one model for each subject, each pose can be computationally expensive,and the model can fail to generalize to unseen subjects and poses when used fordownstream tasks such as motion reconstruction. Upon witnessing the incredibleprowess of deep neural networks [Goodfellow et al., 2016] on a multitude of regres-sion tasks in biomechanics, from motion tracking [Pearl et al., 2023] to joint torque2prediction [Lam and Vujaklija, 2021], we developed a neural inertia model namedSPDNet as a representation of joint-space inertia matrix parametrized by pose andsubject-specific physical properties including body mass and height. We show thata trained SPDNet can predict physically plausible inertia; and that when trainedon datasets of different motions, the predicted inertia can also reveal motion-typedependence of inertia. Further, we show that SPDNet’s predicted matrices yieldlower motion errors than ARB inertia when we applied them to motion reconstruc-tion of gaits from subjects not seen in training. To the best of our knowledge, weare the first group ever attempted to use neural networks for estimating the inertiamatrix or BSIPs from human kinematic and force plate data in a biomechanicalcontext.But this study still has several major limitations. First, while our study showsthat data-driven models can reveal that estimated inertia can be different for walk-ing and running motion, explaining the gap between walking and running inertia isbeyond the scope of this study. Second, our estimation and motion reconstructiononly works on a specific phase in each gait cycle lasting roughly 100 ms, duringwhich velocity change is large and musculocontrol stays constant. Third, our meth-ods require both kinematic data and dynamic data, making large, publicly availabledatasets such as Mahmood et al. [2019] that contain only kinematic data not suit-able for estimation. And thus far we have only evaluated the methods on walkingand running motion, so how they will perform on other motions such as jumping,kicking are yet to be known. Lastly, we have yet to interpret our estimated inertiafrom the standpoint of mass distribution and segment geometry. We believe futurestudies can be carried out to resolve these limitations, so our method can be appliedin a broader scope.3Chapter 2Related WorkInertia estimation of human body is not only an area of interest in biomechan-ics, but also fundamentally connected to the problem of human motion trackingand character control that are of great interest to the computer vision and graphicscommunity: in a typical force-based tracking or control pipeline, the character’sactuation forces are fed into a forward dynamics solver, and the solver computesvelocity and position in the next time step based on the character’s mass matrix aswell as force inputs. In this Chapter, we offer a systematic review of prior work inthe following domains:• Estimation of human body segments’ inertial parameters (BSIPs) or the joint-space mass matrix from biomechanics and computer vision venues;• Body segment models built for analytically constructing the joint-space massmatrix. We also review various downstream tasks in biomechanics, visionand graphics that make use of those models and constructed matrices: solv-ing inverse dynamics, motion tracking and character control.• Datasets available for data-driven inertia estimation.2.1 Inertia Parameter EstimationBiomechanists have sought to accurately identify the inertial parameters of humanbody segments for centuries [McConville et al., 1980]. While people in the dis-4tant past have conceived and executed ingenious methods to measure BSIPs, weconstrain our discussion to methods from recent decades that utilize modern mea-surement and computational techniques. Pioneering work by Dumas et al. [2007],McConville et al. [1980], Young et al. [1983], Zatsiorsky [1985] attempted to esti-mate BSIPs such as the mass, center of mass and principle moment of inertia of livehuman subjects from photometric data and anthropometric measurements. Thesemeasurements are taken when subjects were at static poses but not when they werein motion, and in these early work the geometry of each body segment was notthoroughly examined. Later work derived BSIPs of human subjects from forceplate data [Damavandi et al., 2009, Jovic et al., 2016, Pataky et al., 2003], or kine-matic mocap data [Venture et al., 2008] or both [Ayusawa et al., 2011, Chen et al.,2011, Hansen et al., 2014]. These methods allow measurements to take place whilesubjects are performing actions and thus, detect differences in inertial parametersbetween when a subject is in motion and when the subject is static. Yet two prob-lems persist in these later methods: (1) No canonical standards exist to evaluate theaccuracy of obtained BSIPs [Mungiole and Martin, 1990, Pataky et al., 2003]. Typ-ically, inertial parameters were verified by comparing them with values obtainedfrom prior studies. But since prior studies often did not employ same measurementand estimation method, one may wonder the extent to which such comparisons aremeaningful. (2) They typically employ the articulated rigid body (ARB) assump-tion: body segments were approximated with rigid capsules or cylinders of uniformmass distribution, chained up by joints. While these are reasonable assumptions tomake when identifying inertial parameters of robots [Bonnet et al., 2016, Lee et al.,2019b, Wensing et al., 2017], a human body segment contains bones, tendons, fatand muscles which are deformable, so parameters such as the distribution of masswithin a segment can change during human movement.Another line of work in biomechanics seeks to use MRI data for identifyingBSIPs [Cheng et al., 2000, Martin et al., 1989, Mungiole and Martin, 1990], oridentifying BSIPs along with constructing detailed geometric representation ofeach segment [Sreenivasa et al., 2016]. While BSIPs estimated from MRI scansare comparable to those identified from photometric data and anthrometric mea-surements, MRI-based images also suffer from problems that plagued the priormethods, such as overestimating segment volumes. Moreover, MRI scans can only5be captured when a subject is in a static pose, and procuring these scans can beexpensive.Other than literature in biomechanics, a few recent work from computer visionand graphics have also attempted to estimate inertial parameters of humans in mo-tion. Lv et al. [2016] estimated the mass, center of mass and inertia of each bodysegment of human subjects when performing multiple kinds of motions, includingwalking, running and jumping, by using mocap data, force plate data and pressureinsoles’ data as inputs to their method. They acknowledge that their approxima-tion of body segments as rigid bodies can lead to errors, but offered no validationof the estimated parameters. Yang et al. [2023] attempted to estimate per-segmentmass of humans in motion from monocular videos containing only kinematic in-formation, for their end goal of reconstructing human motion. They confirmedthat estimating per-segment mass indeed improves the quality of their reconstruc-tion. The most recent, relevant work by Zhang et al. [2024] estimated the entirejoint-space mass matrix of simulated human subjects rather than their individualbody segment masses using a neural network. The network takes past joint anglesas input, transforms it through multiple fully-connected layers with activation andfinally produces the mass matrix. Although they showed that their motion recon-struction quality improves upon introducing a physics-based branch that estimatesmass and other physical forces, since inertia estimation is not the focus of the pa-per, they did not either verify the physical plausibility of estimated mass matrices,or confirm estimating the matrices alone can improve reconstruction quality.2.2 Body Segment Models for Downstream TasksIn the majority of biomechanics, vision and graphics literature, researchers con-struct joint-space mass matrix for a humanoid character to solve its dynamics. Thematrix is constructed analytically from body segment models built with geometricprimitives, and the level of detail of such models varies across literature. For solv-ing inverse dynamics, e.g. estimating ground reaction force (GRF) or joint torque(JT), one would use the mass matrix and the character’s kinematic state to solve forforces [Brubaker et al., 2009, Clancy et al., 2023, Park et al., 2022, Ren et al., 2007,Shimada et al., 2021, Venture et al., 2008, Verheul et al., 2022, Yi et al., 2022, Zell6and Rosenhahn, 2015, 2020, Zell et al., 2020]; to solve motion tracking or charac-ter control problem, an integrator would take the constructed mass matrix as wellas external forces applied to the character, compute the character’s joint-space ac-celeration and finally, advance the character to its next state [Andrews et al., 2016,Brubaker et al., 2010, Ga¨rtner et al., 2022, Rajagopal et al., 2016, Yang et al.,2023].The simplest yet most commonly employed is the class of articulated rigid body(ARB) models: each body segment is modelled as a rigid object, often in cylindri-cal or capsular shape, with mass uniformly distributed in the segment [Bergaminet al., 2019, Ga¨rtner et al., 2022, Liu et al., 2021, Lv et al., 2016, Shimada et al.,2021, Venture et al., 2008, Yi et al., 2022, Zell et al., 2020]. The mass of eachsegment either follows population average [Ga¨rtner et al., 2022, Yi et al., 2022,Zell et al., 2020] or is personalized per subject via optimization [Lv et al., 2016,Venture et al., 2008]. These models are easy to build and to actuate for controltasks [Bergamin et al., 2019, Shimada et al., 2021].Another class of body segment model is musculoskeletal models that uses dif-ferent geometric primitives to represent tissues that constitute body segments, e.g.bones, muscles and tendons. In these models, tendons and muscles are representedby elastic strings attached to bones [Park et al., 2022, Rajagopal et al., 2016, Scholzet al., 2016, Seth et al., 2018, Shourijeh et al., 2017, Zhang et al., 2019]. This kindof models is better in terms of anatomical realism, therefore suitable for clinicalanalysis in biomechanics [Rajagopal et al., 2016, Scholz et al., 2016, Zhang et al.,2019] and has been used by professional biomechanical modelling software suchas AnyBody [Damsgaard et al., 2006] and OpenSim [Seth et al., 2018]. Recentwork in graphics have also become interested in this class of models: while suchmodels are muscle-driven and therefore inherently difficult to control, due to sig-nificant advances in neural network-based controllers in recent years we are seeingmore neural network-controlled, musculoskeletal animated characters [Lee et al.,2019a, Park et al., 2022]. Mainstream robotics simulation suites such as MuJoCoalso support simulating this kind of models [Todorov et al., 2012], and open-sourcepart models have been built by the graphics and robotics community [Wang et al.,2022a]. But the major difference between this class of models and ARB mod-els is in control: while most ARB models come with PD (joint angle) or torque7controllers, musculoskeletal models are often muscle-actuated, i.e. a controllerproduces muscle activation levels, and the activation level controls the length oftendons (muscles), so bones rotate around joints [Lee et al., 2019a, Park et al.,2022, Seth et al., 2018, Thelen and Anderson, 2006]. But in terms of geometricconstruction tendons are not distinguished from muscles [Lee et al., 2019a, Parket al., 2022, Seth et al., 2018]; moreover, when looking at inertial parameters or themass matrix derived from those parameters, such models are fundamentally iden-tical to ARB models: the mass of muscles and tendons are lumped to their nearestbones, such that each segment is effectively treated as a single rigid body in inertiaevaluation [Scholz et al., 2016].Biomechanists have noticed the issue with mass lumping, and developed solu-tions to mitigate it. Pai [2010] pointed out that the lumping strategy is problematicbecause (i) the mass of a body segment comes from muscles, not bones; (ii) lump-ing ignores the inertial coupling of muscles to distal joints and thus can lead to in-ertia of distal joints being incorrectly estimated. They developed a skeletal modelconsisting of two bones connected by a revolute joint, and created an “unlumped”musculoskeletal model in which a muscle is an elastic line segment with uniformlydistributed mass connecting two bones. Compared with the inertia obtained froma model with all muscle mass lumped to the bones, the unlumped system’s inertiais significantly different. Later work attempted to resolve this issue by levelingup simulated details: Zhang et al. [2019] models a muscle using multiple Cosseratrods, with each rod having its own elastic properties and mass. But this modelsuffers from scalability issues due to extra degrees of freedom introduced [Zhanget al., 2019]. Building upon Pai [2010], Wang et al. [2022b] further discretizedmuscle as a thread of “mass points” connected via a muscle path that is attachedto bones. Although the “mass points on a thread” model mitigates the issues dueto muscle lumping to some extent, it is nevertheless an approximation to real mus-cles, and in some cases the mass matrix analytically constructed from the modelstill underestimates muscle inertia [Wang et al., 2022b].82.3 Human Motion DatasetsDatasets that can be used to estimate joint-space mass matrices of humans in mo-tion can be broadly divided into two categories: dynamic datasets and kinematicdatasets.Dynamic datasets contain dynamic information such as ground reaction force,and often kinematic information such as joint positions, and the physical dimen-sion, mass of each subject as well. Prior work in biomechanics generally use dy-namic datasets for identifying BSIPs [Ayusawa et al., 2011, Chen et al., 2011,Damavandi et al., 2009, Hansen et al., 2014, Jovic et al., 2016, Pataky et al., 2003].Datasets such as the ones used in Ayusawa et al. [2011], Chen et al. [2011], Hansenet al. [2014] consist of motion clips collected with mocap systems that measure in-door human motion up to high accuracy, and force plate data that contain groundreaction forces. Although these datasets contain abundant amount of informationfrom included human movements, their are small in scale because only the motionof a few subjects and a few types of in-door motions were measured since mocapsystems can only be installed in laboratory settings. Also, these datasets are notpublicly available. But in recent years we see more dynamic datasets becomingpublicly available: Zell et al. [2020] built a dataset consisting of mocap clips andforce plate measurements from 22 subjects; Mourot et al. [2022] built a datasetincluding mocap clips as well as pressure insole data measured on 18 categories ofmotions.On the other hand, many large-scale kinematic datasets are publicly availablefor training visual models to understand human motion [Kuehne et al., 2011, Mah-mood et al., 2019]. The AMASS dataset for example contains 40 hours of motiondata collected from hundreds of subjects covering over ten thousand kinds of mo-tions [Mahmood et al., 2019]. Recent work from computer vision have attemptedto estimate inertial parameters from kinematic data [Yang et al., 2023, Zhang et al.,2024] using deep neural networks. Since dynamic information such as ground re-action force is not available, they performed the estimation by plugging the result-ing mass matrix into a differentiable simulation pipeline, and using a loss functionthat penalizes differences between ground-truth and predicted trajectory to trainthe neural networks.9Chapter 3MethodsIn this Chapter we discuss in detail our methods of estimating the effective inertiaof a human subject in motion, using kinematic and dynamic data. Section 3.1 de-fines effective inertia (matrix) which we intend to estimate. Section 3.2 describesthe specification of the dynamic dataset that we use. In Section 3.3 we introducethe constant inertia model, a non-parameterized representation of the effective in-ertia matrix defined on a per-subject, per-pose basis, and discuss how we optimizedthe model via constrained optimization. In Section 3.4 we introduce neural inertiamodels that define effective inertia matrices with respect to poses and body pro-portions. These are parametrized representations of the effective inertia matrix,and we discuss how we optimized them using gradient descent in the same section.Section 3.5 gives a brief overview of the articulated rigid body (ARB) inertia modelthat we use as a baseline to compare with the former two models. The ARB inertiamodel is a parametric representation of the inertia matrix body mass, body heightand pose; yet unlike the other two models, it is defined by analytical constructionrather than estimation from data. In Section 3.6 we discuss how we plug an inertiamatrix, be it constructed or estimated into a differentiable simulation pipeline toreconstruct the motion of human subjects in stance phase, and then move onto Sec-tion 3.7 to discuss the metrics we used to evaluate the quality of reconstruction andhence, the quality of inertia estimation. We also refer readers to Appendix A fordetails about how we constructed a synthetic dataset and prototyped our estimationmethods on it.103.1 Effective InertiaFigure 3.1: Method overview. (a) Curation of mocap and force plate data andthe dataset partitioning scheme. (b) Selection of data to use for inertiaestimation. An inertia model, be it a neural network or a constant inertiamodel, takes data from the first 100 ms of the initial single support phaseof a gait cycle, and predicts a joint-space inertia matrix correspondingto the initial pose of the cycle. (c) Kinematic model of the dataset.We focus on analyzing inertial properties and reconstructing motion oflower body segments and the torso.Prior work in human motion reconstruction and analysis [Zell et al., 2020,Zhang et al., 2024] typically formulate the dynamic equation of motion of a hu-11man kinematic model in generalized coordinates asM(m,h,q)q¨ = G(m,q)+C(m, q˙)+ τcontact, (3.1)where m is the body mass of a human subject, h is the body height of the subjectM(q) is the d×d (where d is the degree of freedom of the kinematic model) subjectand pose-dependent joint-space inertia matrix of the subject, , q is pose, G(m,q) isthe pose and body mass-dependent gravitational force (GRF), C(m, q˙) is the Corio-lis force, τcontact is ground reaction force and moment. And the inertia matrix M(q)is often habitually constructed from the articulated body (ARB) assumptions—weshowcase the construction from Zell et al. [2020] in Sec. 3.5 and used the inertia asthe baseline. But an analytically constructed ARB inertia cannot capture the com-plex geometry and mass distribution of body segments; moreover, due to lumpingof muscle mass to skeletal mass, the inertia of a joint would only be influenced byits distal segments, and the mass of a muscle will not contribute to the inertia of itsdistal joints, even though it should be inertially coupled to those joints [Pai, 2010].To address the two problems, Verheul et al. [2022], Wang et al. [2022b] took an an-alytical approach by approximating a muscle as a thread of mass points, with massuniformly distributed among the points. We instead take a data-driven approach:given measured body mass m, pose q and GRF, we directly estimate the inertiamatrix M(q); we refer to an estimated inertia that takes account of the complexityof geometry, mass distribution and inertial coupling as an effective inertia.3.2 DatasetWe used the publicly-available human motion dataset from Zell et al. [2020]. Itcontains 318 walking clips and 132 running clips collected from 22 subjects, withdifferent gender and BMI’s; each clip consists of joint poses sampled via a Vi-con Mocap System and contact forces sampled via AMTI force plates [Zell et al.,2020] at 10 ms intervals, from the first half of a gait cycle. Since we are interestedin knowing if inertia is motion-type dependent, we divided the dataset into theWalking Set and the Running Set, and estimated inertia on the two sets separately.Since the inertia matrix also depends on body mass proportion and pose, we furtherpartitioned each of the two sets on a per-subject, per-pose basis: split into subsets,12each made up of gait cycles from one subject, then divided the subset into left-footand right-foot contact set. See Fig. 3.1 (a). Moroever, we exclusively use datafrom the initial single support phase t0 ≤ t < tL in a gait cycle for estimation: seeFig. 3.1 (b) [Bonnefoy-Mazure and Armand, 2015]. This is a period of time duringwhich the ipsilateral foot is in contact with the ground and the contralateral foothas just lifted off, and it lasts for roughly 13% of a gait cycle [Bonnefoy-Mazureand Armand, 2015]. So we pick a uniform window of tL = 100ms for all samplesin both the Walking and the Running Set. We pick this particular window becauseduring this time joint velocity change is sufficiently large such that ignoring veloc-ity dependent forces (e.g. the Coriolis force) and musculo control will not lead tosignificant errors. The dataset has been preprocessed by Zell et al. [2020] such thatthe kinematic model consists of 24 degrees of freedom, including 6 degrees repre-senting root translation and rotation, and 18 lower-body joint angles: see Fig. 3.1(c).3.3 Constant Inertia ModelsWe can estimate a single effective inertia matrix at a particular pose for a particularsubject in a 100 ms-long estimation window, because we can safely assume thatthe matrix stays constant in the window since pose changes are minuscule. Startingfrom dsicretizing the motion of a d-DoF system at an arbitrary time t = ti in thewindow during t0 ≤ ti ≤ tL−1 on the velocity level:Meffective(q˙ti+1 − q˙ti) = hG(qti)+hτcontact(ti), (3.2)where Meffective is a d × d effective inertia matrix that depends on motion type,G is gravitational torque and h is the length of a timestep. We drop the Coriolisterm since it is small relative to other terms. We expect Meffective to be symmet-ric and positive definite, since a non-SPD effective inertia implies a non-positiveeigenvalue, which means if we apply a generalized force that is parallel to its corre-sponding eigenvector, the responding acceleration would be in the opposite direc-tion, which is physically implausible. So when pose and subject are fixed, we canconstruct a simple yet effective statistical model to estimate the effective inertia,by minimizing the difference between predicted and ground-truth per-step squared13impulse f (Meffective), using constrained least squares:f (Meffective) : Rd×d → R= ||MeffectiveX−hτ||2,subject to Meffective ≻ 0,(3.3)where X ∈ Rd×L is the velocity difference matrix:X = [x0,x1, . . . ,xL−1] where xi = q˙ti+1 − q˙ti , (3.4)and τ ∈ Rd×L is the external force vector:τ = [τ0,τ1, . . . ,τL−1] where τi = [G(qti)+ τcontact(ti)]. (3.5)The effective inertia matrix M∗effective that yields fmin is the constant inertia ma-trix we seek for.To evaluate the physical pluasibility of the constant inertia model, we fit tothe Running Set and the Walking Set respectively, and for each of the two setswe fit one model per subject, per ipsilateral foot. This guarantees that gait cyclesused to fit each model share the same body proportions and a roughly consistentpose. But to evaluate the constant inertia model’s ability to generalize to unseenbody proportions and poses and for comparison with neural inertia models, we fit asingle constant inertia model respectively to the Walking Set and the Running Set,such that the fitted effective inertia is averaged across different subjects and poses.In terms of implementation, we use optimization package CVXPY [Diamond andBoyd, 2016] along with CVXPYLayer [Agrawal et al., 2019].We demonstrate that the estimated effective inertia is physically plausible, andthat the effective inertia matrices are both drastically different from ARB inertiaand dependent of the type of motion a gait cycle belongs to. These properties canbe conveniently overlooked by the aforementioned ARB assumptions, but not sowhen we estimate inertial properties from dynamic data.143.4 Neural Inertia ModelsA problem with the constant inertia model as a non-parametrized representation ofeffective inertia matrix is that it does not generalize to different mass distributionsor poses, so an estimated inertia matrix is only physically plausible if fitted to aparticular subject at a specific pose. This makes constant inertia models compu-tationally expensive to estimate, if we ever want to estimate multiple such modelsin order to reconstruct the motion of multiple subjects in different poses. There-fore, we aim to develop neural networks that predict a joint-space inertia matrixparameterized by joint poses and body mass distributions.Figure 3.2: Architecture of SPDNet. First, we feed body mass and heightinto the MLP-based feature extractor to predict an 8-D person vectorp. Next, we feed p along with joint pose q into the MLP-based inertiapredictor to produce d(d+1)2 entries. Finally, we feed these entries to theSPD Layer [Jekel et al., 2022] to make the final prediction SPD.Fig. 3.2 shows the architecture of SPDNet which we developed for inertialprediction. The network predicts a d× d SPD effective inertia matrix from jointpose, body mass m and body height h:Meffective = fSPD(q,m,h), (3.6)where q ∈ Rd are joint angles. It consists of three components: (i) a person vec-tor extractor. This is a simple multilayer perceptron (MLP) network [Goodfellowet al., 2016] with five fully-connected (FC) layers and leaky ReLU (LReLU) ac-15tivation [Nair and Hinton, 2010, Xu et al., 2015]. It takes a subject’s body massand height as inputs and predicts an 8-D latent vector p which encodes the iner-tial and geometric properties of the subject. (ii) an inertia predictor. It is also anMLP network, but with eight FC layers and more neurons in each layer. For thelast FC layer we include d(d+1)2 neurons, so the network predicts the set of inde-pendent entries required for constructing a symmetric inertia matrix. (iii) An SPDLayer [Jekel et al., 2022] which enforces inertia matrice predictions to be SPD. Tofit the network on real-world data with d = 24, we include 24×252 = 300 neuronsin the last FC layer in the inertia predictor. For both MLP networks we includeda descending order of neurons from the first to the last FC layer, following theconvention of MLP network design for regression tasks [Ozates et al., 2023].We train the network using per-step impulse loss which is similar to Eq. 3.3:1N1LN∑j=1L−1∑i=0||Meffective(q˙ j,ti+1 − q˙ j,ti)−h(G(q j,ti)+ τcontact(ti))||2, (3.7)where G(q j,ti) and τcontact(ti) are ground-truth generalized forces from gait cycle j,time ti. In terms of the optimization method, we use mini-batch gradient descent,with the ADAM optimizer [Kingma and Ba, 2014] at the learning rate of 5e− 4,and batch size of 32 gait cycles for 500 iterations.To confirm that our trained models can generalize to unseen poses as well assubjects of unseen mass distribution and heights, we perform leave-one-out crossvalidation: For each of the Walking Set and the Running Set, we create a fold byrandomly pulling one subject out for testing, two subjects for validation and weleave the rest of subjects for training, and we train and test the network over sixfolds. We show, using the ARB inertia model as a baseline, that if we simulate asubject’s motion with inertia matrices predicted from SPDNet, we can attain lowermotion errors than using the ARB inertia model.Next, we present an ablation study to explore the impact of architectural de-cisions. We developed SymmNet fSymm which takes the same inputs as SPDNet,contains the same MLP networks, and constructs a symmetric effective inertia ma-trix from the output of the MLP network just like SPDNet. But it does not pass thematrix to the SPDLayer [Jekel et al., 2022] to make the matrix positive definite. We16also developed UnconNet, an “unconstrained” network whose inertia matrix pre-dictions are not subject to any hard constraints. Its MLP network directly predictsd×d entries of an inertia matrix instead of its unique entries, and it does not con-tain the SPD Layer following the MLP network. For training and testing SymmNetand UnconNet, we use the same hyperparameters used for SPDNet. In Sec. 4.2,we show that enforcing inertia matrix to be SPD via the SPD layer [Jekel et al.,2022] significantly lowers motion errors, while guaranteeing physical plausibility,and that SPDNet outperforms its two counterparts. Then we look into hyperpa-rameter decisions on the SPD Layer—specifically our selection of the Eigen Layerand the ReLU function for positivity enforcement, and discuss why we believe thecombination of the two yields the best performance.The neural networks were implemented in PyTorch [Paszke et al., 2019]. Wecompleted all of our experiments on a desktop with i7-11700 CPU, 64 GB of RAMand NVIDIA RTX 3090 GPU with 24GB of RAM.3.5 The Articulated Rigid Body (ARB) Inertia ModelWhile both the constant inertia model and neural networks need to be trained withdata, we can define an ARB inertia model by analytically construction, when thebody mass, height and the pose of a subject is given, following the approach in[Zell et al., 2020], without relying on state trajectory or dynamic information. Firstwe compute the rotational inertia of each body segment Itorso, Ipelvis, Ithigh, Ishank,Ifoot in its local frame under the following assumptions: (i) the pelvis and torsoare elliptical cylinders; (ii) thighs and shanks are cylindrical cylinders; (iii) feet aresemi-ellipsoids. This gives us the half-body rotational inertia Irot ∈R3Ns×3Ns , whereNs is the number of segments in our ARB skeletal model. Then we put translationaland rotational inertia together into the spatial inertia matrix:Mspatial(q,m,h) =[Itrans(m) 00 Rsegs(q,h)Irot(m,h)Rsegs(q,h)T], (3.8)17where Itrans ∈ R3Ns×3Ns is the diagonal matrix of per-segment masses:Itrans(m) =mtorso 0 . . . 00 mpelvis . . . 00 . . . mleft thigh 00 0 . . . mright foot , (3.9)and Rsegs(q) is the R3Ns×3Ns segment rotation matrix that depends on configurationq. The mass of each body segment is computed by multiplying total body mass mwith Dempster’s body parameters [Winter, 2009], which is estimated from generalhuman population. We can then compute a joint-space, pose and body proportiondependent ARB inertia matrix MARB(q,m,h) for any pose q:MARB(q,m,h) = Jsegs(q)T Mspatial(q,m,h)Jsegs(q), (3.10)where Jsegs(q) is the R6Ns×d segment Jacobian matrix.3.6 SimulationTo reconstruct a subject’s motion in the estimation window, we inject the estimatedinertia matrix M(qt=0) to a forward dynamics pipeline, and obtain pose q overL frames in the estimation window. Specifically, at each frame t we solve forgeneralized acceleration first, using Tikhonov-regularized least squares since thesystem can be highly ill-conditioned:q¨ti = argq¨ min ||Meffective(qˆt=0)q¨t − (G(qti)+ τcontact,ti)||2+λ ||Dq¨ti ||2, (3.11)where λ = 1e−5 is the regularization parameter and D is the d×d identity matrixthat penalizes large joint acceleration along any particular dimension. We thenapply Semi-Implicit Euler integration to update the generalized state:[q˙ti+1qti+1]=[q˙tiqti]+h ·[q¨tiq˙ti+1]. (3.12)183.7 MetricsSince no ground truth inertia matrices that capture non-rigidness and inertial cou-pling are available, to evaluate our trained inertia estimators, we inject the esti-mated inertia into a motion simulation pipeline and measure reconstruction errors,including mean angle error (MAE) and mean per-joint position error (MPJPE).MAE is the predominant metric for evaluating the quality of human motionprediction in the last decade [Lyu et al., 2022]. For a batch of N gait cycles, wecompute MAE asMAE =1N1L1KN∑n=1L−1∑i=1K∑k=1||q7:n,ti,k− qˆ7:n,ti,k||, (3.13)where i is the frame index and k is the joint index. Since the first six degreesof freedom correspond to root translation and orientation, we extract the later 18degrees of freedom for measuring MAE.Mean per-joint position error (MPJPE) Lyu et al. [2022] measures the Eu-clidean distance between ground truth and simulated joint positions:MPJPE =1N1L1KN∑n=1L−1∑i=1K∑k=1||xn,ti,k− xˆn,ti,k||2, (3.14)where xn,ti,k ∈ R3 is the global position of joint k at time ti in gait sample n, obtainedfrom forward kinematics on the predicted pose qn,ti,k.19Chapter 4Results4.1 Inertia Matrices Estimated with Constant InertiaModelsFig. 4.1 shows effective inertia estimates from constant inertia models alongsidewith inertia computed by the ARB inertia model, visualized as box plots. To visu-alize a d×d inertia matrix we first accumulate entries along each row (i.e. lump todiagonal entries), then for each joint aggregate inertia of angles that belong to thejoint with 2-norm. We can make several observations from the figure and see thatinertia estimated by constant inertia models are physically plausible.First, we can see from Fig. 4.1 (a), (b) that our estimated walking inertia isbilaterally symmetric. Furthermore when the ipsilateral foot is the left foot, inertiaof joints on the left leg are larger than the inertia of their counterparts on the rightleg; and when the ipsilateral foot is the right foot, right leg inertia is larger thantheir counterparts on the left leg. This pattern is also apparent in running samples,see Fig. 4.1 (c), (d). We hypothesis that this is due to the role of neural activationon inertia coupling. During walking, muscles of the stance leg before foot fall maybe actuated to a greater amount than the muscles of the contralateral leg, increas-ing the effective inertia of the ipsilateral joints. Similarly, we expect muscles tobe activated to a greater extent during running. This is consistent with the obser-vation that the estimated joint inertia (Constant Inertia model) during running isconsistently greater during running than during walking. More research to test this20hypothesis is still needed.Second, if we look at effective inertia of distal joints, i.e. knee and ankle, wesee that estimated inertia are in general larger than their counterparts constructedunder the ARB assumptions. Most notably, ankle inertia is more apparent whenestimated from data. This is consistent with results from Pai [2010]: when musclemass is unlumped from bone mass the muscle increases inertia of joints distal to it.So the additional estimated inertia is due to shank muscles’ coupling to the anklejoint.What the ARB inertia model fails to capture, and the constant inertia model isable to capture however is the dependence of effective inertia on motion type. FromFig. 4.2 we see that constant inertia’s magnitudes become larger once we switchmotion from walking to running. Yet the ARB inertia model fails to capture sucheffects, as we can tell from the figure that inertia difference is minuscule and onlyexists due to small pose changes from walking to running. Although we can easilydeduce that ARB inertia model cannot capture motion type dependence becauseit is not fitted on data and it is not parametrized by motion type, we cannot tellwhy the estimated running inertia are larger in magnitude, and understanding thatrequires further study.4.2 Inertia Matrices Estimated with Neural InertiaModelsFig. 4.3 compares effective inertia matrices estimated by SPDNet versus thoseconstructed via the ARB inertia model. Again, we see patterns that imply phys-ical plausibility: (i) larger estimated inertia than constructed ARB inertia on an-kle; (ii) bilateral symmetry in magnitude of inertia, i.e. when a leg is in contactwith ground, inertia on its joints are larger than inertia on their counterparts onthe other leg. The symmetry is obvious for walking gaits, but in the case of run-ning gaits, inertia on left-leg joints are larger even if the stance leg is the right leg;see Fig. 4.3 (d). The breaking of bilateral symmetry can be due to that we fitteda single SPDNet to both left and right foot-contact gait cycles. Since stance legis not explicitly used as an input to the network in training, the predicted inertiacan be averaged out by a bit. Comparing walking inertia with running inertia of21SPDNet in Fig. 4.4, we notice again that the gap between walking and runninginertia would be larger if the inertia is estimated from data, than constructed underthe ARB assumptions. But contrary to what we have seen in Fig. 4.2, SPDNet-predicted running inertia are lower in general. Knowing why we see this howeverwould require further analysis.Tab. 4.1 shows the magnitudes of motion errors from the constant inertia model,neural inertia models and the ARB inertia model in testing. We see that our SPDNetcan generalize to walking gaits of unseen poses and mass distributions, and thatSPDNet outperforms the ARB inertia model, the constant inertia model and otherneural inertia models with lower motion reconstruction errors, on both the Walk-ing Set and the Running Set. Also, our neural networks outperform the constantinertia model on test sets, as we expected, because a key advantage of the neu-ral networks over the constant inertia model is that they are parmetrized by bodymass, body height and poses, and thus can generalize to unseen body proportionsand poses at test time. Fig. 4.5 visualizes the distribution of MAE and MPJPE ofeach model over six test subjects selected from the Walking Set and the RunningSet. We can tell from the figure that SPDNet does not outperform other modelssimply because it does better on a few outlier clips. We believe the constant inertiamodel’s particularly bad performance is due to (i) the non-parametrized model’sinability to generalize to unseen body proportions and poses; (ii) the averaging ef-fect. Note that the model is fitted to an entire training set consisting of gait samplesfrom different subjects at different poses; see Sec. 3.3. So the estimated inertiacan be averaged out such that it does not work well if used to simulate a particu-lar subject’s motion with a specific initial pose. In Fig. 4.6 we visualize 150 ms(including 100 ms in the estimation window and 50 ms after the window) of jointangular trajectory reconstructed by SPDNet, the ARB inertia model alongside withground truth trajectories. Notice that the ARB inertia model produces unstable mo-tions when tested on either the Walking Set or the Running Set. On the other hand,SPDNet’s predicted angular trajectories are a lot closer to the ground truth thanARB inertia’s predicted trajectories on both the Walking Set and the Running Set,and its reconstructed motion remains stable until after the estimation window. SoSPDNet’s predicted matrices can generalize well to reconstructing unseen motionfor a short period of time outside window.22Turning to qualitative evaluation: Fig. 4.7 visualizes a motion clip reconstructedby different models. We see that skeletal motion reconstructed with SPDNet is theclosest to the ground truth. But SymmNet and UnconNet fail to produce visuallybetter results than the ARB inertia model. Also, the motion reconstructed withthe constant inertia model is no longer stable after 50 ms, and in terms of visualcloseness to ground-truth motion it is also worse than others.The observation that SPDNet but not SymmNet or UnconNet beats the ARBbaseline motivates us to look into the impact of architectural decisions on our neu-ral networks’ performance. From Tab. 4.1 we notice that enforcing the predictedinertia matrix to be SPD yields much lower motion reconstruction errors than en-forcing the matrix to be simply symmetric, or not enforcing constraints at all, eitheron the Walking Set or the Running Set. Our qualitative evaluation in Fig. 4.7 canalso confirm this. On the other hand, from Tab. 4.1 we notice that enforcing sym-metry alone does not grant much performance boost over UnconNet on the WalkingSet and in fact, makes performance worse than UnconNet on the Running Set. Webelieve a part of the reason that contributes to the performance difference is thatinertia matrices predicted by SymmNet or UnconNet have negative eigenvalueswhich lead to physically implausible behaviors. Looking at Fig. 4.8, we see thatthe inertia matrix predicted by SymmNet contains negative eigenvalues, and thatthe magnitudes of some of those negative eigenvalues are larger than the magni-tudes of their counterparts produced by UnconNet. A large, negative eigenvaluecan lead to problems when we use the matrix to solve forward dynamics. First, aforce applied in the direction of the eigenvector corresponding to the eigenvaluecan lead to a velocity change in the opposite direction, which violates Newton’sSecond Law of Motion, see Fig. 4.9. Second, from Eq. 3.12 we can tell that if apredicted acceleration can be decomposed into a component vector that lies alongthe direction of the eigenvector (and another component that is in a different di-rection), the “eigen-component” can lead to a fairly large predicted force whichamplifies loss. Moreover, we can tell from 4.8 that UnconNet does not naturallypredict symmetric matrices because we can see complex eigenvalues, which arephysically implausible. We can also see from the same figure that the Constant In-ertia model produces matrices which contains slightly negative eigenvalues. Thisis not what we expected given the PSD enforcement in Eq. 3.3, and we believe this23is due to numerical errors in CVXPY [Diamond and Boyd, 2016].Moving onto Tab. 4.2: here we compare the effects of different SPD layer op-tions, including whether we use Cholesky Layer or Eigen Layer, and the positivityfunction [Jekel et al., 2022] we use. We notice that learning with SPDNet is onlyeffective when we use Eigen layer and the ReLU positivity function. It makessense that Eigen Layer outperforms Cholesky Layer, since Eigen layer preserveseigenbases learned by the MLP network from inputs, and such eigenbases are ef-fectively latent vectors that incorporate physical and geometric properties of bodysegments. As far as using ReLU for positivity enforcement, we believe it is moreeffective than the Absolute Value function or the Square function when used withthe Eigen layer because it preserves the ordering of eigenvalues Jekel et al. [2022],such that the MLP network is always penalized for predictions that lead to negativeeigenvalues.24Figure 4.1: Distribution of effective inertia estimated by the constant iner-tia model vs computed by the ARB inertia model over all subjects.Each subject is (a) walking with left foot on ground; (b) walking withright foot on ground; (c) running with left foot on ground; (d) runningwith right foot on ground. The magnitudes are computed by diagonallylumping inertia matrices first, then aggregating over angles (displace-ments) associated with a joint (direction). We plot the distribution ofmagnitudes on log scale (left) and linear scale (right).25Figure 4.2: Distribution of walking vs running effective inertia with the con-stant inertia model and the ARB inertia model. (a) Constant inertiamodel’s predictions for left-foot contact gaits and (b) right-foot contactgaits. (c) ARB inertia model’s constructions for left-foot gaits and (d)right-foot gaits. The magnitudes are also computed by diagonal lump-ing then aggregated. We plot the distribution across all subjects on logscale (left) and linear scale (right).26Figure 4.3: Distribution of effective inertia estimated by SPDNet vs com-puted by the ARB inertia model over all subjects. Each subject is (a)walking with left foot on ground; (b) walking with right foot on ground;(c) running with left foot on ground; (d) running with right foot onground. The magnitudes are also computed by diagonal lumping thenaggregating, then plotted on log scale (left) and linear scale (right).27Figure 4.4: Distribution of walking vs running effective inertia with SPDNetand the ARB inertia model. (a) SPDNet’s predictions for left-foot con-tact gaits and (b) right-foot contact gaits. (c) The ARB inertia model’sconstructions for left-foot gaits and (d) right-foot gaits. The magnitudesare also computed by diagonal lumping then aggregated. We plot thedistribution across all subjects on log scale (left) and linear scale (right).28Subj + Metric SPDNet SymmNet UnconNet Const. M ARB M(a) Walking Set MAE (rad, ↓), MPJPE (mm, ↓)01 MAE 0.019 0.118 0.170 0.189 0.661MPJPE 21.4 121 106 169 22603 MAE 0.031 0.133 0.137 0.180 0.332MPJPE 57.9 115 110 102 19505 MAE 0.031 0.158 0.207 1.02 0.538MPJPE 50.9 105 132 619 22809 MAE 0.034 0.085 0.114 0.214 0.476MPJPE 80.2 73.6 103 338 21317 MAE 0.0397 0.108 0.193 0.482 0.431MPJPE 64.1 99.6 130 280 18220 MAE 0.0435 0.127 0.227 0.345 0.250MPJPE 43.5 127 227 345 250Avg. MAE 0.031 0.124 0.190 0.414 0.475Avg. MPJPE 53.0 107 135 309 216(b) Running Set MAE (rad, ↓), MPJPE (mm, ↓)03 MAE 0.019 0.334 0.118 1.30 0.137MPJPE 27.8 329 161 707 84.504 MAE 0.020 0.789 0.067 1.54 0.536MPJPE 28.9 518 72.1 687 14610 MAE 0.0178 0.165 0.391 2.73 0.213MPJPE 30.3 191 481 1669 12817 MAE 0.026 0.511 0.697 0.939 0.208MPJPE 26.1 395 436 1112 86.218 MAE 0.022 0.283 0.444 1.58 0.305MPJPE 26.9 225 266 844 12620 MAE 0.021 0.173 0.143 1.59 0.16MPJPE 28.5 160 140 909 87.4Avg. MAE 0.021 0.376 0.310 1.61 0.260Avg. MPJPE 28.1 303 260 988 110Table 4.1: Test set MAE and MPJPE of different models for inertia represen-tation. We evaluated each model on six folds; for each fold we use gaitcycles from one subject for testing. (a) Walking Set results. (b) RunningSet results.29Figure 4.5: Box-and-whisker plots of metrics from reconstructing test setmotion clips of the six folds by different inertia models: (a) WalkingSet MAE and (b) MPJPE; (c) Running Set MAE and (d) MPJPE. Left:log scale, right: linear scale.Arch. (Layer, Positivity Function) MAE(rad, ↓)MPJPE(mm, ↓)SPDNet (Cholesky, ReLU) 3.74 2165SPDNet (Cholesky, Abs. Value) 3.82 2968SPDNet (Cholesky, Square) 4.59 2039SPDNet (Eigen, ReLU) 0.031 49.4SPDNet (Eigen, Abs. Value) 1.27 814SPDNet (Eigen, Square) 4.40 1536SymmNet 0.135 120UnconNet 0.192 238Const. M 0.808 428ARB 0.475 216Table 4.2: Mean test set MAE (↓) and MPJPE (↓) of walking gait inertia es-timation from different SPDLayer configurations. We also list the MAEand MPJPE from SymmNet, UnconNet, Constant Inertia Model and theARB Inertia Model for comparison.30Figure 4.6: Joint angles’ mean± one standard deviation vs time from motionclips reconstructed by different models. The joint angles are grouped byanatomical planes they are situated in: e.g. the hip Sagittal angle is thehip-x axis angle shown in Fig. 3.1 (c), and the knee Horizontal angleis the knee-y axis angle shown in Fig. 3.1 (c). An empty plot in thegrid means that angle was not measured in the dataset provided by [Zellet al., 2020].31Figure 4.7: Qualitative comparison of reconstructed motion between neuralnetworks, the constant inertia model and the ARB inertia model. Theground truth motion is measured on Subject No. 17 while running, dur-ing a stance phase with right foot staying in contact with the ground.We compare at three time instances: 10 ms, 50 ms, 90 ms.32Figure 4.8: Eigenvalues of inertia matrices, either constructed or predicted onSubject No. 17 from a running clip with right foot contacting ground.We can tell SPDNet guarantees eigenvalues of predicted matrix beingpositive. Note that we set the minimum acceptable eigenvalue to be1−10 but that value is too small to be printed out in the figure.33Figure 4.9: An inertia matrix with negative eigenvalues can make the sim-ulated human produce physically implausible behaviors. Here to thesimulated human we apply an eigenforce, i.e. an impulse force in thesame direction as an eigenvector corresponding to a negative eigenvalueof the system’s predicted inertia matrix. (a) The generalized unitaryeigenforce applied to the system. (a) The system’s responding acceler-ation q¨. We see the applied force and the induced acceleration are inopposite directions, and that is not physically plausible.34Chapter 5Conclusions and Future WorkWe developed two models, the Constant Inertia Model and SPDNet, for repre-senting subject-specific effective joint-space inertia matrix of the lower body of ahuman in everyday walking and running. The estimated inertia matrices do notmake use of the rigidity assumptions aimed for approximating physical and geo-metric properties of human body segments, incorporate effects of inertial couplingand are motion type dependent. Also, unlike prior models that were analyticallyconstructed based on the ARB assumptions, both models are data-driven and needto be regressed from kinematic mocap data, dynamic force plate data as well asthe body heights and masses of human subjects. While the Constant Inertia Modelis non-parametric and thus should be estimated per-subject, per-pose, SPDNet isparametrized by pose, body height and mass, and it can generalize well to unseenposes and body proportions. Experimental results have shown that inertia matri-ces predicted by SPDNet are symmetric and positive definite, and that makes suchmatrices physically plausible.Compared with some prior work in biomechanics which sought only to esti-mate individual BSIPs, our work seeks to estimate joint-space inertia matrices thatcan be injected into a simulation framework for reconstructing lower-body motionof a human character that were not seen by our inertia model in training. We in-jected matrices predicted by SPDNet into a differentiable simulator to reconstructmotion in the first 140 ms of the stance phase in a gait cycle, either when a subjectis walking or running. We then showed that SPDNet’s matrices yield much lower35motion errors (MAE and MPJPE) than ARB inertia matrices. This result confirmsthat SPDNet is able to capture non-rigid properties of body segments as well as in-ertial effects of a human in motion much better than the ARB inertia model, whenmusculocontrol is constant.Despite what our work has achieved, it sill has many limitations. First of all,we have yet to explain the difference between estimated and running inertia. Sec-ond, character control which is important for analyzing and constructing motion inbiomechanics contexts [Clancy et al., 2023, Lam and Vujaklija, 2021, Ren et al.,2007, Xiang et al., 2011, Zell et al., 2020] or driving complex human motions forgraphical applications [Bergamin et al., 2019, Park et al., 2022, Shimada et al.,2021] is completely absent from our discussion. This limits our estimation andmotion reconstruction to the initial single support phase of a gait cycle, since aftermore time elapses active musculoskeletal control and the joint torque it induceswould be more significant, hence can no longer be ignored. We can implement asimple PD controller as done in prior works on graphics [Shimada et al., 2021],or resort to more complex activation-based controllers used to drive musculoskele-tal models [Park et al., 2022]. Another aspect we can improve is extending ourmethod to work on purely kinematic datasets which are generally larger and moreabundant. This may involve training our neural networks on motion loss rather thanper-step impulse loss which requires force plate data as an input. Finally, we be-lieve interpreting our estimated inertia in terms of segment-wise mass distributionand geometry is possible, and more work can be done to show that we can recon-struct such information from inertia matrices and hence, generate more visuallyand anatomically realistic human characters.36BibliographyA. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter.Differentiable convex optimization layers. Advances in neural informationprocessing systems, 32, 2019. → page 14S. Andrews, I. Huerta, T. Komura, L. Sigal, and K. Mitchell. Real-TimePhysics-Based Motion Capture with Sparse Sensors. In Proceedings of the 13thEuropean Conference on Visual Media Production (CVMP 2016), CVMP 2016,New York, NY, USA, 2016. Association for Computing Machinery. ISBN9781450347440. doi:10.1145/2998559.2998564. URLhttps://doi.org/10.1145/2998559.2998564. → pages 1, 7K. Ayusawa, G. Venture, and Y. Nakamura. Real-time implementation ofphysically consistent identification of human body segments. In 2011 IEEEInternational Conference on Robotics and Automation, pages 6282–6287.IEEE, 2011. → pages 5, 9K. Bergamin, S. Clavet, D. Holden, and J. R. Forbes. Drecon: data-drivenresponsive control of physics-based characters. ACM Transactions OnGraphics (TOG), 38(6):1–11, 2019. → pages 7, 36A. Bonnefoy-Mazure and S. Armand. Normal gait. Orthopedic management ofchildren with cerebral palsy, 40(3):567, 2015. → page 13V. Bonnet, P. Fraisse, A. Crosnier, M. Gautier, A. Gonza´lez, and G. Venture.Optimal exciting dance for identifying inertial parameters of ananthropomorphic structure. IEEE Transactions on Robotics, 32(4):823–836,2016. → page 5M. A. Brubaker, L. Sigal, and D. J. Fleet. Estimating contact dynamics. In 2009IEEE 12th International Conference on Computer Vision, pages 2389–2396.IEEE, 2009. → page 637M. A. Brubaker, D. J. Fleet, and A. Hertzmann. Physics-based person trackingusing the anthropomorphic walker. International journal of computer vision, 87(1):140–155, 2010. → page 7S.-C. Chen, H.-J. Hsieh, T.-W. Lu, and C.-H. Tseng. A method for estimatingsubject-specific body segment inertial parameters in human movement analysis.Gait & Posture, 33(4):695–700, 2011. ISSN 0966-6362.doi:https://doi.org/10.1016/j.gaitpost.2011.03.004. URLhttps://www.sciencedirect.com/science/article/pii/S0966636211000816. →pages 1, 5, 9C.-K. Cheng, H.-H. Chen, C.-S. Chen, C.-L. Lee, and C.-Y. Chen. Segmentinertial properties of chinese adults determined from magnetic resonanceimaging. Clinical biomechanics, 15(8):559–566, 2000. → page 5C. E. Clancy, A. A. Gatti, C. F. Ong, M. R. Maly, and S. L. Delp. Muscle-drivensimulations and experimental data of cycling. Scientific Reports, 13(1):21534,2023. → pages 1, 6, 36M. Damavandi, N. Farahpour, and P. Allard. Determination of body segmentmasses and centers of mass using a force plate method in individuals ofdifferent morphology. Medical engineering & physics, 31(9):1187–1194, 2009.→ pages 1, 5, 9M. Damsgaard, J. Rasmussen, S. T. Christensen, E. Surma, and M. De Zee.Analysis of musculoskeletal systems in the anybody modeling system.Simulation Modelling Practice and Theory, 14(8):1100–1111, 2006. → page 7T. T. Dao. From deep learning to transfer learning for the prediction of skeletalmuscle forces. Medical & biological engineering & computing, 57:1049–1058,2019. → page 1S. Diamond and S. Boyd. Cvxpy: A python-embedded modeling language forconvex optimization. Journal of Machine Learning Research, 17(83):1–5,2016. → pages 14, 24R. Dumas, L. Che`ze, and J.-P. Verriest. Adjustments to McConville et al. andYoung et al. body segment inertial parameters. Journal of Biomechanics, 40(3):543–553, 2007. ISSN 0021-9290.doi:https://doi.org/10.1016/j.jbiomech.2006.02.013. URLhttps://www.sciencedirect.com/science/article/pii/S0021929006000728. →pages 1, 538E. Ga¨rtner, M. Andriluka, E. Coumans, and C. Sminchisescu. Differentiabledynamics for articulated 3d human motion reconstruction. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages13190–13200, 2022. → pages 1, 7I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.http://www.deeplearningbook.org. → pages 2, 15C. Hansen, G. Venture, N. Rezzoug, P. Gorce, and B. Isableu. An individual anddynamic Body Segment Inertial Parameter validation method using groundreaction forces. Journal of Biomechanics, 47(7):1577–1581, 2014. ISSN0021-9290. doi:https://doi.org/10.1016/j.jbiomech.2014.03.004. URLhttps://www.sciencedirect.com/science/article/pii/S0021929014001584. →pages 1, 5, 9C. F. Jekel, K. E. Swartz, D. A. White, D. A. Tortorelli, and S. E. Watts. Neuralnetwork layers for prediction of positive definite elastic stiffness tensors. arXivpreprint arXiv:2203.13938, 2022. → pages ix, 15, 16, 17, 24Y. Jiang, T. Van Wouwe, F. De Groote, and C. K. Liu. Synthesis of BiologicallyRealistic Human Motion Using Joint Torque Actuation. ACM Trans. Graph.,38(4), jul 2019. ISSN 0730-0301. doi:10.1145/3306346.3322966. URLhttps://doi.org/10.1145/3306346.3322966. → page 1J. Jovic, A. Escande, K. Ayusawa, E. Yoshida, A. Kheddar, and G. Venture.Humanoid and Human Inertia Parameter Identification Using HierarchicalOptimization. IEEE Transactions on Robotics, 32(3):726–735, 2016.doi:10.1109/TRO.2016.2558190. → pages 1, 5, 9D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXivpreprint arXiv:1412.6980, 2014. → page 16H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: a large videodatabase for human motion recognition. In 2011 International conference oncomputer vision, pages 2556–2563. IEEE, 2011. → page 9S. K. Lam and I. Vujaklija. Joint torque prediction via hybridneuromusculoskeletal modelling during gait using statistical ground reactionestimates: an exploratory study. Sensors, 21(19):6597, 2021. → pages 1, 3, 36S. Lee, M. Park, K. Lee, and J. Lee. Scalable muscle-actuated human simulationand control. ACM Transactions On Graphics (TOG), 38(4):1–13, 2019a. →pages 1, 7, 839T. Lee, P. M. Wensing, and F. C. Park. Geometric robot dynamic identification: Aconvex programming approach. IEEE Transactions on Robotics, 36(2):348–365, 2019b. → page 5L. Liu, J. L. Cooper, and D. H. Ballard. Computational modeling: Humandynamic model. Frontiers in Neurorobotics, 15:723428, 2021. → page 7X. Lv, J. Chai, and S. Xia. Data-driven inverse dynamics for human motion. ACMTransactions on Graphics (TOG), 35(6):1–12, 2016. → pages 6, 7K. Lyu, H. Chen, Z. Liu, B. Zhang, and R. Wang. 3D human motion prediction: Asurvey. Neurocomputing, 489:345–365, 2022. ISSN 0925-2312.doi:https://doi.org/10.1016/j.neucom.2022.02.045. URLhttps://www.sciencedirect.com/science/article/pii/S0925231222002077. →page 19N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass:Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVFinternational conference on computer vision, pages 5442–5451, 2019. → pages3, 9P. E. Martin, M. Mungiole, M. W. Marzke, and J. M. Longhill. The use ofmagnetic resonance imaging for measuring segment inertial properties. Journalof biomechanics, 22(4):367–376, 1989. → page 5J. T. McConville, T. D. Churchill, I. Kaleps, C. E. Clauser, and J. Cuzzi.Anthropometric relationships of body and body segment moments of inertia.AMRL-TR, pages 80–119, 1980. → pages 1, 4, 5L. Mourot, L. Hoyet, F. L. Clerc, and P. Hellier. Underpressure: Deep learning forfoot contact detection, ground reaction force estimation and footskate cleanup.In Computer Graphics Forum, volume 41, pages 195–206. Wiley OnlineLibrary, 2022. → page 9M. Mungiole and P. E. Martin. Estimating segment inertial properties:comparison of magnetic resonance imaging with existing methods. Journal ofbiomechanics, 23(10):1039–1046, 1990. → page 5V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmannmachines. In Proceedings of the 27th international conference on machinelearning (ICML-10), pages 807–814, 2010. → page 1640M. Nakada, T. Zhou, H. Chen, T. Weiss, and D. Terzopoulos. Deep Learning ofBiomimetic Sensorimotor Control for Biomechanical Human Animation. ACMTrans. Graph., 37(4), jul 2018. ISSN 0730-0301.doi:10.1145/3197517.3201305. URLhttps://doi.org/10.1145/3197517.3201305. → page 1M. E. Ozates, D. Karabulut, F. Salami, S. I. Wolf, and Y. Z. Arslan. Machinelearning-based prediction of joint moments based on kinematics in patientswith cerebral palsy. Journal of Biomechanics, 155:111668, 2023. → page 16D. K. Pai. Muscle mass in musculoskeletal models. Journal of biomechanics, 43(11):2093–2098, 2010. → pages 2, 8, 12, 21J. Park, S. Min, P. S. Chang, J. Lee, M. S. Park, and J. Lee. Generative gaitnet. InACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022. → pages1, 6, 7, 8, 36A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen,Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style,high-performance deep learning library. Advances in neural informationprocessing systems, 32, 2019. → page 17T. C. Pataky, V. M. Zatsiorsky, and J. H. Challis. A simple method to determinebody segment masses in vivo: reliability, accuracy and sensitivity analysis.Clinical Biomechanics, 18(4):364–368, 2003. → pages 1, 5, 9O. Pearl, S. Shin, A. Godura, S. Bergbreiter, and E. Halilaj. Fusion of video andinertial sensing data via dynamic optimization of a biomechanical model.Journal of Biomechanics, 155:111617, 2023. → page 2A. Rajagopal, C. L. Dembia, M. S. DeMers, D. D. Delp, J. L. Hicks, and S. L.Delp. Full-body musculoskeletal model for muscle-driven simulation of humangait. IEEE transactions on biomedical engineering, 63(10):2068–2079, 2016.→ pages 2, 7L. Ren, R. K. Jones, and D. Howard. Predictive modelling of human walking overa complete gait cycle. Journal of Biomechanics, 40(7):1567–1574, 2007. ISSN0021-9290. doi:https://doi.org/10.1016/j.jbiomech.2006.07.017. URLhttps://www.sciencedirect.com/science/article/pii/S0021929006002752. →pages 1, 6, 36A. Scholz, M. Sherman, I. Stavness, S. Delp, and A. Kecskeme´thy. A fastmulti-obstacle muscle wrapping method using natural geodesic variations.Multibody System Dynamics, 36:195–219, 2016. → pages 2, 7, 841A. Seth, J. L. Hicks, T. K. Uchida, A. Habib, C. L. Dembia, J. J. Dunne, C. F.Ong, M. S. DeMers, A. Rajagopal, M. Millard, and Others. OpenSim:Simulating musculoskeletal dynamics and neuromuscular control to studyhuman and animal movement. PLoS computational biology, 14(7):e1006223,2018. → pages 2, 7, 8R. Sharif Razavian, B. Ghannadi, and J. McPhee. A synergy-based motor controlframework for the fast feedback control of musculoskeletal systems. Journal ofbiomechanical engineering, 141(3):31009, 2019. → page 1S. Shimada, V. Golyanik, W. Xu, and C. Theobalt. Physcap: Physically plausiblemonocular 3d motion capture in real time. ACM Transactions on Graphics(ToG), 39(6):1–16, 2020. → page 1S. Shimada, V. Golyanik, W. Xu, P. Pe´rez, and C. Theobalt. Neural monocular 3dhuman motion capture with physical awareness. ACM Transactions onGraphics (ToG), 40(4):1–15, 2021. → pages 6, 7, 36M. S. Shourijeh, N. Mehrabi, and J. McPhee. Forward Static Optimization inDynamic Simulation of Human Musculoskeletal Systems: A Proof-of-ConceptStudy. Journal of Computational and Nonlinear Dynamics, 12(5):51005, 2017.ISSN 1555-1415. doi:10.1115/1.4036195. URLhttps://doi.org/10.1115/1.4036195. → pages 1, 7S. Song, Ł. Kidzin´ski, X. B. Peng, C. Ong, J. Hicks, S. Levine, C. G. Atkeson,and S. L. Delp. Deep reinforcement learning for modeling human locomotioncontrol in neuromechanical simulation. Journal of neuroengineering andrehabilitation, 18:1–17, 2021. → page 1M. Sreenivasa, C. J. G. Chamorro, D. Gonzalez-Alvarado, O. Rettig, and S. I.Wolf. Patient-specific bone geometry and segment inertia from MRI images formodel-based analysis of pathological gait. Journal of biomechanics, 49(9):1918–1925, 2016. → page 5D. G. Thelen and F. C. Anderson. Using computed muscle control to generateforward dynamic simulations of human walking from experimental data.Journal of biomechanics, 39(6):1107–1115, 2006. → page 8E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-basedcontrol. In 2012 IEEE/RSJ international conference on intelligent robots andsystems, pages 5026–5033. IEEE, 2012. → pages 7, 4542G. Venture, K. Ayusawa, and Y. Nakamura. Motion capture based identification ofthe human body inertial parameters. In 2008 30th Annual InternationalConference of the IEEE Engineering in Medicine and Biology Society, pages4575–4578, 2008. doi:10.1109/IEMBS.2008.4650231. → pages 1, 5, 6, 7J. Verheul, S. Sueda, and S. H. Yeo. Muscle inertia during running: A massivechange of moments? In 40th International Society of Biomechanics in SportsConference, 2022. → pages 6, 12J. Verheul, S. Sueda, and S.-H. Yeo. Muscle inertial contributions to ankle kineticsduring the swing phase of running. Journal of Biomechanics, 147:111455,2023. ISSN 0021-9290. doi:https://doi.org/10.1016/j.jbiomech.2023.111455.URL https://www.sciencedirect.com/science/article/pii/S0021929023000246.→ page 2H. Wang, V. Caggiano, G. Durandau, M. Sartori, and V. Kumar. Myosim: Fastand physiologically realistic mujoco models for musculoskeletal andexoskeletal studies. In 2022 International Conference on Robotics andAutomation (ICRA), pages 8104–8111. IEEE, 2022a. → page 7Y. Wang, J. Verheul, S.-H. Yeo, N. K. Kalantari, and S. Sueda. DifferentiableSimulation of Inertial Musculotendons. ACM Transactions on Graphics (TOG),41(6):1–11, 2022b. → pages 2, 8, 12P. M. Wensing, S. Kim, and J.-J. E. Slotine. Linear matrix inequalities forphysically consistent inertial parameter identification: A statistical perspectiveon the mass distribution. IEEE Robotics and Automation Letters, 3(1):60–67,2017. → page 5D. A. Winter. Biomechanics and motor control of human movement. John Wiley& Sons, 2009. → page 18Y. Xiang, J. S. Arora, and K. Abdel-Malek. Optimization-based prediction ofasymmetric human gait. Journal of Biomechanics, 44(4):683–693, 2011. ISSN0021-9290. doi:https://doi.org/10.1016/j.jbiomech.2010.10.045. URLhttps://www.sciencedirect.com/science/article/pii/S0021929010006135. →pages 1, 36B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activationsin convolutional network. arXiv preprint arXiv:1505.00853, 2015. → page 16G. Yang, S. Yang, J. Z. Zhang, Z. Manchester, and D. Ramanan. Ppr: Physicallyplausible reconstruction from monocular videos. In Proceedings of the43IEEE/CVF International Conference on Computer Vision, pages 3914–3924,2023. → pages 6, 7, 9X. Yi, Y. Zhou, M. Habermann, S. Shimada, V. Golyanik, C. Theobalt, and F. Xu.Physical inertial poser (pip): Physics-aware real-time human motion trackingfrom sparse inertial sensors. In Proceedings of the IEEE/CVF conference oncomputer vision and pattern recognition, pages 13167–13178, 2022. → pages6, 7J. W. Young, R. F. Chandler, C. C. Snow, K. M. Robinette, G. F. Zehner, M. S.Loftberg, et al. Anthropometric and mass distribution characteristics of theadult female. Technical report, Civil Aerospace Medical Institute, 1983. →page 5V. Zatsiorsky. Estimation of the mass and inertia characteristics of the humanbody by means of the best predictive regression equations. Biomechanics;proceedings, pages 233–239, 1985. → page 5P. Zell and B. Rosenhahn. A physics-based statistical model for human gaitanalysis. In Pattern Recognition: 37th German Conference, GCPR 2015,Aachen, Germany, October 7-10, 2015, Proceedings 37, pages 169–180.Springer, 2015. → page 6P. Zell and B. Rosenhahn. Learning inverse dynamics for human locomotionanalysis. Neural Computing and Applications, 32(15):11729–11743, 2020. →page 7P. Zell, B. Rosenhahn, and B. Wandt. Weakly-supervised learning of humandynamics. In Computer Vision–ECCV 2020: 16th European Conference,Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16, pages 68–84.Springer, 2020. → pages xi, 1, 2, 7, 9, 11, 12, 13, 17, 31, 36, 45X. Zhang, F. K. Chan, T. Parthasarathy, and M. Gazzola. Modeling and simulationof complex dynamic musculoskeletal architectures. Nature communications, 10(1):4825, 2019. → pages 7, 8Y. Zhang, J. O. Kephart, and Q. Ji. Incorporating physics principles for precisehuman motion prediction. In Proceedings of the IEEE/CVF Winter Conferenceon Applications of Computer Vision, pages 6164–6174, 2024. → pages 6, 9, 1144Appendix APreliminary Experiments on theSynthetic DatasetBefore training and testing our methods on the real world dataset by Zell et al.[2020], we created simple two DoF leg models in MuJoCo [Todorov et al., 2012],generated synthetic datasets of leg-ground contact clips from them, and performedinertial estimation on them using our methods. This allows us to know if the con-stant inertia model or neural inertia model can attain sensible estimations at leaston systems that are simple and easy to debug, before trying them on a 24-DoFsystem that is more complex.A.1 Synthetic Dataset ConstructionWe created two simple rigid double pendulum models to emulate the motion ofa human leg. Each model consists of two capsules and two revolute joints; thetop capsule emulates a thigh with mass set to 0.7 kg; the bottom capsule emu-lates a shank with mass set to 0.3 kg. The joints respectively emulate a hip and aknee. Both models have two degrees of freedom. The tendon-actuated model is aderivative of MuJoCo’s built-in model with six muscle-tendon pairs; we mappedeach pair to a human leg muscle, see Fig. A.1 (b). To emulate GRF/M generatedwhen a human leg touches ground, we make each pendulum contact the groundat a non-zero initial velocity starting from time zero, then sample state trajectory45and contact forces following the impact. Since mass matrix depends on pose, wemake each pendulum contact the floor at different poses by (i) either directly set-ting the initial pose for the unactuated model, (ii) or defining different activationlevels for muscle-tendons and waiting until the tendon-driven model settles to equi-librium poses. To make impact responses from the same initial pose not identicalto each other, we vary the impact velocity and tilt the pendulum’s plane of motionso impacts take place at different tilt angles, see Fig. A.1 (a). Thus, using eachmodel we can create two datasets: one contains simulation clips sampled at dif-ferent poses (the Pose Set), another contains clips sampled at the same pose, butwith segment masses being different in each simulation (the Mass Set). For theunactuated model’s Pose Set, we sampled 11,664 clips in total at 81 poses, so eachpose subset contains 114 clips. For the tendon-actuated model’s Pose Set, we sam-pled 3645 clips in total also at 81 different equilibrium poses, corresponding to 81muscle activation combinations. And for the tendon-actuated model’s Mass Set wesampled data from 81 different thigh-shank mass combinations.A.2 Synthetic Data FittingThe constant inertia model we used for the preliminary experiments is smaller thanthe one fitted on the real-world dataset: it contains only 4 parameters since thejoint space inertia matrix of the 2-DoF tendon-driven leg model has shape 4× 4.We also fit it with constrained least squares. The neural inertia model we used isless developed than SPDNet: it directly predicts inertia from pose q and total bodymass m, without using a person feature extractor to build latent vector p first. Thenetwork contains fewer layers and fewer neurons per layer, since the matrix to bepredicted is smaller in dimension The SPD Layer uses Cholesky Layer. We use ther2 metric to evaluate the quality of impulse fit.We see from Fig. A.2 (a), (b) that constant inertia models which are fitted per-pose, per-mass can attain r2 scores close to 1 on most test subsets. From Fig. A.2(c), (d) we see that the neural network can obtain high r2 scores on most poses ormasses, despite low performance on several subsets.46Figure A.1: Synthetic dataset construction and partitioning. (a) We createtwo leg models to generate synthetic data from: an unactuated doublependulum and a tendon-driven double pendulum. With the unactuatedpendulum, we generate data at different initial poses to obtain differenteffective inertia matrices. For the tendon-actuated model, we applydifferent muscle activations so the initial pose would be different. Wealso set the pendulum’s plane of motion at different tilt angles w.r.t.ground in order to diversify our data. (b) Musculotendious actuatorsthat map to real-world leg muscles.47Figure A.2: R2 scores of effective inertia fits, on synthetic data from thetendon-actuated double pendulum. (a) - (b): Fitting constant inertiamodels on the Pose, Mass Set. (c) - (d): Fitting Neural Network on thePose, Mass Set.48
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Pai, D. K. (Dinesh K.)
Data-driven models of human body inertia
Text