Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Rothwell, Austin
Master of Science - MSc
University of British Columbia
Personalized simulation of human bodies is a long standing goal in many applications, ranging from animation to apparel. Even though personalized geometric models can now be easily acquired, physics-based simulation requires soft tissue properties and their distribution over the person’s body. Here we show that mechanical properties of the human body can be directly measured using a novel hand-held device. We describe a complete pipeline for measurement, modeling, parameter estimation, and simulation. The methods described here can be used to create personalized models of an individual human or of a population. Furthermore, we show how to predict soft tissue properties from widely available 3D geometric models of the human body. To train such a prediction model, we utilize a unique database of registered measurements of body shape and soft tissue properties, acquired from over 70 participants. We use a recently introduced convolutional neural network architecture adapted for 3D surfaces, and train the network to predict the distribution of tissue properties over the 3D human body surface. Once the network is trained, no specialized equipment is required, and soft tissue properties are predicted in minutes. The method can be used with commodity 3D scanners, and even with geometric models downloaded from Internet or created by artists. Our methods make realistic human body simulations available to a wide range of users and applications.
MEASURING, MODELLING, SIMULATING, ANDPREDICTING HUMAN TISSUE PROPERTIESbyAustin RothwellBA HONS Psychology, Minor in Computer Science, University of BritishColumbia, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)December 2019c© Austin Rothwell, 2019The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:MEASURING, MODELLING, SIMULATING, AND PREDICTINGHUMAN TISSUE PROPERTIESsubmitted by Austin Rothwell in partial fulfillment of the requirements for thedegree of MASTER OF SCIENCE in Computer Science.Examining Committee:Dinesh Pai, Computer ScienceSupervisorLeon Sigal, Computer ScienceExamining Committee MemberiiAbstractPersonalized simulation of human bodies is a long standing goal in many appli-cations, ranging from animation to apparel. Even though personalized geometricmodels can now be easily acquired, physics-based simulation requires soft tissueproperties and their distribution over the person’s body. Here we show that me-chanical properties of the human body can be directly measured using a novelhand-held device. We describe a complete pipeline for measurement, modeling,parameter estimation, and simulation. The methods described here can be usedto create personalized models of an individual human or of a population. Fur-thermore, we show how to predict soft tissue properties from widely available 3Dgeometric models of the human body. To train such a prediction model, we utilizea unique database of registered measurements of body shape and soft tissue prop-erties, acquired from over 70 participants. We use a recently introduced convolu-tional neural network architecture adapted for 3D surfaces, and train the networkto predict the distribution of tissue properties over the 3D human body surface.Once the network is trained, no specialized equipment is required, and soft tissueproperties are predicted in minutes. The method can be used with commodity 3Dscanners, and even with geometric models downloaded from Internet or created byartists. Our methods make realistic human body simulations available to a widerange of users and applications.iiiLay SummaryTissue properties, like thickness and stiffness, are vital for creating virtual humantissue simulations. This research demonstrates how to acquire tissue propertiesacross the human body and how to use these properties for realistic human tissuesimulations. In addition, we show how to predict tissue properties across the body,given the shape of the body, allowing personalized tissue properties to be acquiredquickly and easily. This technology assists virtual human body simulation, whichhas applications in a variety of fields, such as animation, apparel, and medicine.ivPrefaceAll work presented was conducted in the Sensorimotor Systems Laboratory atthe University of British Columbia. Methods were approved by the Universityof British Columbia Research Ethics Board [H18-01269].Portions of the Chapter 3 text and Figures 3.1 to 3.5 and 6.2 are used with per-mission from Pai et al., [43], which I am an author of. These sections were partof a collaborative project between myself and multiple other graduate students inthe lab. Other areas of the dissertation include work that has been submitted forpublication to a journal, authored by Austin Rothwell, Ye Fan, and Dinesh Pai. Iwas the lead investigator, responsible for concept formation, identification and de-sign of the research program, testing and improving performance, data collectionand aggregation, software development, data analysis, and manuscript creation. YeFan and Dinesh Pai contributed to concept formation, software development andmanuscript creation.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Tissue Properties and Simulation . . . . . . . . . . . . . . . . . . 11.2 Human Body Models . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Research Direction . . . . . . . . . . . . . . . . . . . . . . . . . 42 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Body Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Physically Based Human Simulation . . . . . . . . . . . . . . . . 72.3 Soft Tissue Modeling . . . . . . . . . . . . . . . . . . . . . . . . 82.4 Estimating Tissue Properties . . . . . . . . . . . . . . . . . . . . 11vi2.5 Geometric Learning . . . . . . . . . . . . . . . . . . . . . . . . . 123 Human Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . 143.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Acquiring Tissue Properties . . . . . . . . . . . . . . . . . . . . . 153.2.1 Skin Probe . . . . . . . . . . . . . . . . . . . . . . . . . 153.2.2 Measuring Human Soft Tissues . . . . . . . . . . . . . . 193.2.3 Motion Capture . . . . . . . . . . . . . . . . . . . . . . . 203.2.4 Marker and Probing Locations . . . . . . . . . . . . . . . 203.2.5 Probe Calibration . . . . . . . . . . . . . . . . . . . . . . 223.2.6 Tissue Property Estimation . . . . . . . . . . . . . . . . . 233.2.7 Parameter Estimation Validation . . . . . . . . . . . . . . 283.3 Body Shape Registration . . . . . . . . . . . . . . . . . . . . . . 283.3.1 Template Definition . . . . . . . . . . . . . . . . . . . . 293.3.2 Skeleton Fitting . . . . . . . . . . . . . . . . . . . . . . . 303.3.3 Geometry Fitting . . . . . . . . . . . . . . . . . . . . . . 323.3.4 Probing Position Identification . . . . . . . . . . . . . . . 333.4 Estimated Parameter Interpolation . . . . . . . . . . . . . . . . . 343.4.1 Prior Creation . . . . . . . . . . . . . . . . . . . . . . . . 363.4.2 Using the Prior . . . . . . . . . . . . . . . . . . . . . . . 373.4.3 Optimizing Kernel Parameters . . . . . . . . . . . . . . . 373.4.4 Breast Thickness Correction . . . . . . . . . . . . . . . . 384 Learning Tissue Properties . . . . . . . . . . . . . . . . . . . . . . . 404.1 CNN Implementation . . . . . . . . . . . . . . . . . . . . . . . . 404.1.1 Data Preparation . . . . . . . . . . . . . . . . . . . . . . 404.1.2 Network Architecture . . . . . . . . . . . . . . . . . . . . 434.1.3 Data Extraction . . . . . . . . . . . . . . . . . . . . . . . 455 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475.1 Training Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 475.2 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . 526 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54viiBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58viiiList of TablesTable 4.1 Description of features used in the convolutional neural network(CNN) input. Each feature can be thought of as a color channelin an image. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43ixList of FiguresFigure 2.1 Complexity of human skin. Not pictured are underlying mus-cles, bones, and tendons, which make the deformation of hu-man skin even more complex. Image from OpenStax College,distributed under a CC-BY 3.0 licence. . . . . . . . . . . . . 10Figure 3.1 The Skin Probe. 1: 3D tracking marker arrangement on the“tentacle”; 2: Single-piece probe “head”, holding the camerasand force sensor; 3: Carbon fiber composite probe stem; 4:Quick-change, magnetic-lock probe tip; 5: Front-facing cam-eras; 6: Front trigger button; 7: Rear input buttons; 8: Er-gonomic handle, housing Arduino micro-controller; 9: Ex-ternally accessible Arduino ISP header (reprogramming port);10: Exit point for probe cables, with internal strain relief clamp. 16Figure 3.2 Cutaway view of the the probe’s sensor “head” section. 1: Em-bedded force/torque sensor; 2: stereolithographic resin print-ing (SLA)-printed probe stem mount, which secures the stem tothe force sensor; 3: Integrated clamps hold the probe camera atan adjustable distance from the tip; 4: The probe tip, showingthe internal channel housing the locking magnets, and pinholeused for securing tips expecting negative loadings; 5: Endo-scope camera; 6: Camera and force sensor cable routing toprobe base . . . . . . . . . . . . . . . . . . . . . . . . . . . 17xFigure 3.3 The probe capture software. 1: Live feed of the forces andtorques applied to the probe tip, showing a recent touch; 2: Vi-sualizer window, displaying the real-time positions of the par-ticipant markers and the skin probe tip through a user-adjustablevirtual camera; 3: Live split view from the three probe cam-eras, showing the probe shaft and tip in-frame, along with theparticipant and several markers. . . . . . . . . . . . . . . . . 18Figure 3.4 Capture volume surrounded by eight Vicon MX cameras. Notethe higher density of Vicon cameras at the far side of the roomto reduce occlusion caused by the experimenter when probing.The green area shows the zone where the experimenter recordsdata from, the blue area shows where the participant is situated. 21Figure 3.5 Raw data captured after probe calibration. Recorded forcesshowing four probe contacts (left) and recorded tip positionwhile probing the sternum and the pectoralis major muscle(right). Note the hysteresis as the probe returns to the initialcontact location. . . . . . . . . . . . . . . . . . . . . . . . . 22Figure 3.6 µ¯ and L0 fit to a force displacement curve from contact on thesternum (a) and belly (b). Blue circles correspond to linear fitafter µ¯ estimation, red circles correspond to fit after L0 estima-tion. The blue line shows the loading phase after smoothingwith a Butterworth filter. A threshold of 2 N is used for fitting. 25Figure 3.6 Examples of measured tissue thickness over 30 female partic-ipants. Thickness values are shown on a scale from 0 cm to 5cm using the jet colorspace. Breast thickness values are addedto measured thickness values post-hoc, based on the geometryof the participant (see Section 3.4.4 for further details). . . . . 27Figure 3.7 Force displacement curves across the torso of a single partici-pant from two measurement sessions. The similarity betweenthe measurement sessions demonstrates that our parameter es-timation method produces consistent results. . . . . . . . . . 28xiFigure 3.8 Front and back of the female template tetrahedral mesh (a,b).Red circles show locator positions on the template. The tem-plate skeleton in the initial skeleton configuration (c). . . . . . 29Figure 3.9 The initial template mesh torso (a). The template mesh af-ter the skeleton fitting (b). The template mesh after geometryfitting (c). The corresponding body scan (d). This is a chal-lenging example, since the scanned body varies greatly fromthe template, the arms are close to the torso, and there are ar-eas of overlapping tissue, however our method still produces areasonable fit. . . . . . . . . . . . . . . . . . . . . . . . . . . 32Figure 3.10 Prior thickness distribution depicted on template, where bluecorresponds to thin areas and red corresponds to thick areas.Note how the thickness map takes underlying skeletal anatomyinto account. . . . . . . . . . . . . . . . . . . . . . . . . . . 36Figure 3.11 Information weights visualized on an example participant (red= 1, blue = 0). The torso values rely on measured data sincemeasurements were taken on the torso, while the limbs rely onprior data since there were no measurement on the limbs. . . . 38Figure 3.12 The subject-specific mesh (a), and the breasts removed versionof the subject-specific mesh (b). Tissue thickness of the breastis calculated by computing the difference between (a) and (b),producing a breast tissue thickness offset (c), which can beapplied to measured or predicted data. . . . . . . . . . . . . . 39xiiFigure 4.1 Transformation of a function from a 3D body surface to a 2Dflat torus. The top left image shows the function defined overan example 3D body surface (tissue thickness in this case) withthe three circles (yellow, blue, purple) defining the cut. Thetop right shows the same function on the same surface, butunravelled as a 2D planar mesh. The bottom left shows the2D positions of vertices, where vertex color corresponds to thethickness value. The bottom right shows the final function im-age, after scattered interpolation to acquire parameters acrossall pixels and tiling to produce a flat torus shape. This trans-formation follows the process of Maron et al. [34]. . . . . . . 41Figure 4.2 Our CNN architecture. Each grey block refers to a convolutionand ReLU layer and each red block is a pooling layer. . . . . . 44Figure 5.1 Tissue thickness prediction for 30 representative triplet sets forone participant. The variability among predictions is expectedas the transformation of thickness values from 3D to 2D resultsin resolution loss for some areas of the mesh. As such, the in-dividual predictions are averaged to acquire the final predictedvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Figure 5.2 Tissue thickness prediction for a participant who was not in-cluded in the training set. The measured tissue thickness forthis participant is compared to the predicted tissue thickness,with the error between measured and predicted values shownon the same scale. The last column demonstrates the breastthickness correction, which is applied after the predictions aremade, to more accurately capture the thickness of the breasts,where measurement is infeasible (see Section 3.4.4 for furtherdetails). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49Figure 5.3 Thickness predictions (a) and stiffness predictions (b) for twobody models that were not part of our experiment. . . . . . . 50xiiiFigure 5.4 Force displacement plots across 15 torso locations. Force-displacement curves using measured tissue properties (black)are compared to using predicted tissue properties (pink). Forcedisplacement curves are produced by our tissue model (seeSection 3.2.6). . . . . . . . . . . . . . . . . . . . . . . . . . 52Figure 5.5 Soft body garment fit simulation using the predicted tissuethickness and stiffness (a). Rigid body simulation using thebody geometry (b). Note the compression of the breasts andthe bulging of tissue around the perimeter of the bra in the softbody simulation. . . . . . . . . . . . . . . . . . . . . . . . . 53Figure 6.1 We describe how to quickly construct personalized soft-bodysimulation models of people, using only 3D geometric modelsof their body shape. We use a convolutional neural network,trained with new high quality data of soft tissue measurementsand registered 3D body scans. At run time, the input to oursystem is only a 3D model of a person’s shape (a), which maybe obtained from any source. Here we see two different bodyshapes downloaded from a web store. The system predicts allthe soft tissue properties required for a finite element simu-lation, such as thickness (b) and stiffness (c). Observe thatthe system predicts reasonable qualitative differences betweendifferent parts of the body, and between the two different bodyshapes. The results may be immediately used in many applica-tions such as virtual garment fitting (d). In these applications,it is essential to use a personalized soft body simulation, sincethe fit depends on how the body changes shape in responseto the garment. Using only the shape model (e) gives incor-rect estimates of how the garment stretches and how the bodylooks. By contrast, the personalized soft tissue simulations (d)enabled by our methods produce more realistic estimates of thedeformed body shape and garment stretch. . . . . . . . . . . . 55xivFigure 6.2 Raw force – displacement curves measured from a location ona biceps muscle. In one condition, the biceps was relaxed;in the other, it was isometrically flexed in the same pose, bysupporting a 6.8 kg weight in the hand. Note that the muscle issignificantly stiffer when active. . . . . . . . . . . . . . . . . 56xvGlossaryAPI application programming interfaceCNN convolutional neural networkDOF degrees of freedomFDM fused-deposition modelingFEM finite element methodFPS frames per secondGPU graphics processing unitHRBF Hermite radial basis functionLBS linear blend skinningMRI magnetic resonance imagingMSE mean squared errorSLA stereolithographic resin printingxviAcknowledgmentsI would like to thank my lab colleagues for the help and advice. In addition, I wouldlike to thank my supervisor for his guidance and feedback. Finally, I would like tothank my parents and girlfriend for supporting and encouraging me. I especiallywant to thank my mom for the extravagant packed lunches and dinners that fuelledmy long days and nights at the lab.xviiChapter 1IntroductionHumans are complex creatures. Whether we are discussing the intricate musclesystems involved in walking, the complicated patterns of neuronal firing that lead toconsciousness, or the elaborate social signals present in human communication, themystery of the human has yet to be fully uncovered. Indeed, many fields of researchare primarily focused on some facet of humans. Psychology investigates the humanmind, biology explores the inner working of the human body, anthropology looksinto the societies humans live in.Computer Science, on the other hand, doesn’t directly demonstrate the anthro-pocentric sentiment that many other fields exhibit. Computer Scientists often studyalgorithms or data structures, instead of muscles or brains. The advancements inComputer Science, however, can and have been used to enhance the study of hu-mans. Whether it is new computational hardware for genome sequencing, or com-plex computational models predicting the effect of humans on our world, com-puters have changed human research. As such, our goal is to use state-of-the-artComputer Science technology to better understand the human body, specifically, inour case, the material properties of human tissue.1.1 Tissue Properties and SimulationBy material properties, we are referring to some intensive mechanical propertyof the material (in this case human tissue), such as stiffness or elasticity, or even1something as simple as thickness. These varying tissue properties across the humanbody are interesting and useful from a basic science point of view, where they couldbe used to describe and categorize different types of tissue, or report how humantissue varies across ages, ethnicities, or body types. These properties, however, canalso be used to create powerful human tissue simulations, which have applicationsin a variety of fields, including computer graphics, industrial design, and medicine.A human tissue simulation is simply a mathematical model with parameters thatdescribe said tissue being exercised by a computer to reproduce behaviour of thetissue in some situation. With that rather cryptic explanation of a human tissuesimulation, let’s discuss a concrete example.In the current day, when an aerospace firm is tasked with designing a newair plane, the engineers do not simply start with a prototype plane that will in-evitably crash, then iterate on the design until the plane seems air-worthy in mostconditions. Rather, the engineers would build virtual planes that are tested strin-gently through simulation of various flight conditions, both common and rare. Thismethod reduces the wasted materials and lives from the intermediate prototypes,and allows for reasonable certainty that the first physical prototype plane will flysafely. It is the rigorous understanding of aerospace material properties that allowsfor these simulations.The aerospace method of design is in stark contrast to the current methodof clothing design. Similar to air planes, clothing design is an iterative process,however, the use of virtual prototypes is limited. Indeed, the process of cutting-stitching-fitting-discarding is cycled until an appropriate fit on a variety of bodiesis acquired. These failed prototypes do not have such grave consequences as in theaerospace industry, however, they do result in wasted materials and wasted time.A simulation paradigm, similar to that of the aerospace industry, would result insavings for the clothing manufacturer, however, it would also result in better fittingclothing, as the garment could be tested on a greater variety of bodies in a greatervariety of situations, benefiting both consumers and producers. Other applicationsof human tissue simulation include; e-commerce, medicine, and computer graph-ics. This method, dependent on virtual humans, is within reach, however, thedevelopment of an accurate and realistic human body model, based on capturedtissue properties, is the missing piece.21.2 Human Body ModelsThe complexity and variability of humans has limited the development of humanbody models. Most human body models in the field of computer graphics arepurely kinematic, meaning that deformation of the body is based on the position ofthe underlying skeleton. Simulations using kinematic human body models cannotaccount for deformation caused by physical interaction, such as gravity or the forceapplied from an external object. Recent advances in physics-based human bodysimulation have successfully dealt with the numerical methods for simulating softhuman tissue, however the constitutive properties needed for these simulations arechosen almost arbitrarily.These physics based simulation methods can produce human tissue deforma-tion that is suitable for movies and video games, where visually plausible resultsare acceptable and utmost realism is not required. Areas such as medicine and in-dustrial design, however, require accurate tissue properties that lead to realistic softtissue behaviour in simulation. For example, surgical and clothing fit simulationswould be ineffective if the simulated tissue separated in an infeasible way whencut with a scalpel or bulged in an unrealistic way upon compression from a tightfitting garment. As such, trustworthy models of the human body are crucial formany areas outside of computer graphics.Despite the importance of physically accurate human body models, only ahandful of researchers have addressed this area. Specifically, development of thesemodels requires the measurement of human tissue deformation, the modelling ofsuch tissues, and the estimation of model properties, which is challenging for avariety of reasons. Human tissue is non-homogeneous, multi-layered, and variesgreatly across the body. It responds to compression in a highly non-linear fashionand slides elastically over underlying surfaces such as muscles, bones, and ten-dons. Furthermore, it varies significantly across multiple people and even changesbased on voluntary control (e.g., by flexing a muscle). As such, the measurementand modelling of such tissue is difficult.There has been significant progress in the last decade in measuring humanbody appearance and shape, thanks to advances in computer vision and other op-tical scanning technologies. However, the spatial distribution of the mechanical3properties needed for contact simulation cannot be easily measured by existingtechniques. The mechanical behaviour of soft tissues is highly non-linear; it isdifficult to extrapolate properties derived from, say, passive videos or ultrasoundelastography, to those that can represent soft tissue compression due to a garmentor wearable device. Despite these difficulties, our goal is to create accurate andrealistic human body models that allow for trustworthy simulations.1.3 Research DirectionThis dissertation is split into two main sections.1. In Chapter 3 we show how material properties of the human body can bedirectly measured using a novel hand-held device. Parameters required for afinite element method (FEM) simulation can be estimated from this data, andused for simulating personalized models of an individual. This work comple-ments the recent advances in numerical methods for soft tissue simulation byproviding data-driven parameter values for more realistic simulations. In ad-dition to enabling new applications for human simulation, analysis of thesemeasurements provides new insights into how the human body actually be-haves.2. In Chapter 4 we demonstrate that tissue properties across the human bodycan be learned from body shape. Specifically, given the 3D geometry of anindividual body, our method predicts the spatially varying tissue propertiesacross that body. These properties can then be directly used in an accurate,personalized, full body simulation. Utilizing this method, no specializedequipment is necessary. All that is needed is a 3D representation of humanbody shape, which can be acquired using a 3D scanner, body measurements,or body images. As such, this method puts the power of realistic body sim-ulation in the hands of the consumer, which has many applications.The contributions of this work are as follows:• We developed the first system for in vivo measurement of spatially varyingmechanical properties needed for simulating contact with the human body.4• We developed a complete pipeline for measuring and modelling contactproperties of soft tissues. This includes a novel skin contact probe for rapidassessment of local contact properties of the skin.• We propose a simple and effective skin model, and provide new data on theeffective thickness of skin over the body.• We capture contact deformation behaviour of human tissue across the bodyand across multiple people, in order to estimate tissue properties using anovel parameter estimation procedure.• We capture the shape of these same people and register a template to eachbody to establish correspondence.• We use the measured material properties across each mesh to create a tissueproperty space model of the human body based on body shape, involvingmodification of a standard convolutional neural network (CNN) frameworkto allow learning on 3D geometry.• Finally, we demonstrate that the tissue property estimates produced from ourmodel are of high accuracy, and can be directly used for realistic human bodysimulations.5Chapter 2Related WorkWe are not aware of any work that has measured the distribution of mechanicaltissue properties across the body, nor are we aware of any work that has learnedthe distributions of these tissue properties across the human body from the shape.However, there has been an enormous amount of literature related to modelinghuman body shape, simulating human tissue, modeling and estimating propertiesof soft tissue, and learning functions from geometry. We will summarize someimportant works from each of these areas.2.1 Body ShapeThe study of body shape was initiated in the field of industrial design, where nor-mative models of the human body were designed, in order to inform decisions sur-rounding design for humans. In 1967, Dreyfuss [17] created representative modelsof the American human population by aggregating anthropometric measurements.Furthermore, Dreyfuss provided design suggestions for products such as desks andchairs based on the normative body models. Since these initial descriptions of thedistribution of body shape, virtual human models have been created for use withsimulations, thereby informing ergonomics and industrial design [5].More recently, there has been increasing interest in modelling the shape spaceof humans in order to predict body shape based on sparse measurements (e.g.,[25, 31, 52]). For example, researchers have devised methods for determining body6shape based on partial 3D scan data, or motion capture markers (e.g., [3, 4, 45]).Most of these methods involve learning some sort of low-dimensional representa-tion of human body shape space, and using this representation to estimate a bodyshape that fits the sparse input. Though these estimates are accurate, they requirespecialized technology, such as 3D scanners or motion capture systems to be use-ful.Other approaches have used sparse tape measurements to predict the shape ofthe human body. For example, Wuhrer and Shu [66] utilized statistical shape mod-els to solve for 3D shape from 1D body measurements. Furthermore, they useda non-linear solver to extrapolate from the statistical shape model to fit measure-ments outside of the range that was used to train the shape space model. Otherresearchers demonstrate similar methods to predict body shape, additionally utiliz-ing gender, weight, and age [16].Finally, there has been a significant amount of work on learning body shapefrom images [7, 11, 14, 22, 44]. These methods use one or more images of thehuman body, and extract either pose or silhouette information, which acts as sparsedata that is used to fit a statistical human shape space model. Since people usu-ally have clothing covering their body, specialized human body shape estimationmethods have been developed to work with baggy clothing [6, 67].2.2 Physically Based Human SimulationThere is a vast literature on Physics based human simulation, one major focus ofwhich is modeling and simulation of musculoskeletal systems to produce believ-able human deformations. One of the first developments in physical simulationof humans was a method that simulated the interaction between objects and thehand of a synthetic character [19]. This method takes into account the active forcesof the fingers on the object, and the reactive forces that the object exerts on thefingers. They demonstrate that this method produces improved contact simulation(i.e., deformation of the fingers and the object). Importantly, this research was thefirst to employ the FEM, which has subsequently been used for a wide range ofhuman Physics based simulation tasks.Other researchers utilized similar techniques to simulate human muscles. For7example Chen and Zeltzer [13] developed a FEM based muscle simulation that uti-lized a biomechanical model to apply non-linear forces to the FEM mesh. Thoughthese simulations were primitive due to computational constraints of the time, thismethod demonstrates the ability for physics based simulation to accurately capturethe forces and motion involved in human deformation. As a consequence, this areaof human anatomy meets computer science was interesting to both animators andbiomechanical engineers.Since these initial applications of physics based simulation to humans, the com-plexity and accuracy has been upped. Researchers now model complex muscu-loskeletal systems and preform large-scale physics based simulations that result inaccurate muscle and tissue deformation [18, 28, 53, 58, 59]. Much of this workis focused on the numerical methods to make such simulations tractable and im-proving the musculoskeletal modeling through use of segmented human datasetsor internal 3D data acquired through magnetic resonance imaging (MRI).These methods have already been used in movies and medicine, including theaward winning Tissue system from Weta Digital; recent software includes Anato-scope, Vital Skin, and Ziva VFX. These simulations, though efficient, anatomicallyrealistic, and visually satisfactory, are generally not based on measured tissue prop-erties. Indeed, for all physics based simulations, parameters that define physicalproperties of the material being simulated must be chosen. These properties aregenerally hand-chosen or based on removed biomechanical measurements. Whilethese variable properties can produce satisfactory simulations that may be suit-able for movies, tissue properties measured in vivo are required for utmost realism,which is required in fields such as medicine or clothing design.2.3 Soft Tissue ModelingHuman skin, which makes up for 16% of the total body weight of an adult [51], hasa myriad of functions. It provides a layer of protection for the internal body, actsas a interface between the inner and outer environments, provides social signals toothers, hosts important biochemical and cellular processes, and provides sensoryfeedback to the body. Given the vital physiological functions of skin, the area oftissue research has seen much progress in the recent years, as engineers, scientists,8and clinicians work together to understand and model the inner workings of humantissue. This work has wide medical, pharmaceutical and commercial applications(see [29] for a review of skin biophysics).Soft tissue is a difficult material to model as it is highly variable across the body.Furthermore, it varies greatly across age, ethnicity, sex, body type, temperature,sun exposure, and many other individual and environmental factors. Skin itself isnon-homogenous and is made up of three distinct layers: The epidermis, dermis,and hypodermis. A number of other structures make up human skin, such as hairfollicles, sweat glands, melanocytes, collagen fibres, and many more (see Figure2.1). As such, to accurately capture the complexity of human skin, Maurel et al.,[36] suggest that physically based tissue models must consider: non-homogeneity,anisotropy, non-linearity, viscoelasticity, multiple layers/components.Soft tissue models fall into two general categories: phenomenological mod-els and structural models. Phenomenological models capture the macroscopic be-haviour of the tissue, while ignoring the underlying structure. That is, there is noexplicit definition of the skin layers or any other components of human skin. Ex-amples include; a model defining a simple stress-strain relationship (e.g., [61]), or asingle lump of FEM mesh with tied parameters (e.g., [9]). These models are simpleand can accurately capture global tissue behaviour, however, the model parametershave no immediate connection to underlying tissue structure.In contrast, structural models explicitly define the components of human tissue(e.g., collagen fibres) and associate specific parameters for each component, andparameters that describe the interactions between components (e.g., [33]). Limbert[29] argued that structural models can be thought of as a collection of interact-ing phenomenological models, and the difference between phenomenological andstructural models is simply a matter of spatial scale. Regardless, the componentsand parameters of structural models tend to have direct physical interpretations.Despite the increased realism, these models require detailed understanding of ge-ometric and constitutive properties of tissue, which is difficult to acquire. Further-more, the complexity of these models requires orders of magnitude more comput-ing power and can easily result in intractable computations.From in vitro animal experiments, where tissue is removed from the animalso that mechanical properties can be deduced, rabbit skin was demonstrated to be9Figure 2.1: Complexity of human skin. Not pictured are underlying muscles,bones, and tendons, which make the deformation of human skin evenmore complex. Image from OpenStax College, distributed under a CC-BY 3.0 licence.a rubber-like material [27]. Similar experiments have been complete with otheranimals in order to approximate human tissue, such as measuring the compres-sive response of porcine adipose tissue [15]. Due to the rubber-like nature of themeasured tissues, engineers often base tissue models off the classic hyperelasticmodels of Rivlin and others [41, 48]. This form of constitutive model is used forour purposes.102.4 Estimating Tissue PropertiesDespite the accuracy of in vitro tissue testing with animal samples, the general-ization of these measurements are unclear. Does the extracted tissue from a rabbitfairly represent the inter and intra variability of human tissue? Likely not, however,these often destructive in vitro methods are not feasible with live human tissue.As such, most work in estimating tissue properties has used passive excitation, inwhich motion of the tissue is introduced by the motion of the human (e.g., jumpingor running), rather than motion caused by an external force.Sifakis et al. [54] used motion tracking markers to track the deformation ofan actor’s face, and estimate material parameters for the volumetric facial musclemodel they created. Since then, many researchers have utilized a similar method,fitting model parameters in a data driven fashion. For example, Pons-Moll et al.[46] used a 3dMD high resolution markerless motion capture system to collect40,000 scans of individuals in motion. Using this human tissue deformation data,parameters were estimated for a modal model and a co-rotated linear elasticitymodel describing passive human tissue properties ([23, 46]). Wang et al. [62] havedone similar work, where Kinect sensors were used to capture tissue motion inorder to estimate material parameters. Though successful, these methods are lim-ited because they can only capture tissue properties related to passive motion. Theproperties related to tissue compression, for example, cannot be accurately eluci-dated from passive excitation. Furthermore, these examples used linear elasticitymodels, which lack accuracy in large tissue deformations, where real human tissuebehaves highly non-linearly.Though many researchers have utilized active excitation (i.e., an external objectapplying force to the sample) to estimate material properties for non-living things,few researchers have estimated material properties of humans using this method.For example, Pai et al. [42] used a robotic arm to apply known forces to inanimateobjects, and measured the deformation of the object to estimate material proper-ties. In addition, a similar method has been used to estimate material propertiesof different types of cloth [37, 38]. Kry and Pai used a similar method on humantissue, where contact force and displacement of the human hand was measuredduring grasping and manipulation in order to estimate the effective compliance of11the joints [26]. In addition, Bickel et al. [8] applied known forces to the humanfacial tissue while capturing deformation in order to fit parameters to a non-linearsoft tissue model. Pai et al. [43] utilized a similar method to Bickel et al. [8], how-ever their handheld probe allowed for more flexible in vivo measurements; modelparameters can be easily estimated across the whole body and across multiple peo-ple. In addition, their Sliding Thick Skin tissue model captures the non-linearity oftissue compression like the model used by Bickel et al. [8], but also allows elas-tic sliding below the soft tissue layer to capture tangential skin motion. Thoughtissue properties across the body can be easily estimated, this approach requires a3D body scanner, a motion capture system, and a precise force sensor, which arespecialized tools not available to consumers.2.5 Geometric LearningGeometric learning refers to learning from data that is in non-euclidean space, forexample, social networks or mesh surfaces. These data must be treated differentlythat standard euclidean data, meaning that traditional techniques cannot be appliedeffectively. Though there are multiple flavours of geometric learning, here we onlyfocus on methods concerned with learning functions over a non-euclidean domain.See the comprehensive geometric learning review by Bronstein et al., [12] for moredetails.Given the development and success of CNNS in the recent years in the domainof computer vision [30], researchers are eager to obtain similar success with 3Ddata, however the application of CNNS to 3D geometry is not trivial due to thelack of a good generic representation. However, researchers have still been suc-cessful at applying CNNS to a variety of tasks, including object classification andsegmentation (e.g., [20, 47, 65]). These implementations utilize either a 2D imagerepresentation of per-vertex features or a voxel representation of the 3D geometry.Though effective, these methods do not utilize topological information, such as therelationship between neighboring points, and lack accuracy.Researchers have experimented with other methods of geometric learning. Forexample, Masci et al. [35] used CNNS on 3D geometry by defining surface patchesbased on geodesic distance and modifying the CNN to ignore patch orientation.12Sinha et al. [56] proposed another approach to geometric learning, where theyconvert a 3D shape into a geometry image. Though successful, their geometryimage approach is not seamless and translation invariant, which reduces the effec-tiveness of the CNN. Maron et al. [34] propose a similar technique, however theirgeometry image is constructed as a flat torus that is seamless and supports a well-defined convolution across the entire 3D surface, which leads to improved results.The method of Maron et al. [34] is adapted for our purposes.13Chapter 3Human MeasurementTo elucidate tissue properties, we must apply known forces to the body, and mea-sure the deformation of the skin. In particular, we need to capture this deforma-tion behaviour across the body, and across a range of people. As such, standardapproaches involving complex set-ups, experimenter expertise, and slow data col-lection, were not viable. Instead, it was necessary that our experiment pipelineallowed for rapid data collection, was comfortable for the participant, and waseasy for the experimenter. Such a system is essential for wide-scale tissue mea-surements, in order to build a database of human skin properties.To learn the distribution of tissue properties across the body from body shape,we first need labelled training data. That is, we need body shapes and correspond-ing tissue property estimates across the body. Furthermore, we need topologicalcorrespondence between each body, represented by a triangle mesh. This per-mits direct comparisons between vertices on body meshes, which is favourablefor learning across multiple people.We first outline details of the participants from the experiment. Next, we ex-plain the methods and tools used to acquire the tissue property estimates across thebody. We then describe the body shape registration process used to achieve topo-logical correspondence between participants, and finally describe the interpolationtechnique used to acquire estimated tissue properties across the registered meshesthat represent the body of each participant.143.1 ParticipantsA total of 76 participants (40 female) participated in the experiment. The datafrom 4 participant had to be excluded from analysis for various reasons, such asequipment error or experimenter error. This left us with a total of 72 good datasets(39 female). We only utilize the female datasets for the current example, as wehad more female participants than males. A comparable method can be applied tomale data. The study was approved by the Behavioral Research Ethics Board ofthe University of British Columbia, and all participants provided written informedconsent.3.2 Acquiring Tissue PropertiesTissue properties are acquired based on the methods of Pai et al., [43], which areincluded below.3.2.1 Skin ProbeThe handheld probe we use for our data collection (Fig. 3.1) is a custom in-housedesign, specifically tailored for the project and produced with rapid prototypingtechniques. The bulk of the design work was carried out in 3D solid CAD software(Inventor 2017, Autodesk, California USA).The probe houses three miniature high-definition RGB cameras (USB Endo-scope, Abask, Taiwan), a six-axis force/torque sensor (F/T Nano 17, ATI IndustrialAutomation, North Carolina USA), three user input buttons, and a micro-controllerboard (Arduino LLC, Turin Italy). The experimenter receives visual feedback dur-ing operation through a user interface (UI) on a separate monitor. This UI is directlycontrollable using the buttons on the probe. The cameras are placed to provide aclear view of the skin patch around the probing point, with overlapping fields-of-view that eliminate all occlusions around the probe except at the point of contact.The design also enables future stereoscopic 3D analysis of the patch’s deformation.With a sensitive force sensor mounted back from the cameras (Fig. 3.2), arelatively long shaft was required to reach the probing point. This had to belightweight, to minimize the shaft mass’ influence on the sensor; compact, to avoidoccluding the camera’s views of the skin patch; and extremely stiff, to ensure a rigid1561743285109Figure 3.1: The Skin Probe. 1: 3D tracking marker arrangement on the “ten-tacle”; 2: Single-piece probe “head”, holding the cameras and force sen-sor; 3: Carbon fiber composite probe stem; 4: Quick-change, magnetic-lock probe tip; 5: Front-facing cameras; 6: Front trigger button; 7: Rearinput buttons; 8: Ergonomic handle, housing Arduino micro-controller;9: Externally accessible Arduino ISP header (reprogramming port); 10:Exit point for probe cables, with internal strain relief clamp.transformation between the probing location and the probe itself, even as forces areapplied to the tip. After experimenting with metal shafts and tubes, we switchedto a lightweight carbon fibre tube, with a custom-made 3D-printed mounting and amagnetically-locking, quick-change tip.In any application where experiment hardware is in direct contact with mem-bers of the public, safety is even more of a concern than usual. While contact forcesin our application are low, we have taken care to avoid sharp edges and exposedelectronics in the probe design, to minimize risk of injury to both the experimentersand participants.16124635Figure 3.2: Cutaway view of the the probe’s sensor “head” section. 1: Em-bedded force/torque sensor; 2: stereolithographic resin printing (SLA)-printed probe stem mount, which secures the stem to the force sensor;3: Integrated clamps hold the probe camera at an adjustable distancefrom the tip; 4: The probe tip, showing the internal channel housingthe locking magnets, and pinhole used for securing tips expecting neg-ative loadings; 5: Endoscope camera; 6: Camera and force sensor cablerouting to probe baseFabricationThe probe was fabricated with a combination of 3D printing technologies. Weused a fused-deposition modeling (FDM) printer (Dimension 1200es, Stratasys,Minnesota USA) to build the larger parts of the probe, such as the handle andmarker tracking assembly. These were made out of ABS plastic, a fairly typicalthermoplastic frequently used for injection-molded parts of consumer products.ABS FDM results in semi-elastic parts which stretch rather than break, and whichare approximately 75% as strong as their injection-molded equivalents. Parts inthe tip assembly, which slot directly together, required finer tolerances than thosewhich could be achieved through FDM printing, so we opted for SLA, using a Form2 printer (Formlabs, Massachusetts USA). The only significant structural element171 23Figure 3.3: The probe capture software. 1: Live feed of the forces and torquesapplied to the probe tip, showing a recent touch; 2: Visualizer window,displaying the real-time positions of the participant markers and the skinprobe tip through a user-adjustable virtual camera; 3: Live split viewfrom the three probe cameras, showing the probe shaft and tip in-frame,along with the participant and several markers.which is not 3D printed is the carbon fibre stem, a hollow tube which was hand-cutto length.Capture SoftwareThe Skin Probe software (Fig. 3.3) serves primarily to monitor data streams fromthe various sensors and Vicon system, and to consolidate these streams into record-ings sampled at up to 120Hz. It also allows the experimenter to execute automatedoptical flow post-processing on the captured video feeds. The software is con-trolled through a graphical user interface written in C++, and using the Qt 5 appli-cation library. An additional 3D view of the capture volume is provided through18our visualizer application, which was written in Python and can display both liveand recorded data from the main application. Another important role of the Skin-Probe software is to calibrate the Skin Probe before experiments: details of thisprocess are discussed in Sec. 3.2.5.We use Vicon Blade to monitor the Vicon system. Blade captures the positionin 3D space of any detected markers in the capture volume, and provides robusttracking of the transformations of rigid “props” like the Skin Probe and calibrationboard. Blade relays this capture data via a streaming application programminginterface (API) to the main Skin Probe application, where we implemented our ownpredictive marker tracking solution for the less-rigid bodies of our test participants.Our probe design, software, and experiment protocols are now thoroughlyproven through hundreds of capture sessions. The hand held nature of the devicelets us quickly and easily take multiple measurements in a single short session,the tracked markers on both the probe and the participant allow us to precisely lo-cate these measurements on the participant’s body, letting us map the participant’stissue in a novel and compelling way. The probe’s extensible design also leavesseveral interesting opportunities for future work.3.2.2 Measuring Human Soft TissuesWe have developed and refined a complete measurement pipeline that addressesthe important constraints of in vivo mechanical measurement of human tissues.First, the measurements are taken by humans holding a small hand held device,and not an intimidating machine. This is important since the probe has to applyforces on the skin. Furthermore, the measurements are taken as the participant isstanding, or while they are sat comfortably on a stool in a relaxed pose, so thatparticipants feel comfortable during data collection. Second, since it is difficult forhuman participants to hold still for extended periods of time, we track both the skinprobe and the participant’s body, in order to compute the precise relative motion ofthe probe with respect to the body, even if the participant moves. This allows usto ’cancel out’ the motion of the participant when calculating displacement of theprobe tip. Third, we measure over large portions of the body. With our system thiscan be done easily, since participant and probe can be repositioned almost freely19within a large volume. This means participants can engage in poses or motions,while having their whole body accessible for the measurement tool.One major difficulty of contact measurement is that we need to measure tissuedeformation close to the contact point. Unlike recent work on dense face and bodymotion capture [24, 64], we cannot use world-fixed cameras due to significant oc-clusion from the probe. This problem is mitigated in our project by the probe’sintegrated cameras, which capture these occluded areas in their overlapping fields-of-view (Sec. 3.2.1). Another difficulty is that the experimenter has to be closeto the participant and probe, and can block the motion capture markers. We min-imized this difficulty by careful placement of the cameras (Fig. 3.4) and also bydesigning a “tentacle” to hold the probe markers in a relatively unobstructed man-ner (Fig. 3.1). Finally, the measurement process is very quick, requiring less than5 seconds per touch, and 30 minutes for data collection after initial setup. This isimportant to avoid participant fatigue and boredom. Furthermore, we display rele-vant information (e.g., motion tracking, force, estimated tissue properties) on twoscreens, so the participant can engage in the experiment.3.2.3 Motion CaptureTo record the position and orientation of the skin probe and the skin patches beingprobed on the participant, an 8-camera Vicon MX motion-capture system, record-ing at 120 frames per second (FPS), is used. Captured data are synchronized withforce measurements and streamed for real-time display and further processing, andsaved to disk (Sec. 3.2.1). To prevent marker occlusion caused by the experimenterobtaining measurements, the density of Vicon cameras is higher toward the oppo-site side of where the experimenter stands (see Fig. 3.4). This allows us to focususe of the available cameras in areas the markers are not occluded.3.2.4 Marker and Probing LocationsRetro-reflective markers are attached to the skin using medical-grade adhesive tape.Experimenters place the markers in pre-defined locations, using palpation and ahuman body diagram to locate the positions. Marker locations are sex-specific; 14markers are attached to males and 13 markers are attached to females. Markers20XYParticipantExperimenterCapture VolumeFigure 3.4: Capture volume surrounded by eight Vicon MX cameras. Notethe higher density of Vicon cameras at the far side of the room to reduceocclusion caused by the experimenter when probing. The green areashows the zone where the experimenter records data from, the blue areashows where the participant is situated.are placed atop bony landmarks. Bony means that the markers aren’t placed onfatty areas (e.g., stomach or breast), so that motion of the participant’s skeletonis captured, rather than the motion of their tissue. In addition, they are placed oneasy to find landmarks across the body, for example, the iliac crests, medial endsof the clavicles, acromions, and spinal vertebrae. This ensures consistent markerplacement across different experimenters and across different participants.On both males and females, 15 locations on the torso are probed. These probinglocations are pre-defined and located in a similar manner to the marker positions.The probing locations are marked on the participant’s skin with a washable pen, toensure accuracy of the probing. Note that the actual locations that were probed ateach trail are known precisely since we track the probe tip with the motion capturesystem (see Section 3.3.4 for further details). The probing locations were chosen toensure local consistency of underlying tissue (e.g., no probing locations are on theborder of muscle and bone), and to highlight the variability of human tissue across21the torso. For example, locations were chosen on the bony regions on the clavicle,muscular regions on the pectoralis major, and fatty regions on the lower abdomen.3.2.5 Probe CalibrationThe probe is calibrated prior to each measurement session, in three steps. First,we estimate the transformation between the probe’s reference frame and the probetip by placing the probe’s tip on the center of our tracked calibration board. Next,we stabilize the force sensor readings (which have an intrinsic non-zero “bias”) bytaring the readings at two orientations: one with the shaft pointing down, to yieldzero lateral forces and zero torques; and one with the shaft held out horizontally, toyield zero axial force.0 8 16Time (s)-3036Force (N)SternumX AxisY AxisZ Axis0 5 10 15Displacement (mm)024Force (N)Pectoralis MajorSternumFigure 3.5: Raw data captured after probe calibration. Recorded forces show-ing four probe contacts (left) and recorded tip position while probing thesternum and the pectoralis major muscle (right). Note the hysteresis asthe probe returns to the initial contact location.The shaft, while lightweight, still has a measurable mass which we wish toeliminate from our readings. We obtain several samples of orientation and force/-torque data by moving the probe through various (arbitrary) orientations, with noexternal loading. From these we estimate the mass and center of mass of the probeshaft, which can subsequently be used to correct the force readings in any orienta-tion.With these calibrations, the combined force and torque (called a “wrench” inmechanics) can be transformed from the sensor frame to the probe tip to accurately22report the actual contact wrench applied to the body (see Figure 3.5). This is doneusing the 6× 6 transpose of the Adjoint matrix of the coordinate transformation.Please see, for example, the textbook by Murray et al. [40], Chapter 2, Section 5,for background on wrenches and their transformations.3.2.6 Tissue Property EstimationWe follow the general methods of Pai et al. [43], whereby tissue properties areestimated by measuring the response of human tissue to active excitation acrossthe torso. However, we use a different material model that takes incompressibilityof human tissue into account more directly, and is also more efficient.We model the tissues as hyperelastic incompressible isotropic Neo-Hookeanmaterials. Note that this is the classical model as introduced by Mooney [39],where incompressibility is imposed exactly as a constraint, instead of an additionalenergy that penalizes volume change (see [57] for a discussion of different Neo-Hookean models). The Mooney Neo-Hookean energy isΨM = µ2 tr(FT F− I). Thisis a simpler model than that of [43] but has the advantage that it is widely im-plemented in computer graphics software, and has some justification from biome-chanics.Shergold et al. [50] showed that the incompressible isotropic hyperelasticOgden model [41] well describes the strain hardening response of pig skin; thisalso models pig fat well [15]. We first explored using this model for soft tis-sue. Consider a small cylindrical “puck” of material with cross-section area Aand rest height L0. It exerts a force f when compressed axially by z to a new heightL= L0−z. The Ogden strain energy density isΨO = 2µα2 (λα1 +λα2 +λα3 −3), whereµ is the shear modulus, λi is the ith principal stretch, and α is the hardening ex-ponent. For uniaxial compression, Comley and Fleck [15] expressed the principalstretches in terms of the axial stretchλ = 1− z/L0, (3.1)and derived analytical expressions for stress when volume is preserved. Their ex-pressions are equivalent to f = −2Aµα (λα−1−λ−(1+α/2)). To get J-shaped strainhardening response upon extension, a high value of α (over 6) is necessary, but for23compression the dominant effect is that of the effective tissue thickness L0, sincethe energy blows up as λ → 0. Therefore we chose a fixed value of α = 2, becausethis makes the Ogden model exactly equal to the Mooney Neo-Hookean model, andreduces the number of parameters to be estimated. This makes the force exerted bythe skin puck to bef =−Aµ (λ (z)−λ−2(z)). (3.2)One advantage of using Eq. 3.2 for parameter estimation is that it directly mapsto the normal forces measured on the skin. It also shows that the two free pa-rameters, µ and L0, contribute to the force very differently, with shear modulus µcontributing linearly and L0 contributing in a highly non-linear fashion.Therefore, we first estimate the linear part of the response for small forces fromthe slope of the force-displacement data. From Eqs. 3.1 and 3.2, it can be shownthatd fdz∣∣∣z=0=3AL0µ def= µ¯. (3.3)We call µ¯ the touch stiffness since it encodes the effective stiffness felt by a gentletouch to the skin, and combines the effects of both material stiffness and the thick-ness of the skin. Once µ¯ is estimated, we can estimate L0 by fitting Eq. 3.2 to theforce-displacement data with non-linear least squares with fixed µ¯ (see Figure 3.6for example fits). Only the loading phase of the touch is used for fitting, and thresh-old of 2 N is applied to avoid fitting to large forces, where data can sometimes benoisy, especially for bony regions. In addition, a Butterworth filter is applied to theloading phase, to remove any displacement jitters caused by motion tracking.Notice that unlike [43] this model does not require any FEM simulation ofthe skin puck to estimate parameter values; only standard curve fitting is used. Inaddition, it has fewer parameters. This makes estimation very fast and robust.Data are captured from 15 locations across the torso. For each probing locationthe parameters of soft tissue thickness and touch stiffness µ¯ are estimated. Oncethe parameters are estimated, sparse data interpolation is used to acquire modelparameters across the template surface, giving us per-vertex tissue property esti-mates, which can be used for learning (see Figure 3.6 for an example of estimatedtissue thickness maps). Importantly, these per-vertex estimates are defined on cor-responding 3D meshes registered to each participant. This registration process is240 10 20Displacement (mm)012Force (N)(a)0 10 20Displacement (mm)012Force (N)(b)Figure 3.6: µ¯ and L0 fit to a force displacement curve from contact on thesternum (a) and belly (b). Blue circles correspond to linear fit after µ¯estimation, red circles correspond to fit after L0 estimation. The blueline shows the loading phase after smoothing with a Butterworth filter.A threshold of 2 N is used for fitting.described in the Section 3.3 and the tissue property interpolation is described inSection 3.4.25(a)26(b)Figure 3.6: Examples of measured tissue thickness over 30 female partici-pants. Thickness values are shown on a scale from 0 cm to 5 cm us-ing the jet colorspace. Breast thickness values are added to measuredthickness values post-hoc, based on the geometry of the participant (seeSection 3.4.4 for further details).27Figure 3.7: Force displacement curves across the torso of a single participantfrom two measurement sessions. The similarity between the measure-ment sessions demonstrates that our parameter estimation method pro-duces consistent results.3.2.7 Parameter Estimation ValidationTo validate our parameter estimation method and tissue model, we capture tissueproperties twice from a single participant. Importantly, we measured the tissueproperties from identical locations on the torso. We compare the force displace-ment curves between data collection sessions and determine that our estimated tis-sue properties are consistent (see Figure 3.7). This demonstrates that our methodproduces reliable tissue property measurements, even when the participant shiftsin pose, or the experimenters probing technique differs. Note that we allowed 20minutes between measurement of the same location, in order to reduce the effectsof hysteresis.3.3 Body Shape RegistrationIndividual human body shape is acquired with a Human Solutions Vitronic VI-TUS 3D body scanner, capable of acquiring 300 points per cm3. On each bodyscan, 30 anatomical landmarks are identified with Human Solutions Anthroscan28(a) (b) (c)Figure 3.8: Front and back of the female template tetrahedral mesh (a,b). Redcircles show locator positions on the template. The template skeleton inthe initial skeleton configuration (c).software, either automatically or by manually identifying body landmarks on thescan surface. These 30 points on the scan, which we call locators, are used for thebody shape registration process. Specifically, a subset of L = 26 landmark locatorsare used for the skeleton fitting, and a subset of M = 13 motion tracking locatorsare used for pose estimation. Once the scan and locator positions are acquired, atemplate mesh is fit to each scan, first just based on locator positions for a coarseregistration, then the geometry of the template mesh is fit to the scan surface torefine the registration. With this procedure, we can represent each body shape withthe same topology (i.e., achieve correspondence between bodies).3.3.1 Template DefinitionA template skeleton containing K = 20 bones is used (see Figure 3.8). The templateskeleton is defined by θs ∈R2K+3K+6 =(θb,θ j,θr), which is comprised of θb ∈R2Kbone parameters, θ j ∈ R3K joint angle parameters, and θr ∈ R6 root translationand rotation parameters. Specifically, θb includes a bone length and bone widthparameter for each bone, θ j includes 3 rotation parameters for each bone, and θrincludes 3 rotation and 3 translation parameters to position the root of the skeleton.Though only the bone length parameter from θb is necessary, the additional widthparameter allows for shape change during deformation, better capturing bodies of29different sizes. The initial skeleton configuration is defined as θs0A rig can be thought of as a function that takes a point in physical space, andmaps it to a different position based on a skeleton configuration θs. With this, apoint can be transformed in physical space withx′ = r(x,θs) (3.4)using linear blend skinning (LBS) or similar. The exact position of x′ given θs isdetermined by the skinning weights of x (i.e., how much the movement of eachbone influences the movement of the point). As such, a rig function using LBS canbe elaborated asx′ = r(x,θs) = R(θr)K∑i=1wiA(θb(i),θ j(i))x (3.5)where R is a rigid body transform, wi is the scalar weight at x associated to the ithbone, and A is an affine transform that includes rotation and scaling.In addition to the template rig, a template tetrahedral body mesh containingN = 34,196 vertices V ∈ R3N is used (see Figure 3.8). This template mesh wasartist generated, and has pre-determined locator positions on it that match locatorpositions on each subject scan. The skinning weights of the vertices and locatorson our template were determined using bounded biharmonic weights [21], and areheld constant throughout the fitting process.Our body fitting algorithm is comprised of two main parts. First, the parametersof θs are estimated to acquire a new skeleton configuration and tetrahedral meshthat is sized for each scanned subject. Second, the positions of the vertices V thatcomprise the template mesh are manipulated in order to approximate the geometryof the subject scan surfaceS .3.3.2 Skeleton FittingFirst the template rig is transformed to fit the subject locator positions. Specifically,the bone length and width parameters of θb need to be estimated, in order to deter-mine a rig shape that fits the subject locator positions. In addition, the joint angleparameters of θ j need to be estimated in order to deform the skeleton to match the30pose of the subject-scan. We also need to estimate the root translation and rotationθr of the skeleton to align the skeleton with the scan. We first solve for θr, anduse the result to then jointly estimate the parameters of θb and θ j. This results in asubject fitted rig, which shares the same topology as the template rig, but is sizedto the subject scan. Once the subject-specific rig is acquired, the initial templatemesh can be deformed to fit the subject-specific rig.The 6 translation and rotation degrees of freedom (DOF) of θr are solved forby minimizing the distance between the L = 26 subject and template landmarklocators using a non-linear least squares solver. Specifically, the optimal values θ ∗rare solved with the following equation, where ls ∈ R3L is the subject scan locatorpositions, lt ∈ R3L is the template locator positions. Importantly, the other θb andθ j parameters of θs are held constant at their initial values in θs0 throughout thisoptimization.θ ∗r = argminθr12|r(lt ,θs)− ls|2 : θ ∗r3 = 0,θ ∗r5 = 0 (3.6)Only the rotation about the y-axis is solved for, since subjects are standing upstraight when scanned, hence θ ∗r3 = 0 and θ ∗r5 = 0. θ∗r is applied to θs before solvingfor the rest of the parameters (θb and θ j).Now the remaining 2K+3K = 100 parameters of θs can be estimated. Specif-ically, the optimal values θ ∗b and θ∗j are estimated by deforming the locators on thetemplate using LBS in order to match the subject scan locator positions. The 3Doffset between template locators after transformation by θs and subject locators isgiven by o(θs) ∈ R3L whereo(θs) = |r(lt ,θs)− ls| (3.7)Gauss-Newton is used to minimize the non-linear least squares problem, whereH ∈R5K is a diagonal matrix containing length, width, and orientation deformationpenalties for each bone. Importantly, the θr values of θs are held constant at θ ∗rduring this optimization process(θ ∗b ,θ∗j ) = argmin(θb,θ j)12||o(θs)||2F +12θTs Hθs (3.8)31(a) (b) (c) (d)Figure 3.9: The initial template mesh torso (a). The template mesh after theskeleton fitting (b). The template mesh after geometry fitting (c). Thecorresponding body scan (d). This is a challenging example, since thescanned body varies greatly from the template, the arms are close to thetorso, and there are areas of overlapping tissue, however our methodstill produces a reasonable fit.Once θ ∗b and θ∗j are solved for, a new subject-specific skeleton configurationis created using the estimated parameters θ ∗s = (θ ∗b ,θ∗j ,θ ∗r ). LBS is then usedto deform the template tetrahedral mesh to fit the subject-specific skeleton withV ′ = r(V,θ ∗s ). This mesh shares the same topology and shape as the template,but is sized to the subject-specific skeleton (see Figure 3.9b). With a mesh thatapproximates the size of the subject, we can now fit it to the subject scan surface,in order to capture the exact geometry of the subject.3.3.3 Geometry FittingBefore the geometry fitting process is started, a distance field is created based onthe scan surface using H = 1,000 samples. Once created, the geometry fitting canproceed, in which the vertices of the tetrahedral mesh fit to the subject-specific rigare transformed to capture the surface of the subject scan, which is represented bythe distance field. By doing so, a personalized tetrahedral mesh is created, whichshares the same topology as the template, but captures the shape of the subjectscan.Ceres [1] is used to solve the non-linear least squares problem, where 2 costfunctions are imposed. First, a distance field cost EH , that serves to minimize the32distance between vertices V ′ on the template being fit and the field representing thesubject scan. The distance field can be any function f (x) : R3→ R where f ′(x) iscontinuous.EH = ∑vi∈V ′f (vi) (3.9)We used a Hermite radial basis function (HRBF) for our distance field [32, 60, 63].Second, an energy cost EE is used to regularize the fitting process, penalizingdeformations that are drastically different than the template. This serves to main-tain mesh regularity. Standard compressible Neo-hookean energy is used, where Fis the deformation gradient, µ = 2, and λ = 0.EE =µ2(tr(FT F)−3)−µ log(det(F))+ λ2log2(det(F)) (3.10)Overall, the vertices are fit to the scan surface by minimizing the followingweighted sum. β = 1 is set manually, whereas α is varied iteratively from 0.1to 10,000 by a factor of 10. This ensures that mesh regularity and locator corre-spondence is maintained at the beginning of the fitting process, when HRBF valuesare very large. As HRBF error values become smaller toward the end of the fittingprocess, α is increased, in order to achieve a close fit to the scan surface.V ∗ = argminVαEH +βEE (3.11)By minimizing this error, the tetrahedral mesh fitted to the subject-specific rigis transformed to capture the geometry of the subject scan, resulting in a subject-specific tetrahedral mesh V ∗ (see Figure 3.9c). In this study, the geometry fittingprocess is only executed on the torso region of the body by pre-defining verticesthat belong to the torso on the original template.3.3.4 Probing Position IdentificationTissue property data captured across the body needs to represented on the subject-specific tetrahedral mesh. As such, the exact probing locations need to be identifiedon the mesh, taking into account shape and pose of each subject. Motion captureis used to track M = 13 markers that are placed on the subject’s torso during data33collection. Motion capture is also used to track the probe tip position t throughoutthe data collection process. This information describes where the subject and probeare at all times during the experiment, allowing us to compute the position of eachprobe touch on the subject-specific tetrahedral mesh.At the start of each probing trial, when the probe first makes contact with theskin, we capture the probe tip position t0 and the positions of the subject’s motioncapture markers ms ∈ R3M. We then solve for skeleton parameters θr and θ j byminimizing the distance between the motion capture markers on the subject, andmotion tracking locators defined on the subject-specific mesh in the scan pose mt ∈R3M. For each trial, we first solveθ ∗′r = argminθr12|r(mt ,θs)−ms|2 : θ ∗′r3 = 0,θ ∗′r5 = 0 (3.12)then use this output to solvec(θs) = |r(mt ,θs)−ms| (3.13)θ ∗′j = argminθ j12||c(θs)||2F +12θTs Hθs (3.14)giving us θ ∗′s = (θ ∗b ,θ∗′j ,θ ∗′r ), which defines a skeleton configuration that matchesthe pose of the participant during each trial. This process is very similar to theinitial skeleton fitting procedure, except there are no bone length and width DOFand the fitting is based on a smaller subset of template locators ms and mt . Next,we transform the tetrahedral mesh based on the trial-specific skeleton configurationV ∗′ = r(V ∗,θ ∗′s ). Finally, we project the probe tip position t onto the surface ofV ∗′ to determine the probing position on the tetrahedral mesh. This process iscompleted for each trial. Since there is correspondence between V ∗ and V ∗′, wecan represent the probing position from every trial on V ∗.3.4 Estimated Parameter InterpolationGiven the subject-specific mesh, estimated tissue properties, and identified prob-ing locations, sparse data interpolation can now be used to acquire tissue propertiesacross all vertices of the subject-specific tetrahedral mesh. At each identified prob-34ing location, we set the vertex to have values of the estimated tissue properties(e.g., thickness).This algorithm interpolates probe measurements τˆ ∈ Rm taken at contact loca-tions c ∈ Rm×3 over mesh vertices v ∈ Rn×3. First, for each measurement locationj, we calculate the distances d j ∈ Rn×3 between the vertices and the contact loca-tion.d j = v− c j (3.15)Next, the probability density function of a multivariate GaussianN (·,µ,Σ) issampled at di j for each vertex i and each contact location j to acquire interpolationweights w∈Rn×m, where µ = 0, Σ∗j is the optimized covariance matrix (see Section3.4.3 for optimization details), and α∗j is the optimized scale factors.wi j = α∗jN (di j,µ,Σ∗j) = α∗j1√(2pi)3 detΣ∗jexp(− 12dTi jΣ∗−1j di j)(3.16)These interpolation weights are then normalized for each vertex i based on thesum over sample locations j, so that the sum of weights for each vertex is 1.w¯i =wi∑ j wi j(3.17)Tissue measurement data is then interpolated across all vertices using the nor-malized weights to acquire measured tissue property values τ˜ ∈Rn for all vertices.τ˜ = w¯T τˆ (3.18)We define φ as a function that produces interpolated tissue property values fol-lowing the outlined procedure, given the probe measurements τˆ , contact locationsc, body mesh vertices v, and kernel parameters θ = [{α,Σ}]m.τ˜ = φ(τˆ,c,v,θ) (3.19)This function is used for optimized kernel parameters (see section 3.4.3)35(a) (b)Figure 3.10: Prior thickness distribution depicted on template, where bluecorresponds to thin areas and red corresponds to thick areas. Note howthe thickness map takes underlying skeletal anatomy into account.3.4.1 Prior CreationWe used an anatomically accurate 3D human body model to create a nominal thick-ness map that could be used as our prior. Specifically, we extracted the skeletonand skin of a selected 3D model, and shrink wrapped the outer skin surface aroundthe underlying skeletal surface. This shrink wrapped skin can be thought of as the’inner body’, in which soft tissue, such as organs, muscles, fat, and skin are foundexterior to. To create the thickness map, the euclidean distance between outer bodyvertices and inner body vertices was calculated. The prior thickness map was cre-ated using Blender 2.80 [10] and is shown in Figure 3.10.363.4.2 Using the PriorTo combine the interpolated data with our prior, we must calculate another set ofweights, which we call the information weights I ∈ [0,1]n, where a value of 1 meansthat the tissue property value of a vertex is completely constrained by the measureddata, and 0 means it is completely unconstrained, and thus takes the value of theprior. First, the weights for each sample location j are normalized based on themaximum weight of that sample location, so that the largest vertex weight for eachsample is 1.wˆ j =w jmaxi wi j(3.20)The weights are then summed for each vertex i, and clamped to be within therange [0,1]. This ensures that vertices that are overconstrained by measurementdata are normalized (i.e., ∑i wˆi j = 1) , and weights that are underconstrained areleft as is (i.e., ∑i wˆi j < 1).I = max(∑jwˆ j,1)(3.21)Finally, the information weights, which can be visualized on the body mesh(see Figure 3.11), are used to combine the interpolated tissue property values withthe prior values τ0 ∈ Rn, in order to acquire the final predicted tissue propertyvalues τ1 ∈Rn across the mesh. Note that refers to element-wise multiplication.τ1 = τ˜ I+ τ0 (1− I) (3.22)3.4.3 Optimizing Kernel ParametersGaussian kernels are used to interpolate the data. For each measurement location,a scale factor α ∈ [0,∞] and a covariance matrix Σ ∈ R3×3 are optimized. Thesekernel parameters are optimized based on prior thickness values τ0 derived fromcalculating the distance between the skin and skeleton of an anatomically accurate3D human model. First, the prior thickness values are scaled to the minimum andmaximum measured tissue thickness values. Next, for all m measurements, kernelparameters θ = [{α,Σ}]m are simultaneously optimized using an iterative least-squares solver scipy.optimize.least squares. Specifically, the differ-37(a) (b)Figure 3.11: Information weights visualized on an example participant (red= 1, blue = 0). The torso values rely on measured data since measure-ments were taken on the torso, while the limbs rely on prior data sincethere were no measurement on the limbs.ence between the prior tissue property data and the interpolated tissue propertydata is minimized. The error term is only calculated for surface vertices belongingto the torso of the body, since measurements are only taken across the torso.θ ∗ = argminθ||τ0−φ(τˆ,c,v,θ)||22 (3.23)Covariance matrices Σ must be positive semi-definite. As such, Choleskyparametrization is used in order to ensure positive definite covariance matrices inthe unconstrained optimization. Values of α are constrained to be greater or equalto zero.3.4.4 Breast Thickness CorrectionTissue property data were not measured on the breast of most female participantsdue to the privacy constraints. As such, the values of tissue properties across thebreast region are simply an interpolation from surrounding measurements, i.e., the38(a) (b) (c)Figure 3.12: The subject-specific mesh (a), and the breasts removed versionof the subject-specific mesh (b). Tissue thickness of the breast is cal-culated by computing the difference between (a) and (b), producing abreast tissue thickness offset (c), which can be applied to measured orpredicted data.true thickness of the breast is not well represented. To fix this problem, we estimatebreast tissue thickness by utilizing the geometry from the body scan of each sub-ject. Specifically, in addition to the template we fit to each scan, we also fit anothertemplate with the breasts removed, where the breast geometry is ignored during thefitting. Now, to estimate breast tissue thickness, we can simply compute the dif-ference between the standard template and the template with the breasts removed(see Figure 3.12), giving us a breast tissue thickness offset, which can be appliedpost-hoc to any measured or predicted data. This additional breast thickness is notused to train the CNN since it can be directly extracted from body shape.39Chapter 4Learning Tissue Properties4.1 CNN ImplementationAt a high level, a CNN working on 3D geometry can be thought of as a map fromsome functions defined over some 3D shape to another function defined over thesame 3D shape. In this case, the multiple input functions are comprised of varioussimple geometric features of the body, which can be easily computed given the 3Dtriangle mesh representing the body shape. The output of the CNN can be thought ofas a some other functions that cannot be simply calculated based on geometry, theestimated tissue properties (tissue thickness and touch stiffness) in this example.Using the terminology of Maron et al. [34], the CNN learns a mapping from aset of ”easy” functions to a set of ”hard” functions defined on the 3D shape. Weoutline the preparation to create input data for the CNN, then describe the structureof the CNN and the learning process, and finally detail the extraction process ofCNN output.4.1.1 Data PreparationAfter the parameter estimation and body fitting process, we have a personalizedmesh representing the shape of each participant and estimated tissue property val-ues across the mesh. Importantly, correspondence is established between eachbody mesh, meaning that per-vertex parameter estimates can be directly comparedbetween participants. This format is not directly useful, as a standard CNN archi-40Figure 4.1: Transformation of a function from a 3D body surface to a 2D flattorus. The top left image shows the function defined over an example 3Dbody surface (tissue thickness in this case) with the three circles (yellow,blue, purple) defining the cut. The top right shows the same function onthe same surface, but unravelled as a 2D planar mesh. The bottom leftshows the 2D positions of vertices, where vertex color corresponds tothe thickness value. The bottom right shows the final function image,after scattered interpolation to acquire parameters across all pixels andtiling to produce a flat torus shape. This transformation follows theprocess of Maron et al. [34].41tecture is defined to work on images. As such, the 3D geometry data should betransformed to a 2D image-like structure, so that a standard CNN can be use tolearn on 3D surface.First, the torso region of each personalized mesh is extracted, since tissue prop-erty data was only collected across the torso. Different body shapes are dealt withby pre-defining torso vertices on the template mesh, ensuring that the torso seg-ments of each participant mesh contain identical vertex topoology. Once the torsomesh is extracted, we adapt the method of [2, 34] to create a 2D representationof the 3D surface. This method first unwraps the 3D surface into a square pla-nar mesh representation, which is defined by a triplet set of three surface points.The 2D square representation of the 3D surface is then duplicated and stitched to-gether until a flat-torus is made. A flat-torus 2D representation is beneficial as theimportant CNN property of translation invariance is maintained. Finally, with the2D mesh representation, an image can be created by interpolating per-vertex valuesover a 2D grid defining the pixel locations. Overall, this pipeline allows us to createimages of functions defined over a 3D surface, parametrized by the positions of thetriplet of surface points (see Figure 4.1). Importantly, the triplet parametrizationallows for multiple 2D images to be created from the 3D representation.Per-vertex geometric and tissue property data can now be fed into a standardCNN. Specifically, ”easy” geometric functions such as vertex position components,vertex normal components, and principal curvatures are transformed to 2D imagesfor each participant, using an image size of 256× 256 (See Table 4.1 for a moredetailed description of features used). Each function over the 3D geometry that isused for learning is called a feature, and can be thought of as a color channel in anRGB image. In the current example, 10 features were used, meaning that a singleinput image is of size 256× 256× 10. Since each triplet set results in a uniqueimage, additional training data from one participant can be acquired by utilizingother triplet sets. This process is akin to data augmentation, a common machinelearning practice when working with sparse data, since each triplet set magnifies apart of the mesh. Here we use 100 randomly chosen triplet sets, meaning that eachparticipant provides 100 images as input to the CNN. This number was chosenempirically.Similar to CNN input, ground truth ”hard” functions over the 3D surface (tissue42# Feature Description Source1 X X component of vertex position N/A2 Y Y component of vertex position N/A3 Z Z component of vertex position N/A4 Xn X component of vertex normal N/A5 Yn Y component of vertex normal N/A6 Zn Z component of vertex normal N/A7 PC1 k1 principal curvature [49]8 PC2 k2 principal curvature [49]9 gaussian gaussian curvature - k1k2 [49]10 mean mean curvature - (k1+ k2)/2 [49]Table 4.1: Description of features used in the CNN input. Each feature can bethought of as a color channel in an image.thickness and stiffness in this example) is transformed to a 2D flat-torus representa-tion using the same set of triplets as the input data. As such, for each 256×256×10input image, there is a corresponding 256×256×2 ground truth image. This alsomeans the CNN output is of size 256× 256× 2. The values of all channels ofinput and ground-truth images are scaled to the range [0,255] given a maximumand minimum feature value. That is, 0 corresponds to the minimum feature valueacross all participants and 255 corresponds to the maximum feature value acrossall participants. This means that all feature values are scaled in the same way foreach participant. Overall, with 39 participants and 100 images per-participant, atotal of 3900 images were used for training and testing the CNN. To gain betterinsights into how well our model generalizes, and given our small set of trainingdata, we implemented leave-one-out cross validation. The results of each trainingsession are then averaged to better capture the predictive performance.4.1.2 Network ArchitectureThe CNN is a mappingf (·,w) : Rm×n×10→ Rm×n×2 (4.1)43Figure 4.2: Our CNN architecture. Each grey block refers to a convolutionand ReLU layer and each red block is a pooling layer.where w are the learned weights and m = n = 256 are the height and width of eachimage. To train this mapping, a mean squared error (MSE) loss function is usedE(w) =∑2k=1∑mi=1∑nj=1 | f (X ,w)i jk−Yi jk|22mn(4.2)where X ∈ [0,255]m×n×10 is an input image, Y ∈ [0,255]m×n×2 is a ground truthimage, k indexes image channels, and i and j index image rows and columns,respectively.Our CNN implementation is relatively standard, including an information com-pression block followed by an up-sampling block (See Figure 4.2). Specifically, weutilize a part of a pre-trained VGG19 model [55], which is a CNN trained on overone million ImageNet1 images. Since this CNN is trained to classify RGB imagesinto categories, two modifications are made to the VGG19 model in order to allowfor more input channels, and allow for image predictions rather than categoricalpredictions. Specifically, the input layer of the VGG is removed, and replacedwith a custom input layer consisting of a convolution and ReLU layer that enablesimages of size 256× 256× 10 to be used. This transforms input images to size256× 256× 64, after which the standard VGG19 model takes over until the final1×1×4096 fully connected layers of the VGG19 model, which are removed and1www.image-net.org/44replaced with up-sampling layers in order to produce an output image of the samesize as the input. Specifically, 3 convolution layers are added, up-sampling from16×16×512 to 256×256×2.Overall, our CNN implementation is similar to Zhang et al. [68], who useda CNN to colorize grey-scale images, however, we keep the pooling layers of theVGG19 and discard the class rebalancing step designed to account for pixel colorrarity. Our CNN was implemented with TensorFlow2. For training the CNN, weused tf.train.AdamOptimizer, with a batch size of 1, a learning rate of1e-5, and all other hyper-parameters set to their default values. To reduce over-fitting, we utilize early-stopping with a patience limit of 10 epochs, meaning westop model training if validation accuracy does not improve over 10 consecutiveepochs. We train the model to a maximum of 50 epochs. Model training is run ona NVIDIA3 Titan X graphics processing unit (GPU) and takes around one hour.4.1.3 Data ExtractionFor each CNN output image, the predicted values can be transferred back to thecorresponding 3D body surface in order to compare to the ground truth values.First, the 4 tiles that comprise the flat-torus image are rotated and averaged toproduce a single prediction per vertex. Next, the average image is sampled at the2D vertex positions that correspond to the 3D vertices in order to capture per-vertex predicted values. Finally, the transformation of these predicted values to the3D surface is trivial due to the correspondence between the 3D surface vertices andthe vertices of the 2D representation.This extraction process, from flat-torus predicted values to 3D surface pre-dicted values, is completed for each image (i.e., each triplet set). As such, thereare 100 predictions per-vertex for one subject. To produce the final prediction fora single participant, we take a weighted average of the per-vertex predicted values.Similar to Maron et al. [34], we used triangle area scale-factor for the weightedaverage when we aggregate predictions across different triplet sets. This ensuresthat predicted tissue properties from areas of the mesh that are magnified in the 2Drepresentation have the most weight. Furthermore, it ensures that areas of the mesh2www.tensorflow.org/3www.nvidia.com45that are minified in the 2D representation (i.e., where the most noise will be), arelightly weighted. With this extraction process, the predicted values are representedon the 3D body surface in the same format as the ground truth values, allowing fordirect comparison. Overall, the data extraction process is comparable to the inverseof the data preparation process. Once the predicted tissue property data is extractedto the body mesh, realistic and personalized soft-body simulations are possible.46Chapter 5Results5.1 Training ResultsThe CNN was trained and validated on datasets from 39 participants to learn tissueproperties: in this example, tissue thickness L0 and touch stiffness µ¯ (see Section3.2.6 for explanation of model parameters). To further demonstrate the CNN, weused two datasets from participants that were not included in the training set. Thesedatasets consisted of 100 input images of size 256× 256× 10 that were createdusing the same triplet sets as the initial training data. This means that the cutof each image (i.e., 2D representation of the 3D function) is consistent across alltraining and testing data, which is essential for prediction with a CNN. With thistest input data, the output of the trained CNN can be compared to the ground truthdata to quantify the accuracy of prediction.Figure 5.1 shows the per-vertex tissue thickness predictions for each input im-age (i.e., each triplet set). Overall, though there is variability between each pre-diction, trends are visible. For example, the belly area is often predicted as muchthicker than the shoulder region. This variability between predictions is not surpris-ing due to the resolution disparities caused by the different triplet positions. Somesets of triplets may result in a 2D representation that magnifies vertices around theshoulder region, while waist regions are represented with a relatively low resolu-tion. When the 2D representations are sampled at pixel locations to produce CNNinput, drastically varying values may be sampled beside one-another in regions oflow resolution, as such, predictions across these regions can be inaccurate.47Figure 5.1: Tissue thickness prediction for 30 representative triplet sets forone participant. The variability among predictions is expected as thetransformation of thickness values from 3D to 2D results in resolutionloss for some areas of the mesh. As such, the individual predictions areaveraged to acquire the final predicted values.48Figure 5.2: Tissue thickness prediction for a participant who was not in-cluded in the training set. The measured tissue thickness for this par-ticipant is compared to the predicted tissue thickness, with the error be-tween measured and predicted values shown on the same scale. The lastcolumn demonstrates the breast thickness correction, which is appliedafter the predictions are made, to more accurately capture the thicknessof the breasts, where measurement is infeasible (see Section 3.4.4 forfurther details).Though each one of these predictions may have regions of inaccuracy due to alow resolution 2D representation, scale-factor averaging the body map predictionsacross all triplets sets reduces the noise. Figure 5.2 shows the averaged tissuethickness prediction over all cuts. When the scale factor is very small (i.e., the areaof a triangle in the 2D representation is much smaller than on the 3D surface), thepredicted value has a small weight. This reduces noise from predictions that arein areas of low resolution, and increases signal of predictions from areas that arerepresented at a reasonable resolution.49(a)(b)Figure 5.3: Thickness predictions (a) and stiffness predictions (b) for twobody models that were not part of our experiment.50The final averaged prediction is compared to measured data to compute a pre-diction error measure (see Figure 5.2). The error body map is scaled to the samerange as the measured and predicted tissue thickness values, demonstrating that therelative error between the measured and predicted thickness values is low. Aver-aging over each cross validation dataset, we achieved a mean per-vertex test errorof 0.3 cm (a relative error of 7.81%), for tissue thickness L0 values. For touchstiffness µ¯ values, we achieved a mean per-vertex test error of 21.6 (a relative errorof 9.31%). Note that touch stiffness is unitless (see Section 3.2.6 for explanationof model parameters). Relative error is calculated by dividing the metric error bythe difference between the maximum and minimum tissue property values. As wecollect data from more participants, which is currently ongoing, we expect to seeimprovements in generalizability.To validate the predicted tissue properties, we compare force displacementcurves using measured parameters versus predicted parameters for an examplebody (see Figure 5.4). This comparison demonstrates that the predicted tissueproperties, used with our tissue model (see Section 3.2.6), accurately captures thevariability of deformation across the torso. As such, body simulations using pre-dicted tissue properties will be similar to that of measured tissue properties.In the examples outlined in this paper, we demonstrate the learning and predic-tion of tissue thickness and stiffness maps across the body. In order to predict otherper-vertex values across the body, the CNN would have to be retrained. Though datapreparation and CNN training can take multiple hours (depending on the machine),the process is identical to the method we have outlined in this paper, and can beeasily applied to learn the distribution of any measured tissue properties across thebody. Once the network is trained, realistic tissue properties can be predicted fornew 3D shapes in minutes. For example, Figure 5.3 demonstrates the prediction oftissue properties for two females that were not included in the training data. No-tice how the trained network is able to account for shape differences between thefemales when predicting tissue properties.51Figure 5.4: Force displacement plots across 15 torso locations. Force-displacement curves using measured tissue properties (black) are com-pared to using predicted tissue properties (pink). Force displacementcurves are produced by our tissue model (see Section 3.2.6).5.2 Simulation ResultsThe predicted tissue properties for each female can be used to create personalizedhuman body simulations. We use the word personalized because the predictedtissue properties are tailored to each body. One common application of such humanbody simulations is garment fitting, where the personalized tissue properties areused to measure goodness of fit on an individual basis. Figure 5.5 demonstratesthis application, where the predicted tissue property values are used for the softbody simulation. Notice how the soft body simulation captures important detailsthat are not present in the rigid body simulation (e.g., breast compression and tissuebulging). Personalized tissue properties are necessary for these realistic soft bodysimulations. Importantly, these tissue properties are simply predictions from thetrained network, rather than measured with specialized equipment by expert datacollectors.52Figure 5.5: Soft body garment fit simulation using the predicted tissue thick-ness and stiffness (a). Rigid body simulation using the body geometry(b). Note the compression of the breasts and the bulging of tissue aroundthe perimeter of the bra in the soft body simulation.53Chapter 6ConclusionsMeasured tissue properties, that are captured in vivo, are essential for realistic hu-man body simulation. We demonstrate a complete-pipeline for measuring, model-ing, simulating, and predicting human tissue, which allows for efficient data cap-ture across the body and across people (see Figure 6.1 for a summary). Our pipelineaddresses the main challenges of large-scale contact data collection, including sim-plicity, comfort, efficiency, and accuracy, and allows for high-quality models ofcontact with the human body, which have a wide variety of applications. Despitethese developments, acquiring realistic spatially varying tissue properties, whichare necessary for realistic human simulations, can be challenging.54Figure 6.1: We describe how to quickly construct personalized soft-body simulation models of people, using only 3Dgeometric models of their body shape. We use a convolutional neural network, trained with new high qualitydata of soft tissue measurements and registered 3D body scans. At run time, the input to our system is only a3D model of a person’s shape (a), which may be obtained from any source. Here we see two different bodyshapes downloaded from a web store. The system predicts all the soft tissue properties required for a finiteelement simulation, such as thickness (b) and stiffness (c). Observe that the system predicts reasonable qualitativedifferences between different parts of the body, and between the two different body shapes. The results may beimmediately used in many applications such as virtual garment fitting (d). In these applications, it is essential touse a personalized soft body simulation, since the fit depends on how the body changes shape in response to thegarment. Using only the shape model (e) gives incorrect estimates of how the garment stretches and how the bodylooks. By contrast, the personalized soft tissue simulations (d) enabled by our methods produce more realisticestimates of the deformed body shape and garment stretch.550 5 10 15 20 25024Force (N)Displacement (mm)RelaxationFlexionFigure 6.2: Raw force – displacement curves measured from a location on abiceps muscle. In one condition, the biceps was relaxed; in the other, itwas isometrically flexed in the same pose, by supporting a 6.8 kg weightin the hand. Note that the muscle is significantly stiffer when active.Indeed, to acquire personalized spatially varying tissue properties, one mustcome into the lab and participate in data collection that takes roughly an hour.Though not an unreasonable request, this method is infeasible for global accessto personalized tissue properties. As such, we demonstrate that tissue propertiescan be learned from the shape of the body, and personalized tissue property valuescan be quickly and easily acquired with prediction from a trained network. Indeed,once the network is trained, realistic spatially varying tissue properties can be ac-quired in minutes without the use of specialized equipment. A 3D representation ofa body, which there are various means of acquiring, is the only necessary input. Assuch, this method removes the requirement of lab presence, and enables realisticand personalized human body simulation for consumers.There are several limitations that we wish to address in future work. In partic-ular, we would like to...• Estimate skin friction, which does not follow a simple Coulomb friction, and56depends on hydration.• Model the dependence of tissue property values on muscle activation. Flex-ion can drastically change tissue properties (see Figure 6.2), and may requirea musculoskeletal inner body simulation.• Model tissue hysteresis, which is present in many sample locations.• Measure more participants to acquire a large enough sample to draw generalconclusions about tissue property variation across the population.• Better account for the anatomy underlying human tissue when estimatingtissue properties. Human tissue lies on top of structures such as bones, ten-dons, and fat, for which there are strong priors regarding the distribution ofthese structures over the body. For example, the shoulder area is bony andthe belly area is fatty for most people. Though we take into account under-lying skeletal anatomy, we would also like to utilize the below-skin muscles,tendons, and fat.• Measure one participant at hundreds of locations across the body to acquirehigh resolution tissue properties across the body, and use this data to informour prior when measuring other participants at sparse locations. This in-formation could be used to improve the sparse data interpolation to acquiretissue property data that more accurately represents underlying anatomy.• Experiment with simpler paradigms of tissue property prediction.• Expand the measurement and prediction of tissue properties to include thearms and legs.Despite these limitations, our presented method for capturing spatially vary-ing tissue properties across the human body improves the accessibility of realistichuman body simulations. Such personalized simulations are applicable in manyfields, including animation and apparel. Our method demonstrates a pipeline forrealizing these simulations without the need for specialized equipment.57Bibliography[1] S. Agarwal, K. Mierle, and Others. Ceres solver. http://ceres-solver.org. →page 32[2] N. Aigerman and Y. Lipman. Orbifold tutte embeddings. ACM Trans.Graph., 34(6):190:1–190:12, Oct. 2015. ISSN 0730-0301. → page 42[3] B. Allen, B. Curless, and Z. Popovic´. The space of human body shapes:Reconstruction and parameterization from range scans. ACM Trans. Graph.,22(3):587–594, July 2003. ISSN 0730-0301. doi:10.1145/882262.882311.URL http://doi.acm.org/10.1145/882262.882311. → page 7[4] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis.Scape: Shape completion and animation of people. ACM Trans. Graph., 24(3):408–416, July 2005. ISSN 0730-0301. doi:10.1145/1073204.1073207.URL http://doi.acm.org/10.1145/1073204.1073207. → page 7[5] N. I. Badler. Real-time virtual humans. In 5th Pacific Conference onComputer Graphics and Applications (PG ’97), 13-16 October 1997, Seoul,Korea, page 4, 1997. doi:10.1109/PCCGA.1997.626166. URLhttps://doi.org/10.1109/PCCGA.1997.626166. → page 6[6] A. Balan and M. J. Black. The naked truth: Estimating body shape underclothing,. In European Conf. on Computer Vision, European Conference onComputer Vision, volume 5304 of LNCS, pages 15–29, Marseilles, France,Oct. 2008. Springer-Verlag. → page 7[7] A. Ballester, E. Parrilla, A. Pie´rola, J. Uriel, C. Perez, P. Piqueras, B. Na´cher,J. A. Vivas, and S. Alemany. Data-driven three-dimensional reconstructionof human bodies using a mobile phone app. International Journal of theDigital Human, 1:361, 05 2017. doi:10.1504/IJDH.2016.10005376. → page758[8] B. Bickel, M. Ba¨cher, M. A. Otaduy, W. Matusik, H. Pfister, and M. Gross.Capture and modeling of non-linear heterogeneous soft tissue. ACM Trans.Graph., 28(3):89:1–89:9, July 2009. ISSN 0730-0301.doi:10.1145/1531326.1531395. URLhttp://doi.acm.org/10.1145/1531326.1531395. → page 12[9] J. E. Bischoff, E. M. Arruda, and K. Grosh. Finite element modeling ofhuman skin using an isotropic, nonlinear elastic constitutive model. Journalof Biomechanics, 33(6):645 – 652, 2000. ISSN 0021-9290.doi:https://doi.org/10.1016/S0021-9290(00)00018-X. URLhttp://www.sciencedirect.com/science/article/pii/S002192900000018X. →page 9[10] Blender Online Community. Blender - a 3D modelling and renderingpackage. Blender Foundation, Blender Institute, Amsterdam, 2019. URLhttp://www.blender.org. → page 36[11] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black.Keep it SMPL: Automatic estimation of 3D human pose and shape from asingle image. In Computer Vision – European Conference on ComputerVision 2016, Lecture Notes in Computer Science, pages 561–578. SpringerInternational Publishing, Oct. 2016. → page 7[12] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.Geometric deep learning: Going beyond euclidean data. IEEE SignalProcessing Magazine, 34(4):18–42, July 2017. ISSN 1558-0792.doi:10.1109/MSP.2017.2693418. → page 12[13] D. T. Chen and D. Zeltzer. Pump it up: Computer animation of abiomechanically based model of muscle using the finite element method.Computer Graphics, 26:89–97, 07 1992. → page 8[14] Y. Chen, D. P. Robertson, and R. Cipolla. A practical system for modellingbody shapes from single view measurements. In British Machine VisionConference, pages 1–11. BMVA Press, 2011. → page 7[15] K. Comley and N. Fleck. The compressive response of porcine adiposetissue from low to high strain rate. International Journal of ImpactEngineering, 46:1–10, 2012. → pages 10, 23[16] F. Danckaers, T. Huysmans, D. Lacko, and J. Sijbers. Evaluation of 3d bodyshape predictions based on features. 6th International Conference on 3DBody Scanning Technologies, 10 2015. doi:10.15221/15.258. → page 759[17] H. Dreyfuss. The Measure of Man: Human Factors in Design. The Measureof Man: Human Factors in Design. Whitney Library of Design, 1967. ISBN9780823073702. URL https://books.google.ca/books?id=5 ceSQAACAAJ.→ page 6[18] Y. Fan, J. Litven, and D. K. Pai. Active volumetric musculoskeletal systems.ACM Transactions on Graphics (TOG), 33(4):152, 2014. → page 8[19] J. Gourret, N. Thalmann, and D. Thalmann. Simulation of object and humanskin formations in a grasping task. ACM SIGGRAPH Computer Graphics,23:21–30, 07 1989. doi:10.1145/74334.74335. → page 7[20] K. Guo, D. Zou, and X. Chen. 3d mesh labeling via deep convolutionalneural networks. ACM Transactions on Graphics, 35(1):3:1–3:12, Dec.2015. ISSN 0730-0301. doi:10.1145/2835487. URLhttp://doi.acm.org/10.1145/2835487. → page 12[21] A. Jacobson, I. Baran, J. Popovic´, and O. Sorkine. Bounded biharmonicweights for real-time deformation. ACM Transactions on Graphics(proceedings of ACM SIGGRAPH), 30(4):78:1–78:8, 2011. → page 30[22] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end recoveryof human shape and pose. In Computer Vision and Pattern Regognition(Conference on Computer Vision and Pattern Recognition), 2018. → page 7[23] M. Kim, G. Pons-Moll, S. Pujades, S. Bang, J. Kim, M. J. Black, and S.-H.Lee. Data-driven physics for human soft tissue animation. ACM Trans.Graph., 36(4):54:1–54:12, July 2017. ISSN 0730-0301.doi:10.1145/3072959.3073685. URLhttp://doi.acm.org/10.1145/3072959.3073685. → page 11[24] M. Klaudiny, S. McDonagh, D. Bradley, T. Beeler, and K. Mitchell.Real-time multi-view facial capture with synthetic training. ComputerGraphics Forum, 36(2):325–336, 2017. ISSN 1467-8659.doi:10.1111/cgf.13129. URL http://dx.doi.org/10.1111/cgf.13129. → page20[25] B.-Y. Koo, E.-J. Park, D.-K. Choi, J. J. Kim, and M.-H. Choi.Example-based statistical framework for parametric modeling of humanbody shapes. Comput. Ind., 73(C):23–38, Oct. 2015. ISSN 0166-3615.doi:10.1016/j.compind.2015.07.007. URLhttp://dx.doi.org/10.1016/j.compind.2015.07.007. → page 660[26] P. G. Kry and D. K. Pai. Interaction capture and synthesis. ACM Trans.Graph., 25(3):872–880, July 2006. ISSN 0730-0301.doi:10.1145/1141911.1141969. URLhttp://doi.acm.org/10.1145/1141911.1141969. → page 12[27] Y. Lanir and Y. Fung. Two-dimensional mechanical properties of rabbitskin-ii. experimental results. Journal of biomechanics, 7(2):171IN9175–174182, 1974. → page 10[28] D. Lee, M. Glueck, A. Khan, E. Fiume, and K. Jackson. A survey ofmodeling and simulation of skeletal muscle. ACM Transactions onGraphics, 28(4):1–13, 2010. → page 8[29] G. Limbert. Mathematical and computational modelling of skin biophysics:a review. In Proc. R. Soc. A, volume 473, page 20170257. The RoyalSociety, 2017. → page 9[30] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks forsemantic segmentation. In 2015 IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 3431–3440, June 2015.doi:10.1109/CVPR.2015.7298965. → page 12[31] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl:A skinned multi-person linear model. ACM Trans. Graph., 34(6):248:1–248:16, Oct. 2015. ISSN 0730-0301. doi:10.1145/2816795.2818013.URL http://doi.acm.org/10.1145/2816795.2818013. → page 6[32] I. Maceˆdo, J. P. Gois, and L. Velho. Hermite interpolation of implicitsurfaces with radial basis functions. In 2009 XXII Brazilian Symposium onComputer Graphics and Image Processing, pages 1–8, Oct 2009.doi:10.1109/SIBGRAPI.2009.11. → page 33[33] M. Marino. Molecular and intermolecular effects in collagen fibrilmechanics: a multiscale analytical model compared with atomistic andexperimental studies. Biomechanics and Modeling in Mechanobiology, 15(1):133–154, Feb 2016. ISSN 1617-7940. doi:10.1007/s10237-015-0707-8.URL https://doi.org/10.1007/s10237-015-0707-8. → page 9[34] H. Maron, M. Galun, N. Aigerman, M. Trope, N. Dym, E. Yumer, V. G.Kim, and Y. Lipman. Convolutional neural networks on surfaces viaseamless toric covers. ACM Transactions on Graphics, 36(4):71:1–71:10,July 2017. ISSN 0730-0301. doi:10.1145/3072959.3073616. URL61http://doi.acm.org/10.1145/3072959.3073616. → pagesxiii, 13, 40, 41, 42, 45[35] J. Masci, D. Boscaini, M. M. Bronstein, and P. Vandergheynst. Geodesicconvolutional neural networks on riemannian manifolds. 2015 IEEEInternational Conference on Computer Vision Workshop (ICCVW), pages832–840, 2015. → page 12[36] W. Maurel, Y. Wu, N. M. Thalmann, and D. Thalmann. Biomechanicalmodels for soft tissue simulation. Springer, 1998. → page 9[37] E. Miguel, D. Bradley, B. Thomaszewski, B. Bickel, W. Matusik, M. A.Otaduy, and S. Marschner. Data-driven estimation of cloth simulationmodels. Comput. Graph. Forum, 31(2pt2):519–528, May 2012. ISSN0167-7055. URL http://dl.acm.org/citation.cfm?id=2318896.2318912. →page 11[38] E. Miguel, D. Miraut, and M. A. Otaduy. Modeling and estimation ofenergy-based hyperelastic objects. In Proceedings of the 37th AnnualConference of the European Association for Computer Graphics, EG ’16,pages 385–396, Goslar Germany, Germany, 2016. Eurographics Association.doi:10.1111/cgf.12840. URL https://doi.org/10.1111/cgf.12840. → page 11[39] M. Mooney. A theory of large elastic deformation. Journal of appliedphysics, 11(9):582–592, 1940. → page 23[40] R. Murray, Z. Li, and S. S. Sastry. A mathematical introduction to roboticmanipulation. CRC Press, 1994. → page 23[41] R. W. Ogden. Large deformation isotropic elasticity–on the correlation oftheory and experiment for incompressible rubberlike solids. Proc. R. Soc.Lond. A, 326(1567):565–584, 1972. → pages 10, 23[42] D. K. Pai, K. v. d. Doel, D. L. James, J. Lang, J. E. Lloyd, J. L. Richmond,and S. H. Yau. Scanning physical interaction behavior of 3d objects. InProceedings of the 28th Annual Conference on Computer Graphics andInteractive Techniques, SIGGRAPH ’01, pages 87–96, New York, NY, USA,2001. ACM. ISBN 1-58113-374-X. doi:10.1145/383259.383268. URLhttp://doi.acm.org/10.1145/383259.383268. → page 11[43] D. K. Pai, A. Rothwell, P. Wyder-Hodge, A. Wick, Y. Fan, E. Larionov,D. Harrison, D. R. Neog, and C. Shing. The human touch: Measuringcontact with real human soft tissues. ACM Trans. Graph., 37(4):58:1–58:12,62July 2018. ISSN 0730-0301. doi:10.1145/3197517.3201296. URLhttp://doi.acm.org/10.1145/3197517.3201296. → pages v, 12, 15, 23, 24[44] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis. Learning to estimate 3Dhuman pose and shape from a single color image. In Conference onComputer Vision and Pattern Recognition, 2018. → page 7[45] L. Pishchulin, S. Wuhrer, T. Helten, C. Theobalt, and B. Schiele. Buildingstatistical shape spaces for 3d human modeling. Pattern Recogn., 67(C):276–286, July 2017. ISSN 0031-3203. doi:10.1016/j.patcog.2017.02.018.URL https://doi.org/10.1016/j.patcog.2017.02.018. → page 7[46] G. Pons-Moll, J. Romero, N. Mahmood, and M. J. Black. Dyna: A model ofdynamic human shape in motion. ACM Trans. Graph., 34(4):120:1–120:14,July 2015. ISSN 0730-0301. doi:10.1145/2766993. URLhttp://doi.acm.org/10.1145/2766993. → page 11[47] R. Qi Charles, H. Su, M. Kaichun, and L. Guibas. Pointnet: Deep learningon point sets for 3d classification and segmentation. pages 77–85, 07 2017.doi:10.1109/CVPR.2017.16. → page 12[48] R. S. Rivlin and D. Saunders. Large elastic deformations of isotropicmaterials. vii. experiments on the deformation of rubber. PhilosophicalTransactions of the Royal Society of London A: Mathematical, Physical andEngineering Sciences, 243(865):251–288, 1951. → page 10[49] S. Rusinkiewicz. Estimating curvatures and their derivatives on trianglemeshes. In Proceedings of the 3D Data Processing, Visualization, andTransmission, 2Nd International Symposium, 3DPVT ’04, pages 486–493,Washington, DC, USA, 2004. IEEE Computer Society. ISBN0-7695-2223-8. doi:10.1109/3DPVT.2004.54. URLhttp://dx.doi.org/10.1109/3DPVT.2004.54. → page 43[50] O. A. Shergold, N. A. Fleck, and D. Radford. The uniaxial stress versusstrain response of pig skin and silicone rubber at low and high strain rates.International Journal of Impact Engineering, 32(9):1384–1402, 2006. →page 23[51] H. Shimizu. Shimizu’s textbook of dermatology. JA KUBU. → page 8[52] C. Shu, P. Xi, and A. Keefe. Data processing and analysis for the 2012canadian forces 3d anthropometric survey. Procedia Manufacturing, 3:3745– 3752, 2015. ISSN 2351-9789.63doi:https://doi.org/10.1016/j.promfg.2015.07.813. URLhttp://www.sciencedirect.com/science/article/pii/S2351978915008148. 6thInternational Conference on Applied Human Factors and Ergonomics(AHFE 2015) and the Affiliated Conferences, AHFE 2015. → page 6[53] W. Si, S.-H. Lee, E. Sifakis, and D. Terzopoulos. Realistic biomechanicalsimulation and control of human swimming. ACM Transactions onGraphics (TOG), 34(1):10, 2014. → page 8[54] E. Sifakis, I. Neverov, and R. Fedkiw. Automatic determination of facialmuscle activations from sparse motion capture marker data. ACM Trans.Graph., 24(3):417–425, July 2005. ISSN 0730-0301.doi:10.1145/1073204.1073208. URLhttp://doi.acm.org/10.1145/1073204.1073208. → page 11[55] K. Simonyan and A. Zisserman. Very deep convolutional networks forlarge-scale image recognition. International Conference on LearningRepresentations, abs/1409.1556, 2014. → page 44[56] A. Sinha, J. Bai, and K. Ramani. Deep learning 3d shape surfaces usinggeometry images. In European Conference on Computer Vision, 2016. →page 13[57] B. Smith, F. D. Goes, and T. Kim. Stable neo-hookean flesh simulation.ACM Trans. Graph., 37(2):12:1–12:15, Mar. 2018. ISSN 0730-0301.doi:10.1145/3180491. URL http://doi.acm.org/10.1145/3180491. → page23[58] S. Sueda, A. Kaufman, and D. K. Pai. Musculotendon simulation for handanimation. ACM Trans. Graph. (Proc. SIGGRAPH), 27(3):83:1–83:8, 2008.→ page 8[59] J. Teran, E. Sifakis, S. S. Blemker, V. Ng-Thow-Hing, C. Lau, andR. Fedkiw. Creating and simulating skeletal muscle from the visible humandata set. IEEE Transactions on Visualization and Computer Graphics, 11(3):317–328, 2005. → page 8[60] R. Vaillant, L. Barthe, G. Guennebaud, M.-P. Cani, D. Rohmer, B. Wyvill,O. Gourmel, and M. Paulin. Implicit Skinning: Real-Time SkinDeformation with Contact Modeling. ACM Transactions on Graphics, 32(4):Article No. 125, July 2013. doi:10.1145/2461912.2461960. URLhttps://hal.archives-ouvertes.fr/hal-00819270. SIGGRAPH 2013Conference Proceedings. → page 3364[61] D. Veronda and R. Westmann. Mechanical characterization of skin—finitedeformations. Journal of Biomechanics, 3(1):111 – 124, 1970. ISSN0021-9290. doi:https://doi.org/10.1016/0021-9290(70)90055-2. URLhttp://www.sciencedirect.com/science/article/pii/0021929070900552. →page 9[62] B. Wang, L. Wu, K. Yin, U. Ascher, L. Liu, and H. Huang. Deformationcapture and modeling of soft objects. ACM Trans. Graph., 34(4):94:1–94:12, July 2015. ISSN 0730-0301. doi:10.1145/2766911. URLhttp://doi.acm.org/10.1145/2766911. → page 11[63] H. Wendland. Scattered Data Approximation: Cambridge Monographs onApplied and Computational Mathematics No. 17, chapter 16.2 -Hermite-Birkhoff interpolation. Cambridge University Press, GB, 2005.ISBN 9780521843355;0521843359;. → page 33[64] C. Wu, D. Bradley, M. Gross, and T. Beeler. An anatomically-constrainedlocal deformation model for monocular face capture. ACM Trans. Graph.,35(4):115:1–115:12, July 2016. ISSN 0730-0301.doi:10.1145/2897824.2925882. URLhttp://doi.acm.org/10.1145/2897824.2925882. → page 20[65] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3dshapenets: A deep representation for volumetric shapes. In Conference onComputer Vision and Pattern Recognition, pages 1912–1920. IEEEComputer Society, 2015. URLhttp://dblp.uni-trier.de/db/conf/cvpr/cvpr2015.html. → page 12[66] S. Wuhrer and C. Shu. Estimating 3d human shapes from measurements.Machine Vision and Applications, 24, 09 2012. → page 7[67] S. Wuhrer, L. Pishchulin, A. Brunton, C. Shu, and J. Lang. Estimation ofhuman body shape and posture under clothing. Comput. Vis. Image Underst.,127:31–42, Oct. 2014. ISSN 1077-3142. doi:10.1016/j.cviu.2014.06.012.URL http://dx.doi.org/10.1016/j.cviu.2014.06.012. → page 7[68] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. InEuropean Conference on Computer Vision, 2016. → page 4565
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Measuring, modelling, simulating, and predicting human tissue properties
Text