UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Position and force trajectory programming within a virtual excavation environment DiMaio, Simon P. 1998

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_1998-0412.pdf [ 11.77MB ]
JSON: 831-1.0064803.json
JSON-LD: 831-1.0064803-ld.json
RDF/XML (Pretty): 831-1.0064803-rdf.xml
RDF/JSON: 831-1.0064803-rdf.json
Turtle: 831-1.0064803-turtle.txt
N-Triples: 831-1.0064803-rdf-ntriples.txt
Original Record: 831-1.0064803-source.json
Full Text

Full Text

POSITION A N D FORCE TRAJECTORY PROGRAMMING WITHIN A VIRTUAL EXCAVATION ENVIRONMENT By Simon P. DiMaio B.Sc. (Electrical Engineering) University of Cape Town, 1995 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE STUDIES DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA September 1998 © Simon P. DiMaio, 1998 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of &&GrfLiCftt t Co^UTE^ G^tCrtH££&"v*fr The University of British Columbia Vancouver, Canada Date 7 QoTD^'St / ^ I f DE-6 (2/88) Abstract A means of automatically reproducing operator demonstrated excavation trajectories has been devel-oped for the automation of repetitive digging tasks and the teleoperation of remote machines. The dynamic adjustment of backhoe end-effector impedance, according to a variable impedance-control strategy, has been proposed for this purpose. This strategy matches machine end-point and envi-ronment impedances by means of a new, generalised duality principle which enables simultaneous position and force control over varying environment characteristics, by observing a specified com-promise between position and force trajectory errors. A virtual excavation environment has been designed and implemented to provide an inexpen-sive, safe, flexible, yet realistic platform for the development and evaluation of such automation and control strategies. The simulator comprises a mechanical impedance model of the excavator manip-ulator, a model for the synthesis of bucket-ground interaction forces, as well as graphical and haptic interfaces for both visual and force feedback. Experimental results show that task demonstration and playback within a virtual excavation en-vironment results in appropriate trajectory tracking throughout a digging motion which encounters environment impedances ranging from free motion to hard contact. < Table of Contents Abstract ii List of Figures vi List of Tables ix Acknowledgments x 1 Introduction 1 1.1 Thesis Objectives and Approach 2 1.2 Thesis Overview 6 2 A Survey of Programming by Human Demonstration 8 2.1 Human Skill Hierarchy . . " 9 2.2 Primitive Motion and Parameter Level Learning . . . 10 2.3 Associative-Model Based Learning Methods 15 2.3.1 Static Linear and Non-Linear Models 16 2.3.2 Probabilistic Models 20 2.4 Task Level Learning 22 2.4.1 Environment Model Based Learning Methods 22 2.5 Steps Toward Automated Excavation . . 23 2.6 Relevance of Programming-by-Human-Demonstration to Excavation Tasks 25 3 Variable Impedance Control for Task Reproduction 27 3.1 Task Recording and Playback within Dissimilar Environments 28 3.2 The Duality Principle and its Effect upon Manipulator Impedance Selection 29 3.2.1 Hybrid Impedance Control 31 in 3.3 Interaction with Variable Environment Impedance using a Generalised Duality Principle 32 3.3.1 Manipulator Impedance Selection for the Minimisation of Trajectory Errors . 32 Static Impedance Selection 33 Dynamic Impedance Selection: A State-Space Approach . . . . 34 3.3.2 Position-Force Error Weighting 40 3.3.3 Online Identification of Environment Impedance • . . . 41 3.3.4 Force/Velocity Ratio for the Approximation of Environment Impedance . . . 42 4 A n Excavat ion Simulator 44 4.1 Simulation Hardware Architecture : 45 4.2 Dynamic Mini-Excavator Model 46 4.3 Dynamic Visual Excavation Environment 50 4.3.1 Environment Initialisation 54 4.3.2 Communication between the Contact and Graphical Environments 57 4.3.3 Graphics Simulation and Rendering 57 4.3.4 Creating a Hole in the Ground Plane > 58 4.3.5 Animating Falling Soil 59 5 B u c k e t - G r o u n d Interact ion Force M o d e l 63 5.1 Modelling Parameters and Assumptions 65 5.2 Interaction Model Force Components . 66 5.2.1 Viscous Friction . 67 5.2.2 Soil Stiffness and Shearing 68 5.2.3 Accumulated Load . 70 5.2.4 Coulomb Friction . . . . . : . . . . 71 5.2.5 Force Component Parameter Selection 72 5.3 Simulation Results - 73 5.4 Force Model Extensions ' . . ' . . . 74 5.5 Model Validation Strategies 76 iv -" 5.5.1 Model Parameter Estimation 77 5.5.2 Operator Performance within a Virtual Environment 81 6 Trajectory Programming and Playback within the Virtual Environment 82 6.1 Environment and Machine Parameters . 83 6.2 Trajectory Playback using a Fixed Machine Impedance 84 6.3 Trajectory Playback using a Variable Machine Impedance 85 7 Conclusions and Future Work 94 7.1 Future Work : 96 Bibliography 98 A The Discretisation of a Second Order Dynamic System 103 A.l Step-Invariant Transformation 103 A.2 Backward Difference Approximation 105 v List of Figures 1.1 Task demonstration and playback within an excavation environment 3 2.2 Task skill hierarchy 10 2.3 Human association between sensed input stimuli and selected output actions. . . . . 15 3.4 Trajectory transfer between demonstration and playback environments 27 3.5 Examples of (a) inertial, (b) resistive and (c) capacitive systems, with electrical ana-logues 30 3.6 Dual-input, single-output functions relating desired position and force with position and force errors , 35 3.7 The dual relationship between manipulator damping and environment stiffness, first over a range of manipulator mass with Be = 500Nm _ 1 s and then over a range of environment damping with Mm = 110kg. In both cases f^dB = 20Hz 38 3.8 The dual relationship between manipulator damping and environment damping, first over a range of manipulator mass with Ke = 39000Nm - 1 and then over a range of environment damping with Mm = 110kg. In both cases f^B — 20Hz 39 3.9 The dual relationship between manipulator damping and environment mass, first over a range of environment damping with Ke — 39000Nm _ 1 and then over a range of environment stiffness with Be = 500Nm _ 1 s . In both cases fzdB — 20Hz. . 40 3.10 The dual relationship between manipulator damping and desired trajectory band-width, with Mm = 100kg, Be = 500Nm _ 1 s and Ke = lOOOONm-1 41 4.11 Virtual excavation environment sub-systems 44 4.12 Excavation simulator system 45 4.13 The instrumented Takeuchi TB035 mini-excavator 46 4.14 The mini-excavator, its components and coordinate frames. 47 vi 4.15 Machine impedance control with mass, stiffness and damping parameters 48 4.16 System signal flow and control between master and slave sub-systems. 49 4.17 Simulation system flowchart. 5 ! 4.18 An exploded view of the excavator model components. 52 4.19 The textured polygonal modules which are added while constructing an excavation trench . 52 4.20 A screen snapshot of the visual environment 53 4.21 The graphical user interface menu. . ; 54 4.22 A view of the virtual environment from within the cabin. 55 4.23 Graphics algorithm flow, indicating scene updates, rendering pipeline and channels. . 56 4.24 System flowchart outlining the graphics algorithm, implemented using SGI Performer 61 4.25 A flowchart showing the contents of the main graphics simulation loop 62 5.26 The parameters describing tooth geometry and orientation 65 5.27 The cutting edge frame and dimensions 66 5.28 Cutting resistance due to viscous friction. . 67 5.29 Stiffness force components which lead to soil shearing 68 5.30 Approximated planar failure surface . 69 5.31 Load forces due to the accumulation of ground material in the bucket. 70 5.32 Ground surface dimensions are updated and stored in Fu(r) 71 5.33 Tangential cutting resistance due to Coulomb friction • • • • 72 5.34 Simulated digging trajectory 73 5.35 Estimated forces during interaction with stiff soil 74 5.36 Estimated digging forces with a soil relaxation factor 75 5.37 Force integration over arbitrary bucket geometries 75 5.38 Bucket load inertia 77 5.39 Digging model parameters are selected in order to describe real measured force/torque data • • 78 5.40 Recursive-least-squares model parameter identification, neglecting shearing effects. . 80 vn 6.41 Force trajectories within a new playback environment 84 6.42 Position and force trajectories under high, fixed machine impedance control 87 6.43 Position and force trajectories under low, fixed machine impedance control 88 6.44 Position and force trajectories under moderate, fixed machine impedance control. . . 89 6.45 Position and force trajectories under variable impedance control within an unchanged environment 90 6.46 Position and force trajectories under variable machine impedance control with a low velocity weighting 91 6.47 Position and force trajectories under variable machine impedance control with a high velocity weighting. 92 6.48 Position and force trajectories under variable machine impedance control with a medium velocity weighting 93 Vlll List of Tables 4.1 Experimentally determined excavator and environment impedance parameters. . . . 48 4.2 Mini-excavator joint limits 49 5.3 Ground-interaction-model parameter identification using batch least squares 79 6.4 Demonstration and playback environment parameters 83 6.5 Machine parameters. . . 83 IX A c k n o w l e d g m e n t s My sincerest grat i tude is extended to Dr Tim Salcudean for his guidance, encouragement and patience throughout this thesis. His clear and insightful ideas have been a constant source of inspiration. I am deeply grateful to my parents and family who have extended their love and encouragement over a great distance, despite the frustrations of e-mail communication and telephone calls across multiple time-zones. It is their motivation and support, shared over the many years of study, that have enabled me to challenge my goals. I wish to thank my friends, many of whom are now scattered throughout the globe,.but who still manage to make the time to keep in touch. Their news, encouragement and the endless stream of humour, have meant a great deal to me. Thanks too to my fellow students arid friends within the Robotics group. The comradery that we have shared both in the laboratory and during recreational outings, has been invaluable. The assistance of the robotics group and departmental secretaries, as well as the technical staff who ensure tha t the many administrative mat ters run smoothly, is also greatly appreciated. Finally, it is the generous financial support provided by the Canadian Commonwealth Scholarship and Fellowship Program tha t has enabled me to study in Canada. For this opportunity, I am eternally grateful. x Chapter 1 Introduction In the construction, mining, agriculture and waste disposal fields, no other tool has been more versatile than the backhoe excavator. A large number of these machines, whose design has remained largely unchanged for decades, are currently in service worldwide. Simplicity and reliability are key factors contributing to the success of this design. A symbiotic relationship is quite evident between the operators of this equipment and the machines themselves, which simply act as an extension of human manipulative ability. The relationship between man and machine is complex, commanding a great deal of experience before acceptable performance is achieved. Because of the complexity of this relationship, the automation or remote operation of such equipment has proved difficult. Such automation promises a number of benefits, including: • automatic task execution, • remote operation within environments which are inaccessible to humans and the elimination of human operators from hazardous environments, • human supervision of multiple semi-autonomous machines and improvements in machine util-isation, • reduced dependence upon operator skills, and • lower operator workload. Excavation tasks such as trench digging result in extremely repetitive operation which places excess physical and mental strain upon the human operator. •1 Chapter 1. Introduction 2 Excavation within environments which are inaccessible, due to physical constraints or safety concerns, require tha t machines be controlled either automatically or from a remote location. Such environments are found in hazardous waste disposal, mining, sub-sea and planetary excavation applications. Profitability of mining activities often rely upon the rate at which raw material, or ore, can be removed from the earth. This can depend upon the number and efficiency of available excavating machines, as well as the number of operators employed. The ability of operators to supervise a number of semi-autonomous machines could therefore be a useful contribution toward increased performance and throughput . From these arguments, it is clear that a transfer of human manipulative skill from operator to machine is necessary in order to realise semi- or fully-autonomous control of excavation equipment, for the purposes outlined above. Unfortunately, due to the complex nature of the control of excavat-ing machines it is difficult to describe a goal, such as desired trench geometry, in terms of primitive digging motions. Experienced human operators are not used to articulating the reasons for their actions and decisions in a way tha t can easily be applied to an automatic controller, as many of their skills are learned over time, becoming seated largely in the sub-conscious. 1.1 Thes i s Objec t ives and Approach The objective of this thesis is to develop a means of acquiring and executing primitive digging motions without explicitly describing actions, but rather by observing human completion of similar motions. The scope of this problem has been restricted to primitive digging motions, such as a single digging action which is controlled at an effectual level. "Programming by Human Demonstration" is a technique whereby task skill can be transferred from a human operator to an automatic machine by means of demonstration rather than by explicit articulation of the task. This strategy is illustrated in Figure 1.1. The human operator guides an excavator through a series of digging actions within a demonstration environment which may consist of a real excavator controlled directly, or a simulated excavator within a virtual excavation Chapter 1. Introduction Demonstration Environment Task Representation and Storage I Application Environment PJ/ J ATime Figure 1.1: Task demonstration and playback within an excavation environment. environment. In both cases the operator controls the machine by means of a joystick or haptic device which measures both desired machine motion and desired force. During task demonstration the actual measured position and force trajectories achieved by the excavator are stored for future replication within an application or playback environment. A number of issues complicate the unsupervised playback of stored actions within a target environment, including: • unknown, non-homogeneous application environments which differ from the demonstration environment, • complex machine dynamics, and • changing task objectives. Similar issues may also be evident during the demonstration phase, where the operator interacts with a joystick rather than with the environment itself. Therefore, the desired manipulator motions and forces applied to the joystick may not be consistent from the digging implement's point of view, resulting in poor machine control. The objective in both cases can thus be described as the desire Chapter 1. Introduction 4 to obtain similar task performance in two distinct, dissimilar environments. Because it is often difficult to adequately model the excavation environment, it is desirable for the purposes of teleoperation or automation to include some level of slave autonomy which would allow an excavator to deviate from the desired digging trajectory in order to respond to unexpected environmental changes during unsupervised periods. Under dynamic cognitive force control [7, 10] the desired motion trajectories are adjusted in response to extreme measured interaction forces by means of a suitable heuristic. Shared-compliance control or impedance control [4, 45, 55], is an-other popular means of relaxing trajectory following, and is achieved by the control of mechanical impedance at the manipulator end-effector. Dynamic cognitive force control assumes the knowledge of a set of actions with which to respond under error conditions, while impedance control depends upon the selection and control of a suitable machine impedance, a task more amenable to automa-tion. Existing impedance control strategies have limited abilities for the dynamic adjustment of endpoint impedance in order to match that of the environment. A new variable impedance con-trol strategy has been implemented to achieve robustness in the face of unmodelled, uncertain task playback environments. A virtual excavation environment has been developed to serve as a simulator upon which au-tomation and control strategies can be evaluated, thus providing an inexpensive, safe and stable research platform [14]. This simulator comprises machine dynamics as an impedance model, a ground-bucket interaction model, a graphical display sub-system and a haptic interface. Although virtual environments have been presented previously, particularly for the purpose of master-side interaction within delayed or low-bandwidth teleoperation applications [5], the author is not aware of any similar implementations within the excavation field. Simpler simulators for human factors studies, employing essentially no environment dynamics, have been developed for other excavator-based machines such as log-loaders [56] and feller bunchers [40]. The primary advantage of such systems has been in providing the operator with visual feedback, and in some cases very limited kinesthetic feedback, during periods in which such sensory information cannot be derived from the remote environment [4]. One difficulty experienced in their design has been in the development of accurate and computationally tractable environment models. For digging, the present consensus is Chapter 1. Introduction 5 that one can expect to obtain only order-of-magnitude force predictions. Indeed, the soil may be non-homogeneous and its mechanical properties poorly defined [48]. Alekseeva, Artem'ev, Bromberg, Voitsekhovskii and Ul'y&nov compiled a set of models in 1972 [1], which had been adopted in various forms by researchers within the field. These include Dombrovskii's equation and the Fundamental Equation of Earthmoving (Reece, 1964 from [48]). The following component forces are common to such models: Cutting Resistance, defined as the force required to overcome soil cohesion during cutting. Soil-Tool friction, defined as the bucket resistance while moving through the soil, due to adhe-sion forces. Soil-Soil friction, defined as the force required to move a soil layer, with respect to an adjacent layer. Bernold [6] drew on both Alekseeva et a/.'s work, and research within the agricultural engineering field, to develop an interaction model based upon similar forces, with the addition of acceleration forces and explicit consideration of a shearing plane. Singh [48] recognised that ground interaction models could never be detailed enough to provide more than order-of-magnitude estimation results. For this reason, he attempted to improve predic-tions through learning or adaptation. His criticism of Zelenin and Alekseeva was that their work was based upon empirical results for specific types of machines and that it is not clear how to extrapo-late their methodology for arbitrary mechanisms. In addition to this, the effects of curved cutting surfaces, compression and shearing of captive soil, uneven terrain of varying depth and varying inclination, are difficult to predict, thus supporting the use of an adaptive estimation technique. While the ground-interaction force models developed to-date have proven useful for bucket de-sign, the characterisation of efficient digging parameters and the estimation of order-of-magnitude digging forces for excavator design, automation and control, they are not useful for the purpose of simulation. A bucket-ground interaction model capable of synthesising qualitatively accurate interaction forces and torques necessary for feedback to human operators and controllers, requires Chapter 1. Introduction 6 the following features which are not suitably addressed by current models: • estimation of all forces and torques acting upon the bucket during digging, rather than simply draught force, • characterisation of interaction forces and torques over arbitrary digging trajectories which include changes in bucket angle and depth, and • a dynamic excavation model, giving instantaneous forces in response to the machine and environment states. A new and intuitive ground interaction model which is based upon stiffness, friction, load and shearing forces, has been developed. Its purpose is to provide qualitatively accurate force feedback during real-time virtual environment interaction, in order to support the testing and evaluation of human task programming for the purposes of machine automation and remote operation. The variable impedance control approach used for task programming, has been evaluated within the virtual excavation simulator. 1.2 Thes i s Overv iew Chapter 2, A Survey of Programming by Human Demonstration: Previous work toward the under-standing of human manipulative skill, machine learning and programming by human demonstration is presented in order to set the stage for this thesis. Chapter 3, Variable Impedance Control for Task Reproduction: The reproduction of demon-strated digging trajectories is achieved through the use of variable manipulator impedance control. In this chapter the concept of duality between environment and manipulator impedances is gener-alised as a means to compromise position and force trajectory errors experienced during trajectory playback in varying application environments. Such uncertainty between demonstration and appli-cation environments is the primary motivation for many of the techniques discussed in Chapter 2. Chapter 1. Introduction 7 Chapter 4, An Excavation Simulator: This chapter outlines the development of a virtual ex-cavation environment required for the testing and evaluation of task programming and playback strategies. Simulator architecture and high-level operation are described. The implementation of machine dynamics, impedance control and the rendering of a realistic real-time virtual environment are discussed in detail. Chapter 5, Bucket-Ground Interaction Force Model: A new bucket-ground interaction force model is presented. Particular attention has been paid to the development of a flexible, physically plausible model which will serve as a good testbed for control techniques developed for application within real physical environments. Chapter 6, Trajectory Programming and Playback within the Virtual.Environment: The demon-stration and playback of digging trajectories, within the virtual excavation environment, is pre-sented. Both the advantages and disadvantages of the proposed technique are identified and dis-cussed, with reference to experimental results. Chapter 7, Conclusions and Future Work: Research results, contributions and the scope for further work are summarised in this final chapter. Chapter 2 A Survey of Programming by Human Demonstration In the past, robot programming for the purpose of automation has demanded complex modelling by expert human programmers. Such programming is often extremely time consuming and costly,, thereby limiting the widespread use of robot technology [13]. In addition, these techniques fre-quently lead to inflexible task programs which are designed to perform a specific task within a specific environment with little tolerance for robot inaccuracies and uncertainties within an un-known and unstructured environment. Often such robot programs and procedures are based upon tasks that have been performed traditionally by humans. A feature of human skill and strategy is its apparent transparency to the human expert. It is often impossible for an individual to describe or to understand the nature of the skills which they employ in order to undertake a particular task. These learned skills are based upon experience and are therefore buried deep within the subconscious. For this reason, a great deal of research has been undertaken in order to investigate the acquisition of specific task skills by means of human demonstration, or Programming by. Human Demonstration (PHD), an apparently intuitive means of robot programming. This "teaching by showing" strategy eliminates the need to explicitly describe particular skill features, but rather transforms the problem into one of skill identification, modelling and replication. Young children begin their lives with little or no knowledge of their environment, or the means for its manipulation. By observing and imitating the strategies exhibited by experienced humans, new skills are learned and subsequently refined by repetition and practice. This is the basis of current research into' machine learning by human demonstration. Over the past number of years, numerous automatic observation, modelling, imitation and refinement strategies have been employed in an attempt to achieve human-like performance and robustness. 8 Chapter 2. A Survey of Programming by Human Demonstration 9 The ability to capture human manipulative skill and knowledge offers a number of appealing benefits, including [43]: • providing an understanding of the processes which intervene between sensed stimuli and human responses, • the capacity to explain human behaviour, • the ability to design stimuli which produce desirable human responses, • the training of humans to exhibit behaviours of interest, or the aiding and training of inexpe-rienced humans, and • the automation of machines and processes. This chapter outlines a number of approaches to skill analysis, understanding and replication which have been proposed or implemented in recent years, with particular emphasis on the transfer of such skills to autonomous or semi-autonomous machines, as is the context of this work. 2.1 Human Skill Hierarchy Researchers within the field of machine learning have described basic task skill in a hierarchical manner as shown in Figure 2.2 [54]. High level goals are defined at the Task Level, and are often represented symbolically, perhaps as an automaton [43]. In contrast, the Primitive Motion Level describes non-contextual component motions which are used to perform a specified task-level command. At the parameter level, primitive motions are described in terms of environment interaction variables, such as position, velocity, force, compliance etc. The cognitive phenomena which constitute this hierarchy within humans are clearly not measur-able entities, instead we rely on a limited set of "proxy" variables to provide direct insight into such phenomena [43]. Proxy variables are those signals and states which are directly measurable and Chapter 2. A Survey of Programming by Human Demonstration 10 Task Level i Primitive Motion Level Parameter Level Figure 2.2: Task skill hierarchy. which represent the human cognitive process in some way. These include a variety of signals which are experienced via human sensory organs, in addition to measurable human actions. Examples of proxy variables or signals include force, torque, position and velocity. Bordegoni et al. [8] note that if we only consider skill at a physical or effectual level, then robots can be taught to perform a sequence of actions which constitute a task. Within this framework, skill lies within the capability of the end-effector and in its ability to coordinate movements. If we wish to consider the skills which would enable a robot to cope with several situations, then we move onto the conceptual skill level, or task level. Since human conceptual skill is developed throughout a lifetime of experience, its transfer to machines by means of human demonstration would take a significant portion of a lifetime. For this reason, all of the machine learning strategies presented within this chapter provide a limited solution to the human-machine skill transfer problem. By combining a priori task knowledge and flexible task models, many of these techniques have succeeded in transferring basic effectual skills, such as constrained motion [20, 3, 29, 13], force control [26], gripping [30] and navigation [16, 39] skills, from human to machine. 2.2 Primitive Motion and Parameter Level Learning At the primitive motion and parameter levels, task variables such as force, position, velocity and compliance become the focus of attention during skill modelling and learning, while their context Chapter 2. A Survey of Programming by Human Demonstration 11 within a larger task goal often becomes irrelevant. In the 1980s, Hirzinger et al. [20, 23] outlined a method for the programming of industrial robotic tasks which essentially enables surface following by means of position/force control. During a training stage, the human operator guides the manipulator end-effector through the task by means of a sensor-ball (a compliant force/torque sensor [19]) which may be mounted either to the robot itself, or to an inertially fixed platform. The former option provides an intuitive interface which conveys manipulator and environment characteristics to the operator. The sensor-ball acts as a 6-DOF force/torque sensor, enabling the human operator to provide force commands in task-space. A robot position controller, based upon state variable feedback, is employed for task-space control of the robot manipulator, while position displacement commands are derived from applied and sensed forces/torques, by means of a stiffness relationship. Forces and torques experienced by the end-effector within its environment are measured and summed with applied operator force commands, yielding a resultant force. This force is decomposed into two perpendicular components, one being in the direction of sensed environment force. Through the choice of a suitable stiffness relationship, these force components are used to compute displacement vectors which drive the manipulator controller. The selection of force/torque components enables the application of different stiffnesses along constrained and unconstrained axes. Hirzinger et al. chose a stiffness roughly equal to that of a human along the unconstrained direction, resulting in responsive free motion, and a stiffness similar to that of the manipulator itself along the constrained axis. Only single planar constraints have been considered, while performance in the presence of multiple constraint directions has not been discussed. An initial task demonstration results in a set of nominal path trajectories, as well as sensed environment forces and torques which are used as a reference for future repetitions. Trajectory corrections can be made during task playback by both the human teacher and the robot itself, by feeding the resultant force/torque derived from actual sensed, recorded original and applied supervisory forces and torques into the manipulator controller as described above. Therefore, while the robot tries to correct its path due to deviations from teaching phase sensory signals, the human operator monitors its behaviour and adds corrections if necessary. This strategy has been termed Chapter 2. A Survey of Programming by Human Demonstration 12 "supervised learning". The use of associative memories has also been suggested in order to coordinate trajectory adjustments. These memories, similar to the associative models which will be described in Section 2.3, are trained to approximate the relationship between measured task variables/signals and human supervisory corrections, thus eliminating such supervision from the execution loop. Hirzinger and Heindl have not discussed the robustness of this programming technique to various types and magnitudes of environment variations. In addition, it is not clear to what extent measured error forces in the direction of programmed free motion will affect position or velocity tracking. It appears that this technique favours the minimisation of force errors over that of position or velocity errors. This has been necessary due to their consideration of only hard environment constraints. The Space Robot Technology Experiment (ROTEX) [11, 22, 21, 23] has demonstrated the utility of these tele-sensor-programming techniques for both teleoperation and off-line programming. Tele-sensor-programming has been used within a virtual predictive environment to overcome the problem of large time delays under teleoperation. The human operator interacts with a virtual environment in order to produce robot programs which can be played back at the remote slave manipulator. Applications include mechanical assembly, servicing (e.g. grasping a floating satellite) and "Orbital-Replaceable-Unit" exchange. Asada and Izumi [3] have developed an automatic program generation strategy which is applica-ble to complex environments with multiple constraints. Their strategy is based upon hybrid control concepts and focuses on the determination of distinct position and force controlled well as their corresponding command references, based upon data measured during human demonstration. Discontinuities or characteristic changes in measured position and force signals are used to identify a sequence of task segments, each of which will be represented as a hybrid motion. The direction of motion within each sub-task is used to determine a constraint force vector, which in turn indicates which axes are to be force controlled. By combining measured position and force trajectories, as well as learned control mode switches, their algorithm was able to produce a simple "Hybrid Po-sition/Force Control Program", capable of producing autonomous control of a manipulation task. While significant assumptions regarding the nature of this task were made, it is noted that this Chapter 2. A Survey of Programming by Human Demonstration 13 technique will accommodate curved surface constraints, or those which are not perpendicular to each other. A critical shortcoming of the hybrid control technique is found in the switching of control modes as constraint conditions-change. The manipulator usually makes contact with a constraint while still in position control mode, resulting in an undesirable impact force. Rapid mode switching often results in undesirable impact forces, particularly if switching occurs too early, or too late. Asada and Izumi suggest that such impacts may be alleviated by slowing down approach speeds and by decreasing control stiffness gradually when switching from position to force control modes. A refined robot programming algorithm, based upon that of Asada and Izumi [3], is presented by Kosuge et al. [29]. Rather than generating a hybrid control strategy for task execution, this algorithm compiles a set of compliance laws, each of which act over sequences of multiple task segments. Two levels of control are derived during the data analysis-and-interpretation stage: a high-level description of discrete contact state transitions which describe the sequence of task actions, and a compliance law for each state. Further analysis of selected state transitions, as well as measured motion signals, focuses on the identification of compliant motion control laws, based upon damping control, which are consistent with the demonstrated task. Under damping control, v = u + Bf, (2.1) where v is end-effector velocity, u is the reference velocity, / is the force and moment applied to the end-effector and B is a damping matrix. Given the end-effector velocity prior to contact, v = u, and contact force / , a suitable damping matrix, which is consistent both before and after impact, can be determined. Kosuge et al. show that a single damping matrix can be chosen to satisfy a sequence of multiple contacts as long as the set of contact normals is orthogonal. The sequence of contact state transitions is divided into maximal length segments, eachof which can be described by a single damping control law. This strategy results in smooth transitions between states, without frequent controller changes. Relaxation of the contact orthogonality requirement results in some ambiguity in determining the constraint surface normals necessary for the deduction of a suitable Chapter 2. A Survey of Programming by Human Demonstration 14 control law. Only simple surface contacts are considered. In addition, the authors do not discuss the issue of environment uncertainty and its effect on program robustness. Delson and West [13] have presented a means to identify a compliance controller for each signifi-cant segment within a manipulation task, which will succeed despite bounded variations in workpiece location. By analysing position and force data measured during multiple human demonstrations of a single task, in the presence of known environment variations, one can determine a range of permissible position and force errors. Since the overall task is known to succeed in the presence of these errors, it is proposed that an automated manipulator should also succeed, provided that trajectory errors are less than those of the human demonstrator and that environment variations are no larger than those experienced during demonstration. They outline the identification of a compliance controller which will result in position and force errors, relative to nominal trajectories, that are less than those of the human operator, in the presence of similar environment variations. Intuitively this corresponds to the use of demonstration inconsistencies in identifying task aspects for which robot performance is not critical. In this work, manipulation tasks are assumed to consist of a sequence of straight line segments which are performed in a quasi-static fashion. A unique, fixed task frame is selected for each sub-task segment and is based upon orthogonal directions of motion and constraint force. The authors also appear to have assumed that the human operator is unable to determine and adapt to workpiece rotations. This ability to adapt may effectively reduce force errors in the latter stages of the task. Ikeura and Inooka [25] have investigated the feasibility of using iterative demonstrations in order to refine a desired manipulation task. In addition, they show that iterative task modifications can be made in a lower dimensional subspace than that of the task itself. For example, a task which is defined by 3-DOF motion, can be successfully modified by the demonstrator, one workspace axis at a time. Experimental results show that while task errors are gradually reduced throughout the iterative process, high frequency artifacts are produced within the position trajectories, due to difficulties in making high speed trajectory modifications. Late correction responses that are out of phase with the perceived trajectory error, result in high frequency oscillations. It is shown that a Chapter 2. A Survey of Programming by Human Demonstration 15 human operator is capable of reducing motion errors with frequencies of up to approximately 2Hz. Thereafter, compensation attempts serve only to exacerbate existing errors. While their approach to robot task teaching does not include any consideration of force/torque trajectories, or environment uncertainty, Ikeura and Inooka do take advantage of the operator's ability to improve task efficiency through iteration. It is shown that such iterative modifications can be performed one dimension at a time, however no comparison is made which indicates that such a strategy is beneficial. In addition, it is doubtful as to whether such a per axis (workspace) modification technique would be intuitive with multiple degrees-of-freedom as well as with both position and force trajectories, which may not be decoupled. 2.3 Associative-Model Based Learning Methods Many skill transfer approaches attempt to model the association between sensed input stimuli and selected output actions which are measured during human task demonstration, in an attempt to capture the processes that intervene between human sensing and human action. This model is depicted in Figure 2.3. Numerous modelling strategies have been evaluated [43], including linear, Output Action r \ w w Environment Hurpan 4 % W " \ * Sensory Input Figure 2.3: Human association between sensed input stimuli and selected output actions. non-linear, distributed, stochastic and symbolic models, with varying degrees of success. Most of these techniques assume some a priori knowledge of the task configuration and objectives prior to demonstration and use measured input/output data to identify model or task parameters. Two particularly important design decisions present themselves during this analysis, namely the Chapter 2. A Survey of Programming by Human Demonstration 16 choice of model architecture and the selection of both input and output signals. Model structure is determined by a number of factors including environment assumptions, existing knowledge of the task characteristics, as well as the type and nature of measured signals. The following categories characterise the relationships between these signals [43]: Linear relationship: There is evidence (see [43]) that humans develop linear models of non-linear phenomena if the approximation is adequate. Non-Linear relationship: These are classified as either functional or lumped relationships. A func-tional non-linear relationship can be expressed as an analytical function, while a lumped relationship includes features such as limits, saturation and hysteresis. Probabilistic relationship: Transitions between signal states are stochastic in nature. Distributed relationship: Parameters which are used to describe a model relationship are them-selves dependent upon the system variables, resulting in relationships within relationships. The signals themselves are chosen based upon input/output dependence,and identifiability [43]. In addition, Grudic and Lawrence [16], Masliah and Albrecht [35] note that in order for there to be effective skill transfer from human to machine, it is necessary that the robot have identical sensing abilities as those of the human operator during task demonstration [16]. In other words, the human should demonstrate a skill using only the robot's sensory perspective. 2.3.1 Static Linear and Non-Linear Models Liu and Asada [32] have proposed a method of skill acquisition which is based upon parameter association between a task process model and a tool holding compliance model, effectively correlating process state and task information with the operator's task performance strategy. Each model, characterised by functions D and P with parameters a and b, describes the dynamic relationship Chapter 2. A Survey of Programming by Human Demonstration 17 between measured interaction force and tool velocity, and is derived analytically (2.2, 2.3): Human Grasping Model: V(F, V; a) = 0 (2.2) Task Process Model: P{F, V\ b) = 0 , ' (2.3) where F and V are measured tool force and velocity, while a and b are human compliance model and process model parameters respectively. It is proposed that, given a set of adequately detailed models, as well as an accurate means of identifying time varying model parameters, it is possible to form an associative mapping between these parameters over multiple task demonstrations. Once such an association has been established, it can be used to map a suitable set of parameters within the manipulation parameter space, given a set of measured process parameters. Experimental results obtained during a deburring task are used to demonstrate the implementa-tion and effectiveness of such a strategy. Associative mappings are formed by means of both linear models and non-linear neural networks, the latter proving to be most successful at characterising complex mappings. It is shown that when essential features are missing from either the process model or the manipulation model, parameter mappings become inconsistent. This is evident as the association of a single point within the process parameter space, with two distinct points within the manipulation parameter space. In practice, Lipshitz's condition has been used in order to determine the extent to which human skills can be consistently transferred from demonstration results. It is assumed that the performance of the human operator is consistent within the model framework, providing a single unique mapping between process and manipulation parameters. This was found to hold true only for highly skilled, expert operators, while the consistency of unskilled operators improved with task experience. The successful performance of actual tasks, by means of the learned associative mapping, has not been verified. A method of tele-teaching, developed by Kunii and Hashimoto [30], enables a robotic system to learn an effective grasping strategy based upon the determination of a multidimensional function which describes the relationship between sensed inputs and reflexive grip adjustments. The purpose of this technique is to compensate for modelling errors and environment uncertainties, resulting in Chapter 2. A Survey of Programming by Human Demonstration 18 a stable grip. Sensed inputs include grasp configuration, contact points between fingers and object, grasping force errors and motion of the grasped object. Their strategy effectively learns the reflexive behaviour of a human, in order to compensate for grasp errors, which are determined by compar-ing actual measured grasp forces to those estimated by an ideal model. Complex manipulations involving significant grasp changes, such as changes in the number of gripper contact points, are accommodated by switching between multiple reflex functions, based upon sensed contact forces and a hybrid dynamic control scheme similar to that proposed by McCarragher [37]. Each of these multidimensional functions are learned through the training of a suitable Neural Network or Radial Basis Function. While this technique effectively learns and represents those dextrous skills which a human uses to effect grasping, it does require some a priori knowledge of the environment, in the form of a detailed task model. Kaiser et al. [26] have relaxed the heavy assumptions on model structure which are a feature in both [32] and [30]. They consider the skill acquisition problem by analysing and modelling the relationship between a set of perceived stimuli and the operator's resulting actions, as well as considering questions regarding data pre-processing and skill refinement. The authors have selected only the input stimuli which are actually responsible for the operator's output actions, by using the notion of entropy as described in Shannon's information theory. By minimising the entropy or conditional entropy of a particular output action trajectory, given an input sensory trajectory, it is shown that suitable input/output dependence can be determined. This dependence eliminates any irrelevant inputs, thus simplifying modelling, improving robustness and aiding to improve consistency between measured inputs and outputs. In [26] a single Time-Delay Radial-Basis-Function network is used to model the association be-tween significant sensory inputs and action outputs which result from a single execution strategy. Further skill optimisation is realised by the implementation of an adaptation algorithm which em-ploys an adaptive model reference control scheme. For this it has been assumed that a reference or reinforcement signal is available for adaptation. Chapter 2. A Survey of Programming by Human Demonstration 19 These concepts are demonstrated in a contact experiment in which a human operator guides a Puma manipulator by means of a passive 6-DOF joystick. Position and force measurements, obtained during the demonstration of a simple contact motion, are used to train a model network. In experiments the model network exhibited a significant steady-state force error during playback, requiring further refinement, adaptation and thus human intervention. The model network describes a single, low-level skill which forms part of an overall complex task, including multiple strategies. The integration of single strategy model networks into high-level tasks has not been discussed. Both the entropy-based signal pre-processing and skill refinement algorithms proposed in this work, rely on significant human supervision and some understanding of the task objectives. It also appears that a strictly monotonic relationship is assumed in the modelling of the input/output function. The Self Partitioning, self Organising and dimensionality REducing framework (SPORE), a non-parametric function approximation model, has been used by Grudic and Lawrence [16] for the transfer of human skill to robotic systems. SPORE is particularly suited to describing high dimensional functions. The authors argue that in order to accomplish a complex, real-world task, the human expert is likely to use a large number of sensory inputs, including vision and tactile information. The SPORE approximation technique is highly flexible with respect to the number of function inputs, while remaining practical to implement. Approximation of a multi-input, single output function is achieved iteratively by adding layers of simple, low degree basis polynomials. The addition of such layers ceases once an approximation error metric falls below a suitable threshold. In experiments, SPORE is shown to successfully model the relationship between human visual input (1024 colour pixel inputs) and human navigation control outputs during a "tree tending" exercise. Measured sensory inputs and control actions are used during the iterative construction of. a SPORE approximation. The resulting model successfully replicates the demonstrated skill, enabling the robotic system to operate autonomously. While the SPORE network is capable of approximating complex non-linear functions with arbi-trary accuracy, its structure provides no understanding of the demonstrated human skill, but simply Chapter 2. A Survey of Programming by Human Demonstration 20 acts as an associative model. It effectively recognises a set of input patterns and associates these with a particular action. The questions of input/output signal dependency or consistency have not been addressed in the selection of function inputs. 2.3.2 Probabilistic Models During the execution of a repetitive task, a human operator does not employ identical sets of actions for each task iteration, yet performance during each of these iterations does represent the same underlying skill. Yang, Xu and Chen [60] have noted that human manipulation performance is doubly stochastic in nature. Both the underlying unmeasurable mental state, or intention, and the resultant physical actions of the human operator, should be characterised as stochastic processes. Skill learning within this context is therefore the process of uncovering the exact nature of the skill, from recorded data. The Hidden Markov Model (HMM) provides a means to characterise such a doubly stochastic function with an underlying syntactic or task structure, measurable only through a set of stochastic variables. Such models have been applied to speech recognition, task manipulation analysis, skill learning and other phenomena which appear to be stochastic in nature, but stem from a state-like, goal oriented mental process. For example Hannaford and Lee [17] have observed rapidly changing force/torque signals during tele-manipulation tasks, which appear random despite arising from a deterministic mental process. It is supposed that these apparently stochastic signals are a high-gain function of the operator's mental state and include rapid actions which are performed for the purpose of exploration and identification of the task. Such identification is necessary in order to compensate for limited sensing abilities and to improve the operator's knowledge of task configurations and dynamics. The Hidden Markov Model consists of a set of states, interconnected through probabilistic tran-sitions (a Markov chain). Each state is associated with one or more output probability distribution functions. In their model, Hannaford and Lee have associated these Markov states with the opera-tor's mental states or sub-goals, each including probability functions which describe the stochastic Chapter 2. A Survey of Programming by Human Demonstration 21 nature of the force/torque signal within its sub-goal. Yang et al. [60] have proposed the use of HMMs to provide &'most likely performance as a measure of human skill, facilitating the selection of the best action sequence based upon previously measured action data. In more recent work [59], Xu and Yang make the distinction between action skill and reaction skill. Action skill is the process of determining how to act once the task goal is known, while reaction skill is the ability to respond to on-line sensory feedback while completing the desired task. In [60], Yang, Xu and Chen described an approach to action skill learning by means of Hidden Markov Models. In [59], Xu and Yang seek to establish a mapping between sensory input and control, or reaction output, thus describing reaction skills. The control strategy is partitioned into a finite set of control patterns, each of which is represented by a Hidden Markov Model. Recorded feedback and control signals are used to train each control pattern model, resulting in a trained model bank. At any instant during task execution the measured task history, including both feedback input and feed-forward control, are used to obtain the probability of each control pattern model. A suitable control action is then determined based upon the most probable control pattern, or a weighted combination of the most probable control patterns. In this way, the HMM is used as a means of recognising the current control strategy in order to act accordingly. The HMM is a feasible means of skill representation when the system. is stochastic and non-stationary in nature. Its compact parametric structure results in fast convergence during the learning procedure and can be generated using existing knowledge of the physical process at hand. Potential problems related to the HMM approach have also been observed: • For the discrete HMM, continuous measured signals are quantised and represented by a finite set of symbols. The resulting observation matrix or symbol codebook contributes a significant computational overhead. • Current methods require that model parameter updates be made using large signal histories, necessitating largely off-line methods. Chapter 2. A Survey of Programming by Human Demonstration • 22 • The effects of environment uncertainty and discrepancies on HMM validity after learning, are not discussed. For instance in Yang et a/.'s work, the human operator may respond to such uncertainty, thus altering the most likely performance criteria. 2.4 Task Level Learning Preceding sections have discussed the learning of low-level skills which are useful in generating primitive motions and parameters. Other techniques focus upon acquiring task-level skills or the segmentation of complex tasks into a set of primitive motions. 2.4.1 Environment Model Based Learning Methods Models which are based upon some a priori knowledge of the environment or task structure are often employed in order to ensure a fixed, known learning framework. High level models rely upon extensive environment models which enable task level teaching, while primitive motion or parameter level teaching enables the learning of fundamental manipulation skills. The strategies employed on each level are quite different, therefore complete task capture will require learning on at least two levels. Task-centric skill learning aims at capturing a sequence of high level sub-tasks without regard for the operator's low-level motion strategies. Complex sensing abilities, detailed environment models and known manipulator motion strategies are substituted at these primitive levels in order to facilitate intuitive teaching within a high-level environment. Takahashi and Ogata [53] present a task teaching paradigm, based upon human demonstration within a virtual environment. The operator interacts with symbolic objects within this virtual teaching environment, by means of a hand gesture interface device (Dataglove™), while receiving simulated visual feedback. This is similar to the work of Lloyd and Pai [33], who use a simple graphical interface which enables the operator to specify and preview grasping, motions and contact actions which constitute Chapter 2. A Survey of Programming by Human Demonstration 23 a desired task. The goal of this environment is to present a "task-centric" view of robot programming to the non-expert operator, with little or no emphasis on the robot which will ultimately realize the task. During the execution phase, the recorded sequence of task commands is interpreted using task-dependent interpretation rules, as well as manipulation properties embedded within each component object. The teaching environment is mapped into the task environment and task-level operations are interpreted into low-level robot commands. Mechanical constraints of the robot are considered in order to re-plan the task sequence if necessary. Takahashi, Ogata and Muto [54] show that exploratory or reflexive motions can be learned by observing the demonstration of a variety of similar tasks, each varying in its level of complexity. Such motions do not contribute to the completion of a task on a symbolic task level, but rather to the human operator's ability to perform complex tasks efficiently and robustly without acute sensing abilities. In a peg-in-hole experiment it is shown that the exploratory or reflexive motions employed during each task demonstration can be determined by eliminating the general symbolic strategy evident from the union of motions for all task iterations. The segmentation of task measurements into distinct sub-tasks, or primitive motions, has been studied by McCarragher [37, 36] and Wang et al. [58, 15]. They have developed qualitative analysis techniques and "intelligent filters" for the segmentation of position, force, stiffness, and mechanical impedance trajectories measured during human demonstration of complex assembly tasks. The combination of primitive motions into high-level task descriptions remains outside the scope of this thesis, however it is noted that the mechanics of high-level learning is quite different from that of primitive motion or parameter level learning. Little work has been undertaken to combine these two areas of study. 2.5 Steps Toward Automated Excavation "Dynamic Cognitive Force Control" refers, in excavation, to a class of automation strategies which limit sensed interaction forces through dynamically adjusting digging trajectory depth and speed [12, Chapter 2. A Survey of Programming by Human Demonstration 24 7, 10]. Bullock, Apte and Oppenheim [12] describe this as "digging by feel", and used observations of human operators.to design such an adaptive strategy. A human operator will adjust the angle-of-attack, penetration depth and angle of removal, based upon varying ground characteristics and forces. They have divided the task of automated excavation into two separate objectives, namely the tactical objective, which deals with short-term tasks and reactions; and the strategic objective which is concerned with the long-term goal (usually based upon geometry). Their solution is therefore the combination of both force-cognitive excavation, as well as spatial task planning. The proposed tactical approach is to: . 1. Command the manipulator under position control. 2. Sense forces and process at a higher level. 3. Alter the intended digging trajectory. They note that force, hybrid and impedance control techniques cannot be solutions to the force cognitive excavation problem, since they are signal-level approaches, where it is difficult to incorporate task objectives. Bullock et a/.'s results show complex behaviour, similar to that expected of a human operator, and success under a broad range of soil conditions. Bodur et al. [7] also observed the human operator's use of interaction force, sensed using cabin jerk, hydraulic pressure (displayed upon a pressure gauge) and the deformation of hydraulic hoses, to adjust the digging trajectory. Their strategy is similar to that presented by Bullock et al. and prescribes the following sequence of events upon exceeding force limits: 1. Halt horizontal bucket motion. 2. Rotate the bucket outward by approximately 15° in order to compensate for torque strain at the bucket joint. 3. Raise the Bucket by 1cm. 4. Move inward at a 15° slope while monitoring forces. Chapter 2. A Survey of Programming by Human Demonstration 25 5. Upon reaching the predefined trajectory depth, without exceeding a force limit, correct the bucket angle and continue digging horizontally. Bradley and Seward [10] have used a similar force cognitive strategy in their efforts with the LUCIE project. One interesting feature is their use of velocity vector control. Here, a velocity vector is generated at each time interval, which will move the manipulator to the subsequent target point, by the next time interval. These endpoint velocities are roughly translated into cylinder velocities and applied. The actual measured velocity is then recorded and used to compute a velocity discrepancy vector, which is taken as a form of software force feedback. Field experiments using this system, have demonstrated excavation speed comparable to an average human operator, and quality comparable with tha t of an expert operator. 2.6 Re levance of P r o g r a m m i n g - b y - H u m a n - D e m o n s t r a t i o n t o Excavat ion Tasks In this chapter, a large variety of techniques for the transfer of manipulative skill from humans to robots have been outlined and evaluated, many of which are clearly not suitable within the context of excavation. Due to the complexity of the conceptual reasoning required to undertake a complex task consisting of multiple sub-tasks or motions, it is sensible to begin with the consideration of only parameter level skill, or the ability to coordinate movements on a physical or effectual level. Once mastered, these can be incorporated into existing task-level learning algorithms or trajectory path planners. Those associative modelling techniques which have been reviewed [32, 30, 26, 16, 17, 60, 59] require at least a rudimentary model framework which is dependent upon both environment and op-erator characteristics. Non-linear mappings within this framework are generated by neural network-based models and probabilistic structures such as the Hidden Markov Model. These are often com-putationally demanding, require exhaustive training input and it is often unclear how to incorporate higher level task goals in a flexible manner. Environment-model based learning methods rely upon either extensive environment models or. the means to automatically generate such models in real-time [53, 33, 54]. Due' to the nature of Chapter 2. A Survey of Programming by Human Demonstration 26 excavation within time-varying, non-homogeneous environments, this approach is not feasible. Other approaches designed specifically for "learning" or programming at a parameter or effec-tual level [20, 23, 3, 13] are more promising for applications in excavation. The use of end-effector compliance or impedance adjustment is common in these techniques, resulting in relatively good task replication within uncertain environments. Subsequent chapters describe a method for the demonstration and playback of excavation trajectories using similar concepts. This method em-ploys an impedance controller which overcomes problems due to hybrid control-mode switching, as experienced by Asada and Izumi [3], the force tracking bias experienced by Hirzinger et al. [20, 23] and slow off-line compliance controller design employed by Delson and West [13]. It is suitable for manipulator control within uncertain, varying environments which present a wide range of mechan-ical impedance, such as that found in excavation, rather than environments which include only rigid contact surfaces. In addition, the role of input signals such as desired end-effector position and force is clear, such that higher-level goals can easily be incorporated if these can be expressed in terms of position and force excavation trajectories. For example the "maximum extraction of soil per hour" is a high-level goal which can be presented at the primitive motion level by specifying appropriate desired digging trajectories, generated by a high-level task planner. Chapter 3 Variable Impedance Control for Task Reproduction The strategies outlined in Chapter 2 each aspire to the same goal, namely to enable an automatic machine to mimic human skill and performance within an environment, after human activity has been observed/measured in a similar environment. It has been noted that human skill can be observed at different levels, however this project concentrates upon the reproduction of human task performance on a parameter level. Issues pertaining to the playback of position and force trajectories, measured during human task demonstration, are discussed within this chapter. Two distinct environments are considered, namely the demonstration and playback environments, as shown in Figure 3.4. Demonstration Environment Trajectory Extraction, Representation and Storage ~i *• r** Playback Environment Figure 3.4: Trajectory transfer between demonstration and playback environments. A human operator interacts with a demonstration environment which may be either real or virtual, generating both desired position and force trajectories. A robot interacts with a playback environment, attempting to mirror human performance measured in the demonstration environment. 27 Chapter 3. Variable Impedance Control for Task Reproduction 28 In this sense, the problem becomes that of trajectory tracking, or replication, between environments. 3.1 Task Recording and Playback within Dissimilar Environments The reproduction of measured position and force trajectories within identical demonstration and playback environments is in most cases a trivial task since these trajectories remain consistent within the playback environment. In most applications there is significant disparity between these environments, which may itself not remain constant throughout the task. It is for this reason that numerous programming by human demonstration techniques attempt to model a relationship between environment variation or uncertainty, and human corrective action. It is noted that the demonstration phase itself may contain two dissimilar environments, namely that which the human operator experiences (usually a joystick or other haptic interface) and that which the robot experiences. Similar disparity may also be experienced within teleoperation systems in which "transparency" is not observed between master and slave environments [31]. The concepts presented within this section may also be relevant under such conditions. On a parameter level, it is clear that the instantaneous position and force relationship within a task environment is consistent only for a single environment impedance. Impedance, in the general sense, can be considered to be the relationship between effort and flow. Both mechanical impedance, „ force effort -'mechanical — velocity flow (3-4) and its electrical analogue, _ voltage _ effort ^electrical — , — a i ^O.OJ current now clearly employ effort and flow in the same way. Consider the single axis interaction with an en-vironment with mechanical impedance Ze, which constrains the relationship between velocity and force, x and F respectively (3.6): . F = Zex. (3.6) Chapter 3. Variable Impedance Control for Task Reproduction 29 If environment impedance Ze can be approximated by a second-order linear relationship with mass Me, damping Be and stiffness Ke terms, then: F(s) = (Mes + Be + !y)sx{s) . (3.7) Velocity and force trajectories measured during interaction with an environment characterised by impedance Ze may not be consistent within a second environment of impedance Z'e. Therefore, it is not possible to reproduce both velocity and force trajectories simultaneously within this second environment. A force error F is experienced if only the measured velocity is followed in the playback environment, F=(Ze- Z'e)x , (3.8) while a velocity error x is experienced if only measured forces are considered: ~i=i~z:~~k)F- (3-9) Subsequent sections describe a means to balance these errors in order to effect a satisfactory compromise during task playback in uncertain and time-varying environments. 3.2 The Duality Principle and its Effect upon Manipulator Impedance Selection Anderson and Spong [2] have noted that the type of manipulator control strategy that is employed depends fundamentally upon the characteristics of the environment with which it interacts. Ac-cording to Anderson and Spong: "The manipulator should be controlled to respond as the dual of the environment". They characterised environments as inertial, capacitive, or resistive, as follows: • A system with impedance Z(u) is inertial iff |2"(0)| = 0. • A system with impedance Z{UJ) is resistive iff |Z(0)| = c, where 0 < c < oo. ' • A system with impedance Z(u) is capacitive iff |Z(0)| = oo. Chapter 3. Variable Impedance Control for Task Reproduction 30 x f T - > - •*^L ^ • ^ W W - ^ IT .IX w / //////// / b (b) (c) Figure 3.5: Examples of (a) inertial, (b) resistive and (c) capacitive systems, with electrical ana-logues. Examples of each impedance category, with electrical circuit analogues, are shown in Figure 3.5. An inertial system exhibits zero impedance at steady state, thus acting as an effort source, while a capacitive system has infinite impedance, acting as a flow source. Capacitive and inertial environ-ments represent dual impedances, due to their inverse relationship, while a resistive environment is self-dual. If a simple manipulator impedance control strategy is employed along a single axis, then: fo{s) + fe(s) = Zms(x(s) - x0{s)) , (3.10) where xQ and x are desired and actual end-effector position, while /o is the desired applied force and Zm is the manipulator endpoint impedance. An environment force fe results from interaction with an environment impedance: fe(s) = -Zesx(s) . (3-11) Actual manipulator position and applied force can then be expressed as a function of desired position and force by combining Equations (3.10) and (3.11). sx(s) -SXQ(S) + Zm + Zt 7o(«) Zm + Ze fe{s) = -— ^~fo{s) ~ ^ e mr7 SX0{s) Zm + Ze Zm + Ze (3.12) (3.13) Chapter 3. Variable Impedance Control for Task Reproduction 31 If the environment is found to be inertial, so that Ze(0) = 0, then zero steady-state position error is achieved as long as manipulator end-point impedance is capacitive, that is Zm(0) = oo. Conversely, if the environment is found to be capacitive, so that Ze(0) = oo, then steady-state force tracking can be attained if manipulator impedance is inertial, that is Zm(0) = 0. A resistive environment can be position controlled if Zm(0) = oo, or force controlled if Zm(0) = 0. Therefore, the duality condition ensures that in any manipulator-environment relationship there is no more than one position source and no more than one force source. This enables the tracking of a given motion or force trajectory in Cartesian space by selecting a suitable manipulator impedance based upon measured environment impedance. One such strategy, developed by Anderson and Spong, is Hybrid Impedance Control. 3.2.1 Hybrid Impedance Control Hybrid position/force control [42] is the result of a realisation that both the position of a manipulator end-effector and the contact force which it exerts upon the environment along one axis, cannot be controlled independently. Under this strategy, the task space is split into two subspaces, namely position-controlled and force-controlled subspaces. This leads to an intuitive approach to force control which recognises the distinction between force and position control subspaces, however it fails to recognise the relevance of manipulator impedance. By controlling the amount of mechanical compliance, or "impedance" in the manipulator, contact forces with the environment can be regulated. This is known as Impedance Control [24]. While it does consider the effects of manipulator impedance during robot-environment interaction, it is used primarily as a position-control scheme, with small adjustments made to react to contact forces, thus softening position tracking. Therefore, little attempt is made to follow a commanded force trajectory, thereby ignoring the distinction between force and position controlled subspaces. Hybrid Impedance Control [2] is the combination of both Hybrid and Impedance Control, in that manipulator impedances are selected within position and force controlled sub-spaces, based upon environment impedance and hence the duality principle. Until now, environment impedance has been classified as either inertial, or capacitive, suitable for either position or force control. Fixed Chapter 3. Variable Impedance Control for Task Reproduction 32 position and force control sub-space impedances are selected, based upon the task at hand, and do not change during task execution. In the next section, the notion of dual impedances is extended in order to obtain a general principle of duality which is not limited to discrete, passive inertial and capacitive impedances, but rather accounts for a continuous range of dynamic impedances and their duals. 3.3 Interaction with Variable Environment Impedance using a Generalised Duality Principle In general, the environments which we encounter as humans tend to be complex, non-homogeneous, time-varying systems which cannot be classified as either inertial or capacitive. Most current man-ufacturing and assembly applications result in either unconstrained, free-space manipulator motion or constrained motion against hard contacts. The nature of these extreme environment impedances allows one to adopt either position or force control in a well.defined and consistent manner, how-ever during interaction with intermediate environment impedances it is not quite as clear as to whether the manipulator should be position or force controlled. This is the case when considering softer, deformable environments such as liquids, soil, soft metals, human and animal tissue, etc. In these scenarios, the manipulator encounters an environment impedance which may range between the inertial and capacitive extremes. In addition, environments are commonly categorised only according to their steady state behaviour within this model. Inertial and capacitive environment impedances are defined only by their DC or steady state values. This section describes a generalised duality principle which guides the selection of a manipulator impedance function in order to realise a compromise between position and force tracking in the dynamic sense. 3.3.1 Manipulator Impedance Selection for the Minimisation of Trajectory Errors Given an environment which is neither un-impeding nor constraining, it is not clear whether to track the given position trajectory or the given force trajectory. It seems natural in this case to make a compromise between position and force tracking by minimising the sum of their respective Chapter 3. Variable Impedance Control for Task Reproduction 33 errors. A weighted sum-of-squared-errors (3.14) can be used as an error norm for this purpose [18]: E = W(x0-x)2 + (f0-f)2 = Wx2 + f2 . (3.14) By minimising this error with respect to the manipulator impedance function, a compromise between position and force control is achieved. A scalar weighting W is used to normalise displacement and force units while shifting the relative error sensitivity between position and force errors. For example, if SI units are used throughout, a value of W = 1000000 means that a position tracking error of lmm and a force tracking error of IN result in the same contribution to total error E. For simplicity, impedance control along a single axis will be considered here and it is assumed that the controlled manipulator is described by: Mms2x{s) = f0(s) + /e(s) - (Bms + Km)(x(s) - x0{s)) , ' (3.15) where Mm is the effective mass of the manipulator end-effector, Bm and Km are its desired end-point damping and stiffness factors. It is assumed that only the manipulator stiffness and damping behaviour can be adjusted within its controller, while the mass Mm is fixed. The target environment is approximated by means of a first order linear relationship characterised by damping and stiffness factors Be and Ke: fe(s) = -(Bes + Ke)x(s) . (3.16) Static Impedance Selection By substituting (3.16) into (3.15) under static conditions, the manipulator position x and applied manipulator force fe are expressed in terms of desired position and force, xo and /o, as well as environment and manipulator stiffnesses, Ke and Km respectively: *W = K \Kfo{®+KKZKXQ{® ( 3 - 1 7 ) /e(0) = 7 r ^ / o ( 0 ) + / ^ * o ( 0 ) . (3-18) Chapter 3. Variable Impedance Control for Task Reproduction 34 The resultant combined error and its derivative with respect to manipulator stiffness becomes: dE _ ( /Q + xQkm \f x0 /Q + x0Km \ Km ~ \Km + Ke X°)\Km + Ke (Km + Ke)2) , J KeXQ Ke{f0 + X0Km)\ +2\Km + Ke (Km + Key ) ( ^ ° j This error can be minimised with respect to manipulator stiffness Km by setting dE • W y r - = 0 and obtaining that Km = — or /o = Kexp • (3.21) Therefore, the "optimal" manipulator stiffness is proportional to the inverse of environment stiff-ness. The duality condition is satisfied for both Ze(0) = Ke = 0 and Ze(0) = Ke = oo, while being extended over all intermediate impedance values. At the second solution the error is always zero, independent of the choice of manipulator impedance Km, since the relationship between de-sired trajectories /o and xo is consistent with environment impedance Ke. This occurs when the demonstration and playback environments are identical. Dynamic Impedance Selection: A State-Space Approach It is useful to extend the notion of duality from static to dynamic systems, in order to determine the relationship between higher order manipulator and environment impedance terms. Since the concept of "inverse" no longer applies as it does in the static case, a new formulation is required. Position and force trajectory errors, x and fe are once again expressed in terms of desired position and force, as well as manipulator and environment impedance parameters: X{S) = Mms2 + (Bd + Be)s + (Kd + Ke)fo{s) BdS + I<d . . , . —x0{s) - x0(s) Mms2 + {Bd + Be)s + {Kd + Ke) 1 Ms) M m s 2 + (Bd + Be)s + {Kd + Ke) Mds2 + Bes + Ke Mms2 + (Bd + Be)s+ (Kd + Ke) x0(s) (3.22) Chapter 3. Variable Impedance Control for Task Reproduction 35 /«w Bes + Ke Mms2 + {Be + Bd)s+ (Ke + Kd) BdBes2 + (BeKd + KeBd)s + KeKd Mms2 + (Bd + Be)s + (A'd + Ke) Mms2 + fles + Kd ' Mds2 + {Be + Bd)s+ (Ke + tfa) M * j fidjBeS2 + (BeA'rf + KeBd)s + A'eA'd a;o(s) + /o(«) Mms2 + (Bd + 5e-)s + (Kd + Ke) x0(s) (3.23) A combined error measure is derived by summing the weighted error norms \\x\\2 and ||/e||2i where \m2^{~fjH(ju)\2du (3.24). This effectively computes the RMS error response' when the system is driven by white noise. E = W\\xU + \\fe\\2 (3.25) It is convenient to compute each 2-norm from a state-space system representation. Position and force error relationships are written as two dual-input, single-output functions, as shown in Figure 3.6, and expressed in controller canonical state-space form. V l A3,c: l " l w l rVlDnV^O 2L '2v- '2 -»• X f Figure 3.6: Dual-input, single-output functions relating desired position and force with position and force errors. Chapter 3. Variable Impedance Control for Task Reproduction 36 This requires tha t each input-output transfer function be strictly proper, which is achieved by applying a first-order filter to both desired position and force signals (3.26). 27r/3d£ / o ( * ) • = x'0{s) = s + 27rf3dB ^hdB Ms) x0(s) (3.26) s + 2nf3dB Input feedthrough terms (ABCTD state-space model) are avoided in order to prevent an error-bandwidth which is unbounded. The minimisation of an error E which is driven by non-bandlimited inputs would demand a non-bandlimited manipulator impedance. This is not physically plausible, therefore it is intuitive to low-pass filter desired position and force inputs. Both x and / e can then be expressed as two-input, single-output systems, each consisting of two linear transfer functions. These are of the form: x0(s) y(s) b\s2 +b\s+bl bjs2+bls+bl s 3 + a l s 2 + a l s + a l s 3 + a 2 s 2 + a 2 s + a 2 Ms) (3.27) where y(s) represents either x or fe. Each of the position and force error relationships are expressed in State-Space form: x = Ax + Bu y = CTx , (3.28) by means of the controller canonical form, in which matrices A, B and C are expressed as follows: A = B = C = -a\ 1 0 0 0 0 . 1 0 0 0 _ b\ b\ -a\ 0 1 0 0 0 0 0 0 1 b\ -a\ 0 0 0 -0 0 0 0 0 0 b\ b\ 0 0 0 -a\ 1 0 T bl] 0 0 0 - « 2 0 1 T 0 0 0 -a 0 0 (3.29) (3.30) (3.31) Chapter 3. Variable Impedance Control for Task Reproduction 37 Given such a state-space representation, the system 2-nprm can be computed [9]: \\H\\l = Tr(BT neATtCTCeAtdtB\ (3.32) The dynamic inverse relationship, constituting duality in the general sense, is computed by min-imising the combined error norm \\E\\2, with respect to manipulator and environment impedances. A Newton-based gradient search algorithm, available within the MatLab Optimisation Toolbox, has been selected for this purpose, the results of which are shown in Figures 3.7, 3.8, 3.9 and 3.10. Each graph illustrates the relationship between "ideal" manipulator damping Bm and machine mass Mm, environment damping Be, environment stiffness Ke or desired trajectory bandwidth. fzdB as these parameters are varied over ranges which may be typical during excavation. In this analysis, the error norm E is minimised with respect to only machine damping Bm-Minimisation with respect to manipulator mass Mm tends to result in poor convergence or near zero mass. For this reason the machine mass term should be chosen manually as the lowest physically realisable value and not left as a minimisation parameter. In addition,-machine stiffness Km has been selected based upon static conditions, as discussed previously in Equation 3.21 and is not included as a parameter for the minimisation of E. In general the error norm can be minimised with respect to Mm, Bm and Km, however physical constraints as well as static performance have been considered here. "Optimal" manipulator damping decreases monotonically with both environment stiffness and damping, in accordance with the inverse nature of duality already experienced during the analysis of static impedances. Increased machine mass Mm demands higher machine damping, while manipulator impedance selection is particularly sensitive to the bandwidths of desired force and position commands, /o and Xo, which are determined by the.first order filters described in Equation 3.26. Both desired force and position filters have identical bandwidths. The manipulator 's ability to track force and position setpoints is dependent upon the rate of change, or equivalently the bandwidth, of these signals. A 2-norm is computed using white noise input sources with infinite bandwidth. Therefore without suitable input filtering, the minimisation of error E is dominated by bandwidth requirements. This Chapter 3. Variable Impedance Control for Task Reproduction 38 6000 5000 -. 4000 | 3000 m 2000 1000 0 | | | | k M = 960kg 8 =S00Nm"'s 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 X104 0.5 1 1.5 Figure 3.7: The dual relationship between manipulator damping and environment stiffness, first over a range of manipulator mass with Be = 500Nm_1s and then over a range of environment damping with Mm = 110kg. In both cases f3ciB = 20Hz. is evident in Figure 3.10, which indicates a drop in manipulator damping with increased input bandwidth. Lower damping results in increased manipulator bandwidth in response to the wider command trajectory bandwidth. The result obtained under static conditions can be shifted to any fixed frequency, u:-W \juZm\ -I Mm (JLJ)2 + Bm (ju) + Km | = \j"Ze W (3.33) (3.34) \Me{jw)2 + Be(jv) + Ke\ Therefore, the relationship between environment and machine impedances is dependent upon fre-quency. By employing a 2-norm, as shown in Equation 3.25, the total energy of error E is minimised over a bandwidth determined by input filters (3.26) and is thus not optimal at all frequencies within this bandwidth. Chapter 3. Variable Impedance Control for Task Reproduction 39 50 100 150 200 250 300 350 400 450 500 550 600 Figure 3.8: The dual relationship between manipulator damping and environment damping, first over a range of manipulator mass with Ke = 39000Nm_1 and then over a range of environment damping with Mm = 110kg. In both cases fodB = 20Hz. From Equations (3.22) and (3.23), the resulting coupled system consisting of second order ma-nipulator dynamics and first order environment dynamics has the characteristic equation: Mms2 + (Be + Bd)s + {Ke + Kd) = 0 (3.35) If the manipulator is stable and the environment exhibits non-negative damping and stiffness characteristics, then the coupled system is guaranteed stable. In practice, where higher order and non-linear terms may be present in both the manipulator dynamics and environment, care should be taken to ensure stability of the coupled system. In general, this analysis is not restricted to first order environment and manipulator dynamics, although the resulting relationships and interactions between higher order manipulator and envi-ronment terms become extremely complex and costly to compute. A worthwhile extension to the impedance models considered here, may be: {M0s2 + Bos)x(s) = f0(s) + fe(s)-{Mms2 + Bms + Km){x(s)-x0{s)) fe(s) = -{Mes2 + Bes + Ke)x{s) , (3.36) Chapter 3. Variable Impedance Control for Task Reproduction 40 100 200 300 400 500 600 700 800 900 u ! : K'i'lSOOONm"' : -1000' ' ' ' ' 1 ' ' 1 ' 100 200 300 400 500 600 700 800 900 Mm[kg] Figure 3.9: The dual relationship between manipulator damping and environment mass, first over a range of environment damping with Ke = 39000Nm-1 and then over a range of. environment stiffness with Be = 500Nm -1s. In both cases f3(iB = 20Hz. where Mo and Bo are fixed manipulator mass and damping factors which cannot be eliminated due to physical constraints, while Mm, Bm and Km constitute desired manipulator impedance parameters which may be adjusted through appropriate control. 3.3.2 Position-Force Error Weighting The selection of a weighting factor which trades-off position and force tracking errors may be guided by a number of factors which are dependent upon the task at hand. A simple interpretation of this weighting is as the ratio between force and displacement resolution. For example, if a force tracking error of IN is as significant as a position tracking error of 1mm, then a weighting, W, of 10002 = 1000000 may be appropriate. This resolution is clearly task dependent and will differ markedly between, for example, ground excavation and surgical applications. In cases where there are environment impedance discontinuities, such as at a hard unilateral contact surface, it may be desirable to relax position tracking constraints in order to avoid excessive Chapter 3. Variable Impedance Control for Task Reproduction 41 0 10 20 30 40 50 60 70 80 90 100 Figure 3.10: The dual relationship between manipulator damping and desired trajectory bandwidth, with Mm = 100kg, Be = 500Nm -1s and. Ke = lOOOONm-1. contact force. This is similar in concept to a guarded move [34], where manipulator velocity is decreased in anticipation of impending contact.. If the general location of such an interface is known prior to impact, then the weight W can be reduced for this purpose. Until now, only a scalar position-force error weighting has been considered, while dynamic adjustment is most likely to be more effective in complex tasks. By replacing the scalar weighting W with a suitable low-pass filter, position tracking can automatically be reduced during periods of rapid environment impedance fluctuation, as is evident during contact transitions. 3.3.3 Online Identification of Environment Impedance Manipulator impedance which is based upon duality requires some means of determining environ-ment impedance. This is a relatively simple task when this impedance is classified as either "soft" or "hard", inertial or capacitive. In the general case, where the manipulator is to respond to a continuously changing environment impedance, it is necessary to employ some form of environment impedance identification. At least two requirements are at issue in achieving such identification: Chapter 3. Variable Impedance Control for Task Reproduction 42 • Identification should occur quickly, such that the manipulator impedance can be adjusted in order to track changes in the environment. For instance, during the transition from a low impedance environment to a high impedance environment, it is necessary to decrease manipulator impedance before excessive contact forces develop. • Any identification algorithm should be capable of executing in real-time. During excavation, identification based upon standard algorithms such as least squares, is not capable of computing accurate environment impedance estimates rapidly enough for manipulator impedance matching. Therefore, a simplified impedance matching strategy which is loosely based upon the generalised duality principle discussed previously, has been developed for the purpose of excavator control [44]. 3.3.4 Force/Velocity Ratio for the Approximation of Environment Impedance A rough environment impedance estimate can be inferred from the relative magnitudes of instanta-neous interaction force and velocity [44]. Consider a manipulator impedance which can be controlled between a minimum stiffness, A"m, and a maximum stiffness, Km. Interpolation between these ex-tremes, based upon an effort/flow impedance relationship, is achieved by: a n d | / e | > fmin G\vm\ l/el = Rm +-Xm(Km — Km) and G\vm\ > fmin l/el G\vm\ -. 1 = Rm + —Am{Km — Km) u>m remains unchanged , (3.37) where G is a scale factor which may represent the magnitude of a nominal impedance, or the ratio of average expected environment force to average expected manipulator velocity (similar in concept U\fe\>G\vm\ Am If|/e|<GK| Am Else Chapter 3. Variable Impedance Control for Task Reproduction • 43 to position error weighting W). The threshold force, / m t r i is imposed in order to avoid ill-defined Am when both the velocity and force are small. When | / e | ^> G |u m | (contact with a high impedance environment), the manipulator will have a low impedance and will track force commands. When G|u m | 3> | / e | (free motion), then the manipulator will have a high impedance and will track position commands. At all intermediate environment impedances the manipulator impedance is interpolated in the spirit of the generalised duality principle, without complex environment impedance identification. This strategy enables only an impedance magnitude adjustment, which has been interpreted to produce a stiffness adjustment. Additional identification, capable of distinguishing between environment impedance components such as stiffness and damping, can be used to augment the manipulator impedance adjustment as described in Section 3.3.1. For the multi-dimensional case, this scheme can be applied along each coordinate of the manip-ulator space. This chapter has developed the means to limit position and exerted force trajectory errors of a robot end-effector interacting within an environment of uncertain mechanical impedance. This is achieved by varying the impedance of the robot end-effector according to the dual of environment impedance. Such variable impedance has been applied to a backhoe excavator located within a virtual excavation environment. Subsequent chapters describe the design and implementation of this environment, while Chapter 6 presents the results of task demonstration and playback experiments conducted therein. Chapter 4 An Excavation Simulator This chapter describes the design and implementation of a virtual excavation environment which allows an operator to perform digging tasks with a simulated backhoe excavator [14]. The operator interacts with a joystick, or haptic interface, in order to provide machine bucket-endpoint motion and force commands, and to experience force feedback generated by a ground interaction model. Both the virtual excavator and its environment are graphically rendered and viewed upon a computer screen. Simulator sub-systems are related as shown in Figure 4.11. „ visual state t g 9 B | jet&annn&ant** position & force _ ,_ 7 force Graphics Rendering Joystick Controller ground state excavator state position & force force > Excavator Dynamic position force-^ Ground Interaction Figure 4.11: Virtual excavation environment sub-systems. Position and force commands are generated by the action of the operator on a dynamic joystick, which is capable of exerting force feedback to the human hand. A task-space dynamic model of the excavator determines its motion as a function of position and force commands provided by the joystick controller, and.an environment force. Environment forces and torques are estimated, based upon virtual excavator motion and soil characteristics, by a ground interaction model which is described in Chapter 5.. Excavator and soil states, derived from the excavator dynamic model and ground interaction 44 Chapter 4. An Excavation Simulator 45 model, are used to render a graphical environment which serves as visual feedback. 4.1 Simulation Hardware Architecture The simulator system comprises three significant hardware components, namely a Silicon Graph-ics Iris workstation, a real-time VME system including a SPARCengine processor running the VxWorks™ operating system and a haptic interface, as arranged in Figure 4.12. Computational Visual Environment Contact Environment MagLev Joystick Figure 4.12: Excavation simulator system. complexity and real-time execution requirements for each simulator sub-system excluded the use of a single processor. The excavator dynamic model, environment interaction model, as well as the joystick control law, are computed on the SPARC processor, while the visual environment is rendered by an SGI workstation. Two haptic devices have been employed as an operator interface, a 6-DOF magnetically levitated (MagLev) joystick [47] and a twin pantograph device [49]. The Ma-gLev joystick is most suited for velocity control, while the twin pantograph device, with its larger workspace, has proved useful under position command control. The virtual environment state is communicated from the SPARC processor to the SGI for graphical rendering, through a serial port. The following section describes the machine end-point dynamic model which is based upon experiments undertaken on a real hydraulic mini-excavator. This model is computed by the SPARC Chapter 4. An Excavation Simulator 46 processor in real-time. 4.2 D y n a m i c Mini-Excavator Mode l A Takeuchi TB035 mini-excavator, shown in Figure 4.13, has served as the model for the virtual excavation environment. The specifications, as well as the instrumentation of this machine, are Figure 4.13: The instrumented Takeuchi TB035 mini-excavator. outlined in [52]. Bucket endpoint coordinates are conveniently described within the workspace by means of cylindrical coordinates {#i, r, z} and bucket angle a, as shown in Figure 4.14. Polar base-frame coordinates have been chosen due to the rotated planar action of the manipulator. Articulated boom, stick and bucket components are constrained to operate within a single plane, the orientation of which is determined by the cabin angle 0\, with respect to the base-frame. Bucket endpoint position is described by means of four independent parameters, namely cabin rotation &i, height z (from the ground), radial distance r (from the cabin rotation axis) and bucket orientation a (with respect to the horizontal). Angle of attack, otattack, is often used to describe bucket orientation, since it clearly indicates the angle of the cutting teeth. The relationship between Chapter 4. An Excavation Simulator Figure 4.14: The mini-excavator, its components and coordinate frames. angles a and aattack is constant, governed only by bucket geometry. Throughout the remainder of this thesis, angle a refers to the angle-of-attack aattack-General kinematic and dynamic characteristics of backhoe machines have been presented in [27], [57] and [28], while a specific description of the relationship between hydraulic actuator displace-ments, joint angles and four-DOF cylindrical workspace coordinates, is described in [52]. Using position-based impedance control, the bucket endpoint dynamics in the vertical plane of the arm can be shaped into a relationship that is reasonably well approximated by (4.38), as described in [45]. Mms2x(s) = f0(s) + fe(s) + (Bms + Km)(x0{s) - x(s)) (4.38) Mm, Bm and Km are desired mass, damping and stiffness matrices, x = [ r z a ] T and xo are the actual and desired bucket position and orientation, while /o and fe are the commanded and actual forces and torques exerted by the bucket on the environment, as shown in Figure 4.15. It has been shown in [45] and [46] and proven in practice that such impedance is indeed achievable with Mm equal to the excavator mass, and measured stiffness and damping parameters in the ranges shown in Table 4.1. These impedance parameters were utilised within the virtual environment as part of a discrete dynamic model based upon (4.38). The discretisation of this model is described in Appendix A. Here, XQ, /O and fe act as inputs to the model, which produces x, the actual Chapter 4. An Excavation Simulator 48 Figure 4.15: Machine impedance control with mass, stiffness and damping parameters. task-space position/orientation of the excavator bucket endpoint. The environment damping and stiffness parameters, also shown in Table 4.1, were measured during actual interaction normal to a typical ground surface, and were used to guide the scaling of ground interaction model components. The environment force / e , is dependent upon the environment state, bucket position and bucket velocity. Table 4.1: Experimentally determined excavator and environment impedance parameters. Parameter Excavator Environment M 100kg -B SOOONm^s 500Nm_1s K 104 - 105 Nm"1 > l O ^ m " 1 A signal flow diagram, detailing the interaction between the haptic device, excavating machine and environment model, is shown in Figure 4.16. The position command control scheme is outlined in this diagram. Both commanded position and force, xo and /o, are fed forward to the excavator, while the resultant position and force, x and / / , are fed back to the operator via a haptic device. Manipulator workspace limits are effected in joint space, as shown in Figure 4.16, resulting in a Chapter 4. An Excavation Simulator 49 Manipulator Workspace Limits Manipulator Dynamics f Kvs + Kp *&V -^-» Inverse Kinematics JF-4 L Direct Kinematics ^ Hf xj Hx * s Haptic Device Environment Figure 4.16: System signal flow and control between master and slave sub-systems. workspace boundary force, /workspace- Joint limits for the mini-excavator have been reported in [52] and are provided in Table 4.2. The resultant force fed back to the operator, via the haptic device, is Table 4.2: Mini-excavator joint limits. Joint Angle 02 03 04 Minimum - l ,085rad -2.6927rad -2.6843rad Maximum 1.2075rad -0.6102rad 0.5182rad the sum of this workspace force and the computed environment interaction force, fe. The excavator impedance can be adjusted as a function of the environment model, as discussed in [44]. Figure 4.17 further clarifies the real-time operation of the contact environment, by way of a high-level system flowchart. Commanded-position upon the joystick corresponds to desired ma-chine velocity and must therefore be integrated to compute desired machine end-point position. This rate control strategy is employed with the MagLev joystick. During the task demonstration phase, environment interaction forces and torques are fed back directly to the MagLev joystick as stiffness (in order to maintain system stability [41]), while during playback these forces are used in Chapter 4. An Excavation Simulator 50 conjunction with endpoint velocity to adjust the machine end-effector impedance. The following section outlines the design of a graphical environment which provides the operator with visual feedback. 4.3 Dynamic Visual Excavation Environment Visual feedback is delivered with an SGI workstation equipped with the VGX, 02 or Octane SE graphics engines. Both the excavator model and dynamic environment are displayed within a realistic, yet responsive visual environment. IRIS Performer™ [51], a graphics library built upon OpenGL, was used to implement a scene rendering pipeline complete with a graphical user interface. The primary design issues in the construction of this graphical environment were: • the modelling and animation of an articulated excavator manipulator and • the implementation of a dynamic, deformable digging environment. In designing and implementing an animated visual environment there must be a trade-off between realism and speed. For immersive graphical environments, the latter requirement is particularly important in order to prevent "simulator sickness", a symptom of the lack of causality between oper-ator commands and visual response [38], usually caused by latency. Since the operator's viewpoint is not continually changing during digging operations, we have elected to provide some detail, at the expense of motion speed and smoothness. Scale models of each of the five excavator components were drafted individually and assembled as a hierarchy of dynamic coordinate systems within the graphics scene tree, as shown in Figure 4.18. Visual models are added to the scene tree by means of IRIS Performer library functions. The excavator configuration, generated by the real-time system, is transmitted to the graphical environment, in terms of workspace coordinates, by a serial communications interface. Joint angles, computed using the excavator inverse kinematics, are used to orient each manipulator link with respect to the previous link frame, taking advantage of the coordinate system hierarchy provided through IRIS Performer's DCS objects. • i Chapter 4. An Excavation Simulator 51 ( Start ) Update Machine Impedance Compute New Machine Position Initialise Excavator and Ground States Initialise Haptic Device r Input Position and Force Applied to the Haptic Device Threshold, Limit and Scale Velocity and Force Setpoints Integrate Desired Velocity = Desired Position yes Machine Position = Desired Position ^ Inverse Kinematics Check Joint Limits Forward Kinematics Compute Workspace Limit Forces Compute Ground Interaction Forces and Torques Compute New Ground and Load States Set Haptic Interface Stiffness Proportional to Environment Force no Playback? yes Figure 4.17: Simulation system flowchart. Chapter 4. An Excavation Simulator 52 Figure 4.18: An exploded view of the excavator model components. The world environment consists of a ground plane which is capable of deformation during digging. As the excavator bucket removes soil from the ground, a trench is generated within this plane, complete with textured soil, clearly reflecting environment state to the operator. A polygonal mesh-based ground plane proved to be prohibitively time consuming to render. Because of this it was necessary to expand the trench by adding polygonal modules as required. A three polygon module, shown in Figure 4.19, is added as the bucket tip makes progress through the ground. Although this technique does produce a realistic looking trench, it does still require the generation Ground Coloured Mask Soil Textured Inner Face JL 3-Polygon Module / Bucket Trajectory Figure 4.19: The textured polygonal modules which are added while constructing an excavation trench. Chapter 4. An Excavation Simulator 5:5 of a hole within the ground plane. This has been implemented by rendering the trench graphics in front of the ground plane, while leaving the trench visible in spite of being submerged below ground level. This is achieved by clearing the depth buffer (z-buffer) after ground plane generation, but prior to trench rendering, effectively forcing the ground plane to the back of the scene. The outer faces of the trench are hidden by blending them into the ground plane using matching colouring, while the inner faces are appropriately textured. Additional features, such as a shadow and soil which falls to the ground as the bucket is relieved of its load, have been included in order to provide the operator with valuable depth cues, and to enhance visual feedback realism. Figure 4.20 shows both the graphical excavator and environment during digging. Figure 4.20: A screen snapshot of the visual environment. Chapter 4. An Excavation Simulator 54 A graphical-user-interface menu which allows the operator to select different viewpoints within the visual environment, has been included as shown in Figure 4.21. In addition to providing external and internal cabin views such as those shown in Figures 4.20 and 4.22, it is possible to select views which aid the operator when the bucket is not visible from the cabin due to visual obstructions. This is a feature which can be particularly useful in "augmented reality" applications. Figure 4.21: The graphical user interface menu. The flowchart shown in Figure 4.24 (page 61) outlines the graphics algorithm. 4.3.1 Environment Init ial isation There are three important environment initialisation stages which are required before graphics rendering may proceed. The first of these is the allocation and initialisation of memory resources which are to be shared between three parallel processes, namely Application, Traversal/Cull and Draw Processes. The application process's role, is to update the scene, which is stored as a tree of geometric elements, according to simulated dynamics and other available input such as ground state. During rendering, this tree is traversed in order to determine relative depth and occlusion of objects, as well as to disregard (Cull) objects which are not within the field-of-view. The draw Chapter 4. An Excavation Simulator 55 Figure 4.22: A view of the virtual environment from within the cabin. Chapter 4. An Excavation Simulator 56 process projects the 3-D scene onto the image plane (with perspective) and paints (fill, shade) the frame buffer. Since each of these processes make use of scene information and contents, it is necessary to ensure that these entities are shared. Once such shared memory allocation has taken place, a process fork occurs. During the second initialisation stage, models and geometric objects are added to the scene tree. The mini-excavator consists of five independent components, each modelled within its own coordinate frame, which are added to the scene tree as a hierarchy of Dynamic Coordinate Systems (DCSs). In this way, each manipulator link can be translated and rotated with respect to its immediate predecessor, rather than to the world/base frame. At this stage, a single graphics rendering pipeline and associated channel are created and linked to a new graphics window. The relationship between application, scene, rendering pipeline and channel, is shown in Figure 4.23. The software rendering pipeline contains Traversal/Cull and Draw Display Device Figure 4.23: Graphics algorithm flow, indicating scene updates, rendering pipeline and channels. processes, as described earlier. A Channel contains properties such as viewer position, orientation and field-of-view. A pipe renders the scene, as viewed by the Channel. In general, there may be multiple pipes and channels per pipe, for multiple views of the scene; however in this application a single pipe-channel configuration was sufficient. Once initialisation is complete, a serial communications process is spawned and the graphics simulation loop begins. Chapter 4. An Excavation Simulator 57 4.3.2 Communication between the Contact and Graphical Environments Because of hardware limitations and interface issues, it has been necessary to divide graphics and contact simulation environments over two independent computing platforms. Graphics rendering is achieved using an SGI workstation, while contact dynamics are computed on a SunSparc based real-time system. This architecture requires that there be communication between the SunSparc and SGI systems. Since the real-time dynamics simulation is not dependent upon graphics rendering, it was necessary to provide only unidirectional communication from the Sun to the SGI. A standard three wire serial link was implemented. A parallel communication process is spawned in order to decouple the graphics loop from the serial data acquisition loop. This alleviated problems due to sluggish serial ports and limited flow control. 4.3.3 Graphics Simulation and Rendering The main graphics simulation loop begins as soon as a communication sub-task has been successfully spawned. This is the heart of the visual environment in which the scene is updated in response to environment and excavator state (input from the dynamics simulation). An outline of the loop is shown in Figure 4.25 (on page 62) as it is executed on the SGI graphics workstation. Scene updates occur during the frame draw interval, during which traversal/cull and draw processes are busy with the next frame image: 1. Excavator state (bucket endpoint position and orientation), is updated using data received by the communications process. Access to this data is regulated by means of a semaphore. 2. If ground material has been displaced by the bucket, then the scene is updated to indicate a hole in the ground plane. 3. Soil is shown to fall out of the loaded bucket, if it is oriented appropriately. 4. Bucket height and ground contact are easier to judge with the aid of a shadow. Due to Chapter 4. An Excavation Simulator 58 rendering limitations, only a single rectangular shadow of the bucket has been implemented. At each iteration of the simulation loop, this partially transparent shadow polygon is positioned and sized according to the height and orientation of the bucket. The shadow swells with increased height from the ground plane, while its centre shifts as the bucket rotates. 5. Inverse Kinematics calculations are performed in order to obtain manipulator joint angles which are used to orient the hierarchy of dynamic coordinate frames, each containing a link model. 6. The user may adjust the visual viewpoint by means of a mouse and the GUI panel. Termination of the simulation loop is signalled by the user through the GUI panel, keyboard event handler, or by means of window controls. 4.3.4 Creating a Hole in the Ground Plane The primary purpose of the virtual environment implemented here is to provide realistic visual and force feedback during digging operations. It is therefore essential that trenches and holes be generated graphically during excavation. A preliminary solution to the terrain deformation problem was to represent the ground plane as a regular rectangular mesh whose vertices could be positioned independently. In this way, a hole could be generated by simply displacing vertices vertically, according to the bucket trajectory. Unfortunately, the large number of polygons required to suitably model such terrain proved computationally prohibitive. An alternative strategy is indicated in the Figure 4.19. Here, a hole is constructed by adding polygonal modules, as the bucket cuts through the soil. In this way, graphics elements are added only as required, eliminating unnecessary vertices which lead to increased computation within the rendering pipeline. While this strategy leads to efficient trench generation, it is still necessary to create an opening in the ground plane, in order to expose the hole. A simple, yet non-ideal solution to this problem is to shade the outside of the polygonal trench using the same colour as that of the ground plane, while the inside is covered by an appropriate soil texture. In addition to this it is necessary to clear the Z-buffer after drawing the ground plane, but prior to drawing trenches. This ensures that the ground does not occlude Chapter 4. An Excavation Simulator 59 any holes. The double-sided trench has been implemented by employing superimposed pairs of polygons, rendered using both front and back-face culling. One criticism of this technique is the fact that the ground plane has no depth after the Z-buffer has been cleared and is thus no longer capable of occluding any other submerged entities. Soil volume excavated from the ground is shown to be collected within the bucket by means of a triangular volume which grows as displaced volume increases. Each of the dynamically evolving graphics elements, including the bucket shadow, soil volume and trench, are added to the scene tree as Geometry Objects, or Geodes [51], each containing multiple geometry sets endowed with specific properties such as colour and texture. Since these objects can be created and manipulated dynamically, they are ideal for environment modelling during excavation. 4.3.5 Animating Falling Soil This feature has been added due to overwhelming demand by persons who have operated the sim-ulator! Previously, excavated ground material would remain stuck within the bucket, regardless of its orientation angle. This turned out to be a source of great concern for operators, who resorted to a vigorous bucket shaking strategy in an attempt to dislodge stubborn soil. Once the orientation angle of a soil laden bucket exceeds —67.5° , a short animated sequence begins, showing the soil as it falls from the bucket to the ground. The falling soil volume is represented by a diamond-shaped, soil textured solid. As it falls onto the ground, the diamond shape is deformed until it settles as a triangular heap. The free-falling soil is accelerated toward the ground, in order to provide a sense of realistic motion. This chapter has presented the realisation of a virtual excavation environment which will serve as an experimental platform for the testing of digging task demonstration and playback. In addition to this, such an environment may be suited to other applications including: • operator training within an excavator simulator, • operator feedback during teleoperation, . Chapter 4. An Excavation Simulator 60 • the evaluation of machine interfaces and controls within a flexible virtual environment, and • augmented reality. The following chapter describes a new bucket-ground interaction model which has been developed in order to estimate interaction forces and torques fe in real-time. Chapter 4: An Excavation Simulator 61 C START (Graphical Environment) D Initialise Performer Libraries. Allocate and Initialise Shared Memory. Process FORK L Application (Scene Update) T: Traversal Cull 1 Pipeline i Draw Initialise Scene Tree: Initialise Graphics Rendering Pipe & Application Window. Create and Initialise a Channel on the Pipe. Initialise GUI Panel & Input Event Handler. Spawn Child Process Simulation Loop IX Serial Communications: Receive and Store Excavator State from Sun 1E. I Free CPUs. Terminate Input Event Handler. Free Semaphores & Kill the Communications Process. Terminate Parallel Processes (APP, CULL, DRAW). C STOP " > Figure 4.24: System flowchart outlining the graphics algorithm, implemented using SGI Performer Chapter 4. An Excavation Simulator (START ^ \ ^ • (Simulation Loop) _y Synchronise (Wait for next frame) Latency Dependent Updates * Update Light Source & Viewpoint Positions Initiate Frame Draw T Latency Independent Updates Update Excavator State Variables. Release Semaphore. Update the Bucket Shadow. Compute Inverse Kinematics. Translate and Rotate DCS components. Compute Viewpoint using the latest Mouse Input. Update GUI. Expand Hole Drop Soil Figure 4.25: A flowchart showing the contents of the main graphics simulation loop Chapter 5 Bucket-Ground Interaction Force Model In simulating, controlling, or automating the backhoe digging action, it is clearly necessary to employ a model to describe ground interaction forces. Such a model cannot completely describe digging interaction because of the non-homogeneity of soil and other ground materials. Changes in soil density, texture and moisture make such modelling extremely difficult. .Because of this only a few simple interaction models, based upon the assumption of soil homogeneity, have been developed. Their purpose has been to create qualitative models which are able to predict digging forces within an order-of-magnitude under controlled conditions. Dombrovskii's equation (given in [1]^ describes static draught force as: Draught Resistance = kbh +/J,N•+ e(l + qnp)qkn , (5.39) where b and h are the width and thickness of the soil slice, k is the cutting resistance factor, fi is the coefficient of friction between bucket and soil, TV is the downward pressure of the bucket upon the soil, e is the soil-soil friction coefficient, qnp is the ratio of soil prism volume (the mound of soil travelling in front of the cutting edge), to the bucket volume, q is the bucket volume (capacity), 63 Chapter 5. Bucket-Ground Interaction Force Model 64 kn is the ratio of soil in the bucket to bucket volume. The Fundamental Equation of Earthmoving (Reece, 1964 from. [48]) is similar in form to Dom-brovskii's equation: Draught Resistance = (pgd2Np + cdNc + qdNq)w , (5.40) where . • p is the density of the ground material, g is acceleration due to gravity, d is cutting depth, c is the cohesion coefficient of soil, q is the surcharge pressure acting on the soil surface, w is the tool width. The coefficients Np, Nc and Nq are parameters which are a function of both soil and bucket char-acteristics. These parameters, as well as those within Dombrovskii's equation have been determined empirically. Alekseeva et al. used such equations to evaluate the effects of bucket geometry and orientation upon digging efficiency. Referring to Figure 5.27, Alekseeva et al. claim tha t "efficient digging" is realised when: • a is no less than 3° - 7°, • j3 is greater than 20° - 22° in moist soils and greater than 22° - 25° for heavy, stony soils, • 8 is at least 28° - 30°, in general 8 ranges between 25° and 55° during efficient digging operations, • tooth height a is no more than 40% to 50% of the soil slice thickness, h. Chapter 5. Bucket-Ground Interaction Force Model 65 1 h Figure 5.26: The parameters describing tooth geometry and orientation. Other experiments showed the effects of tooth geometry. A narrower tooth requires less penetration force and was found to generally reduce draught force (horizontal force), while the optimal tooth spacing to tooth width ratio, was found to be approximately 2.5. A longer tooth increases cutting resistance, as well as depth, while allowing more soil to slip between the teeth. Numerous plots of model parameters have been produced, based upon experiments performed under efficient digging conditions with constant angles-of-attack and digging depths. Such results are unsuitable for the real-time estimation of digging forces, either for an outdoor application or within a simulated environment. This section describes a new bucket-ground interaction model which is capable of estimating dynamic forces and torques over arbitrary digging trajectories in real-time. Simulation results are provided, as well as ideas for model identification, extension and refinement. 5.1 Modelling Parameters and Assumptions A flat cutting surface of length socket, which penetrates the ground surface by sp(r,z,ot) metres, is assumed. This penetration measurement depends upon bucket endpoint depth, orientation and the existing ground surface profile, Fu(r), as shown in Figure 5.27. The cutting edge is located at bucket endpoint position [r z]T, while the cutting surface is inclined at angle a with respect to the fixed task frame. Planar motion of the bucket endpoint can be expressed using vertical and Chapter 5. Bucket-Ground Interaction Force Model 66 Figure 5.27: The cutting edge frame and dimensions. radial velocities, vz and vr, as well as an angular velocity, ua, which are computed by numerical differentiation. A ground interaction-force model has been developed, which is based upon viscous friction, Coulomb friction, stiffness and load forces. Some of these components have been employed and verified in other models [1, 6, 34], but are generalised here by considering arbitrary bucket angles and trajectories. Three of the four force components are computed relative to the bucket coordinate frame, which consists of normal, tangential, and orientation components, as shown in Figure 5.27. Each interaction component, its origin and implementation, is discussed in the following sections. 5.2 Interaction Model Force Components Four fundamental forces have been modelled with respect to both bucket and soil states. These include: • viscous friction or damping • soil stiffness and shearing forces • load accumulation forces Chapter 5. Bucket-Ground Interaction Force Model 67 • Coulomb friction, each of which will be described in detail 5.2.1 Viscous Friction Viscous friction between cutting and soil surfaces, due in part to adhesive properties between bucket and ground materials, results in a force which opposes motion (5.41, 5.42), as shown in Figure 5.28. I tt \ ^p^-> Figure 5.28: Cutting resistance due to viscous friction. fbr fbz n -R(a)B(r,z,sp)R(ay B(r,z,sp) = (5.41) (5.42) (5.43) = -sp(r,z,a) ba0a , sp(r, z, a)bto 0 0 sp(r, z,a)bn0 The damping constants are a product of material-dependent nominal damping factors, bto and bno, as well as contact surface area which is assumed to be proportional to the penetration depth, sp. Forces are computed with respect to a bucket-oriented coordinate frame with axes tangent and normal to the cutting surface. A rotation matrix, -R(cc), describes the transformation between this bucket frame and an inertial world coordinate frame. Chapter 5. Bucket-Ground Interaction Force Model 68 Damping forces are expected to be greatest in the tangential bucket direction, since both the cutting surface and bucket sides contribute to viscous friction. Resistance to motion in the normal direction results from side and edge contacts. In addition, both normal forces and orientation torques may include significant damping due to environment compression. 5.2.2 Soil Stiffness and Shearing A stiffness force term (5.44) is computed relative to a stiffness centre, {rc, zc, ac}: fkt Jkn RT{a)k r,- r -kQ(a- ac) , (5.44) where k and ka are translational and angular stiffness factors. Both Bernold [6] and Malaguti [34] have considered the addition of soil shear failure, attributable to the breaking of cohesive bonds within the ground, as shown in Figure 5.29. Cssx Failure. Surface Figure 5.29: Stiffness force components which lead to soil shearing. We can consider the cutting action to be a series of soil body failures. As the cutting edge moves through the ground, soil displacement and the accumulation of soil stiffness forces result in simultaneous shearing, or failure along a surface within the earth, caused by the breaking of cohesive Chapter 5. Bucket-Ground Interaction Force Model 69 bonds between soil particles. The shape of this passive soil failure surface is described by Terzaghi (see [34]) as a logarithmic spiral, the angle of which depends upon the soil's internal angle of friction, cutting edge form and orientation. These logarithmic spiral shearing surfaces are approximated by planar surfaces in the Wedge Theory of Passive Soil Failure, as described in [34] (see Figure 5.30). The angle that this approximated planar surface makes with the cutting surface is dependent upon cutting edge form and soil properties. Figure 5.30: Approximated planar failure surface. A further simplification is made here by considering a planar failure surface which develops perpendicular to the cutting blade and is located between the teeth endpoints and the ground surface, as shown in Figure 5.29. Soil failure can be implemented by maintaining an estimate of the shearing force threshold, Swdp, where w is the bucket width, 5 is a soil characteristic and dp is the distance from bucket tip to soil surface along the normal to the cutting surface. Shearing occurs as soon as the opposing normal stiffness force equals or exceeds the current shearing force threshold, that is: If fkn > Swdp, then (5.45) At this time, soil stiffness is relieved and the centre of stiffness is reset to the current bucket endpoint position. The ground surface profile Fu(r), shown in Figure 5.32, is also updated to reflect the change in soil surface dimensions. It is noted that this model component is most plausible for dry, non-compressible materials. Wet soil, for instance, would tend to compress and deform rather than to fail, or shear. For this reason, a lagging stiffness centre is used to simulate soil relaxation Chapter 5. Bucket-Ground Interaction Force Model 70 and is implemented by means of a first order low pass filter, as shown in Equation (5.46). = RT{a) rc(s) zc(s) l TrtS+l 0 TrnS+1 R{a) r{s) z(s) (5.46) Independent first order soil relaxation time constants, Trt and Trn are maintained for motion tangential and normal to the cutting surface. Parameter Ts is the simulation sample period. This ensures that stiffness forces result from cutting surface motion integrated over finite time. In [34] the distance between subsequent soil failures has been shown to be particularly sensitive to cutting edge depth and is implemented by modulating a soil cohesion factor by means of a depth dependent sawtooth function. By contrast in Equation (5.45), bucket position and orientation (including depth), as well as soil state, account for periodic soil failures in a clear, physically intuitive manner. 5.2.3 Accumulated Load Load material which accumulates within the bucket during digging results in two force components, namely load weight and accumulation force (see Figure 5.31). The latter results primarily from churning effects of soil caught within the bucket [1], resulting in increased draught resistance as shown in Equation (5.47). Figure 5.31: Load forces due to the accumulation of ground material in the bucket. Chapter 5. Bucket-Ground Interaction Force Model 71 € • Vload -p-9 • Vload _ Tic = ^Sbucket {flz cos a - fir sin a) (5.47) Here, p represents soil density, — g is acceleration due to gravity and Vioad is the volume of soil accumulated within the bucket, computed according to (5.48): Vioadit) = / w(Fu(r) - z{t))dr , (5.48) Jr(t) where {Fu(r) - z(t)) is the bucket endpoint depth below the ground surface at time t. This surface profile is updated as the bucket moves through the earth, accumulating material, as shown in Figure 5.32. The soil accumulation force is modelled as being proportional to bucket soil volume, with e r(t) rmax Figure 5.32: Ground surface dimensions are updated and stored in Fu(r). being a bucket geometry characteristic. The orientation torque Tia is computed from the resulting normal load force component which is assumed to act at the centre of the cutting edge, Sbucket • 5.2.4 Coulomb Friction Soil-bucket contact force normal to the cutting surface results in a resistive force which opposes tangential bucket motion, as shown in Figure 5.33. This force has been modelled as Coulomb friction (5.49): fir flz Chapter 5. Bucket-Ground Interaction Force Model 72 Figure 5.33: Tangential cutting resistance due to Coulomb friction. fct = -^TL\\hn + fkn + fln\ = ~/*T—r|/n | , ( 5 - 4 9) H M where fi is the coefficient of friction between soil and bucket and vt is the tangential bucket velocity. 5.2.5 Force Component Parameter Selection Each of the four bucket-ground interaction forces presented in this section includes one or more factors which are determined by both bucket and soil characteristics such as: • soil texture and consistency, • soil moisture content, • soil density, • bucket geometry, and • cutting surface and tooth geometry. For instance shearing, traditionally a feature of metallic and crystalline solids, is most applicable to dryer, more brittle ground material, while a moist, fine soil would present more viscous properties. Softer ground is also more deformable, with less tendency to shear along well defined failure planes. The relaxation of soil stiffness centres simulates this type of behaviour. Chapter 5. Bucket-Ground Interaction Force Model 73 Estimated bucket-ground interaction forces, based upon a range of different soil properties, are presented in Section 5.3. 5.3 Simulation Results The estimated ground interaction forces shown in Figures 5.35 and 5.36 were generated for the bucket trajectory illustrated in Figure 5.34. The bucket endpoint follows a circular path with a constant cutting surface angle with respect to the direction of motion. Bucket endpoint velocity magnitude remains constant throughout this trajectory. Radial Position, R [m] Figure 5.34: Simulated digging trajectory Both vertical and radial interaction forces are shown for two different force models, corresponding to distinct soil consistencies. The simulation results illustrated in Figure 5.35 were computed using high stiffness coefficients and shearing thresholds with little stiffness-centre relaxation, corresponding to hard, brittle ground material. The effects of increased stiffness relaxation are shown in Figure 5.36. Chapter 5. Bucket-Ground Interaction Force Model ,74 10000 _ 8000 ' z g- 6000-£ g 4000-01 2000 -Radial and Vertical Digging Forces :::z 2000 -I 0-§ -2000 -u. 15 e -4000 --6000 -10 15 Time [s] Figure 5.35: Estimated forces during interaction with stiff soil. Ground interaction forces generated during operator control within the virtual environment are shown in Figure 6.41. 5.4 Force Model Extensions A number of refinements follow directly from the interaction model described in this section, namely: • the integration of model forces and torques over arbitrary cutting surface areas corresponding to different bucket geometries, • the computation of soil load inertia, and • soil-soil friction along a failure surface. Interaction forces and torques presented thus far have been computed with respect to a single point, the bucket endpoint, or teeth. In reality, interaction forces are experienced throughout the cutting surface and are dependent upon bucket rotation and local soil properties. For this reason we Chapter 5. Bucket-Ground Interaction Force Model 75 Radial and Vertical Digging Forces (Lagging Stiffness Centre) £. o -I -2000 -u. I -4000 -10 15 Time [s] Figure 5.36: Estimated digging forces with a soil relaxation factor. can integrate the local interaction of infinitesimal cutting surface areas with the ground, as shown in Figure 5.37. f(X,t) Figure 5.37: Force integration over arbitrary bucket geometries. Consider a patch of the bucket surface dS in contact with the environment. The patch has coordinates X = [rp zp]T with respect to the fixed task frame and is moving with velocity v(X, t) at time t. Patch velocity is calculated from the linear velocity of the bucket vertex and angular bucket Chapter 5. Bucket-Ground Interaction Force Model 76 velocity d: v(X, t) = v(X0,t) + a(t) x{X- X0)(t) , . (5.50) where Xo is the position of the cutting surface endpoint, or origin. Viscous, Stiffness and Coulomb forces act upon this surface: nx,t) = fbr fbz + fkr fkz + Jcr Jcz (5.51) The resultant force and torque acting upon the bucket can then be computed by integrating over the contact surface S(t) at time t: fs(t) = [ f(X,t)dX Js(t) rs(t) = [ [(X-X0)(t)xf(X,t)]dX Jstt) (5.52) (5.53) Model parameters used in computing local force f(X, t) may be varied over the cutting surface in cases where the soil is non-homogeneous. This also takes into account any uneven ground surface which may have been partially excavated, resulting in varying penetration depths across the width of the cutting surface. By simply adjusting the integral surface described above, it is possible to compute the interaction of arbitrary bucket geometries, as shown in Figure 5.37. Inertial force resulting from the acceleration of load material as it moves up the cutting surface soon after shearing occurs, as well as cohesive friction along a soil failure surface, can also be included as illustrated in Figure 5.38. 5.5 Model Validation Strategies Quantitative model validation refers to the numerical comparison of simulated and measured force data, while qualitative validation concerns itself with high-level comparisons of functional relation-ships. Ideas for both qualitative and quantitative model validation are discussed in this section, including: Chapter 5. Bucket-Ground Interaction Force Model 77 Figure 5.38: Bucket load inertia. • model parameter estimation and cross validation using measured interaction force data, • verbal feedback from experienced operators, and • operator evaluation and ranking within a simulated environment. 5.5.1 Model Parameter Estimation The interaction force model could be validated by attempting to compute model parameters which accurately describe real, measured digging data. A first step toward such validation is to examine model parameter identifiability, as illustrated in Figure 5.39. Model identifiability is established if its soil parameters can be identified solely from simulated output force/torque data. Quantitative validation may then be performed by employing the same identification strategy with real, measured ground interaction forces and torques. Consider the force/torque vector F computed by a function / which takes model parameters 0, bucket position and orientation X(t), and model states S(t) as input: F = f(d,X(t),6(t)). (5.54) Parameter identification proceeds by attempting to minimise an estimation error, with respect to estimated model parameters: min f \f(60,X(t),6(t))- f(6,X(t),5(t))}2dt , (5.55) 6 JTJ, Chapter 5. Bucket-Ground Interaction Force Model 78 Soil PaVpmeters [ba bM u E | p * T„ Tm ] I V r Interaction Model Parameter Identifier Bucket & Soil States 1 ~»n \ , J rV J* 1 Simulated Environment Force/Torque Estimation Error Figure 5.39: Digging model parameters are selected in order to describe real measured force/torque data. where 6Q and 6 are the nominal and estimated model parameter vectors. The error minimisation is performed over the time interval T\ ^ t ^ T^.' Following the bucket-ground interaction model presented within this chapter, bucket tangential and normal forces are computed by: ft Jn (0 = fbt + fct + fit + fkt Jbn i Jin T" Jkn (*) (5.56) By neglecting soil shearing effects, or considering only soft, viscous soils, this model can be expressed as: Jn (s) = -btospxt(s)s - A*|f^ fj§r + ecos{a)Vioad - psin{a)gVload - ^ ' f f " -bn0spxn(s)s + esin(a)Vioad - pcos(a)gV!oad - kTf*^s (5.57) where xt and xn are bucket endpoint coordinates expressed in the bucket frame (as shown in Equation 5.58), and the Laplace operator is used for notational convenience. xt RT{a) . (5.58) Here the stiffness force component, with a lagging centre of stiffness, has been derived using Equations 5.46 and 5.44. From Equation 5.46, bucket displacement from the stiffness centre can be Chapter 5. Bucket-Ground Interaction Force Model 79 written as: rc- r _ zc- z _ («) = ' R(a) » Trts+l 0 0 1 Trns+1 RT(a) -1 0 _ 0 1 _ 1  > > r z 00 By substituting Equation 5.59 into the translational stiffness Equation 5.44, this becomes: fkt Jkn (s) = RT(a)k rr — r Zr — Z («) = k I TrtS+l 0 0 1 Tts+1 0 TrnS+1 0 fcT„ T„s+1 . RT(a) - i2T(a) 1 0 0 1 (5.59) (s) (5.60) (5.61) The resulting model shown in Equation 5.57 is linear in parameters and therefore amenable to parameter identification by means of least squares OT recursive least squares [50]. Table 5.3 and Figure 5.40 show the results of such identification using 1800 samples of simulated environment interaction forces generated by the typical bucket trajectory shown in Figure 5.34, over a time interval of 18 seconds. All model force components were included, with the exception of soil shearing. Table 5.3: Ground-interaction-model parameter identification using batch least squares. Model Parameter ho bno M e P k .. Tt Tn Simulated Value SOOOONm^s 10000Nm_1s 0.9 lOOOONm-3 2000Nm-4s-'2 lOOOOONm-1 0.100s 0.500s Estimated Value 30000Nm-1s 10000Nm-1s 0.9 lOOOONm-3 2000Nm-4s-2 102020Nm-1 0.095s 0.495s Chapter 5. Bucket-Ground Interaction Force Model 80 I 5000 ;• ° E Z-5000 Q. -10000 3 2 n vA.J : ...li: i \l ! iiii 500 1000 1500 Sample {Fs = 100Hz) Z 5000 c m 10000 2 5000 CO 0 0.6 0.4 0.2 0 ( 12 — 10 'E 6 ) , 500 1000 1500 X10 ; : : rr 500 1000 1500 Sample (F =100Hz) Figure 5.40: Recursive-least-squares model parameter identification, neglecting shearing effects. Parameters 6^ 0) bno, /j., e and p converge to the actual simulated values, while parameters k, Trt and Trn converge to within 5% of their nominal values. The estimation errors for these parameters may be attributable to: • a simulated bucket trajectory which does not result in sufficient excitation of all model com-ponents, or • coupling between model parameters. The non-linear nature of the soil shearing phenomenon, included within the bucket-soil interac-tion model presented in this chapter, creates complications for the estimation of its parameters from real or synthetic data. In particular, there is no guarantee that the linear least squares algorithm will work. Other parameter estimation schemes based upon the minimisation shown in Equation 5.55, including gradient-search based methods, may be useful in identifying all model parameters Chapter 5. Bucket-Ground Interaction Force Model 81 in the presence of hard soils which exhibit shearing effects during digging. Preliminary tests using Matlab's Newton based gradient search algorithm have.not been successful. Data pre-processing may aid parameter identification in the presence of discontinuous soil failure or shearing, however these techniques are the subject of future work. 5.5.2 Operator Performance within a Virtual Environment Feedback from human perception of bucket-ground interaction forces within a simulated environ-ment provides a qualitative means of model validation. Despite the lack of direct force feedback by modern machine controls, an experienced operator is aware of the relationships between digging action and interaction forces. Hydraulic pressures displayed within the cabin, engine pitch and cabin motion provide an indirect means of force sensing. Therefore, the operator's verbal response to simulated digging forces provides,some insight into model validity. A second qualitative validation technique involves the use of multiple human operators of varying levels of proficiency. If an operator ranking is known, based upon a set of performance criteria and results obtained within the field, then validation is achieved by obtaining the same ranking from their performance within a simulated digging environment. The challenge here is to refine a set of performance criteria which is both meaningful and accessible within both real and simulated environments. This chapter has described the design and implementation of a bucket-ground interaction model which will provide force feedback during digging task demonstration and playback. It has been shown that the "linear-in-parameters" component of the model is readily identifiable using least-squares. Extensions to include the non-linear or shearing component will be required in the future. The following chapter describes the human demonstration and machine playback of a single digging motion within the virtual excavation environment. Chapter 6 Trajectory Programming and Playback within the Virtual Environment The virtual excavation environment described in previous chapters has served as a test-bed for the trajectory demonstration and playback scheme described in Chapter 3. This multi-degree-of-freedom contact environment exhibits a large range of impedances dependent upon digging trajec-tory, therefore necessitating a strategy.for the demonstration and successful playback of excavation tasks. The non-homogeneous nature of ground material often results in marked differences be-tween demonstration and playback environment characteristics, resulting in the trajectory tracking problem described in Section 3.1. During task demonstration, the operator commands a simple digging motion within the virtual excavation environment described in Chapters 4 and 5. Manipulation of the MagLev joystick results in velocity commands which drive the bucket endpoint, while simulated bucket-ground interaction forces are fed back to the human operator's hand as joystick stiffness. The visual environment is rendered graphically and observed upon a computer monitor. Transparency between the joystick and digging environments is not considered during the demonstration phase. It is assumed that the human operator compensates for any mismatches, in order to achieve the desired task objective. Bucket endpoint position and force trajectories are stored during operator demonstration for subsequent playback. This chapter presents the results of trajectory playback within an environment which differs significantly from the demonstration environment. The velocity/force ratio based variable impedance control scheme, presented in Section 3.3.4, is employed for this purpose. 82' Chapter 6. Trajectory Programming and Playback within the Virtual Environment 83 6.1 Environment and Machine Parameters In order to demonstrate the trajectory replication problem within dissimilar environments, and the manner in which this problem has been addressed, two markedly different environments have been selected. The demonstration environment is relatively soft and viscous, exhibiting no stiffness or shearing characteristics. In contrast, the playback environment is hard, with pronounced shear-ing and significantly higher viscosity. The environment parameters are shown in Table 6.4, while interaction forces experienced during a typical digging trajectory are shown in Figure 6.41. Excavator control has been facilitated by means of the machine dynamic model described in Equation 4.38, using the impedance parameter ranges presented in Table 6.5. Table 6.4: Demonstration and playback environment parameters. Parameter bto bno k Jshear e P M Trt T J-m Demonstration 40000Nm-1s 30000Nm_1s ONm"1 ONm,-2 lOOOONm-3 1000Nm-4s-'2 0.9 0.1s 0.5s Playback 80000Nm_1s eOOOONm^s SOOOONm-1 20007Vm-2 20000Nm-3 1500Nm-4s-'2 0.9 0.5s 1.0s Table 6.5: Machine parameters. Parameter Mass, M Stiffness, K Damping Factor, Q Value 100kg 1000 - lOOOOONm"'2 1.0 Chapter 6. Trajectory Programming and Playback within the Virtual Environment 84 4000 2000 --2000 -4000 -6000 1 1 : |v- — ~ i i i >\.N. \ ' "v-i i ^ _ _ _ _ _ _ _r" / / : i 8 10 12 14 16 18 20 22 Soft Environment Hard Environment 14 16 Time [seconds] 22 Figure 6.41: Force trajectories within a new playback environment 6.2 Trajectory Playback using a Fixed Machine Impedance The effects of machine impedance selection during trajectory playback are shown in Figures 6.42, 6.43 and 6.44, each of which includes plots of desired and actual positions and forces', as well as machine impedance. Under high impedance control (M = 100kg, K = lOOOOONm-1, B = 6324Nm_1s), the machine endpoint is essentially position-controlled, while force errors have little effect. Figure 6.42 shows that while the bucket endpoint trajectory has been reproduced throughout the digging task, gross Chapter 6. Trajectory Programming and Playback within the Virtual Environment 85 force errors are experienced during excavation. Impact and digging forces more than 4.5 times those demonstrated, or up to 4000N, are observed. Under low impedance control ( M = 100kg, K = lOOONm-1,. B = 632Nm _ 1 s ) , the machine endpoint is essentially force controlled, while position errors have little effect. Figure 6.43 shows that while desired environment forces have been reproduced, significant position errors of up to 0.5m are experienced. Under moderate impedance control ( M = 100kg, K = lOOOONni"1, B = 2000Nm _ 1s) , the machine endpoint is both partly force controlled and partly position controlled. Figure 6.44 shows that while neither desired position nor force have been reproduced accurately, their respective errors are moderate, with position errors of up to 0.19m and force errors no more than 2000N. 6.3 Trajectory Playback us ing a Variable Machine I m p e d a n c e It is evident from the previous section tha t the selection of extreme machine impedances can result in undesirably large errors in either position or force trajectories. The use of an intermediate impedance facilitates the trade-off between position and force control, however its selection is dependent upon environment impedance characteristics. In this section the machine endpoint impedance is varied according to the force/velocity ratio discussed in Section 3.3.4 and detailed by Equation 3.37. This allows the machine to respond to the environment impedance in a dual manner. In the case where demonstration and playback environments are identical, perfect position and force trajectory tracking is achieved, independent of machine impedance. This is illustrated in Figure 6.45. ' - . . , . " ' . ' A low velocity gain or nominal impedance, G —. 100, favours the selection, of low machine impedance during ground contact, as shown in Figure 6.46. A high velocity gain, G — 100000, : favours the selection of high machine impedance during environment contact, as shown in Fig-ure 6.47. In both cases, excavator endpoint behaviour has switched between high impedance or effectively position control during free-motion, and a lower impedance during.digging, thereby lim-iting position and force trajectory errors. The choice of velocity gain G affects the relative error Chapter 6. Trajectory Programming and Playback within the Virtual Environment 86 distribution or weighting. For the purposes of excavation, it is desirable to reproduce demonstrated trajectories while remaining within the force capabilities of the machine. Since the realisable con-strained motion at each point within the excavator's workspace is dependent upon its attainable force envelope, it is clear that the control of both machine endpoint position and force is relevant. A velocity gain of between G = 800 and G = 1000 ensures both position and force control, in favour of position control. This is shown in Figure 6.48. Two notable features are evident in Figures 6.46 and 6.48. In both simulations manipulator impedance along the z-axis fails to return to maximum after the bucket returns to free-space (soon after time = 20s). At this time the desired bucket endpoint velocity begins decreasing while there is still significant load weight within the bucket. Consequently machine impedance drops and remains low. While subsequent desired end-effector motion would result in high bucket velocities and thus an increase in machine impedance, it is clear that difficulties arise under static conditions where desired motions result in low end-point velocity. Under such conditions machine impedance declines until scaled endpoint velocity falls below the force threshold, that is G|t>m| < fmin-Both the radial and vertical force trajectories shown in Figure 6.48 experience a surge shortly after the bucket teeth break contact with the earth at time = 20s. The loss of contact results in a force discontinuity, causing rapid acceleration of the bucket endpoint and thus elevated machine endpoint impedance (due to impedance adjustment). Increased endpoint impedance contributes to further acceleration toward the desired position trajectory. Since at this time the cutting surface remains partially submerged, viscous friction force magnitudes increase rapidly with end-effector velocity, but do not result in a significant reduction in machine impedance. Environment forces stabilise as machine position converges upon the desired trajectory. It is noted that this phenomenon is more a feature of the ground interaction model which can produce severe force discontinuities. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 87 IT °3 DC _^, / <t ?ooo 0 -2000 -4000 --6000 b 10000 ^ 5000 -10 x10 10 5 -5 Desired Value Actual Value 10 15 20 25 10 15 20 25 Time [seconds] Figure 6.42: Position and force trajectories under high, fixed machine impedance control. Chapter 6. Trajectory Programming and Playback within the Virtual Environment N N cc 08 cc 2 O od «_ 3 1000 0 -1000 -2000 -3000 E z 4000 z "-° 2000 o 0 10-5 -x10 10 -Desired Value Actual Value 10 10 10 15 20 25 15 20 25 15 20 25 Time [seconds] Figure 6.43: Position and force trajectories under low, fixed machine impedance control. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 89 2000 -2000 -6000 10 5 -xlO 10 -5 -Desired Value Actual Value 10 15 20 25 10 20 25 10 15 20 25 Time [seconds] Figure 6.44: Position and force trajectories under moderate, fixed machine impedance control. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 90 N N X <« 0C Z 0 0 3 1000 0 -1000 -2000 -3000 z o 4000 -2000 0 X10 10 5 0 -X10 10 Desired Value Actual Value 10 10 10 r 15 15 Time [seconds] 20 20 20 25 25 25 Figure 6.45: Position and force trajectories under variable impedance control within an unchanged environment. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 91 N 08 N 1000 0 -1000 -2000 -3000 6000 4000 h 2000 0 x10 10 0 -x10 10 Desired Value Actual Value 1... . 1 . . . . . > - ^ r v ~ , . . [ . 1 -lypfftf*1^ ' s r ^ ^ ( — s\, . ^ i . . _ ' \ , jz^J •• i i f - i 10 20 10 15 T 20 ' 10 15 20 10 15 20 Time [seconds] 25 25 25 1 - 1 [ 1 1 1 25 Figure 6.46: Position and force trajectories under variable machine impedance control with a low velocity weighting. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 92 N < * N 0. 2000 z 0 o * -2000 -*•' -4000 --6000 10000 ^ 5000 E z •I 1 _ — _ _*. -. ^w \ ; 1 1 . S - ^ -< . ' \ - • . i "^ j r I I I '*x->—^ I x10 10 5 -Desired Value Actual Value 10 15 20 10 15 20 Time [seconds] 25 1 • i jr - \ / -\ • • • • • • • • • " ^ - 1 •  25 Figure 6.47: Position and force trajectories under variable machine impedance control with a high velocity weighting. Chapter 6. Trajectory Programming and Playback within the Virtual Environment 93. -8000 10000 10 5 -£ Z 0 -x10 10 5 0 Desired Value Actual Value 10 ^J*~_ 10 Time [seconds] 15 T" 20 20 . 25 25 Figure 6.48: Position and force trajectories under variable machine impedance control with a medium velocity weighting. Chapter 7 Conclusions and Future Work A variable impedance control strategy has been used to reproduce operator generated excavation trajectories within a virtual excavation environment which has been developed for this purpose. A generalised principle of duality between manipulator and environment impedances provides the framework for this impedance adjustment, which allows the simultaneous control of both position and force trajectories in varying environments by creating a compromise between position and force tracking errors. The ability to reproduce operator generated digging motions has the potential to provide a number of benefits, including: • automatic task execution, • remote operation within environments which are inaccessible to humans and the elimination, of human operators from hazardous environments, • human supervision of multiple semi-autonomous machines and improvements in machine util-isation, • "" • reduced dependence upon operator skills, and • lower operator workload. A virtual excavation environment has been developed for the purpose of controller design and evaluation. This has been used as a demonstration environment as well as a playback environment for the reproduction of operator demonstrated tasks, due to the repeatability of experiments within a flexible and stable environment. The simulator comprises a number of sub-systems, including: • machine dynamics with the ability to simulate variable manipulator impedance, 94 Chapter 7. Conclusions and Future Work 95 • a new ground-bucket interaction force model which is capable of synthesising dynamic forces and torques during arbitrary digging trajectories, • a realistic 3-D, graphically rendered virtual visual environment for operator feedback, and • a haptic interface capable of feeding estimated interaction forces, directly to the human hand. Experimental results show that the appropriate adjustment of machine manipulator impedance during digging enables the control of both position and force errors, via a suitable compromise. This enables an automatic machine to operate within an unknown, time-varying environment by adjusting its impedance on a continuous basis, according to the sensed environment impedance and a position-force error weighting. The contributions of this thesis can be summarised as follows: • A method for the operator demonstration and machine playback of primitive digging motions within an uncertain environment. The concept of Robot Programming by Human Demon-stration (PHD) or Teaching by Showing is extended for the programming of digging motions. These motions result in the deformation of ground materia^ resulting in challenges quite dif-ferent from those experienced in applications within environments consisting exclusively of rigid contacts. • The use of variable mechanical impedance and a new generalised duality principle for simulta-neous position and force control within environments which consist not only of rigid contacts, but rather continuously varying environment impedance. • A unique virtual excavation environment which provides both realistic kinesthetic and visual feedback during the simulated interaction between a backhoe machine and the ground. • A bucket-ground interaction force model which is capable of computing real-time force and torque estimates during arbitrary dynamic digging trajectories. This is a significant improve-ment over prior soil cutting and digging models which are predominantly static and limited to only draught motions and fixed cutting angles. These are therefore not suitable for simulation Chapter 7. Conclusions and Future Work 96 purposes within a virtual excavation environment, thus prompting the development of the new interaction force model. 7.1 Future Work There is significant scope for the advancement of the ideas presented within this thesis. A clear step ahead would be in moving from experiments within a virtual environment to evaluation within a physical environment. This produces a new set of challenges, including the need for excavator instru-mentation and compensation for non-linear machine dynamics. Additional avenues for investigation include: • The improved on-line estimation of ground impedance for effective machine impedance ad-justment. At present, a force/velocity ratio is being used, but has failed during periods in which measured forces and velocities are either small or inconsistent. Perhaps a multi-level identification approach can be developed, thus ensuring adequate convergence speed as well as accuracy. • Automatic extraction of the position error weighting W from position and force trajectory data measured during human task demonstration. • Implementation of guarded moves during periods of environment impedance discontinuity, such as those experienced during transitions between free-space motion and ground contact. This can be achieved by temporarily forcing a low position error weighting W and reducing playback velocity. • Proof of stability during machine impedance adjustment. Although end-effector impedance is adjusted over a stable range, the act of adjustment has not been proven stable. No instability has been evident within simulations, however an analytical stability proof or stability bounds may be useful. v • An investigation of the coupling effects between multiple degrees of freedom, each operating under variable impedance control. Chapter 7. Conclusions and Future Work 97 • The incorporation of learned primitive digging motions into higher level task learning or path planning systems. • Refinement of the virtual excavation environment to include motion, sound and stereoscopic feedback for a completely.immersive simulator interface. • Better modelling of machine manipulator dynamics, including non-linear behaviour such as position dependent hysteresis, friction and inertia. The validity of the ground interaction force model, employed within the virtual excavation envi-ronment, has not been at issue within this work, due to an interest only in the disparity between demonstration and playback environments, as well as qualitative relationships and feedback. How-ever, such validation is of particular importance for use within a teleoperation master interface or for virtual task programming aimed at physical application environments. The development of those parameter identification strategies introduced in Section 5.5, as well as non-linear regression techniques, are starting points for such work. Suggestions for the extension and generalisation of the interaction force model have also been provided. Bibliography [1] T.V. Alekseeva, K.A. Artem'ev, A.A. Bromberg, R.I. Voitsekhovskii, and N.A. UPyanov. Ma-chines for Earthmoving Work - Theory and Calculations. Mashinostroenie Publishers, Moscow, 1972. [2]. Robert J. Anderson and Mark W. Spong. Hybrid Impedance Control of Robotic Manipulators. IEEE Journal of robotics and Automation, 4(5):549-556, October 1988. [3] Haruhiko Asada and Haruo Izumi. Automatic Program Generation from Teaching Data from the Hybrid Control of Robots. IEEE Transactions on Robotics and Automation, 5(2):166-173, April 1989. [4] A.K. Bejczy and W.S. Kim. Predictive Displays and Shared Compliance Control for Time Delayed Telemanipulation. Int. Workshop on Intelligent Robots and Systems, July 1990. [5] Antal K. Bejczy, Paolo Fiorini, Won Soo Kim, and Paul Schenker. Toward Integrated Operator Interface for Advanced Teleoperation under Time-Delay. Proc. Int. Conf. on Intelligent Robots and Systems, pages 563-570, 1994. [6] Leonard E. Bernold. Experimental Studies of Mechanics of Lunar Excavation. J. of Aerospace Eng., 4(1) I9-22, January 1991. [7] M. Bodur, H. Zontul, A. Ersak, A.J. Koivo, H. 0 . Yurtseven, E. Kocaoglan, and G. Pa§amehmeto</lu. Dynamic Cognitive. Force Control for an Automatic Land Excavation Robot. In Proc. of the 7th Mediterranean Electrotechnical Conf., Part 2, pages 703-706, Turkey, 1994. [8] Monica Bordegoni, Umberto Cugini, and Caterina Rizzi. Haptic Augmentation Simulation supporting teaching Skill to Robots. Source unknown, pages 123-128. [9] Stephen P. Boyd and Craig H. Barratt. Linear Controller Design - Limits of Performance. Prentice Hall, 1991. [10] David A. Bradley and Derek W. Seward. Developing Real-Time Autonomous Excavator - The LUCIE Story. In Proc. of the 34</l Conf. on Decision and Cont., pages 3028-3033, December 1995. [11] B. Brunner, G. Hirzinger, K. Landzettel, and J. Heindl. Multisensory Shared Autonomy and Tele-Sensor Programming - Key Issues in the Space Robot Technology Experiment ROTEX. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2123-2139, July 1993. [12] Darcy M. Bullock, Sudheer M. Apte, and Irving J. Opperheim. Force and Geometry Contraints in Robot Excavation. In Engineering, Construction and Operations in Space II: Proceedings of Space '90, pages 960-969, Albuquerque, New Mexico, USA, 1990. 98 Bibliography 99 [13] Nathan Delson and Harry West. Robot Programming by Human Demonstration: Subtask Compliance Controller Identification. Proceedings of the 1993 IEEE International Conference on Intelligent Robots and Systems, pages 33-41, July 1993. [14] S. P. DiMaio, S. E. Salcudean, C. Reboulet, S. Tafazoli, and K. Hashtrudi-Zaad. A Virtual Excavator for Controller Development and Evaluation. Proceedings of the IEEE International Conference on Robotics and Automation, pages 52-58, May 1998. [15] Sean Graves, Qi Wang, Wim Witvrouw, and Joris De Schutter. An Environment for Compliant Motion Programming by Human Demonstration. IEEE International Conference on Intelligent Robots and Systems, pages 1579-1585, 1996. [16] Gregory Z. Grudic and Peter D. Lawrence. Human-to-Robot Skill Transfer Using the SPORE Approximation. IEEE International Conference on Robotics and Automation, pages 2962-2967, April 1996. [17] B. Hannaford and P. Lee. Hidden Markov model analysis of force/torque information in tele-manipulation. International Journal of Robotics Research, 10(5):528-539. [18] Keyvan Hashtrudi-Zaad. Variable Impedance Control: A Geometric Approach. Internal Report - University of British Columbia, November 1997. [19] G. Hirzinger. Robot-Teaching via Force-Torque-Sensors. Sixth European Meeting on Cybernet-ics and Systems Research,. EMCSR '82, April 1982. [20] G. Hirzinger and J. Heindl. Sensor programming - a new way for teaching a robot paths and forces/torques simultaneously. RoViSeC 3, 3rd International Conference on Robot Vision and Sensory Controls, pages 549-558, November 1983. [21] G. Hirzinger, J. Heindl, and K. Landzettel. Predictive and Knowledge-Based Telerobotic Con-trol Concepts. Proceedings of the IEEE Conference on Robotics and Automation, pages 1789-1777, May 1989. [22] G. Hirzinger, J. Heindl, K. Landzettel, and B. Brunner. Multisensory shared autonomy- a key issue in the space robot technology experiment ROTEX. Proceedings of the IEEE Conference on Intelligent Robots and Systems, pages 221-230, July 1992. [23] G. Hirzinger and K. Landzettel. Sensory Feedback Structures for Robots with Supervised Learning. Proceedings of the IEEE Conference on Robotics and Automation, pages 627-635, March 1985. [24] Neville Hogan. Impedance Control: An Approach to Manipulation. Journal of Dynamic Systems Measurement and Control, pages 1-23, 1985. [25] R. Ikeura and H. Inooka. Manual Control Approach to the Teaching of a Robot Task. IEEE Transactions on Systems, Man and Cyberbetics, 24(9): 1339-1346, September 1994. [26] M. Kaiser, A. Retey, and R. Dillmann. Robot Skill Acquisition via Human Demonstration. 1th International Conference on Advanced Robotics, 1995. Bibliography 100 [27] A.J. Koivo. Kinematics of Excavators(Backhoes) for Transferring Surface Material. J. of Aerospace Eng., 7(l):17-32, January 1994. [28] A.J. Koivo, M. Thoma, E. Kocaoglan, and J. Andrade-Cetto. Modeling and Control of Ex-cavator Dynamics During Digging Operation. J. of Aerospace Eng., pages 10-18, January 1996. [29] Kazuhiro Kosuge, Toshio Fukuda, and Asada. Acquisition of Human Skills for Robotic Sys-tems. Proceedings of the IEEE International Symposium on Intelligent Control, pages 469-474, August 1991. [30] Yasuharu Kunii and Hideki Hashimoto. Tele-teaching by Human Demonstration in Virtual Enviroment for Robotic Network System. Proceedings of the IEEE International Conference on Robotics and Automation, pages 405-410, April 1997. [31] Dale A. Lawrence. Stability and Transparency in Bilateral Teleoperation. IEEE Transactions on Robotics and Automation, 9(5):624-637, October 1993. [32] Sheng Liu and Haruhiko Asada. Transferring Manipulative Skills to Robots: Representation and Acquisition of Tool Manipulative Skills Using a Process Dynamics Model. Transactions of the ASME - Journal of Dynamic Systems, Measurement and Control, pages 220-229, June 1992. [33] John E. Lloyd and Dinesh K. Pai. Extracting Robotic Part-mating programs from operator In-teraction with a Simulated Environment. International Symposium on Experimental Robotics, pages 1-12, 1997. [34] F. Malaguti. Soil Machine Interaction in Digging and Earthmoving Automation. Proceedings of the l l " 1 International Symposium on Automation and Robotics in Construction, pages 187-192, ' 1994. [35] Maurice R. Masliah and Robert W. Albrecht. The Mobile Robot Surrogate Method for De-veloping Autonomy. IEEE Transactions on Robotics and Automation, 14(2):314-320, April 1998. [36] Brenan J. McCarragher. Force Sensing from Human Demonstration: Stiffness, Impedance and Kinesthetic Sensibility. Proceedings of the IEEE International Conference on Intelligent Robots and Systems, pages 1226-1233, 1994. [37] Brenan J. McCarragher. Force Sensing from Human Demonstration Using a Hybrid Dynamical Model and Qualitative Reasoning. IEEE Conference on Robotics and Automation, pages 557-563, May 1994. [38] J. Michael Moshell, Brian S. Blau, Bruce Knerr, Donald R. Lampton, and James P. Bliss. A Research Testbed for Virtual Environment Training Applications. In Annual Virtual Reality International Symposium, pages 83-89, Seattle, WA, USA, 1993. IEEE. [39] Michael C. Nechyba and Yangsheng Xu. On the Fidelity of Human Skill Models. Proceedings of the IEEE International Conference on Robotics and Automation, pages 2688-2693, April 1996. Bibliography 101 N.R. Parker, S.E. Salcudean, and P.D. Lawrence. Application of Force Feedback to Heavy Duty Hydraulic Machines. In Proc. IEEE Int. Conf. on Rob. and Auto., pages 375-381, Atlanta, USA, May 1993.' N. Parker. Application of force feedback to heavy duty hydraulic machines. Master's thesis, University of British Columbia, October 1992. M. H. Raibert and J. J. Craig. Hybrid Position/Force Control of Manipulators. Journal of Dynamic Systems, Measurement and Control, 102, June 1981. William B. Rouse, John M.Hammer, and Charles M. Lewis. On Capturing Human Skills and Knowledge: Algorithmic Identification. IEEE Transactions on Systems, Man and Cybernetics, 19(3):558-573, May 1996. S.E. Salcudean, K. Hashtrudi-Zaad, S. Tafazoli, S.P. DiMaio, and C. Reboulet. Bilateral Matched Impedance Teleoperation with Application to Excavator Control. IEEE International Conference on Robotics and and Automation, 1998. S.E. Salcudean, S. Tafazoli, K. Hashtrudi-Zaad, P.D. Lawrence, and C. Reboulet. Evaluation of Impedance and Teleoperation Control of a Hydraulic Mini-Excavator. Proc. 5th Int. Symp. on Experimental Robotics, June 1997. . S.E. Salcudean, S. Tafazoli, P.D. Lawrence, and I. Chau. Impedance Control of a Teleoperated Mini Excavator. Int. Conf. for Advanced Robotics '97, July 1997. S.E. Salcudean, N.M. Wong, and R.L. Hollis. Design and Control of a Force-Reflecting Teleop-eration System with Magnetically Levitated Master and Wrist. IEEE Transactions on Robotics and Automation, 11(6) :844-858, December 1995. Sanjiv Singh. Learning to Predict Resistive Forces During Robotic Excavation. International Conference on Robotics and Automation, pages 2102-2107, 1995. L. Stocco, S. E. Salcudean, and F. Sassani. Mechanism Design for Global Isotropy with Appli-cations to Haptic Interfaces. In Proc. ASME Winter Annual Meeting, pages 115-122, Dallas, Texas, November 1997. Karl J. Astrom and Bjorn Wittenmark. Computer-Controlled Systems: Theory and Design. Prentice Hall, 1990. Silicon Graphics Computer Systems. IRIS Performer™ - Programmer's Guide. Silicon Graph-ics Inc., 1995. Shahram Tafazoli. Identification of Frictional Effects and Structural Dynamics for Improved Control of Hydraulic Manipulators. PhD thesis, University of British Columbia, January 1997. Tomoichi Takahashi and Hiroyuki Ogata. Robotic Assembly Operation based in Task-Level Teaching in Virtual Reality. Proceeding of the IEEE International Conference on Robotics and Automation, pages 1083-1088, May 1992. Bibliography 102 [54] Tomoichi Takahashi, Hiroyuki Ogata, and Shin yo Muto. A Method for Analyzing Human Assembly Operations for use in Automatically Generating Robot Commands. Proceedings of the IEEE International Conference on Robotics and Automation, pages 695-700, 1993. [55] Y. Tsumaki, Y. Hoshi, H. Naruse, and M. Uchhiyama. Virtual Reality Based Teleoperation which Tolerates Geometrical Modeling Errors. In Proceedings of Intelligent Rob. and Sys., pages 1023-1030, Osaka, Japan, 1996. IEEE. [56] U. Wallersteiner, P. Stager, and P.D. Lawrence. A Human Factors Evaluation of Teleoperated Hand Controllers. In International Symposium on Teleoperation and Control, Bristol, England, July 1988. [57] P.K. Vahaand M.J. Skibniewski. Dynamic Model of Excavator. J. of Aerospace Eng., 6(2):148-158, April 1993. [58] Qi Wang, Joris De Schutter, Wim Witvrouw, and Sean Graves. Derivation of Compliant '.' Motion Programs Based on Human Demonstration. Proceedings of the 1996 IEEE International Conference on Robotics and Automation, pages 2616-2621, April 1996. [59] Yangsheng Xu and Jie Yang. Towards Human-Robot Coordination: Skill Modeling and Trans-ferring via Hidden Markov Model. IEEE International Conference on Robotics and Automation, pages 1906-1911, 1995. [60] Jie Yang, Yangsheng Xu, and Chiou S. Chen. Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics. IEEE Transations on Robotics and Automation, 10(5), October 1994. ' , Appendix A The Discretisation of a Second Order Dynamic System Numerous techniques are available for the discrete approximation of continuous systems. Issues such as stability, passivity and accuracy play an important role in the selection of a suitable ap-proximation. Two different methods are described in this appendix, either of which can be used for the discretisation of machine and ground interaction dynamics. A.l Step-Invariant Transformation This approximation ensures correct values at sampling instants if the input signal is a piecewise constant signal that changes only at the sampling instants. Computer controlled systems provide such stepped setpoints. Consider a second-order linear system described by its Laplace transformation: Ms2x(s) = f0(s) + fe(s)-{Bs + K)(x(s)-x0{s)) (A.62) This is a dual-input, single-output function with inputs, /o and XQ, and output x. Parameters M, K and B represent mass, spring and damping factors. Expressed as an input-output relationship: 103 Appendix A. The Discretisation of a Second Order Dynamic System 104 The first term of this relationship can be discretised (with sample time T) by way of a Z-transformation = ( l - r ' ) z where, Ms2 + Bs + K s(s- n)(s- r2) a s — r\ k2 h r2. (i--0 zk\ + Zkn _z- 1 ' a r - e T r i bp + bjZ-1 + b2z~2 1 + a\z~l + a2Z~2 + zk3 - PTr2 z — e ri = -B + VB2 -- AMK 2M -B - VB2 -- AMK ' 2 *1 k2 k3 a\ «2 = = = = = 2M 1 rxr2 1 ri(ri -r2) 1 r2{r2 - rx) - ( e T r i + e T r 2 ) e r i + r 2 b0 — kl + k2 + k3 bi = -{(kl + k3)eTri + (ki + k2)eTr' + k2 + k3) b2 = k1eT^ri+r^ + k2eTr2 + k3eTri (A.64) (A.65) The second term is discretised in a similar fashion: Bs + K Ms2 + Bs + K = ( l - z - ^ Z h { h s s — r\ + s - r2 Cp + C\Z * + c2z 2 1 + a\z~l + a2z~2 (A.66) Appendix A. The Discretisation of a Second Order Dynamic System 105 where, r\r2 . Bn + A' «2 = 5 r 2 + K Co — h + h + h Cl = _ ( ( / 1 + / 3 ) c r n + ( Z l + / 2 ) e T r 2 + / 2 + /3 )_ c2 = lieT^+r^ + l2eTr2 + l3eTri (A.67) By re-combining both terms, the following discrete difference equation is obtained: Xk = -a iZfc- i -a2Xk-2 (A.68) + M / 0 + /e)fc + M / 0 + /e)fc-l + W o + / e H - 2 (A.69) + co{xo)k + ci(xo)k-i+c2(xo)k-2 (A.70) Unfortunately, underdamped systems result in complex roots, r\ and r2, which is undesirable for implementation purposes. This problem is avoided by employing a backward difference ap-proximation. Although this does not guarantee correct values at each sampling instant, it does accommodate under-damped systems without the need for complex computation. A.2 Backward Difference Approximation A transfer function represents a differential equation. It is therefore possible to approximate the derivatives with a forward or backward difference. Using the backward difference: dx(t) x(t)-x(t-T) sx(s) ^ - A 2 « W ^ L (A.71) This can be expressed in terms of a Z-transformation which relates the Laplacian operator, s, with Appendix A. The Discretisation of a Second Order Dynamic System 106 discrete operator, z: x{s)s' x(t) -x(t-T) z- 1 zT x{z) (A.72) (A.73) By substituting this function of z for operator s in Equation A.63, a pulse-transfer function is derived: x{z) T2(/o + /e) (M + TB + T2K) - (2M + TB)z~1 + Mz~2 ' ' [BT + KT2)-BTz-x + {M + TB + KT2) - (2M + TB)z~1 + Mz~2 (A.74) (A.75) This in turn is easily converted to a difference equation suitable for implementation within a digital system. (2M + BT)xk^ - Mxk-2 + T2(f0 + fe)k + (BT + KT2)(x0)k - BT{xQ)k^ (A.76) xk = M + BT + KT2 


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items