Applied Science, Faculty of
Biomedical Engineering, School of
DSpace
UBCV
Dupont, Bertille
Master of Applied Science - MASc
University of British Columbia
Today’s prostheses provide numerous intuitive features including several gripping patterns and movements guided by the user’s intentions. Nevertheless, users remain quite dependent on visual and auditory cues when trying to adjust grip forces, and are still unable to use their touchscreens with prosthetic fingertips. To approach this problem, researchers have proven that sensory feedback can help amputees to be faster and more accurate when accomplishing a task. However, there is still a need for a low-cost and adaptable sensory feedback add-on, requiring minimal modifications to the hand. This thesis presents the development of a low-cost sensory feedback system, including a stimulation device and a force sensing fingertip that comprises a capacitive sensor. The novelty of this research is the ability of the fingertip to activate touchscreens, and the full fingertip integration to the prosthetic hand, under the prosthetic glove – easy for the user to maintain. The glove integrated sensor capacitively detects forces over an area of 0.64 cm2. This detection is linear up to 1.6 N, with a resolution of 0.01 N. The second part of this thesis presents the implementation of shear detection, in addition to touch and pressure, using a modified sensor on a flexible printed circuit board. Able to detect touch, pressure, and shear simultaneously, the sensitive fingertip can send sensory feedback to the user. The last part of this thesis presents the implementation of two sensory feedback: one using an armband including a servo motor pushing on the user’s stump (mechanotactile feedback), and one using vibration motors hidden in the socket (vibrotactile feedback) – two feedbacks that are preferred based on a survey that we conducted. It was found that vibrotactile feedback would be more efficient due to its complete integration to the hand and its ability to send shear feedback. The next steps are the implementation of Bluetooth connections for wireless stimulation and the development of a user iv interface for real-time calibration. An ethics approval to test the device on patients was granted and will be put into practice to understand the benefits of the device and gather users’ opinions.
TOUCHSCREEN-FRIENDLY FINGERTIP SYSTEM FOR TACTILE FEEDBACK IN PROSTHETIC HANDS by Bertille Dupont B.Sc., University of Nevada, Reno, 2018 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Biomedical Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2020 © Bertille Dupont, 2020 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: Touchscreen-friendly fingertip system for tactile feedback in prosthetic hands submitted by Bertille Dupont in partial fulfillment of the requirements for the degree of Master of Applied Science in Biomedical Engineering Examining Committee: John D. W. Madden, Electrical and Computer Engineering Supervisor Hendrik F. M. Van der Loos, Mechanical Engineering Supervisory Committee Member Karon MacLean, Computer Science Supervisory Committee Member iii Abstract Today’s prostheses provide numerous intuitive features including several gripping patterns and movements guided by the user’s intentions. Nevertheless, users remain quite dependent on visual and auditory cues when trying to adjust grip forces, and are still unable to use their touchscreens with prosthetic fingertips. To approach this problem, researchers have proven that sensory feedback can help amputees to be faster and more accurate when accomplishing a task. However, there is still a need for a low-cost and adaptable sensory feedback add-on, requiring minimal modifications to the hand. This thesis presents the development of a low-cost sensory feedback system, including a stimulation device and a force sensing fingertip that comprises a capacitive sensor. The novelty of this research is the ability of the fingertip to activate touchscreens, and the full fingertip integration to the prosthetic hand, under the prosthetic glove – easy for the user to maintain. The glove integrated sensor capacitively detects forces over an area of 0.64 cm2. This detection is linear up to 1.6 N, with a resolution of 0.01 N. The second part of this thesis presents the implementation of shear detection, in addition to touch and pressure, using a modified sensor on a flexible printed circuit board. Able to detect touch, pressure, and shear simultaneously, the sensitive fingertip can send sensory feedback to the user. The last part of this thesis presents the implementation of two sensory feedback: one using an armband including a servo motor pushing on the user’s stump (mechanotactile feedback), and one using vibration motors hidden in the socket (vibrotactile feedback) – two feedbacks that are preferred based on a survey that we conducted. It was found that vibrotactile feedback would be more efficient due to its complete integration to the hand and its ability to send shear feedback. The next steps are the implementation of Bluetooth connections for wireless stimulation and the development of a user iv interface for real-time calibration. An ethics approval to test the device on patients was granted and will be put into practice to understand the benefits of the device and gather users’ opinions. v Lay Summary Being an upper-limb amputee comes with daily difficulties. Although prosthetic limbs can help, there is no sensation, making it challenging to manipulate objects. An additional modern problem is the inability to activate touchscreen technologies with prosthetic fingertips. Even though commercial prosthetic hands provide numerous features to help task accomplishment, these are still missing affordable sensory feedback, giving users information on what they are touching. This thesis presents the development of a sensory feedback system consisting of a touchscreen-friendly fingertip device simultaneously detecting touch, pressure, and shear, and a motor to translate the sensor information to the user through vibrations or pressures on the skin. The system is fully integrated into a commercial prosthetic hand, permitting the fitting of a prosthetic glove over it, and hence requiring minimal care from users. The next step is to explore the benefits of this device in future work with users. vi Preface The research work presented in this thesis is an application of the work of Research Associate Dr. Mirza Saquib us Sarwar on soft capacitive sensors, and under the supervision of Professor John D. W. Madden. The project is a collaboration with the Barber Prosthetics Clinic, a collaboration established by the author. It was partly funded by an NSERC Engage grant with Barber Prosthetic Clinic. David Moe and Brittany Pousett at Barber provided the context and motivation for the project. The majority of the work was conducted, written and presented by the author, with technical guidance and editorial feedback from Dr. Sarwar and Dr. Madden. In total, two ethics approvals were granted for this research (H19-01120 and H19-01870). In Chapter 2, the sensor fabrication is an extension of the soft capacitive sensor work of Dr. Sarwar, with the implementation of a new dielectric structure and electrode architecture designed by the author. The electronics and coding of the system were designed by the author, with the help of Dr. Sarwar for the printed circuit board, Emily Lee (NSERC USRA) for the coding, and Kieran Morton (Masters student) for battery components and coding. All components were designed by the author. The development, mechanical characterization, and analysis of the fingertip system were conducted by the author. In Chapter 3, the design of the new sensor for the detection of touch, pressure, and shear on a PCB was developed by a team led by Dr. Sarwar, but also including the author, Leanna Hogarth (NSERC USRA), Neil Fan (UBC USRA), Kota Chang (Coop), and Kieran Morton. The flexible PCB was designed by Kota Chang and Amarpreet Powar (NSERC USRA), under the supervision of the author and Dr. Sarwar. The mechanical characterization of the sensor was fully conducted by the author and analyzed with the help of Dr. Sarwar. vii In Chapter 4, ethics approval was obtained to conduct the survey. The survey was designed by the author with the help of prosthetists Brittany Pousett and David Moe from Barber Prosthetics, Dr. Karon MacLean, and input from Dr. John Madden. All components and ideas in Chapter 4 originate from the author’s work. Bluetooth implementation is ongoing by the author and Kieran Morton, and was started by Emily Lee, under the supervision of the author. The testing proposal was designed by the author with valuable suggestions and guidance from Dr. MacLean and her group, along with Kieran Morton, Dr. John Madden, and Andres Velez Lopez. Ethics approval was obtained for this, and remains available for future work. viii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables .............................................................................................................................. xiii List of Figures ............................................................................................................................. xiv List of Abbreviations ............................................................................................................... xviii Acknowledgments ...................................................................................................................... xix Dedication .....................................................................................................................................xx Chapter 1: Introduction ................................................................................................................1 Background and Definitions ........................................................................................... 1 1.1.1 Amputation ................................................................................................................. 3 1.1.2 Prosthetic Hands ......................................................................................................... 5 1.1.3 Prosthesis Evolution.................................................................................................... 8 Restoring Sensory Feedback ......................................................................................... 11 1.2.1 Sensing Stimuli Using Soft Sensors ......................................................................... 11 1.2.2 Sending a Correlated Sensory Feedback ................................................................... 13 1.2.2.1 Invasive Sensory Feedback ............................................................................... 14 1.2.2.2 Non-Invasive Sensory Feedback....................................................................... 16 1.2.2.2.1 Modality-Mismatched Feedback ................................................................. 17 1.2.2.2.2 Modality-Matched Feedback....................................................................... 19 ix Interacting with Touchscreen Technologies ................................................................. 21 Motivation ..................................................................................................................... 23 Chapter 2: The Fingertip System ...............................................................................................25 Introduction ................................................................................................................... 25 Overview ....................................................................................................................... 28 Components and Fabrication ........................................................................................ 29 2.3.1 The Capacitive Sensor .............................................................................................. 29 2.3.1.1 Working Principle ............................................................................................. 29 2.3.1.1.1 Materials ...................................................................................................... 30 2.3.1.1.2 Dielectric ..................................................................................................... 30 2.3.1.1.3 Electrode Architecture................................................................................. 35 2.3.1.2 Fabrication ........................................................................................................ 36 2.3.1.3 Electronics......................................................................................................... 39 2.3.1.4 Results ............................................................................................................... 40 2.3.2 The Fingertip Support and its Connections .............................................................. 53 2.3.2.1 3D printed fingertip........................................................................................... 54 2.3.2.2 Integration and Connections ............................................................................. 55 2.3.2.3 Soft cover .......................................................................................................... 58 2.3.2.4 Ground plane ..................................................................................................... 59 2.3.2.5 Prosthetic Glove ................................................................................................ 61 Results and Discussion ................................................................................................. 62 2.4.1 Touch and Pressure Detection .................................................................................. 62 2.4.2 Hand Integration ....................................................................................................... 73 x 2.4.3 Touchscreen Interaction ............................................................................................ 74 2.4.4 Cost ........................................................................................................................... 76 2.4.5 Challenges and Future Work .................................................................................... 77 Conclusion .................................................................................................................... 78 Chapter 3: Normal Force and Shear Sensing............................................................................81 Introduction ................................................................................................................... 81 Overview ....................................................................................................................... 82 Implementation of Shear in the Sensor ......................................................................... 83 3.3.1 Electrode Architecture and Dielectric ....................................................................... 83 3.3.2 Characterization ........................................................................................................ 86 3.3.2.1 Pressure measurements ..................................................................................... 87 3.3.2.2 Shear Measurements ......................................................................................... 90 Fingertip/Curved surface implementation .................................................................... 99 3.4.1 PCB Design and 3D Printed Support Modifications ................................................ 99 3.4.2 Curved Surface Characterization ............................................................................ 100 3.4.2.1 Gripper Design ................................................................................................ 101 3.4.2.2 Pressure and Shear Experiment ...................................................................... 102 3.4.3 Discussion ............................................................................................................... 105 Conclusion .................................................................................................................. 107 Chapter 4: Stimulation Feedback and Proposed User Study ................................................109 Introduction ................................................................................................................. 109 Overview ..................................................................................................................... 111 Gathering Prosthesis Users’ Opinions ........................................................................ 112 xi Stimulation Feedback.................................................................................................. 116 4.4.1 Components and Working Principle ....................................................................... 116 4.4.1.1 The Microcontroller ........................................................................................ 119 4.4.1.2 The Feedback Mechanism .............................................................................. 119 4.4.1.2.1 Mechanotactile Feedback .......................................................................... 120 4.4.1.2.2 Vibrotactile Feedback ............................................................................... 126 4.4.2 Calibration............................................................................................................... 128 4.4.3 Challenges ............................................................................................................... 128 Proposal: Testing the Device ...................................................................................... 129 4.5.1 Procedures ............................................................................................................... 130 4.5.1.1 Recruitment and Conditions ........................................................................... 130 4.5.1.2 Pre-Experiment Survey ................................................................................... 130 4.5.1.3 Calibration....................................................................................................... 131 4.5.1.4 Task Selection ................................................................................................. 131 4.5.1.5 Post-Experiment Interview ............................................................................. 133 4.5.2 Plan for Analysis ..................................................................................................... 133 Conclusion .................................................................................................................. 136 Chapter 5: Conclusions .............................................................................................................139 Summary ..................................................................................................................... 139 Future Work and Limitations ...................................................................................... 142 Bibliography ...............................................................................................................................145 Appendices ..................................................................................................................................157 Appendix A - Chapter 2 .......................................................................................................... 158 xii A.1 Stiffness Calculations.............................................................................................. 158 A.2 Additional Sensor Data ........................................................................................... 160 A.3 Additional fingertip data ......................................................................................... 162 Appendix B - Chapter 3 .......................................................................................................... 163 B.1 Setup Specifications ................................................................................................ 163 B.2 Base Sensor Data .................................................................................................... 164 B.3 Mathematics ............................................................................................................ 166 B.4 Additional Sensor Data ........................................................................................... 168 Appendix C - Chapter 4 .......................................................................................................... 173 C.1 Survey ..................................................................................................................... 173 C.2 Pre-Experiment Survey ........................................................................................... 179 C.3 Template for Interview Post-Experiment................................................................ 181 xiii List of Tables Table 2-1: Requirements for the sensitized fingertip system. ...................................................... 27 Table 2-2: Sensor and fingertip system specifications. ............................................................... 72 Table 2-3: Summarized cost of each components ....................................................................... 77 Table 3-1: Shear and normal resolution comparison in literature............................................... 106 Table 4-1: Survey anthropometric data for 16 participants ........................................................ 113 Table 4-2: Thresholds relating touch and pressure intensities (PWM) to the vibration motor specifications (voltage, frequency, and RPM) ........................................................................... 127 Table 4-3: Examples of results that can be obtained during the Box and Blocks test using fragile blocks.. ........................................................................................................................................ 134 Table 5-1: System features summary .......................................................................................... 142 xiv List of Figures Figure 1-1: Skin Sensations. ........................................................................................................... 2 Figure 1-2: Levels of upper-limb amputations ............................................................................... 4 Figure 1-3: Flowchart representing the different prostheses with their working principle, disadvantages, advantages, and cost ............................................................................................... 6 Figure 1-4: Prosthesis components. ................................................................................................ 7 Figure 1-5: Prosthetic evolution from the first discovered artificial hand until today.................. 10 Figure 1-6: Working principle of a parallel plate capacitive sensor. ............................................ 12 Figure 1-7: Flowchart depicting the different types of sensory feedback. ................................... 14 Figure 1-8: Extraneural and Intraneural Electrodes ...................................................................... 15 Figure 1-9: 3 Volt input vibration motor [44] ............................................................................... 18 Figure 1-10: Mechanical Crank Feedback .................................................................................... 20 Figure 1-11: Multi-function tactors which can display the sense of contact, pressure, vibration, shear force, and temperature ......................................................................................................... 20 Figure 1-12: Mutual-capacitance working principle ..................................................................... 22 Figure 2-1: Fingertip system including the sensitive fingertip and its electronics (microcontroller) on the SensorHand Speed from Ottobock ..................................................................................... 29 Figure 2-2: Capacitance changes depending on the dielectric microstructure ............................. 31 Figure 2-3: Sensor microstructured dielectric showing the height, size, and pitch of the pyramids....................................................................................................................................................... 31 Figure 2-4: Pitch Experiment ........................................................................................................ 32 Figure 2-5: Mathematical model parameters for integration. ....................................................... 34 xv Figure 2-6: Stiffness/Elastic Modulus as a function of y .............................................................. 35 Figure 2-7: Fabrication of the sensor. ........................................................................................... 38 Figure 2-8: Flowchart of the sensor readout circuit ...................................................................... 39 Figure 2-9: Picture of the fabricated sensor .................................................................................. 40 Figure 2-10: Pictures of the experiment displaying the indenter driven by the BOSE analyzer up and down and pressing on the sensor. ........................................................................................... 41 Figure 2-11: Sensor pressure sensitivity experiment .................................................................... 42 Figure 2-12: Change in Capacitance of the sensor versus Pressure .............................................. 44 Figure 2-13: Stress-strain curve for a force applied of 0.3 N. ...................................................... 45 Figure 2-14: 500 mg load experiment ........................................................................................... 46 Figure 2-15: Sensor maximum pressure experiment .................................................................... 48 Figure 2-16: Percent change in capacitance of the sensor vs. pressure for ten load cycles of 2 N........................................................................................................................................................ 50 Figure 2-17: Sensor capacitance behavior under a step 25 kPa stress applied for 10 minutes ..... 52 Figure 2-18: Burgers model showing the four elements and the behavior of the strain under a step stress .............................................................................................................................................. 53 Figure 2-19: Fingertip system ....................................................................................................... 54 Figure 2-20: 3D printed support ................................................................................................... 55 Figure 2-21: Integration of the fingertip system ........................................................................... 56 Figure 2-22: 3D printed microcontroller housing that enables the microcontroller to be integrated into the prosthetic hand ................................................................................................................. 57 Figure 2-23: Integration of the microcontroller 3D printed housing ............................................ 58 Figure 2-24: Soft cover ................................................................................................................. 59 xvi Figure 2-25: Effect of a ground plane ........................................................................................... 60 Figure 2-26: Conductive fabric wrapping the fingertip and acting as a ground plane. ................ 61 Figure 2-27: Pressure sensitivity experiment on flat sensor (green), curved sensor (blue), curved sensor wrapped with a ground plane (red), and entire fingertip including the fitting of the prosthetic glove (magenta) ............................................................................................................ 64 Figure 2-28: 100 kPa stress experiment on the fingertip system .................................................. 67 Figure 2-29: Fingertip system % change in capacitance vs. pressure. .......................................... 68 Figure 2-30: Fingertip capacitance behavior under a 100 kPa step stress applied for 10 minutes then relaxed for 50 min ................................................................................................................. 69 Figure 2-31: Step experiment on the fingertip system .................................................................. 71 Figure 2-32: Fingertip Integration ................................................................................................ 74 Figure 2-33: Touchscreen Experiment.......................................................................................... 75 Figure 3-1: Base Sensor top view ................................................................................................. 84 Figure 3-2: New shear sensor design ............................................................................................ 85 Figure 3-3: Mechanical Characterization Setup ........................................................................... 86 Figure 3-4: Pressure Characterization ........................................................................................... 89 Figure 3-5: Shear data in the x-direction ...................................................................................... 91 Figure 3-6: Shear data at different normal loads of 0.3 N, 0.6 N, 0.9 N, and 1.2 N ..................... 93 Figure 3-7: Shear characterization curve ...................................................................................... 96 Figure 3-8: Normal strain characterization curves ....................................................................... 98 Figure 3-9: FLEX-PCB design and implementation .................................................................. 100 Figure 3-10: Gripper Design ....................................................................................................... 102 Figure 3-11: Cup of water experiment ........................................................................................ 104 xvii Figure 4-1: Ranking of sensory feedback including first, second, and third choice. .................. 115 Figure 4-2: Forward and Feedback Loops .................................................................................. 118 Figure 4-3: Mechanotactile feedback system ............................................................................. 121 Figure 4-4: Mechanotactile stimulation feedback system .......................................................... 121 Figure 4-5: Diagram depicting five possible indenter designs ................................................... 124 xviii List of Abbreviations CDC – Capacitance to Digital Converter FINE – Flat Interface Nerve Electrode FSR – Force Sensitive Resistor IES – Inverse Efficiency Score JND – Just Noticeable Difference LIFE – Longitudinal Intrafascicular Electrode PCB – Printed Circuit Board PDMS – Polydimethylsiloxane PE – Proportion of Errors RT – Reaction Time SNR – Signal-to-Noise Ratio STD – Standard Deviation TIME – Transverse Intrafascicular Multichannel Electrode xix Acknowledgments I would like to thank my supervisor Dr. John Madden for enhancing my skills and my curiosity, and for all the support he has given me in all aspects of my life. I express my thanks to my committee, Dr. Hendrik F. M. Van der Loos and Dr. Karon MacLean, for showing interest in my work and taking the time to discuss it. I present my gratitude to Brittany Pousett and David Moe, part of Barber Prosthetics Clinic, for their generous collaboration and their sharing of prosthetic passion. I would also like to particularly thank Mirza Saquib us. Sarwar for our hours of brainstorming and late-night work. I want to thank Gursel Alici for his input on my project and his positive support. I thank my colleagues for their help, support, and for making work such a pleasant environment. I would like to thank Adam Fraser for his help on funding my research and his precious suggestions. I offer my gratitude to the School of Biomedical Engineering at the University of British Columbia, for giving me the opportunity to move to Canada, a place that I can now call home, and to the Natural Sciences and Engineering Research Council of Canada (NSERC) for funding my research. Endless thanks to my friends, always encouraging me all over the world, and to Elise and Clémentine for their help on figures. Finally, my deepest gratitude goes to my family, with particular thanks to my parents and my sister, who have never stopped supporting me in all my decisions and have showed me my path. xx Dedication I dedicate this thesis to my sister Clémentine, my parents, and my soon-to-be-known nephew/niece soon-to-be-born. This will never be enough to show how much I owe you, but hopefully would contribute to an advance in the prosthetics field, bringing a little bit of joy to those in need. 1 Chapter 1: Introduction The first chapter explains amputation and the complications that come with it, including the loss of tactile sensation. In this chapter, we will see how prostheses, throughout the years, have tried to replace a human hand, implementing new features but at significant cost. We will introduce some researches that have been done to detect tactile stimuli using soft sensors, and how these detected feelings are being transmitted to the prosthesis user. Finally, we will define the need for touchscreen interaction ability in prostheses and the main motivations to provide a low cost and touchscreen-friendly sensory feedback system, to help amputees in their everyday tasks by easing prosthesis use. Background and Definitions Human skin has many remarkable functions. It protects and thermoregulates the body, secretes essential substances such as sweat and sebum, and senses a wide variety of stimuli such as temperature or pressure. The different layers of the skin form a whole sensor network made of cutaneous mechanoreceptors (Figure 1-1) that relay tactile information to the brain to activate other body systems, allowing humans to accomplish fine tasks and manipulations. 2 Figure 1-1: Skin Sensations. This diagram represents in yellow the different types of mechanoreceptors in the layers of the skin. Each of them senses information that is transmitted to the parietal lobe of the brain, acting in consequence. Information obtained from Johansson et al. [1] and Culbertson et al [2]. This sensory feedback provides information on what our hand is touching, detecting pressures varying from 10 Pa to 1 MPa while differentiating sharpness, roughness, shapes, dimensions, and temperatures [3]. Just by using the sense of touch through their fingertips, humans attain an acute perception of their environment, shaping actions and decisions of everyday life. In 2008, in the United States alone, 541,000 persons were living with minor or major upper-limb loss [4]. When losing a hand, after a trauma, cancer, an infection, a vascular disease, or 3 congenital deficiencies [5], tactile feedback is lost. Bionic technologies are now trying to fill out this gap, but still lack affordable intuitive control and sensory feedback. An additional low-cost sensory feedback feature could help amputees to achieve fine manipulation, reduce the required visual attention for completing a task, and hence, live their lives more independently from their disability. 1.1.1 Amputation Amputation comes from the Latin noun Amputatio from the verb amputare, signifying to cut off or cut away [6]. Before the 17th century, amputation was described as “dismembering”, and defined in 1617 by the surgeon John Woodal as “the most lamentable part of surgery” [7]. Methods improved over the years, with the use of ligatures in the 16th century, the invention of the tourniquet by Morel in 1674, and the introduction of antiseptic techniques in 1867 by Lord Lister [8]. Reconstructive surgery should respect the patients’ needs religiously, ethically, ethnically, and professionally. It is a heavy intervention affecting the skin, muscles, nerves, vessels, bones, and joints of the limb as well as its surrounding area. After surgery, follow-up evaluations of pain, psychological situation, social reintegration, skin conditions, activities adaptation, as well as rehabilitation, are needed [9]. Upper-limb amputation levels are as follows: partial hand, wrist disarticulation, below elbow (transradial), elbow disarticulation, above elbow (transhumeral), shoulder disarticulation, and forequarter [9]. These are depicted in Figure 1-2. 4 Figure 1-2: Levels of upper-limb amputations. Information obtained from Sheehan et al.[9] The most common upper-limb amputation is through the forearm (transradial), representing 31% of total procedures performed between 1993 and 2006 [9]. After amputation, some amputees might still ‘feel’ the missing limb. This phenomenon is called phantom limb. Phantom pain describes the pain felt on this phantom limb [10]. When using a prosthetic hand, the amputee can develop a sense of ownership over the artificial hand, embodying the device, and reducing the phantom pain. Prosthetic hands can restore some lost functions, reequilibrate the body, and even reduce phantom pain through sensory feedback [11]. 5 1.1.2 Prosthetic Hands There are two categories of prostheses, excluding finger and partial amputation cases: active and passive. These are subdivided into other categories described in Figure 1-3 [12], [13], [9]. Passive hands are mainly used to balance the body and to serve a cosmetic purpose. Thus, amputees tend to be attracted to active hands, giving back the gripping ability that they have lost – especially for non-congenital deficiencies. Even though battery-powered devices are heavier than body-powered ones, they are more intuitive and less demanding – they do not activate by the movement of another part of the body. In this thesis, we will focus on myoelectric prostheses, controlled by electrical signal from muscle contraction, as these are the most advanced devices in the prosthetics market. 6 Figure 1-3: Flowchart representing the different prostheses with their working principle, disadvantages, advantages, and cost. Information obtained from Sheehan et al. [9] 7 Most myoelectric prostheses are composed of a prosthesis mechanism – including the gripping system and the electronics, a silicone inner hand (giving the shape of a hand), and a prosthetic glove (giving the artificial hand a real-life appearance), as depicted in Figure 1-4. They are fitted to the residual limb of the amputee using a socket, designed to correctly balance the forces on the stump and to ensure control through a comfortable interaction between the human and the machine [9]. Power is provided by a battery. For transradial amputees using a myoelectric prosthesis, the hand opens and closes using two electromyography (EMG) sensors, detecting the contraction of on either side of the residual arm. Prosthesis components are displayed in Figure 1-4. Figure 1-4: Prosthesis components. The hand mechanism screws into the forearm, which houses the power supply. The inner hand gives a hand shape to the system, and the cosmetic glove protects the entire system and gives a life-like appearance to the artificial hand. Adapted with permission from Ottobock [14]. 8 Artificial hands offer a wide variety of features for a certain cost. Over the years and through historic events, research, and innovation for the prosthetics field have gain interest. The next section illustrates how history and the emergence of prosthetics companies have given birth to sophisticated artificial hands, but important features such as sensory feedback are still missing at an affordable price. 1.1.3 Prosthesis Evolution Upper limb prostheses have come a long way since the first known artificial hand. The very first appearance of prostheses in history was during the 4th century BC. Iron hands became useful for soldier amputees to hold a shield and protect themselves during battle [5], [15]. This discovery led to new designs, including spring-loaded hands such as the hand “Le Petit Lorain” by the surgeon Ambroise Paré in the 16th century [5], [16]. In 1818, Peter Baliff introduced the concept of body-powered prostheses, moving the hand by transmitting the tension of leather straps due to the motion of the trunk or shoulder girdle. Sauerbruch took advantage of this new technique to design in 1916 an artificial hand controlled by the movements of the upper arm [17]. This new prosthesis type remained expensive to produce due to its small niche market [5]. However, the high numbers of amputees resulting from the two world wars led to a considerable increased demand in artificial hands and legs, bringing prosthesis innovation and development to the next step [18]. A large number of rehabilitation amputee programs were created, numerous associations were developed, and prosthetics companies were formed to provide hands to veterans (e.g. Ottobock created in 1919 [19]). New designs including the Bowden cable body-powered prosthesis (ameliorating the bulky straps to a single solid cable) [20] and myoelectric hands were introduced after World War 2 finished [5]. Later on, at the beginning of 2000s, big companies (e.g. Ottobock, 9 Ossur, Open Bionics, and Mobius Bionics) started to implement new features including several grip patterns, better intuitive control, lighter weight and lower cost with 3D printing techniques [21], [22], [23], or even simple vibration feedback [24]. However, there is still a long way to go before actually replacing our human hand at an affordable price [25]. This development path is summarized in Figure 1-5. 10 Figure 1-5: Prosthetic evolution from the first discovered artificial hand until today. 11 All of these events gave rise to current prosthetic hands, creating technologies that are now compared to the ones seen in movies. New technologies using 3D printing are now creating more affordable prostheses such as the Hero Arm created last year by Open Bionics and available for $3,000 [13]. The use of this technique could help third world countries to provide affordable and customizable prosthetic hands to amputees. Despite the technological progress in prosthetic hands, the burden that comes with wearing a prosthesis has pushed many users to limit the use of their artificial hand to certain tasks, or completely abandon it [26]. According to a 2019 Australian survey, users request lighter and cheaper devices with a chosen appearance, including a more intuitive control and sensory feedback [27], rather than bulky hands or hooks demanding high visual attention and cognitive load. There is a real need to take prosthetic innovation further by adding new features to give sensory feedback, provide touchscreen-friendly fingertips, implement intuitive control, and enhance embodiment while limiting weight and cost. Restoring Sensory Feedback 1.2.1 Sensing Stimuli Using Soft Sensors Over the years, sensors have been developed to mimic human skin and sense different stimuli such as touch, pressure, temperature, humidity, or shear [28]. They may also go beyond the capabilities of the skin by sensing proximity – though this is effectively done by skin via hairs. They are now flexible and stretchable, and even buckle like skin. The prosthetics field has shown interest in these new sensing technologies to provide tactile feedback to amputees by providing them an artificial skin and hence, haptic feedback. The working principle of these sensors can be 12 piezoresistive (transducing a change in the resistance due to an applied stress), piezoelectric (generating a voltage due to an applied pressure), or capacitive (detecting a change in capacitive coupling) [28]. In this work, a capacitive sensing approach is used due to its possible adaptation to detect different stimuli including proximity, touch, pressure, and shear, by varying the size and shape of the electrodes (electrode architecture). A parallel-plate capacitive pressure sensor consists of two electrodes separated by a non-conductive material called a dielectric. As the two electrodes separated by a distance, d, and of area, A, move closer together due to a force F, the thickness d of the dielectric decreases, increasing the capacitance according to the equation 𝐶 = 𝜖𝐴𝑑= 𝜖0𝑘𝐴𝑑 (𝜖 being the absolute permittivity, 𝜖0 being the vacuum permittivity, and k the dielectric constant). Figure 1-6 demonstrates the working principle of a capacitive sensor. Figure 1-6: Working principle of a parallel plate capacitive sensor. 𝑪 = 𝝐𝑾𝑳𝒅 = 𝝐𝑨𝒅 𝑪 = 𝝐𝑾𝑳𝒅−∆𝒅 = 𝝐𝑨𝒅−∆𝒅 13 To be applied as an artificial skin, dielectric and electrodes are made of flexible and stretchable materials. Elastomers are often used for dielectrics materials due to their high strains and deformability. Lipomi et al. demonstrated the high stretchability of a polydimethylsiloxane (PDMS) sensor, keeping a high conductivity at a 150% strain [29]. Another research study has shown that sensors fabricated with EcoflexTM (Smooth-on, Inc.) can be stretched up to at least 460%, enhancing the possibilities for highly sensitive stretchable sensors [30]. This work uses EcoflexTM as a dielectric material due to its low cost, its commercial availability, and its high deformability (allowing the two electrodes to get closer together). Electrodes are usually made of conductive materials such as carbon nanotubes, silver nanowires, liquid metals, carbon black composites, or hydrogels [28]. The sensitivity of these new technologies targets the detection of medium pressure ranges (10 to 100 kPa) for object manipulation and low pressures (<10 kPa) for a light touch [3], which comprises the range of physiological sensing. Micro-structured capacitance sensors can detect pressures as low as 3 Pa [31]. Dielectric structures (impacting its thickness and stiffness), electrode architectures (impacting the area of overlap between the parallel electrodes), and a large choice of materials allow capacitive sensors to be easily customizable, detecting several stimuli such as proximity, light touch, pressure, and shear. Placing them on a fingertip of a prosthesis is a way of detecting the presence of an object, assessing the gripping force exerted on it, or feeling it slipping between fingertips. This is done by analyzing the change in capacitance. The change can then be interpreted to send sensory feedback to the prosthesis user. 1.2.2 Sending a Correlated Sensory Feedback Various methods have been used to translate an input stimulus from a sensor to tactile feedback. These techniques can be invasive, implicating skin break to stimulate nerves directly, or 14 non-invasive, requiring the interpretation of the restored feedback by the user [32]. Figure 1-7 summarizes the different sensory feedback systems, which will be described in this chapter. Figure 1-7: Flowchart depicting the different types of sensory feedback. Both invasive and non-invasive techniques have demonstrated the benefits of tactile feedback, improving task accuracy and time [25], reducing phantom pain [11], and helping prosthesis embodiment. These two methods are described below. 1.2.2.1 Invasive Sensory Feedback Peripheral nerve stimulation is one of the invasive sensory feedback methods. It uses two types of electrodes: extraneural electrodes and intraneural electrodes [33], as shown in Figure 1-8. 15 Figure 1-8: Extraneural and Intraneural Electrodes: (A) Cuff electrode wrapping the nerve, (B) FINE electrode squashes the nerve for greater fascicle proximity, (C) LIFE electrode going parallelly through the nerve, (D) TIME electrode going perpendicularly through the nerve, (E) Microelectrodes array. Adapted from [33] and [34]. Extraneural electrodes (such as flat interface nerve electrodes - FINE or cuff electrodes [35]) wrap the forearm nerves to stimulate nerve fascicles using electrical waveforms with “customized pulse width, pulse amplitude, and interpulse interval” [36]. Intraneural electrodes pierce the nerves for a greater sensitivity with a smaller input stimulus. It uses longitudinal interfascicular electrodes (LIFE - implanted parallel to the nerve fiber), transverse intrafascicular multichannel electrode (TIME - transversally go through the nerve), or multi-electrode arrays (network of needles penetrating the nerve such as the Utah Slanted Electrode Array) [35]. 16 Targeted Muscle Reinnervation (TMR) is another type of invasive sensory feedback technique. It transects a peripheral nerve of the amputated hand to connect it to an intact muscle in the body, usually the chest. The motor nerves previously driving the hand now allow the contraction of the chest, permitting a very intuitive control of a myoelectric hand through EMG [37]. Moreover, sensory nerves formerly connected to the hand are arranged to innervate the chest skin. After some time, a tactile, temperature, or pain stimulus on the chest skin can be felt on the missing hand – in other words, any stimulus perturbing the chest skin would stimulate the redirected hand nerve, allowing the subject to feel the signal as if it was on the hand [37]. This technique reduces the risk of potential infection and/or rejection due to electrodes. However, the differentiation between motor control and sensory feedback might be difficult due to their proximity, where afferent signals (feedback) might interfere with efferent signals (control motor) [33]. Several studies have demonstrated that invasive methods can improve speed and accuracy during tasks due to naturalistic sensory feedback [38],[36]. This empowers self-confidence, self-image, and prosthetic embodiment, enhancing daily-use [38]. However, invasive methods remain expensive, require intensive surgery and follow-up trainings, and have not yet undergone long-term tests, or been tested on congenital amputees. This is a barrier to prosthesis users, who demand sensory feedback at minimum cost. 1.2.2.2 Non-Invasive Sensory Feedback Non-invasive feedback techniques can either use a different feedback stimulus to interpret the original input stimulus, called modality mismatched feedback or sensory substitution (e.g. 17 activation of a vibration motor to represent a detected touch/pressure), or use modality-matched feedback to directly convey the detected touch/pressure to another location in the body (e.g. activation of a motor pushing on a specific part of the body to translate a detected touch/pressure) [33], [39]. These have the advantage of being implemented as add-ons to prostheses, and can give a real perception of the detected stimulus. 1.2.2.2.1 Modality-Mismatched Feedback Vibrotactile, electrotactile, auditory, and augmented reality feedback are used to convey input information from the sensor detecting the stimulus, to the prosthesis user as a different modality. This requires the interpretation of the user to differentiate between different types of stimuli. Vibrotactile stimulation uses a small vibration motor placed on the skin to indicate a detected stimulus. This technique is used commercially by the Luke Arm. The low-cost, low-power consumption, and small size of the vibration actuators (Figure 1-9) make them interesting to implement in a prosthesis [25]. Using one or several actuators [40], with a certain position [41], and activated at different amplitudes, pulse frequencies, burst widths, or inter-burst intervals [42], allows the perception of different stimuli, but could, however, lead to reduced sensation in the stimulated area [43]. This method has shown improvement in grasping force [41] and a better sense of embodiment over the artificial hand [44]. 18 Figure 1-9: 3 Volt input vibration motor [45] Transcutaneous electrical nerve stimulation non-invasively stimulates the radial and ulnar nerve of the residual arm. Small electrical signals trigger the depolarization and repolarization of the nerve cells, leading to the initiation of an action potential. Three main parameters can affect the perceived sensation: the current amplitude, the pulse width, and the pulse frequency [46]. Chai et al. have shown that, on 11 forearm amputees, modulating the current density (from 0.2 10-2 mA/cm2 to 1.2 10-2 mA/cm2), the pulse width (0 µs to 400 µs), and the frequency (1 Hz to 20 Hz) allowed six sensory modalities to be perceived such as touch, pressure, or pain [46], [47]. Damian et al. also studied the potential benefit of this stimulation for slip feedback [48]. Despite the versatility of this technique, the effect of skin condition (e.g. sweat, hairs), the difficult placement of the electrodes, the bulkiness of the device, and the possible interference between EMG sensors and electrotactile electrodes, add to the prosthetic care burden. Audio feedback uses the sense of hearing as a substitution of the detected stimulus. Sound frequency [49] or sound type (e.g. piano or cello triads) [50] can be used to identify the stimulus location, and volume to convey a pressure intensity [49]. This feedback can be easily understood by the user, but can also be easily affected by outside noise during real-life implementation [39]. 19 Visual display can also be used to communicate “EMG signals, aperture, force, and contact” as developed in GLIMPSE, a Google Glass application [51]. The display uses colors and segmented horizontal bars that get filled up as the grip force intensity increases. This type of feedback can help grip force control, but significantly reduces the amount of time needed to complete a task. As sensory feedback is mostly needed to reduce the load for visual attention, this method is less applicable for everyday life prosthetic use. 1.2.2.2.2 Modality-Matched Feedback Modality-matched feedback directly translates the tactile stimulus detected by the sensor at the fingertips of the prosthesis to the user. If the input signal is a touch, the conveyed feedback will be a touch [33]. This allows the user to directly feel the stimulus felt on the fingertip of the prosthesis. Mechanotactile feedback can transmit tactile information as well as the intensity, the degree, the magnitude of pressure exerted by the fingertips of the artificial hand on an object. This educates the central nervous system on how to interpret new afferent signals to different locations corresponding to touch and pressure sensations. Recognized stimuli can be normal or tangential forces [52] conveyed by servo motors [53], 3D printed actuators [54], pneumatic systems [55], or more complex devices involving skin stretch [56],[43] on the residual arm of the individual. Stephens-Fripp et al. examined the impact of reproduced shear and normal force by mechanical cranks with specific locations and orientations, as shown in Figure 1-10. Six different grip were recognized at 80% accuracy [57]. 20 Figure 1-10: Mechanical Crank Feedback, (a) Mechanical Crank, (b) Crank location on cross-section of arm, (c) Placement on the arm. [57] © [2018] IEEE Some researchers have even demonstrated the transmission of several types of stimuli using a multi-haptic device. Kim et al. developed a multi-functional instrument (Figure 1-11) displaying touch, pressure, shear, vibration for surface roughness, and temperature in a closed-loop manner through different types of actuators, accelerometers, force and torque sensors [58]. Figure 1-11: Multi-function tactors which can display the sense of contact, pressure, vibration, shear force, and temperature: (a) 6-bar mechanism with skewed parallelogram and 10 mm brushed motor (P10), (b) 6-bar mechanism with gear and 10 mm brushed motor (G10), (c) 6-bar mechanism with gear and 6 mm brushless motor (G6). [58] © [2010] IEEE. 21 Despite the accurate translation of the detected stimulus to the residual skin of the prosthesis user, these mechanotactile devices are usually bulky and sometimes noisy [25], which goes against users’ desire to reduce the size and weight of their artificial hand [59]. The progress in micro-actuators, such as high-frequency trilayer conducting polymers design [60], could provide precise feedback with a wide range of frequencies for a very small size, allowing the fit of several of them in a prosthesis socket. Sensory feedback techniques can reduce training time and improve task performance by providing grip force and tactile information [61]. Many studies describe sensory feedback as a way of reducing phantom pain [11]. However, invasive sensory feedback devices have not been proven efficient in real-life applications and the required intensive surgery might deter patients. Non-invasive devices mostly use bulky off-the-shelf sensors, involving sensory feedback systems that add volume and weight to the artificial hand. In addition to the lack of sensory feedback, most prostheses do not provide touchscreen-friendly fingertips. This deficiency can impede multi-tasking and prevent social communication. This thesis focuses on the creation of a non-invasive and touchscreen-friendly fingertip device detecting stimuli to send a correlated tactile feedback while being integrated in the prosthesis. Interacting with Touchscreen Technologies Touchscreen technologies surround us. They are used in smartphones, tablets, ATMs, rehabilitation, credit card readers, kitchen appliances, and even in cars. Most available prostheses 22 (such as passive hands or the Ottobock SensorHand Speed) do not provide touchscreen-friendly fingertips. This becomes a problem for unilateral amputees when it comes to multitasking, such as answering a phone call while keeping the other functioning hand free. Bilateral amputees, using prostheses on each side, are incapable of interacting with touchscreens. In today’s society, this inability can become a real social handicap and reduce many everyday life activities that require touchscreen activation. This section explains how these screens work and why most prostheses are unable to activate them. Touch sensing methods use either infrared (using arrays of infrared LEDs), resistance (using a voltage divider where the touch is detected by the contact of two electrodes separated by a dielectric layer), or capacitance. Capacitive touchscreens dominate because they permit multi-touch, provide better optical properties, and have a higher sensitivity for light touch detection [62]. Today’s technologies use mutual-capacitance. The working principle of this method is described in Figure 1-12. Figure 1-12: Mutual-capacitance working principle: (A) Without the presence of a finger, (B) When a finger is approaching. 23 Two co-planar electrodes form a capacitor due to the electric field coupling between them. This determines the capacitance reading between the two electrodes to be CE. The presence of a finger or grounded metal will “steal” some of the electric field, decoupling the electric field between the sense and the drive electrode. This will thus decrease the capacitance CE by the amount CM, where CM depends on the relative position of the finger and the electrode. Prostheses, often covered by a silicone prosthetic glove or another insulating layer, do not offer ground connection to the fingertip. When approaching a touchscreen, the artificial hand has no coupling effect on the electric fields between the two electrodes. The capacitance remains the same, meaning that the finger was not detected. Connecting the fingertips of a prosthesis to ground would allow the interaction with touchscreen technologies. The fingertip device developed in this research (Chapter 2) is connected to ground, providing the ability to activate touch sensing devices. Motivation Prostheses are technologies allowing those who have lost capabilities to recover function. However, the complications that come with having a prosthesis limit users to wearing it for specific tasks only or completely abandon it [26]. Sophisticated prostheses, providing better features and easier control, remain extremely expensive. Most amputees cannot afford advanced prostheses such as the BeBionic, the iLimb, or the Luke arm (which provides sensory feedback). Poorer countries cannot provide sufficient healthcare for amputees to receive even passive prosthetic hands [63]. Therefore, all across the world, most amputees do not have access to artificial hands, or use either passive hands or simple myoelectric hands having the feature of opening and closing 24 on an EMG basis. These existing and more affordable prosthetic arms currently available on the market are lacking sensory feedback and cannot activate touchscreen technologies, an important skill in today’s world. They are difficult to control and have a high gripping strength, making them capable of crushing many objects. Thus, to control these hands, users require a lot of practice. Without sensory feedback, they have to look at their artificial limb, relying on visual and auditory cues to exert an accurate grip force and position at every step of object manipulation. To address this need, various methods have been used to provide tactile feedback, while trying to fulfill important properties, such as providing a continuous and correlated feedback that can be adjustable and easily interpreted [64]. These approaches have their advantages and drawbacks, as described earlier, but users are still looking for simple tactile feedback, easily adaptable, and affordable. This research aims to provide a user-friendly (no additional weight, effort, or volume) and low-cost sensing fingertip that detects touch, pressure (Chapter 2), and shear stimuli (Chapter 3), to provide sensory feedback (Chapter 4) and could be easily adapted to many prosthetic hands. The developed system allows prosthesis wearers to use touchscreen technologies, without altering the marketed artificial hand. 25 Chapter 2: The Fingertip System Sensing technologies attract interest in various fields including skin-attachment devices for health and exercise monitoring, wearables for video-games, or artificial skin for robots and prostheses. The use of these devices can restore parts of lost sensations such as touch or pressure for amputees. However, reasonably priced sensory feedback for prosthetic hands is currently missing on the market. To help their everyday lives and minimize the prosthesis cost for amputees, innovative features, including sensory feedback, need to be durable, low-cost, easily implemented, and adaptable to different types of hands. Introduction Many amputees use myoelectric hands, including the SensorHand Speed from Ottobock, due to their affordable cost and functionalities. The components of these hands are described in Chapter 1. Most prosthetic hands are essentially numb, meaning that they do not provide any tactile feedback to the user. Only a few commercially available hands provide such feedback to users at a significant cost, including the LUKE arm that uses a pressure sensor to convey tactile feedback. To address this gap, many researchers seek out new technologies that would detect tactile stimuli and provide information to the prosthesis user. For example, Clemente et al. propose sensitized thimbles that are fitted to the fingertips of a prosthesis [32]. Other researchers implement force-sensitive resistor sensors (FSR) or capacitive sensors to prosthetic fingers using double sided-tape [65] or shape an electronic skin made of conductive and piezoresistive fabrics around the artificial fingers to capture a change in resistance [47]. This can be easily adaptable to multiple designs but cannot be used in real-life applications due to the external wires and electronics. 26 For prosthesis users, everyday activities bring in potential damage to add-on sensing systems. Such devices need to be protected or encapsulated from outside factors such as heat, water, dust, high forces, or sharp objects. The prosthetic glove, described in Chapter 1, protects the artificial hand. Although this protection is essential, the tightness of the glove and the limited space within the hand create a challenge to integrate sensing fingertips. Rosenbaum-Chou et al. integrated an FSR sensor on the thumb of a hand, coated it with silicone to keep the sensor in place, and fitted a prosthetic glove over it [66]. Segil et al. replaced a BeBionic phalanx with a 3D printed one including barometric pressure sensors and infrared proximity sensors [67]. Although these approaches allow the fit of a prosthetic glove, there is still room for improvement. The ability for prosthetic users to activate touchscreens is still missing, and innovative sensors could allow the detection and feedback of touch, pressure (Chapter 2), and shear (Chapter 3) simultaneously while being integrated to the hand. A low-cost, touchscreen-friendly, and easily modifiable add-on sensitive fingertip could help amputees to have a more precise grip force over manipulated objects. To achieve this, the system in use needs to meet some requirements listed in Table 2-1. 27 Table 2-1: Requirements for the sensitized fingertip system. We developed a touchscreen-friendly 3D printed phalanx support to integrate a highly sensitive capacitive sensor and its electronics. This provides a compact, low-cost, and adaptable system that does not require any care from the prosthesis user, as it is powered by the prosthesis battery, and protected by the prosthetic glove. In this chapter, we will first present the capacitive sensor, its working principle, fabrication, and performance. We will then describe all components of the fingertip system and how they link together. We will finally discuss the results obtained for touch and pressure detection of the sensitized fingertip, as well as its ability to activate touchscreens. Chapter 3 will follow up on the implementation of shear detection in this fingertip. 28 Overview Chapter 2 presents a fingertip system that can detect touch (pressure ≤ 1 kPa [68]) and pressure stimuli, and interact with touchscreen technologies while being completely integrated into a prosthetic hand under a glove. This is accomplished using a 3D printed fingertip that hosts a highly sensitive capacitive sensor with its electronics, replacing the existing fingertip of the inner hand (prosthetic components are depicted in Figure 1-4, page 7). A 3D printed “soft cover” wraps this system, protecting the sensor from getting damaged while fitting the glove. This is then covered by a grounded conductive fabric to enable touchscreen activation. All components were designed and developed by the author, with the help of Mirza us Sarwar for the printed circuit board. Fitted under the glove, the system detects small pressures or touches, showing a change in capacitance of 0.7 % per kPa, and sends an input signal to activate a sensory feedback device via a microcontroller (also integrated in the hand). This system could be adapted to different sizes and types of prostheses. Here, the design fits myoelectric hands such as the SensorHand Speed from Ottobock discussed in Chapter 1 (Figure 2-1). 29 Figure 2-1: Fingertip system including the sensitive fingertip and its electronics (microcontroller) on the SensorHand Speed from Ottobock. The hand can then be fitted with a prosthetic glove. Components and Fabrication 2.3.1 The Capacitive Sensor The first component of the fingertip system is the capacitive sensor. This sensor is used to detect very small pressures, interpreted as touches (up to 1 kPa [68]), to medium pressures (up to at least 100 kPa for object manipulation [3]). Here we describe our sensor’s working principle, including several factors affecting its sensitivity, how we fabricate and connect it, and how it performs. 2.3.1.1 Working Principle In this thesis, we describe a parallel-plate capacitive sensor that is flexible, bendable, and stretchable. It detects small pressures and operates while bent (e.g. on a curved surface such as a 30 finger pad). Several factors determine its sensitivity: 1) the materials used for fabrication, 2) the dielectric structure, and 3) the electrode architecture. 2.3.1.1.1 Materials Electrodes are made using a mixture of conductive carbon black and non-conductive elastomeric material (EcoflexTM 00-30, from Smooth-On US). The two electrodes are separated by a dielectric made of EcoflexTM and are coated on the outside by EcoflexTM. These materials and the fabrication method (described later in this chapter) are based on the work of Mirza us Sarwar [62]. 2.3.1.1.2 Dielectric As described in Chapter 1, a pressure applied on the sensor brings the two electrodes closer together, leading to an increase in capacitance according to the equation 𝐶 = 𝜖𝐴𝑑 . Thus, if the dielectric separating the electrodes is softer, the distance d between the two will be more easily deformed, detecting smaller pressures. Literature shows a lot of interest in dielectric structures, including dielectrics encapsulating bubbles [62], or fabricated with square or pyramidal pillars [69]. Deng et al. stated that pyramids generate a more sensitive sensor than cylinders or cuboids pillars (Figure 2-2 (A)) [69], and that the smaller and higher the pyramids, the greater the sensitivity (Figure 2-2 (B)). The idea was also proven by Tee et al., where the compressibility of the dielectric was shown to be greater when using a structure with a greater sidewall angle slopes (i.e. pyramids instead of cuboid pillars) [70]. 31 Figure 2-2: Capacitance changes depending on the dielectric microstructure: (A) The capacitance-pressure characteristics of the sensors with different pillar-type microstructures, (B) The capacitance-pressure characteristics depending on the sizes of the pyramids [69]. Thus, when implementing pyramidal pillars in a sensor dielectric, several aspects need to be taken into consideration: the height of the pyramids, their size, and the pitch between them (Figure 2-3). These would affect the thickness and softness of the dielectric, impacting the sensor sensitivity. Figure 2-3: Sensor microstructured dielectric showing the height, size, and pitch of the pyramids. 32 We have experimented with different pitches to investigate the effect on sensitivity. Two sensors were fabricated with dielectrics using the same pyramid sizes and heights, but having a pitch of 0.8 mm difference (Pitch B – 1.4 mm pitch resulting in four pyramids per row, and Pitch A – 2.2 mm pitch resulting in three pyramids per row). Five sinusoidal force cycles of 1 N were applied at a frequency of 0.1 Hz using a 9 mm by 9 mm indenter attached to Bose Electroforce dynamic mechanical analyzer and a load cell, recording both force and displacement data, as depicted in Figure 2-4. As the same applied force compresses the sensor with a larger pitch (Pitch A) more (i.e. larger displacement), the sensitivity of this sensor is greater. Figure 2-4: Pitch Experiment (A) Displacement of the indenter (magenta and blue) when a force of 1 N (black) is applied to both sensors (with pitches A and B), (B) Change in capacitance of both sensors (magenta and blue) on the same set of data. 33 In this work, the chosen dielectric is structured with pyramids of size 1.3 mm, 1.6 mm height, and 1.4 mm pitch. Even though the 2.2 mm pitch displayed better results, the fabricated sensor is too fragile for our application. Thus, the pitch of 1.4 mm was chosen as four pyramids per row makes the dielectric deformable enough to detect small pressures, but also robust enough. The top of the pyramids was slightly flattened to ensure the bonding between the top of the pyramids and the bottom layer. This structure satisfies both softness and robustness of the dielectric to ensure the sensor sensitivity and durability. To understand the behavior of this dielectric, we calculated the stiffness of each pyramid as a function of the elastic modulus E. To do so, we modeled the pyramid as several layers, each layer representing a spring. The entire pyramid is then modeled as several springs in series. Each spring has a stiffness 𝑘 =𝐸𝐴𝑦0 where A is the area of the base of the pyramid of sides w, and 𝑦0 is the height. The total stiffness is then k𝑡𝑜𝑡𝑎𝑙 = (∑1𝑘𝑖)−1𝑦0𝑖=0 . By integrating this formula and using parameters depicted in Figure 2-5, we find that: 𝐤𝐭𝐨𝐭𝐚𝐥 = 𝐰𝟐𝐄 (𝟏𝐲−𝟏𝐲𝟎) Equation 2.1 34 Figure 2-5: Mathematical model parameters for integration. Detailed calculations can be found in Appendix A.1. The parameter y represents the height of the part of the pyramid modeled and integrated as a spring. If 𝑦0 is 1.6 mm and y is 1.4 mm, then the calculated stiffness represents the stiffness of the pyramid of 1.6 mm height, with a peak cut off at 1.4mm. Equation 2.1 shows that as y goes to zero, the stiffness goes to infinity, and as y reaches the height of the pyramid 𝑦0, the stiffness tends to be zero (Figure 2-6). This proves that a specific force needed to displace the top surface of the sensor with respect to the bottom one will be small when the displacement is small but larger as the displacement increases. This means that the change in capacitance will be smaller and smaller as the pressure applied to the sensor increases. This non-linear stiffness property allows the detection of very small forces but also permits the dielectric to withstand high forces. 35 Figure 2-6: Stiffness/Elastic Modulus as a function of y 2.3.1.1.3 Electrode Architecture The size of the electrodes is another important factor affecting the sensitivity of the capacitive sensor. This was determined to fit on the finger pad of the 3D printed fingertip described in a later section. The sensor has to be big enough to cover enough area on the finger, and small enough to detect small/sharp objects. In the case that the sensor is too large, a large force applied with a sharp object covering a very small area would barely increase the capacitance. In the case that the sensor is too small, pressure on the side of the fingertip would not be detected. After many trials, the size of the electrodes was defined to be a square of 9 mm by 9 mm to comply with these specifications. This can be modified to implement an array including several small electrodes wrapped around the finger pad to detect multiple pressure points. However, this would imply the electronics to be bigger and make it difficult to place the printed circuit board in the vicinity of the sensor to limit parasitic capacitance. Also, considering that the sensor is for sensory feedback, 010203040500 0.5 1 1.5Stiffness / Ey [mm]Stiffness/E vs. Change in y36 multiple pressure points might be difficult to translate to the user and might require a lot of training on the user’s side to comprehend the stimulation feedback. 2.3.1.2 Fabrication The fabrication of the sensor is a four steps process described in Figure 2-7. A 3D printed mold is used to shape the dielectric including the pyramid of 1.3 mm base size, 1.6 mm height, and 1.4 mm pitch as described above, and pattern the top electrode. A petri dish is used to fabricate the other part of the sensor (Figure 2-7 (1)), including the bottom electrode. The non-conductive EcoflexTM 00-30 material (Smooth-on, Inc) and the conductive material H30253 Carbon Black Super P® Conductive (Alfa Aesar, Thermo Fisher Scientific) are used to fabricate the sensor. This fabrication follows the main steps of capacitive sensor fabrication described by Mirza us Sarwar [62] – with the difference being the use of pyramidal structures here instead of columns. The first step (Figure 2-7 (2)) is filling the 3D printed mold with EcoflexTM 00-30, previously degassed to remove any air bubbles. The elastomer is doctor bladed, meaning that a flat surface is drawn across the mold surface to remove the excess of material and make the surface flat. The mold including the EcoflexTM is degassed to remove bubbles formed in the pyramidal holes. More EcoflexTM is immediately poured in the mold and doctor bladed to ensure the flatness of the 1.8 mm layer. In parallel, degassed EcoflexTM 00-30 is spun on a petri dish at 1400 rpm for one minute (Figure 2-7 (2)). The resulting thickness of this layer is ∼200 μm. EcoflexTM is cured at 60°C for 20 min. This layer is used to encapsulate the sensor and protect the carbon black electrode described in the next step. 37 The second step (Figure 2-7 (3)) is to pattern the carbon black electrodes on each part using a hand-cut mask outlining a 9 mm x 9 mm square with 2 mm x 13 mm rectangular strips, in a transparency film. The electrodes are made conductive by mixing carbon black particles with EcoflexTM in the ratio 1:10 by weight. This 100 μm thick layer is cured at 60°C for 20 min. Then, stripes of carbon black electrodes are covered by transparency film. Degassed EcoflexTM 00-30 is spun on the petri dish and on the mold at 1400 rpm for one minute (Figure 2-7 (4)). The transparency film prevents the EcoflexTM from covering the carbon black stripes, leaving them exposed for connections to the printed circuit board (PCB). EcoflexTM is cured at 60°C for 20 min on the mold, but left uncured on the petri dish. The last step of the process is to peel the molded dielectric off the mold and bond it to the uncured EcoflexTM of the petri dish. Figure 2-7 (5) shows a side, top, and 3D view of this process. Electrodes are aligned manually, trying to perfectly put the squares on top of each other. After curing the sensor for 20 min at 60°C, the sensor is peeled off and cut at the desired dimensions (Figure 2-7 (6)). Its resulting thickness is 2.6 mm. The electrodes are 9 mm x 9 mm, and the electrical contacts are 13 mm long. The repeatability and accuracy of the sensor can be improved using laser-cut masks, a more developed alignment method, doctor-blade machines, and spray-coating method to pattern electrodes. 38 Figure 2-7: Fabrication of the sensor (1) 3D printed mold and petri dish, (2) first layer of EcoflexTM, (3) pattern electrodes, (4) spin on another layer of EcoflexTM, (5) bond the dielectric to the petri dish, (6) peel off the sensor and cut off the excess of EcoflexTM. 39 2.3.1.3 Electronics A capacitance to digital converter (CDC) AD 7745 reads out the capacitance from the sensor. The connection is made by pressing the exposed carbon black stripes onto conductive pads on the back of the 19 mm by 9 mm PCB, and reading the charge of each electrode (Figure 2-8). The total capacitance between the two parallel electrodes is sent to a microcontroller Piksey Pico (Bits N Blobs Electronics [71]) that is connected to a serial port of a computer. This allows the powering of the microcontroller as well as the reading of the capacitance values using an algorithm written in Arduino. The microcontroller can also be powered using an external battery, and the data can be directly sent out to an output device such as a haptic system. Figure 2-8 displays the block diagram of the readout circuit and its components. Figure 2-8: Flowchart of the sensor readout circuit 40 2.3.1.4 Results The resulting sensor is shown in Figure 2-9. This sensor is used to detect pressure, including very small pressures (≤ 1 kPa [68]) that can be interpreted as touches. The flexibility of the sensor permits its utilization on flat and curved surfaces, including finger pads. Figure 2-9: Picture of the fabricated sensor To determine the pressure sensitivity, a load was applied to the sensor using a Bose Electroforce dynamic mechanical analyzer connected to a load cell. The area of the indenter pushing on the sensor was 11 mm by 11 mm. The experimental setup is shown below in Figure 2-10. All experiments below were conducted and analyzed by the author. 41 Figure 2-10: Pictures of the experiment displaying the indenter driven by the BOSE analyzer up and down and pressing on the sensor. The displacement of the indenter is measured by the BOSE system, and the force applied is measured by a load cell screwed above the clamped indenter. Five sinusoidal 0.3 N load cycles were applied to the sensor at a frequency of 0.1 Hz. The sensor is first pressed by a 0.08 N force to ensure contact between the indenter and the entire top surface of the sensor. The data is plotted in Figure 2-11. The capacitance changes with the change in force applied (Figure 2-11 (A)), meaning that the sensor responds correctly to pressure. In the same manner, the capacitance follows the displacement of the indenter, which is the displacement 42 of the top surface of the sensor with respect to the bottom of the sensor. However, we observe in Figure 2-11 (B) that the needed displacement to obtain the same load of 0.3 N increases from the first cycle (i.e. 0.43 mm) to the last cycle (i.e. 0.61 mm). This change in displacement demonstrates the viscoelasticity of the material and is discussed later. Figure 2-11: Sensor pressure sensitivity experiment: (A) Force applied on the sensor with the percent change in capacitance, (B) Indenter displacement with the percent change in capacitance. This data is also graphed in Figure 2-12 (A) to calculate the pressure sensitivity of the sensor. It displays the percent capacitance change as a function of pressure, where the maximum pressure applied is 3.7 kPa. The stepped nature of the percent change in capacitance versus the force applied is due to the load cell resolution, switching to step forces when required input forces 43 are small. This is also seen in the stress-strain curve (Figure 2-13). The sensitivity of the sensor is 4.14 % per kPa, obtained by fitting a linear model Figure 2-12 (A). The sensitivity to displacement is 52.45 % per mm (Figure 2-12 (B)) – obtained with the same method, with a small percent error due to the non-linearity of the curve depicted in Figure 2-12 (B). One hypothesis that might explain the non-linearity of this curve is the change in relative permittivity k as the sensor is being compressed – the dielectric is then composed of more EcoflexTM 00-30 (k= 2.8 [72]) than air (k =1.0006 [73]). This supposes that the percent change in capacitance is increasing more with higher strains due to the equation 𝐶 = 𝜖𝐴𝑑= 𝜖0𝑘𝐴𝑑 , explaining the curvature of the curve. Another hypothesis is that this can be due to the viscoelastic properties of the materials, where the capacitance actually follows the stress-strain curve (Figure 2-13). 44 Figure 2-12: Change in Capacitance of the sensor versus Pressure. The pink line represents the linear regression of the data, having a gradient of 4.14. 45 Figure 2-13: Stress-strain curve for a force applied of 0.3 N. To visualize the sensitivity of the sensor, a 500 mg (corresponding to a load of 4.91 x 10-3 N) acetaminophen tablet was gently placed on the sensor as shown in Figure 2-14. This corresponds to a pressure of ∼60 Pa over the area of the sensor (9 mm x 9 mm). The noise was calculated to be 0.0028 pF or approximately 0.3% of the total capacitance values. The change in capacitance due to the tablet pressure was 0.8%. This shows that the sensor can detect pressures as small as ∼60 Pa. 46 Figure 2-14: 500 mg load experiment (A) On the right; unloaded sensor, on the left; screenshot of the capacitance data from the Arduino serial monitor, (B) On the right; sensor loaded with the acetaminophen tablet, on the left; screenshot of the capacitance data from the Arduino serial monitor. We have applied a higher load of 10 N (corresponding to 123 kPa) at 0.1 Hz to completely compress the sensor and investigate the maximum pressure that the sensor can withstand. We used the same set up as the experiment displayed in Figure 2-10. The data is shown in Figure 2-15 and displays that the capacitance follows better the displacement (Figure 2-15 (B)) rather than the force (Figure 2-15 (A)), which is theoretically expected, knowing that that the capacitance changes as a function of the distance between the two electrodes. However, we can still observe that the 47 capacitance does not perfectly match the displacement. One theory that might explain this is the creep of the EcoflexTM outside of the electrodes, affecting the displacement of the indenter but not of the electrodes relative to each other. This experiment proves that the sensor can repeatedly undergo high loads, saturate, but still remain usable and sensitive. Figure 2-15 (B) demonstrates that the sensor saturates (i.e. reached the limits of the CDC range) at approximately 2.1 N (26 kPa). This corresponds to a displacement of 1.22 mm in Figure 2-15 (A). However, when fitted under the glove the sensor will experience a much smaller pressure due to the stiffness of the glove. This is analyzed in a later section. As described by Equation 2.1, the stiffness of the dielectric increases as the two outside layers are getting closer. This explains the smaller increase of the displacement under higher loads (Figure 2-15 (A)). 48 Figure 2-15: Sensor maximum pressure experiment: (A) Force applied on the sensor with the percent change in capacitance, (B) Displacement applied on the sensor with the percent change in capacitance, (C) Displacement and force data from the BOSE. 49 The same experiment was done but applying a 10 load cycles of 2 N to the sensor, corresponding to a pressure of 25 kPa, which is the maximum pressure (full scale) that the sensor can withstand before saturation (as described above). This data, presenting the percent change in capacitance as a function of pressure, is shown in Figure 2-16. The rest of the data can be found in Appendix A.2, Figure A.2- 1. This experiment aims to find the signal-to-noise ratio (SNR), non-linearity, hysteresis, repeatability error, and resolution at full scale. The non-linearity of the sensor was calculated to be 2.9 % for one cycle and 6 % over ten cycles by finding the maximum difference between the linear fit in black and the capacitance data in magenta. Additionally, the hysteresis error was found to be 3.4 % for one cycle, and 13.1 % for 10 cycles by finding the maximum difference in percent change in capacitance at a given pressure. The repeatability error was calculated to be 1.7 % by taking the maximum difference in change in capacitance at each cycle peak over ten cycles. Lastly, the signal-to-noise ratio (SNR) was calculated using Equation 2.2: 𝑺𝑵𝑹 = 𝟐𝟎 𝒍𝒐𝒈 (|𝑺𝒊𝒈𝒏𝒂𝒍||𝒔𝒕𝒅(𝑵𝒐𝒊𝒔𝒆)|) Equation 2.2 The change in capacitance here at full scale is 1.561 pF, and the noise standard deviation (std) is 0.0.0034 pf (by taking the standard deviation of the baseline capacitance before the first cycle), resulting in an SNR of value of 53.2 dB. Finally, the resolution of the sensor, meaning the smallest change that can be detected, is 25 Pa, calculated by finding the minimum pressure that the sensor can detect that is greater than noise. Creep information is explained later. 50 Figure 2-16: Percent change in capacitance of the sensor vs. pressure for ten load cycles of 2 N. To investigate further how the sensor behaves under high loads, we used the same setup described above to evaluate the creep of the sensor. We have applied a step load of 2 N (corresponding to ∼25 kPa) at 0.1 Hz to compress the sensor until right before saturation occurs, at about 2.1 N, as stated earlier. This load was held for ten minutes, then released to give 50 minutes to the sensor to relax. The data is displayed in Figure 2-17. Figure 2-17 (C) displays a characteristic of viscoelastic materials, where the strain first follows the stress (elastic response), but then takes some time to reach its maximum under a step stress (viscous response). The creep is calculated to be 5.9 % over the ten minutes step stress. See Figure 2-17 (C). After relaxation, we observe that the strain goes back down (elastic response) but does not recover fully (delayed elastic response). This can be modeled using a combination in series of the Maxwell and the Kelvin Voigt models, associating four elements – two dampers and two springs, pictured in Figure 2-18. 51 However, one could observe that the capacitance recovers faster than the strain (Figure 2-17 (B)). One hypothesis to explain this can be the creep of the top and bottom EcoflexTM layer of the sensor, encapsulating the electrodes, and affecting the displacement of the indenter independently from the change in distance between the electrodes. The non-recovered region might become a problem as it can affect the baseline, which would then need to be re-calculated. This can be investigated in future work, by conducting experiments with different materials or designing an adaptive filter (such as the Kalman filter [74]) accounting for this change. 52 Figure 2-17: Sensor capacitance behavior under a step 25 kPa stress applied for 10 minutes (A) % Change in Capacitance corresponding to the pressure applied, (B) Change in force and displacement from the BOSE mechanical analyzer, (C) % Change in capacitance corresponding to the displacement applied. 53 Figure 2-18: Burgers model showing the four elements and the behavior of the strain under a step stress. Reprinted from [75] ©2020, with permission from Elsevier. The sensor itself has a working range of 0 Pa to 25 kPa, with a sensitivity of 4.14 %, and a resolution of 25 Pa. This range includes the human light touch sensitivity of 100 Pa [3], which is important as the sensor needs to be able to detect touches (i.e. very small pressures ≤ 1 kPa) under a prosthetic glove that is quite stiff. It has a maximum creep of 5.9 % for a stress of ∼25 kPa. Moreover, the sensor has a non-linearity of 7.5 %, a hysteresis error of 1.9 %, and a repeatability error of 1.7 %. These values are listed later in Table 2-2. To be integrated in a prosthetic hand, we designed a fingertip system that hosts the sensor and its electronics, covered by the marketed prosthetic glove, sold with the Ottobock hand. 2.3.2 The Fingertip Support and its Connections We developed a fingertip system to protect and hold the sensor in place, provide the ability to activate a touchscreen, and have a close connection between the sensor and the PCB to minimize 54 parasitic capacitance (unwanted changes in capacitance due to surroundings). All components are described below, and the performance of the sensitized fingertip is discussed. 2.3.2.1 3D printed fingertip We designed a 3D printed fingertip that replaces the existing numb inner hand fingertip. This part is printed in Durable material (FormLabs, Canada [76]), enabling some deformations. The fingertip design hosts the sensor and the PCB as shown in Figure 2-19. Figure 2-19: Fingertip system including (a) the sensor, (b) the PCB, and (c) the 3D printed support The sensor is glued on the rounded finger pad of the support (Figure 2-20 (A)). The exposed carbon black stripes go through the cavities to lie flat on the back of the fingertip (Figure 2-20 (A)) and make connections with the 19 mm by 9 mm PCB that is tightly clamped on the back of the support (Figure 2-20 (B)). The deformable Durable material is essential here to avoid the 55 breaking of the clamping mechanism. This fingertip is designed to slide over the prosthetic finger of the hand mechanism (Figure 2-20 (C)). Figure 2-20: 3D printed support (A) Front of the fingertip where the sensor is glued onto the finger pad and the carbon black stripes go through the cavities, (B) Back of the fingertip where the PCB is tightly clamped to press the pads on the carbon black stripes and ensure the connection, (C) Cross-section of the fingertip showing where the hand mechanism finger fits. 2.3.2.2 Integration and Connections The sensitized fingertip is fitted onto the hand mechanism, as shown in Figure 2-21 (A). This part replaces the inner hand fingertip, cut beforehand (Figure 2-21 (B)). The system can then be fitted with the prosthetic glove. As a reminder, the prosthesis components are described earlier in Figure 1-4 on page 7. 56 Figure 2-21: Integration of the fingertip system (A) ) picture showing the fingertip device connected to the prosthetic hand’s mechanism, (B) Components integrated into the prosthetic hand including the (a) the sensitive fingertip, (b) the cut inner hand, and (c) the hand mechanism. The microcontroller also needs to be integrated within the hand. The Piksey Pico was chosen due to its small size (20.3 mm by 20.3 mm) and functionalities (inter-integrated circuit I2C containing a serial data line and a serial clock line for capacitance reading, Arduino pins to facilitate the coding, pulse width modulation pins to modulate the intensity of the feedback correlated to the capacitance read – this is discussed in Chapter 4). Its dimensions permit it to fit on the back of the hand mechanism, under the inner hand. To hold the microcontroller in place, we designed a 3D printed housing, as shown in Figure 2-22. 57 Figure 2-22: 3D printed microcontroller housing that enables the microcontroller to be integrated into the prosthetic hand, showing (A) left-side view, (B) front view, (C) right-side view. The housing attaches to the hand mechanism using a screw, the slot contains the microcontroller, and the wires go from the microcontroller to the PCB fingertip device as shown in Figures 2-23 and 2-32. This part is screwed in next to the black box containing the prosthesis electronics, including the battery connection of the prosthesis that is used to power the microcontroller, as pictured in Figure 2-23. The microcontroller is tightly held to the hand mechanism and does not add any volume to the prosthesis (Figure 2-23 (A)). This location allows easy access to the prosthesis battery connections inside the prosthesis electronics box, used to power the microcontroller (Figure 2-23 (C)). 58 Figure 2-23: Integration of the microcontroller 3D printed housing: (A) Attachment of the microcontroller housing using the existing screw that attaches the prosthesis electronics, (B) Picture showing the prosthesis electronics containing the battery connections, (C) Picture showing the wires coming out of the microcontroller to connect to the PCB and existing battery. The battery powering the prosthesis is located in the forearm. The hand screws into the forearm where conductive sliprings allow the power to pass through the wrist. The prosthesis battery (at least 6 Volts) provides power to open and close the hand. Some power is diverted to power the microcontroller just by connecting the ground and voltage-in pins of the microcontroller to the power pins of the prosthesis electronics located on the hand mechanism – as described above. 2.3.2.3 Soft cover A soft cover wrapping the fingertip holds the sensor in place, and protects it from any damage due to the fitting of the tight prosthetic glove. This part is 3D printed using the Elastic 59 material from FormLabs [76]. The cover shown in Figure 2-24 (A), presses softly on the sensor to make sure that the entire surface of the sensor is in contact with the cover (Figure 2-24 (B)). Figure 2-24: Soft cover (A) Soft cover, (B) Soft cover wrapping the sensor on the fingertip. 2.3.2.4 Ground plane In addition to the parallel-plate capacitance reading, the two electrodes couple due to fringe fields, as shown in Figure 2-25. This is called projected capacitance. An approaching finger or grounded metal then acts as a third electrode, decoupling these fringe fields and hence decreasing the total capacitance (Figure 2-25 (A)) [62]. This is very interesting as the drop in capacitance can detect human proximity. However, this data can become difficult to interpret as an increase in capacitance can be due to a finger moving away from the sensor or a pressure after approaching and touching the sensor. Also, in the case of a prosthetic sensory feedback system, the differentiation between humans and grounded metal will be needed and conveyed back to the user. This can be done using visual cues, but the feedback should not go off when the fingertip approaches grounded metals – the goal would be to have a proximity stimulation only when approaching humans. This can be investigated in future work. 60 In this research, we cancel the proximity effect and focus only on touch and pressure feedback – shear feedback is investigated in Chapter 3. To block the electric field coupling with an approaching finger and hence decreasing the capacitance (Figure 2-25 (A)), a ground plane is added to the sensor acting as a shield (Figure 2-25 (B)). Electric fields couple with the ground plane, but none couple with a finger or a grounded metal. Figure 2-25: Effect of a ground plane (A) A finger approaching decouples the electric field and decreases the capacitance, (B) The ground plane decouples the electric field and forms a shield around the sensor, disabling proximity detection. This allows the user to touch anything without affecting the capacitance reading and thus, the feedback. This shielding is achieved using a layer of conductive fabric connected to ground. The fabric is cut to the right dimensions using a template fitting the fingertip. It is then glued around the soft cover described above (Figure 2-26). The fabric is connected to the ground pin on the PCB placed on the back of the fingertip, using double-sided copper tape. 61 Figure 2-26: Conductive fabric wrapping the fingertip and acting as a ground plane. This additional layer also allows touchscreen interaction. The ground plane wrapping the fingertip decouples the electric fields between the drive and the sense electrodes on a touchscreen device (as described in Chapter 1, see Figure 1-12). The capacitance decreases, indicating the touch of the grounded fabric. The fingertip system including the sensor, 3D printed housing, soft cover, and ground, permits the detection of tactile stimuli and the interaction with touchscreen technologies. It needs to be protected – e.g. by the existing prosthetic glove provided by the prosthetic hand manufacturer – to avoid any possible damage and thus avoid additional care from the user. 2.3.2.5 Prosthetic Glove The prosthetic glove is an essential part of the hand. It protects it from dust, water, or other external dangers. The proposed fingertip system needs to be fully integrated in the hand and use the glove as a protection against possible damages caused during everyday activities. However, the glove that comes with the prosthetic arm is thick (1.3 mm) and has a higher elastic modulus 62 than EcoflexTM (1 MPa for silicone-rubber vs. 0.125 MPa for EcoflexTM [77]). This causes the system to be less sensitive when fitted under the glove. The performance of the entire system is described in the next section. Results and Discussion 2.4.1 Touch and Pressure Detection The fingertip sensitivity was tested using a load cell and a Bose Electroforce dynamic mechanical analyzer. All the followed experiments have small human errors – the glove covers the fingertip and hence hides the sensor, making it difficult to be precise when setting up the experiment. We applied five cycles of a sinusoidal force of 1 N peak to peak amplitude (offset force of 0.1 N, held for at least 45 seconds before starting the testing) using an 81 mm2 indenter as pictured in Figure 2-27 (D). This data is shown in Figure 2-27 (A), (B), and (C). The experiment was first done on the sensor laid flat (green curve). Then, the sinusoidal force was applied on the sensor glued onto the 3D printed fingertip, laying on the curved finger pad surface (blue curve, Figure 2-27 (D)). The experiment was done a third time on the fingertip including the sensor, the soft cover, and the fabric ground plane (red curve). Lastly, a finger portion of the prosthesis glove was fitted to the system (magenta curve, Figure 2-27 (E)) and the entire system was subjected to the force cycles. When laid flat (green curve), the sensor follows the force applied by the 81 mm2 indenter (Figure 2-27 (B), dashed black curve), meaning that the change in capacitance responds directly to the pressure applied (green curve) until at least 12 kPa. When laid on a curved surface (blue curve), the area activated is now 64 mm2 as we kept the same flat indenter. This affects the change 63 in capacitance as a smaller fraction of the total area of the top electrodes is being pushed towards the bottom electrode (Figure 2-27 (B), (C)). Thus, the linear portion of the percent change in capacitance is shorter. As the pressure increases, the change in capacitance reduces due to the increase in stiffness of the pyramidal structured dielectric, as described in Section 2.3.1.1.2, page 30. The addition of the ground (red curve) reduces the percent change in capacitance as the baseline capacitance is changed (i.e. the ground steals some of the electric field, as explained in Chapter 1), and the ground plane comes closer to the top electrode as the pressure increases. Finally, the fitting of the glove (magenta curve) considerably reduces the sensitivity of the fingertip due to its thickness, higher elastic modulus, and its bonding to the fingertip. Figure 2-27 (A) shows that the displacement of the indenter needed to achieve a normal force of 1 N is less once the glove is fitted (magenta curve), proving that the glove is stiffer than the sensor. Moreover, the stiffer properties of the glove distribute the force applied over the entire area of the finger pad, contributing to a smaller change in capacitance overall. Despite the fitting of the glove, the sensor still senses small pressures (around 1 kPa) and achieves a 10 % change in capacitance for 1 N force. The linear fit of the magenta curve shown in Figure 2-27 (C) has a gradient of 0.7, verifying the pressure sensitivity of the fingertip to be around 0.7 % per kPa. 64 Figure 2-27: Pressure sensitivity experiment on flat sensor (green), curved sensor (blue), curved sensor wrapped with a ground plane (red), and entire fingertip including the fitting of the prosthetic glove (magenta). (A) Displacement and force applied data, (B) % Change in capacitance and force data, (C) % Change in capacitance vs. pressure, (D) experiment setup for the curved sensor, (E) experiment setup for the entire fingertip. 65 The stiffness of the glove reduces the pressure sensitivity of the system but acts as a protection against dust, heat, water, and high grip force. To test the reliability of our fingertip system under medium pressures, we applied ten sinusoidal force cycles of 6.4 N (100 kPa) at 0.1 Hz using the Bose Electroforce dynamic mechanical analyzer, as depicted in Figure 2-28 (A), (B), and (C). This set of data shows that the system follows the force applied rather than the displacement of the indenter. This might be due to the viscoelastic properties of the sensor and the glove, as well as a potential gap existing between the sensor and the glove. In this case, the displacement of the indenter does not have the time to return to zero because of the creep caused by the glove. Figure 2-29 displays the percent change in capacitance as a function of pressure for all ten cycles. Here, the capacitance increases when undergoing pressures up to at least 100 kPa. This capacitance change is linear up to around 25 kPa (2 N force on an area of 64 mm2). Then, the increase in capacitance is less, due to the increase in stiffness of the sensor (as expected when looking at Figure 2-6, page 35), as well as the glove being bonded on either side of the sensor. It is important to note that, as mentioned at the beginning of this section, many errors arise from the setup of the experiment – the glove is hiding the fingertip device, making it difficult to align with the indenter. Nevertheless, this experiment shows that system can withstand high pressures (100 kPa) and remain stable. This data was used to calculate the errors and resolution of the system. The hysteresis error is 9.6 % after one cycle and 18.7 % over 10 cycles, found by the maximum difference in percent change in capacitance at a given pressure. The repeatability error is 1.6 % found by the maximum difference in percent change in capacitance between cycle peaks. The SNR was calculated to be 70 dB using Equation 2.2. The non-linearity error is 4.4 % in the linear range. This was found by 66 reproducing the same experiment in the linear range, applying load cycles of 2 N to the system, and fitting a linear model. This data can be found in Appendix A.3, Figure A.3- 1. In the non-linear range (full scale), the non-linearity error is 17.4 % for one cycle. Lastly, the resolution of the system is 0.17 kPa, calculated by finding the minimum change in capacitance that is greater than noise. 67 Figure 2-28: 100 kPa stress experiment on the fingertip system (A) % Change in capacitance corresponding to the pressure applied, (B) Change in force and displacement from the BOSE mechanical analyzer, (C) % Change in capacitance corresponding to the displacement applied. 68 Figure 2-29: Fingertip system % change in capacitance vs. Pressure. We investigated how the system behaves under a step stress of 100 kPa. Using the same Bose Electroforce dynamic mechanical analyzer, we applied that stress on the sensor and held it for ten minutes. The stress was then relaxed and the measurements were maintained for 50 minutes. The data is shown in Figure 2-30. This clearly shows the viscoelastic properties of the material, displaying creep. This is more than the creep of the sensor itself, displayed in Figure 2-17, page 75. The system experiences a creep of 22.9 % for a stress of 100 kPa. Although this is significant, the sensor does not really follow this pattern. This can be explained by the creep due to the glove, how it is bonded to the fingertip system, or due to potential gap existing between the glove and the sensor. 69 Figure 2-30: Fingertip capacitance behavior under a 100 kPa step stress applied for 10 minutes then relaxed for 50 min, (A) % Change in capacitance corresponding to the pressure applied, (B) Change in force and displacement from the BOSE mechanical analyzer, (C) % Change in capacitance corresponding to the displacement applied. 70 The viscoelasticity of the system (viscoelasticity of the glove and the sensor) might become a problem when applying high pressure and wanting to detect a touch soon after. Using the same setup (Figure 2-27 (E), page 64), we applied a step stress of 100 kPa that was held for five seconds – considering that the fingertip is implemented with sensory feedback, the user would know that this pressure is too high and so would release in five seconds. Another stress step of 5 kPa was applied after ten seconds and for five seconds. We repeated this sequence five times (Figure 2-31). The change in capacitance follows the displacement of the indenter where creep is observed for both displacement and capacitance (Figure 2-31 (C)). However, the capacitance seems to recover faster than the strain. Again, this can be explained by creep and a potential gap existing between the sensor and the glove, where the sensor recovers at a different rate than the glove. This of course also depends on the bonding of the glove to the fingertip. One last observation is the repeatability error of each cycle, where all the capacitance changes for each stress level remain within 1% of each other. This experiment was also conducted on the sensor alone, lying on a flat surface, with stress steps of 25 kPa followed by 1.2 kPa. This data can be found in Appendix A.2, Figure A.2- 2. Comparing the two sets, sensor on the fingertip under the glove vs. sensor on a flat surface, gives an idea of the effect of the glove on the sensor. Although compressed by a pressure of 100 kPa for five seconds, a following pressure of 5 kPa is still sensed by the fingertip system, displaying an increase in capacitance of 14.4 % from the baseline at the beginning of the experiment, or 5.7% after 10 seconds of the release of the 100 kPa stress (Figure 2-31 (A)). This change in baseline will need to be investigated in future work to predict changes due to high stresses. 71 Figure 2-31: Step experiment on the fingertip system: (A) % Change in capacitance and pressure data, (B) Force and displacement data, (C) Displacement and % change in capacitance data. 72 Fitted under the glove, the proposed fingertip has a working range of 0 kPa to at least 100 kPa, with a resolution of 0.17 kPa or ∼10 mN. It can detect touches as well as high pressures, allowing users to feel the environment around them. All specifications, including the ones from the bare sensor laid on flat surface, can be found in Table 2-2. Nevertheless, challenges remain and more characterizations need to be achieved. This is discussed in a later section. Table 2-2: Sensor and fingertip system specifications. 73 2.4.2 Hand Integration The sensitized fingertip is fully integrated into the prosthetic hand. This is very important as users should be able to use their prosthetic hand without thinking about protecting the sensitized fingertip. Users sometimes fully immerse the hand in water or use it for specific tasks such as gardening or manual work. The prosthetic glove protects the prosthesis from any dust or water likely causing damage to the hand. Thus, to ensure durability and the easy-care of the system, the fingertip device needs to be fully integrated within the hand and protected by the prosthetic glove. The microcontroller and its connections need to first be placed on the hand mechanism. The inner hand can then be fitted, the index fingertip cut and replaced by the fingertip system that is positioned on the hand mechanism finger, where it is soldered to the microcontroller wires. Figure 2-32 represents how the device is located within the hand. The fit of the sensitized fingertip will be done by the prosthetics clinic. The system does not impede the fit of the inner hand (Figure 2-32 (C)). It only occupies empty space within the hand and thus does not add any volume to the initial hand. The device is powered using the prosthesis battery and any output, such as sensory feedback, can be sent via Bluetooth (future work). 74 Figure 2-32: Fingertip Integration: (A) Bare hand mechanism, (B) Hand mechanism with the sensitized fingertip device, occupying empty spaces only within the inner hand, (C) Final product including the fingertip system fitted to the hand mechanism and covered by the inner hand, this will be protected by the prosthetic glove. The device, protected by the prosthetic glove, does not prevent prosthesis users from doing certain activities. The pressure detection was tested before and after the prosthetic hand was dipped into water, showing that the glove permits the system to work underwater. Some other testing needs to be done to test the device under different temperatures and at higher pressures to ensure its reliability under all real-life situations. 2.4.3 Touchscreen Interaction To understand the importance of such a feature, we conducted a survey (Appendix C.1, protocol number: H19-01120) to investigate the perceived benefits of sensory feedback and touchscreen activation ability for prosthetic users. Participants were asked to rank this need from 75 0 to 10, 0 being not beneficial at all, 10 being very beneficial. Among 16 participants, the benefit of a prosthesis enabling touchscreen interaction was rated 5.6/10, including 6 participants of age 19-29 rating this need 8.8/10. Knowing this, the fingertip was designed to enable touchscreen interaction using a ground plane connection. Users might need to press down a little to make sure that the grounded fabric cover is close enough to the touchscreen technology to activate it. This new feature permits users to continue to use their functional hand while simply answering a phone call with their prosthesis, for example. Figure 2-33 shows some snips of a video recording a prosthetic hand typing a phone number using the integrated fingertip system. It is also very important for bilateral amputees to have the ability to interact with their touchscreen technologies, as they are wearing a prosthetic hand on both hands. Figure 2-33: Touchscreen Experiment. Video screenshots of a prosthetic hand with the fingertip system under the glove typing a phone number. 76 2.4.4 Cost One main goal of this research is to provide a feedback add-on at low cost. All materials and processes were chosen carefully to minimize the cost of the device. The description below explains the cost to produce one sensor. This would need to be further evaluated to give an accurate cost estimation when mass-produced, including fabrication cost, and satisfying the potential market of ∼550,000 upper limb amputees in the United States only [4]. The sensor uses EcoflexTM- 0030 that can be purchased in gallons for approximately $168. The fabrication requires no more than 20 grams of EcoflexTM, costing ∼$0.90. Carbon black costs ∼$428 for 100 g. The two electrodes use 0.4 g of carbon black, costing ∼$1.70. The Durable material from FormLabs costs $175 per liter and the elastic material for the soft cover costs $199 per liter. The 3D printed finger and microcontroller house use ∼12 ml of material costing ∼$2, and the soft cover costs ∼$0.50. The microcontroller is purchased from Bits and Blobs for $11, the PCB costs ∼$12 including the CDC chip cost of ∼$11. Including extra costs from wires, solder, glue, and conductive fabric, the total cost of the materials to produce the fingertip system reaches a cost of $49. This cost is detailed below in Table 2-3. This does not include fabrication cost and profit. 77 Table 2-3: Summarized cost of each components 2.4.5 Challenges and Future Work Despite the functionality and integration of the sensitized system, some challenges remain and improvements can be made. The sensitivity of the devices is found to vary significantly between devices. In three nearly identical sensors, the baseline capacitance value may vary by roughly 9%, with relative changes in capacitance per kPa differing by about 1% at low pressures. The sensor can be manufactured using laser-cut masks, a doctor-balding machine, and a better alignment method to improve fabrication consistency and reduce needed software modifications. Moreover, the viscoelastic properties of the material might cause a problem over time. If the user 78 often compresses the sensor using the maximum grip force of 100 N, the sensor might partially deform. Different materials with stiffer properties, such as PDMS, can be used to fabricate the system and investigate the change in creep and percent errors. Another way of approaching this challenge would be through software development (e.g. implementing adaptive filters [74]). Indeed, if the baseline capacitance is modified or if the sensor does not recover after high pressures (i.e. due to the viscoelasticity properties of the sensor material and the glove), the baseline will need to be recalculated to evaluate the stress applied by knowing the capacitance. This is crucial to send accurate sensory feedback to the user. Future work includes further characterization of the system including working temperature range, repeatability experiments to find the cycle life of the device, and the investigation of the effect of the position of the glove. Conclusion We have developed a low-cost fingertip device capable of detecting small pressure of 1 kPa (that can be interpreted as touches), withstanding pressures of 100 kPa for object manipulation, and activating touchscreen technologies while being protected under a prosthetic glove. This compact device is composed of a highly sensitive sensor and a PCB, combined on a 3D printed fingertip, replacing the existing numb silicone inner hand fingertip. A CDC reads the capacitance data and sends it to a microcontroller located at the back of the prosthetic hand and powered by the prosthesis battery. Fully integrated inside the prosthetic hand, the device does not add any volume or significant weight to the hand, allowing users to benefit from a sensory feedback in real-life applications without the extra care needed. 79 For future work, the microcontroller needs to be redesigned to reduce its size even more. This would allow a Bluetooth module to be added within the hand, wirelessly sending an output sensory feedback signal to the user. Furthermore, the fingertip system can be added to the prosthetic thumb in addition to the index fingertip. This would give an acute grip force perception by reading out the pressure on both sides of the manipulated object. In addition to touch and pressure, we took the fingertip system a step further to detect shear. Chapter 3 describes the redesign of the electrode architecture, the use of a FLEX-PCB, and the modification of the 3D printed support, to implement shear detection. All touch, pressure, and shear information are sent back to the user using two types of feedback – mechanotactile and vibrotactile feedbacks, described in Chapter 4 – to allow users to “feel” all three types of stimulus. 80 81 Chapter 3: Normal Force and Shear Sensing Shear sensing is an ability that helps us to interact with our environment. Prosthesis users could benefit from this ability to adjust their prosthesis grip force and avoid objects to slip between their fingertips, or to simply feel their fingertip sweeping across a surface. Introduction Touch and pressure feedback can empower the use of prosthetic hands (i.e. provide better accuracy and speed as discussed in Chapter 2) as it permits users to measure the grip force with which they are squeezing manipulated objects. Shear feedback is another interesting feature that can help amputees in their everyday lives; it gives the possibility to artificial hand users to “feel” when an object is slipping between the prosthetic fingertips, evaluate the required squeezing force to open a bottle, or simply experience a caress. Shear detection is thus an ability that is beneficial to object manipulation [78]. Restoring shear sensation may also help to simply regain the feel of sweeping fingertips across a surface or when playing a musical instrument. Some exploration of shear sensing in prosthetic fingers has been done. On the commercial side, the myoelectric hand SpeedHand Speed from Ottobock offers a thumb sensor (SUVA) that detects shear to automatically adjust the grip force – useful for example when holding a cup of water being filled up [79], [80]. The sensor was not intended to be used for sensory feedback – as it automatically regulates the grip force – and impedes users to consciously control their hand, limiting its embodiment. 82 A few researchers have studied pressure and shear detection ([81], [82], [62]), or slip feedback (using mechanotactile feedback [83] or electrical stimulation for slip speed feedback [48]). But, to our knowledge, none include both the detection and feedback of touch, pressure, and shear simultaneously for prosthetic hands, and no prosthetic hand on the market proposes such combined abilities. To integrate these features, we build on the work from Sarwar et al. [62] to develop and characterize an updated version of the sensitized fingertip described in Chapter 2, including a soft capacitive sensor detecting touch, pressure, and shear. This additional component aims to permit prosthetic users to detect and feel shear to accurately apply a minimum sufficient grip force to the grasped object, and enhance fingertip sensations through shear detection and feedback. Overview Sarwar et al. developed a capacitive sensor that can detect proximity, touch, pressure, and shear using a specific dielectric that buckles like skin, an electrode architecture allowing pressure and shear detection in the x and y directions, and materials that can be easily stretched or deformed [62]. Based on this work, we developed a fingertip that includes a capacitive sensor with a specific dielectric structure and electrode architecture to detect touch, pressure, and shear. This sensor connects to a flexible PCB to minimize the bulkiness of the fingertip and to ultimately completely integrate it under the glove. In this chapter, we will first describe the new sensor including the dielectric structure, the electrode architecture (developed by a team led by Mirza us Sarwar including the author), as well as the sensor pressure and shear characterization on a flat surface (developed by the author and 83 analyzed with the help of Mirza us Sarwar). Then, we will present the modifications of the fingertip system (done by the team) to include a FLEX-PCB. Finally, we will describe how we tested the sensor in a real-life task and define how the sensor and the entire system can be fully characterized in the future, including curved surface characterization. Implementation of Shear in the Sensor The sensor described in Chapter 2 only detects pressure stimuli (direction z). To include shear detection in x and y, we propose an electrode architecture using five electrodes instead of two (four pads directly on the PCB at the bottom of the sensor, one at the top above the dielectric). The dielectric is structured with cuboid pillars to maintain stiffness in compression while enabling increased compliance to shear. We characterized the sensor using a mechanical setup including an indenter connected to a load cell that measures how much force and torque are applied to our sensor. 3.3.1 Electrode Architecture and Dielectric Sarwar et al. proposed an architecture to detect proximity, touch, pressure, and shear using a single sensor. It consists of one big square electrode partially overlapping four smaller square electrodes [62]. A picture of the sensor (we will call it “base sensor”) is shown in Figure 3-1 (B). This architecture allows the detection of pressure and shear in all directions. Figure 3-1 (A) represents a diagram of the sensor that depicts the four electrodes at the top of the sensor, and the bottom electrode in black. When an x-direction shear force is applied to the sensor, electrodes 1 and 3 move in +x, increasing the area of overlap between electrode 1 and the bottom electrode, and decreasing the area of overlap between electrode 3 and the bottom electrode. Thus, the 84 capacitance of 1 increases, and the capacitance of 3 decreases. The same phenomenon happens in the y-direction with electrodes 2 and 4. When a pressure is applied, all four electrodes increase in capacitance simultaneously. Figure 3-1: Base Sensor top view (A) Base sensor diagram representing the four colored electrodes on top of the black bottom electrode as well as x and y-axes, (B) Picture of the fabricated shear sensor. Normal forces are applied in the Z-direction, and shear in the X and Y directions. One issue with this architecture is the closeness of the traces, adding parasitic capacitance (e.g. the blue trace causes the percent change in capacitance to be less due to its proximity with the black trace). We developed another version of the sensor to ground each trace: the four small electrodes of size 2 mm x 2 mm are directly on the PCB where a ground plane was added to shield any unwanted fringe fields. This sensor is depicted in Figure 3-2. 85 Figure 3-2: New shear sensor design (A) Shear sensor diagram depicting the four bottom colored electrodes directly placed on the PCB and the top electrode patterned on top of the EcoflexTM dielectric, (B) the fabricated PCB shear sensor, showing the sensor region with the bonded dielectric and carbon black electrode (top), and the sensor electronics (bottom region). The dielectric, made of pillars, is fabricated using the method described in the previous chapter (Section 2.3.1.2, page 36), where degassed EcoflexTM is poured into a mold including square pillars of 2 mm size, 1 mm height, and 2 mm pitch. Then, is it degassed, doctor-bladed, and cured for 20 minutes at 60°C. The electrode composed of EcoflexTM and carbon black composite at a ratio of 1:10 by weight is patterned on the mold filled with EcoflexTM using a mask cut at the dimensions of the electrode of size 6 mm x 6 mm. Ecoflex TM is then degassed and spun on top of the electrodes and the PCB at 1400 rpm. The dielectric and electrodes are then bonded directly onto the PCB, on the uncured Ecoflex TM where the top electrode is aligned by hand with the PCB electrode pads. The sensor is cured for 60 minutes at 60°C. Its final thickness is 1.2 mm not including the PCB. 86 3.3.2 Characterization The characterization setup, designed and fabricated by the author, uses a ThorLabs NanoMax 3-axis stage (moving in the x, y, and z-directions by 4 mm maximum), and an ATI Nano 17 6-axis force sensor (load cell). The specifications of the ATI Nano 17 and the ThorLabds NanoMax stage are described in Appendix B.1. The torque and force ranges (up to maximum 12 N for the force and 120 N.mm for the torque) limits our experiments. Characterization with higher forces will need to be conducted with a different load cell. The setup is shown in Figure 3-3 below. The sensor PCB is clamped on the sensor holder and can be moved along the x and y axes using screws. This permits the sensor to be placed right under the load cell. The adapter holds the load cell in place and is fixed to the immovable fixture. It can also be tilted in the x or y directions using screws to ensure a perfect flatness between the load cell and our sensor. A square indenter of area 196 mm2 is screwed to the load cell to press on our sensor. Figure 3-3: Mechanical Characterization Setup. Each component is presented on the diagram on the left and in the picture on the right. 87 Each experiment, conducted and analyzed by the author under the supervision of Dr. Sarwar, consists of first aligning the sensor by eye to try to match the axes of the load cell and the stage with the axes of our sensor. The second step is to move the stage upwards so that the sensor gently touches the load cell (we applied a force of 0.1 N approximately). Pressure and shear measurement can start from there. One challenge for the sensor is to determine if shear and pressure can be independently measured. The approach to separate the two is based on separating capacitive change due to compression from those due to lateral movements, as will be described below. 3.3.2.1 Pressure measurements After applying a small load of 0.1 N to the sensor – to ensure that that the entire surface of the indenter was in contact – we gradually moved the stage up and back down from 0.15 mm to 0.45 mm with steps of 0.10 mm (corresponding to pressures of 1.9 kPa, 3.16 kPa, 4.54 kPa, and 5.92 kPa). Figure 3-4 (A) shows first the force and torque data of the load cell, then the percent capacitance changes of our sensor, and finally, the percent change in capacitance with the pressure applied. The top yellow (Fz) curve shows the steps in normal force corresponding to the stage moving up and pressing on the sensor against the load cell, with shear forces (Fx and Fy) remaining largely unchanged, as expected. There is some shear applied in the y-direction, as can be seen from the small upward steps in the dark orange curve, showing that the applied force is not purely normal (first graph). Some torque is also evident (second graph), which has been minimized by adjusting the tilt, but is difficult to completely remove. The capacitances from all four top electrodes increase, as expected on compression (third and fourth graphs). Figure 3-4 (B) plots the percent 88 change in capacitance beyond the initial force of 0.1 N. These two graphs show that all electrodes behave the same way when pressed down uniformly. The spread of the lines can be due to torque (small incline of the top surface of our sensor or a slight tilt in the load cell) or to a difference of capacitance (the baseline capacitance, Co, is slightly different for each electrode due to different connection lengths, soldering differences, alignment, fabrication errors, and other parasitic effects, which shift relative slope, and can also influence sensitivity). Appendix B.2, Figure B.2- 1 displays the same graphs for the previous sensor, showing the benefits of the shielded electrode pads on the PCB – reducing trace parasitic capacitance that causes larger discrepancies between electrode responses, as seen in graphs (B) and (C). 89 Figure 3-4: Pressure Characterization (A) The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the pressure applied by the load cell in black, (B) Percent change in capacitance for each displacement in the z-direction, (C) Percent change in capacitance for each pressure in the z-direction. The sensitivity of this sensor is 4.00 % per kPa – when a linear model is fit to Figure 3-4 (C). This is lower than the sensor described in Chapter 2 (4.14 %) due to the cuboid structured dielectric replacing the pyramidal one. The pressure resolution of this sensor is 0.2 kPa or 39 mN. This is again poorer than the sensor described in Chapter 2 (25 Pa or 2 mN), because of the stiffer dielectric, and the higher noise due to the addition of electronics. 90 3.3.2.2 Shear Measurements After applying an initial load of 0.1 N, we moved the stage up to 0.11 mm to apply a load of 0.3 N (3.1 kPa) on our sensor. This corresponds to ∼1.5 kPa, which slightly higher than a touch of 1 kPa [68]. Then, we repeatedly moved the stage in one direction (x or y), back and forth, five times. We did this experiment for all +x, -x, +y, and -y directions. This experiment shows the reliability of the sensor. Figure 3-5 below displays the force and torque values of the load cell, the change in capacitance of our sensor, and the change in capacitance with the corresponding shear force. The first graph shows the force applied on the load cell in all directions (Fz, Fy, Fx), due to the sensor on the stage moving up and applying pressure on the load cell. The first step corresponds to the initial load of 0.3 N in the yellow curve. This pressure remains quite constant during the shear experiment – small changes are due to the soft nature of the sensor implying small uniformities in torques. The changes in the blue line are due to the stage moving in the negative x-direction of the sensor, and so shearing the load cell due to friction. The second graph shows that the torque in the y-direction is changing, which is expected as the shear tilts the load cell due to friction. Again, very small changes in torque are seen in the x-directions, but this is difficult to remove to apply a perfect normal force. This set of data shows that the capacitance of electrode 3 increases and the one of electrode 1 decreases as expected, while 2 and 4 remain unchanged. The rest of the data can be found in Appendix B.4, Figure B.4- 1, Figure B.4- 2, and Figure B.4- 3. 91 Figure 3-5: Shear data in the x-direction. The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the shear force in black experienced by the load cell in the x-direction. A second experiment was conducted to show how the sensor behaves when undergoing shear at different normal force loads. We first moved the stage up to apply a normal load of 0.3 N on our sensor. Then, the stage was moved in the x-direction, back and forth, by 0.1, 0.2, 0.3, and 0.4 mm. This was done for all directions and normal loads of 0.3 N, 0.6 N, 0.9 N, and 1.2 N – including the 0.1 N initial force. The data is displayed in Figure 3-6 – depicting the change in 92 capacitance depending on the force and displacement beyond the initial force of 0.1 N (i.e. plotting the percent change in capacitance only due to shear). Each graph presents the data for each normal force applied using a different marker (e.g. stars for 0.3 N). The goal is to show that the magnitude of normal force does not affect the change in capacitance due to shear. Ideally, when shearing in the x-direction, the green lines representing the capacitances of electrode 3 at different normal forces should match and go up at the same angle that the capacitances of electrode 1 in blue go down, and the magenta and red lines should be perfectly flat. This should be the case for all directions. Here, the capacitances of the electrodes not involved in the shear are not changing much (e.g. magenta and red for the x-direction). All changes in capacitance due to different forces are relatively close to each other. In the x-direction, the spread of the green and blue lines as well as the non-flatness of the red and magenta lines might be due to noise or small errors in the experimental setup (tilt in the load cell or the sensor, possible micro-slips between the load cell and the sensor when shearing). The same goes for the y-direction. These errors have a magnitude of about 1 %. The characterization setup is currently re-designed with machined parts to see if these errors are due to the setup or the sensor itself. This set of data shows that the shear data is independent of the normal pressure, and that the sensor exhibits a shear sensitivity of ∼3 % per kPa. The shear resolution of the sensor is ∼0.4 kPa or ∼80 mN. As a reference and to display the progress of the sensor, the base sensor shear data can be found in Appendix B.2, Figure B.2- 2, showing that the new shielded sensor exhibits a better shear sensitivity (∼2 % per kPa shear sensitivity for the base sensor), fewer discrepancies between each electrode, and a better consistency in all directions. 93 Figure 3-6: Shear data at different normal loads of 0.3 N, 0.6 N, 0.9 N, and 1.2 N (A) Percent change in capacitance for the +x-direction as a function of displacement and shear force, (B) -x-direction data, (C) +y-direction direction, (D) -y-direction data. 94 Using a mathematical model developed by M.S. Sarwar and presented in Appendix B.3, we can estimate the shear and the force strains independently from each other. This is done by calculating the capacitance using the equation 𝐶 = 𝜖𝐴𝑑 before and after applying forces to the sensor. This allows us to estimate the magnitude of normal and shear strains being applied to the sensor using the capacitance values of each electrode in real-time. The calculated shear displacement is C2C′1−C1C′2C′2+C′1, scaled by a factor 𝜖𝑊𝜆𝑑, where 𝜀 is the dielectric constant, W is the width of the electrode, λ is the shear displacement, and d is the distance between the electrodes (C represents the initial capacitance and C’ is the capacitance after pressure and shear). This equation estimates the shear displacement independently from the normal strain applied, as explained in Appendix B.3. Figure 3-7 plots this calculated shear displacement, using this equation, based on the capacitance of each electrode, versus the actual shear displacement. In Figure 3-7(A), the four lines going up are the calculated shear displacement for each normal force (e.g. normal force of 0.3 N in orange) using the capacitance of electrodes 1 and 3 only as a function of the shear displacement of the stage of values 0.1, 0.2, 0.3, and 0.4 mm. The four other lines are the calculated shear displacement for each normal force for electrodes 2 and 4. These switch in Figure 3-7 (B), as now electrodes 2 and 4 are the ones involved in the shear. In the x-direction (Figure 3-7(A)), we can see that electrodes 1 and 3 are the ones involved, and 2 and 4, for the y-direction, are not. As these two graphs plot the calculated versus the actual shear displacement independently of the normal force, we expect the lines for each normal force to perfectly match. Thus, a good shear displacement estimation can be done if the data can be fit in two single lines going through zero, and with one of them being flat. The spread of the lines is due 95 to small errors including misalignment during testing and fabrication, tilt and misalignment in the setup, as well as soldering effects causing changes in the baseline capacitance for each electrode. The calculated shear displacement versus shear force can be found in Appendix B.4, Figure B.4- 4. 96 Figure 3-7: Shear characterization curve displaying the calculated shear strain, based on the capacitance value of each electrode, as a function of the actual shear displacement applied. This shows that shear can be independently estimated and predicted from each electrode capacitance (A) Applied x-axis estimated shear strain, (B) Applied y-axis estimated shear strain. 97 We also calculated the value of the normal strain applied when the sensor is being sheared to be ΔC1 +ΔC2 +ΔC3 +ΔC4 ΔC′1+ΔC′2+ΔC′3+ΔC′4, scaled by a factor ηd, where η is the change in distance between the electrode and d is the initial distance between the electrode. This equation estimates the normal strain independently from the shear displacement, as explained in Appendix B.3. Results showing calculated normal strains using this equation, based on the electrode capacitance, versus actual normal displacement are displayed in Figure 3-8. These four plots (representing all four axes) aim to see if the calculated normal strain using the capacitance values of each electrode, independently from the shear displacement, can provide an estimation of the actual normal displacement experienced by the load cell – in this case 0.11, 0.23, 0.33, and 0.45 mm. In Figure 3-8 (A), for example, the four lines present the calculated normal strain for four different shear displacements using capacitance data plotted versus the normal applied strains of 0.11, 0.23, 0.33, 0.45 mm corresponding to normal forces 0.3, 0.6, 0.9, and 1.2 N. These four shear displacements of 0.1, 0.2, 0.3, and 0.4 mm are applied to test the degree of shear independence (e.g. orange line present the value of the calculated normal forces for a shear displacement of 0.1 mm as indicated in the legend, and so on). Ideally, the estimated normal capacitance component plotted on the vertical axis, will be independent of shear applied. In this case, if the data can be fitted with a single straight line that goes through zero, then this makes the estimation of strain simple. This is almost the case in Figure 3-8, showing that by knowing the capacitance of each electrode we can estimate the normal strain applied on the sensor This is not perfect due to small errors in the characterization setup, data analysis, and sensor fabrication, but remains in a 95% confidence interval when finding the standard deviation. The same graphs presenting the 98 calculated normal force as a function of the normal displacement can be found in Appendix B.4, Figure B.4- 5. Figure 3-8: Normal strain characterization curves of both x and y directions displaying the calculated normal strain, based on the capacitance value of each electrode, as a function of the actual normal displacement. This shows that the normal strain can be estimated and predicted independently from shear. 99 After characterizing the sensor on a flat surface, the next step is to redesign the fingertip system described in Chapter 2 to fit the new sensor. Fingertip/Curved surface implementation The described shear sensor needs to be implemented and characterized on a curved surface to fit a prosthetic fingertip, which is typically rounded. We modified the fingertip device described in Chapter 2 to include the new shear sensing version of the sensor (described above) on a FLEX-PCB that minimizes the required space and ensures touch, pressure, and shear detection. 3.4.1 PCB Design and 3D Printed Support Modifications As this sensor uses five sensing pads, the PCB described in Chapter 2 needed to be modified. As a reminder, the sensor described in Chapter 2 includes two traces that wrap around the fingertip to make a connection with the PCB clamped on the back of the fingertip. However, having the new sensor directly on the PCB makes this impossible. To overcome this issue, we decided to use a FLEX PCB that integrates four square pads and traces as shown in Figure 3-9 (A). The dielectric including the top electrode is bonded directly to the PCB using uncured EcoflexTM spun on the sensor, placing the dielectric, and letting it cure for 60 minutes at 60°C (Figure 3-9 (B)). The dimensions are the same as the previous sensor on the rigid PCB, displayed in Figure 3-2. In this design, the bottom part hosting the electrodes and the traces in the middle section are flexible. The top part is rigid as the CDC chips cannot bend. This PCB is designed to specifically fit on a fingertip, where the bottom square part including the electrodes can wrap on a curved surface and the top rigid part can fit on the back of the fingertip. We updated the fingertip 3D printed support described in Chapter 2 to meet these requirements (Figure 3-9 (C)). 100 Figure 3-9: FLEX-PCB design and implementation (A) Raw FLEX-PCB with dimensions including four electrode pads that form the bottom layer of the capacitive fingertip sensor (top end), and the rigid portion of the PCB that houses the electronics, and sits on the back of the finger (where the fingernail would be), four flexible beams electrically connect the electronics to the sensor. (B) Soldered FLEX-PCB including the shear sensor (dielectric placed over the four electrodes and fifth electrode patterned on top), (C) Diagram of the updated fingertip system of diameter 18 mm, including the sensor on the FLEX PCB integrated into a 3D printed holder. Left view showing the fingertip pad, and right view showing the ‘fingernail’ side holding the electronics. The soft 3D printed cover described in Chapter 2 has also been modified to fit the fingertip displayed above. The conductive fabric still wraps the entire system to shield electric fields and permit touchscreen interaction. 3.4.2 Curved Surface Characterization We designed a gripper system to put our new sensor system to practice and regulate the grip force in real-time. The gripper consists of a gear system mounted on a stepper motor. This 101 characterization includes providing force feedback during the grasping of a Styrofoam cup as an initial test case to show the working principle of the sensor. 3.4.2.1 Gripper Design The gripper design is displayed in Figure 3-10. The assembly consists of two 3D printed gears including the thumb and index finger (made of Clear Material from FormLabs). One gear is mounted to the shaft of a stepper motor (QMOT QSH4218-35). The stepper motor is controlled using an Arduino connected to a stepper motor driver (EasyDriver 12779 from Sparkfun ®) and its power supply. Both gear shafts are mounted on ball bearings. The fingertip holds the sensor as displayed in the previous section. It is a detachable part to allow change in the grip angle or the sensor itself. 102 Figure 3-10: Gripper Design 3.4.2.2 Pressure and Shear Experiment We conducted an experiment to simulate a real-life situation and test normal force and shear detection simultaneously. This consists of the gripper holding an empty cup getting filled up with water. Figure 3-11 shows snapshots of this experiment recorded by a camera, with normal force and shear measurements on the right. The grippers clamp the Styrofoam cup on the left and right sides, with the fingertip sensor located on the right gripper, at the right-hand end of the four black 103 stripes. The stripes are the four flex PCB interconnect beams. Next to the photos are the normal strain and shear data calculated independently using the equation derived in Appendix B.3, as a function of time (shear displacement= C2C′1−C1C′2C′2+C′1 for the x-direction, shear displacement = C4C′3−C3C′4C′4+C′3 for the y-direction, and normal strain = ΔC1 +ΔC2 +ΔC3 +ΔC4 ΔC′1+ΔC′2+ΔC′3+ΔC′4). The left-to-right direction (x-axis) corresponds to a shear in the upward direction, hence the shear calculated using the capacitance of top and bottom electrodes when looking at the picture. The opposite direction corresponds to a shear in the downward direction. Shear data measured here is intended to demonstrate the concept, rather than gather precise quantitative results. Comments on the right describe each step of the experiment. First, the gripper is not touching anything and so the sensor is not showing any changes in capacitance (Figure 3-11 (A)). In Figure 3-11 (B), the cup is gripped (normal strain applied). Here, the sensor detects a shear in the upward-front direction. This is due to the geometry of the cup. As water is poured into the cup (Figure 3-11 (C), (D)), the shear in the downward direction is increasing due to the increased downward load provided by the mass of water that shears the contact with the fingertips. The calculated normal strain changes too due to the geometry of the cup applying a different force as it is getting heavier, and due to some noise and parasitic capacitance due to the water in the cup and the electronics surrounding the sensor. At the end of the experiment, the cup is half full, normal strain has decreased due to the geometry of the cup and noise, and the changes due to shear show that the cup is heavier (Figure 3-11 (E)). 104 Figure 3-11: Cup of water experiment showing changes in capacitance due to changes in pressure and shear while the cup is being gripped and filled up with water. The first graph represents the estimated normal strain as a function of time. The second graph represents the estimated shear strain where the horizontal axis is the shear in the up and down direction in the picture, and the vertical axis is the shear in and out the picture. 105 3.4.3 Discussion A demonstration is given of the operation of the multi-axis force sensor on a curved surface, showing the detection of normal and shear strain. Characterization still needs to be conducted to accurately evaluate the pressure and shear sensitivity of the sensor on a curved surface. The fingertip system needs to be reduced in size to be able to fit in a prosthetic glove. A full characterization under the glove needs to be completed. The same setup described in Section 3.3.2 can be used to do so. Moreover, the dielectric needs to be investigated further to improve shear and pressure resolution. A comparison with the literature is provided in Table 3-1. Sarwar et al. utilizes a dielectric structure, including square and cross pillars, that buckles like skin when a shear force is applied [62]. A sensor fabricated with this dielectric offers a normal force resolution of 100 mN – which is poorer than the resolution of the sensor described in Chapter 2 (2 mN) and Chapter 3 (39 mN) – and a shear force resolution of 20 mN – which is better than the shear resolution of our sensor (80 mN). This geometry would be very intriguing in future work to implement onto a prosthetic hand as a complete artificial skin, acting as a sensing prosthetic glove. Also, an investigation on shear sensitivity with pyramidal structured dielectric may also prove interesting, as the pyramidal sensor shows higher sensitivity, especially at low force levels. This may lead to better results, including a better pressure resolution, but this structure can also bring in some additional non-linearity. Another factor that could be investigated is the addition of a “bump” at the top of the sensor to detect a shear [84], [82]. The bump translates the shear force or displacement into a normal 106 displacement, leading to a change in capacitance between the top and bottom electrodes. One study [84] uses this technique for shear and normal force detection and shows a better resolution that our sensor. One limitation of integrating the shear sensor into prosthetic hands is the fit of the glove. The “bump” approach could help to enhance the shear detected by the sensor under the prosthetic glove. Another problem that may arise is the partial deformability of the sensor after high pressures. Other materials such as PDMS can also be investigated to potentially increase the normal force range of the sensor to withstand higher forces including the maximum gripping force of the Ottobock prosthetic hand of 100 N [85]. This material can also enhance shear resolution, just as seen on the work done by Charalambides et al., using a pillar-based approach. Table 3-1: Shear and normal resolution comparison in literature Reference Shear Resolution (mN) Normal Resolution (mN) Sarwar [62] 20 100 Lee [84] 3 3 Charalambides [86] 50 190 Chapter 2 N/A 2 Chapter 3 80 39 107 Finally, evaluation needs to be done in a realistic environment. In the future, it might be integrated into a prosthetic arm, with feedback provided to the user, and used during the performance of common manipulation tasks, to determine the potential of the sensor system. Conclusion Chapter 3 describes the implementation of shear detection in the fingertip system detailed in Chapter 2. To our knowledge, the combination of both normal force and shear for sensory feedback in prostheses has never been researched before. This work represents a step towards the integration of both. First, the electrode architecture of the capacitive sensor was modified to include five electrodes instead of only two, hence permitting the detection of normal force and shear in all directions simultaneously. Four electrodes are directly patterned on a PCB, with shielding added to minimize parasitic capacitance due to traces. A mechanical setup was developed to thoroughly characterize the sensor on a flat surface. The pressure sensitivity of the sensor was determined to be 4 % per kPa with a 3 % per kPa shear sensitivity. This is an improvement over the sensor described by Sarwar et al., which demonstrated a pressure sensitivity of 1.3 % per kPa and shear sensitivity of 2.2 % per kPa [62]. Secondly, the rigid PCB was replaced by a FLEX PCB to reduce the electronics volume and wrap the four electrodes on the PCB around the finger pad. A gripper was designed to implement the sensor and test it in a real-life-like situation. Positioned on the index fingertip of the gripper, the sensor identified normal and shear forces due to the gripping of a cup being filled with water. 108 Future work on this sensor could include the full mechanical characterization of the sensor on a curved surface, as well as the redesign of the fingertip to reduce its size so it fits under the prosthetic glove. This advanced fingertip permits the detection of multiple stimuli and can be configured to enable sensory feedback for prosthesis users. Chapter 4 describes how these sensed stimuli are sent to the user, and how, in the future, the entire sensory feedback can be tested to understand its benefits and improve its design. 109 Chapter 4: Stimulation Feedback and Proposed User Study The sense of touch, or haptics, is our gateway to physical interaction with the environment around us. It is how we feel and manipulate objects, type a thesis, grab a strawberry or a water bottle, or even shake hands. Due to congenital deficiencies, trauma, or diseases, amputees are mainly dependent on other senses to accomplish tasks. To restore this lack of sensation, we have developed a sensory feedback system for prostheses that conveys a stimulation correlated to the perceived stimulus on the fingertip. In this chapter, we describe the development of pressure feedback conveyed by the fingertip in chapter 2. Also, we suggest how shear feedback can be implemented to the system. Finally, we propose a set of experimental tests to conduct a user study in the future. Introduction Upper limb amputees are deprived of the sense of touch from either one or both hands. This lack of haptic information makes them quite dependent on their sight and their sense of hearing while using a prosthesis. Users have to focus their attention on their artificial hand when picking up an object, lifting or rotating it, and releasing it. “I’ve broken a lot of things, so all right. I am like a child. I really have to watch what I do because… then that’s fine, but if I look the other way while lifting it, it will break. Then it doesn’t matter if it is porcelain or plastic”, said a prosthesis user in an interview conducted by Wijk et al. [87]. This cognitive burden explains many prosthesis rejections or abandonment [88]. Users need prostheses that are more intuitive, demand less training, and feel like an extension of their body. Among others, users ask for affordable sensory feedback [88], [59], [44]. 110 Research using non-invasive sensory feedback techniques have been successfully helping task manipulation performance by improving grip force accuracy and reducing time completion by providing more information. Antfolk et al. demonstrated that pressure detected from the fingertips directly translated to the residual arm of the amputee (modality-matched feedback) allowed participants to discriminate two levels of pressures [55]. Rosenbaum-Chou et al. showed that vibrotactile feedback can improve light grip force accuracy by 129%, and be helpful at home [66]. Dietrich et al. proved that electrotactile stimulation improved manipulation of fragile or soft objects, and decreased prosthesis attention and even phantom pain [11]. Audio feedback can also be helpful to detect touch, pressure, or even different types of textures. Lundborg et al. studied this feedback on prosthesis users and individuals suffering from nerve injuries. Microphones were placed on the fingertips of each participant, and fed back the acoustics of the object touched. This method substituted the sense of touch with the sense of hearing (modality-mismatched feedback), allowing participants to differentiate four textures and indicate on which fingertip the touch was applied. Mechanotactile, vibrotactile, electrical, or audio stimulation have all revealed that non-invasive feedback could help amputees in their everyday lives. They can either translate the detected stimulus to another part of the body or substitute the touch by another sense, just as deaf people use sight to communicate, or blind people use touch to read. Touch and pressure feedback have been studied by many researchers. However, few are completely integrated into the artificial hand, and none, to our knowledge, propose normal force and shear feedback simultaneously. This chapter explores mechanotactile and vibrotactile stimulation feedback based on the choice of users revealed in a survey, and details a testing proposal to later investigate sensory feedback benefits. 111 Overview We choose a non-invasive feedback method to reinstate pressure and shear sensation without having to undergo any type of surgery. The goal of this device is to provide pressure and shear feedback allowing a better task manipulation through a precise grip force while reducing visual and hearing attention. This chapter starts with the exploration of prosthesis users’ opinions on sensory feedback (protocol number: H19-01120). We surveyed 16 amputees to understand how sensory feedback could be beneficial in their everyday lives and what would be the best feedback method in their opinion. After getting a positive response on sensory feedback, we decided to pursue a mechanotactile feedback using a servo motor, as well as a vibrotactile feedback using vibration motors, considering the survey statistics. Each feedback approach includes a calibration method to meet the user’s needs. The two systems are ready to be calibrated and tested on prosthesis users for pressure feedback. Shear feedback and Bluetooth connections are currently under revision. Both techniques have their advantages and drawbacks, but the vibrotactile feedback is promoted due to its accuracy, its application for touch, pressure, and shear simultaneously, and its complete integration to the prosthesis. To test the device and improve it, we designed specific tasks including both feedback systems to get patients’ opinions on each of them and reveal the benefits of sensory feedback. Ethics approval was granted to perform those experiments (protocol number: H19-01870). These can be conducted in future work. 112 Gathering Prosthesis Users’ Opinions To get an insight from prostheses users, we decided to conduct a survey giving us more information on what users are looking for in sensory feedback (protocol number: H19-01120). The purpose of this survey is to get some understanding of how being able to feel touch and pressure on prosthetic fingers could affect amputees’ lives. We also wanted to know what type of feedback would be preferred between mechanotactile, vibrotactile, electrotactile, or audio feedback, using a ranking question. This survey is included in Appendix C.1. Anthropometric data was first collected to find correlations between sensory feedback preferences and age, level of amputation, cause of amputation, and the type of prosthesis used. 16 participants completed the survey. Participants characteristics are displayed in Table 4-1 below. 113 Table 4-1: Survey anthropometric data for 16 participants Age 19-29 6 30-49 7 50+ 3 Non-responded 0 Level of Amputation Partial hand/finger 2 Wrist disarticulation 1 Transradial 7 Elbow disarticulation 2 Transhumeral 1 Shoulder disarticulation 3 Non-responded 0 Cause of Amputation Congenital 12 Trauma 4 Infection 0 Vascular 0 Cancer 0 Non-responded 0 Type of Prosthesis Use Passive 8 Body-powered 2 Battery-powered (myoelectric) 5 Non-responded 1 We first asked participants to define specific tasks they are struggling with while using their prosthesis. Seven participants mentioned several issues such as precise object manipulations (“tying ribbons”, “wrist manipulation movements”, “typing on a keyboard”, “working with wrenches”, “playing pool”, “cutting a steak”). One participant mentioned the difficulty of grasping and releasing objects quickly (“driving- grasping a steering wheel and being able to release quickly”). One congenital amputee mentioned that the additional length of the prosthesis is actually what makes them struggle with some daily tasks. Two congenital amputees stated that 114 they only wear prostheses for cosmetic purposes as they see the artificial hand as a hindrance to their “ability to do things”. We then asked how they think that tactile information could help them. Body and battery-powered prosthesis users gave a precise description defining factors such as the reduction of the focus/attention needed to manipulate objects (“I don’t have to pay as close of attention to grab things”), the decrease of visual attention (“I could do more things without looking or concentrating as hard”), the avoidance of damaging objects, and the improved efficiency of the prosthesis used by being quicker to achieve tasks and more informed on the object properties (“I think it would help me by making me more aware of the various textures and surfaces of the objects that I would be grabbing. This would in turn help me decide how much force I should apply to the specific object with the prosthetic hand in order to avoid damaging the object”). Four of the eight passive prosthesis users answered this question, describing that a sensory feedback could help them to know when the prosthesis is just touching something, giving them a potential “greater feeling of connection with the article” and safety information if the prosthesis is “coming into contact with cold, hot, etc.”. One passive hand user mentioned that it would be hard to benefit from the feedback since the cause of amputation was congenital and they do not know what the sense of touch is from this missing arm. To get a more quantitative sense of sensory feedback interest for amputees, we invited participants to rank the benefits of sensory feedback on a scale from 0 to 10 (0 being not beneficial at all, and 10 being extremely helpful in every task). Passive hand users responded with an average of 4.625/10, body-powered had an average of 5.5/10, and battery-powered averaged 7.25/10 (with 115 scores of 10/10, 8/10, 5/10, 10/10, and 6/10). This was expected as battery-powered users would be the ones benefiting most from sensory feedback due to grip control ability. After describing the working principle of each sensory feedback mechanism, we asked participants to rank their preferred options on how they would like feedback to be sent. Results are shown in Figure 4-1. Vibrotactile and mechanotactile feedback appear to be the most interesting stimulations for participants, especially for myoelectric users, where two participants chose vibration feedback as their first option, and two chose mechanotactile feedback as their first choice. Figure 4-1: Ranking of sensory feedback including first, second, and third choice. Overall, this survey shows a positive interest in sensory feedback, especially mechanotactile and vibrotactile feedback. Even though 12 out of 16 amputees were born without a hand, they are still considering the potential benefit of sensory feedback when using a passive, body-powered, or myoelectric prosthesis. 116 Based on this survey, past research findings, and the ease of integration, we decided to proceed with the implementation of two stimulation feedback mechanisms: mechanotactile and vibrotactile. Stimulation Feedback 4.4.1 Components and Working Principle This feedback comprises the fingertip system discussed in Chapter 2, a microcontroller processing the data and sending it to the feedback system, and a sensory feedback system actuated depending on the received data from the microcontroller. Figure 4-2 depicts each component. When using a myoelectric transradial prosthesis, users first contract muscles in their residual arm, with the intention of closing or opening the hand. EMG electrodes located in the prosthetic socket detect these myoelectric signals and send them as an input to the control system. The control system interprets these signals corresponding to a specific intention (opening/closing), and actuates the hand mechanism accordingly. This “forward-loop” is pictured in blue in Figure 4-2. Without sensory feedback, users mainly rely on visual and hearing cues, making sure that they are applying the correct grip force on an object, which can become difficult when the object is fragile and non-deformable. The proposed system gives sensory feedback, providing tactile information from the index fingertip to the user. The sensitized fingertip described in Chapters 2 detects touch and pressure. This data is then processed by the microcontroller, decoding the capacitance data and translating it into a specific feedback input – a vibration frequency (vibrotactile), or a specific motor input, to apply a pressure on the residual arm (mechanotactile). The feedback is activated, non-invasively 117 stimulating the skin at the intensity corresponding to the pressure magnitude of the stimulus. This “feedback-loop” is represented in green in Figure 4-2. With sensory feedback, users have a better perception of the gripping force they are applying on an object with the prosthesis and thus can accordingly adjust the contraction of their muscles in the residual arm that controls this gripping force. Previous research has shown that sensory feedback reduces cognitive load by simply reducing the need for visual and hearing attention [25]. 118 The components of our proposed feedback system are described below: Figure 4-2: Forward and Feedback Loops. Diagram representing the forward-loop (in blue) controlling the prosthesis (©Ottobock [14]) via EMG electrodes, and the feedback- loop (in green) giving tactile information (©clipartmax.com) to the user to adjust grip force. 119 4.4.1.1 The Microcontroller The microcontroller is an important component carrying information from the sensor in the fingertip system to the feedback system. To relay the capacitive sensor data to the feedback system, we selected the Piksey Pico (Bits and Blobs [71]) microcontroller, which includes these features: - A connection to a compatible power source (Vin pin, external input supply 7 V – 18 V) - A USB connection to calibrate the device depending on the user’s specifications - An I2C communication interface with serial data lines and serial clock lines to read the capacitance sensor data - Pulse width modulation pins (PWM) to adjust the sensory feedback depending on the sensor data (e.g. depending on the intensity of the stimulus) - A small size of 20.3 mm by 20.3 mm to fit in the hand. Also, this microcontroller is Arduino-compatible. Thus, all code is written using the Arduino platform. 4.4.1.2 The Feedback Mechanism As previously mentioned, we decided to explore both vibrotactile and mechanotactile stimulation. The two systems have to meet some requirements. First, the system needs to non-invasively stimulate the skin of the user almost instantaneously to provide real-time feedback and allow fast grip force adjustment. This feedback needs to be adjustable and calibrated by the user, depending on comfort and thresholds. Once calibrated, the system needs to send feedback 120 corresponding to the pressure intensity collected by the capacitance sensor. Lastly, the sensory feedback needs to be integrated into the prosthesis as much as possible. 4.4.1.2.1 Mechanotactile Feedback We first developed a mechanotactile feedback using a lever arm connected to a servo motor, a 3D printed part holding the servo, and an armband maintaining the servo close to the residual arm of the user, inspired by Antfolk et al. [53] and Stephens-Fripp et al. [57]. As the servo rotates, the lever arm applies a pressure and a small skin-stretch on the user’s stump, at a previously calibrated intensity related to the pressure intensity sensed by the capacitance sensor on the fingertip system. Shear feedback using this method will be investigated in future work. The servo is a GoTeck GS-5511MG servo motor of size 40.8 mm x 20.1 mm x 38 mm. It weighs 55 g, has a speed of 60°/0.23s, and a standing torque of 13 kg-cm at an applied voltage of 6 Volts. This torque is important as some users might need high forces to feel the feedback. A lever arm, of a specific design, discussed later, is attached to the servo as shown in Figure 4-3, and rotates as directed by the microcontroller. 121 Figure 4-3: Mechanotactile feedback system. (A) Lever Arm attached to the servo motor, (B) Top view of the servo when the lever arm is at 0°, (B) Top view of the servo when the lever arm is at 30°, (C) Top view of the servo when the lever arm is at 55°. A 3D printed part displayed in Figure 4-4 is used to keep the servo in place. The stretchable armband (Valken V-TAC Team Armband) fits into the slots of the hosting part and wraps around the user’s arm to be attached using Velcro (Figure 4-4 (D)). Users can choose the preferred location of the stimulation (i.e. higher or lower, closer or farther from the biceps). Figure 4-4: Mechanotactile stimulation feedback system (A) 3D model of the 3D printed part hosting the servo motor, (B) Picture of the servo placed in the hosting system, (C) Picture of the assembled system, (D) Picture of the system placed on a user’s arm. 122 During pilot testing with the device, we found several challenges arising from this technique. Firstly, servo motors are bulky and add additional weight and volume to the overall prosthesis. These would be even bigger when activated via Bluetooth, as the battery and controller would also need to be attached to the armband. Secondly, some users might not feel the stimulation enough, demanding a higher feedback pressure on the skin. This might irritate the skin after some time, especially when translating high pressures sensed by the fingertip. Thirdly, the pressure felt on the area stimulated might fade away. This is due to the skin and tissues habituating to the applied pressure, just like we can forget the weight of an object that we carry in our pocket. Lastly, when applying a high torque, the lever arm pushes the servo away from the skin, obligating the armband to be tightened, leading to discomfort and possible skin irritation. To minimize discomfort and improve the efficiency of the stimulation by demanding less torque, we designed several lever arms that can be easily interchanged by the user when using the stimulation device. The first design includes a small pad (Figure 4-5 (B)), inspired by the work done by Antflok et al. [53], that distributes the pressure evenly on the subjected area. The second design represents a hemisphere (Figure 4-5 (C)), modifying the pressure felt depending on the force. This circular design might be more comfortable due to rounded edges and better force distribution. The third design includes small pyramids (Figure 4-5 (D)), resulting in an increased pressure in several small areas. The goal of this design is to trigger the stimulation at a smaller torque. The last design targets the challenge of feeding back higher pressures. As described earlier, high pressures on the skin might result in inflammation in the pressed area directly, or due to the tightness of the armband needed to keep the servo in place while rotating the lever arm at higher angles to translate higher pressures. This last design includes two indenters (Figure 4-5 (E)) 123 applying pressure at a different location depending on the intensity of the stimuli. Two thresholds can then be determined – touch and medium pressures. The servo motor rotates in the clockwise direction (stimulating area b in Figure 4-5 (E)) when the fingertip system senses a touch (first threshold), and continues to turn clockwise as the pressure increases until the second threshold. When the pressure hits the second threshold (set medium pressure), the servo rotates in the counter-clockwise direction, stimulating area a in Figure 4-5 (E). Users can choose from these designs, depending on the feeling and comfort they would get from each of them. Designs #1, #2, and #3 can also be adapted to design #4. More designs, including soft 3D printed ones, can be explored in future work. 124 Figure 4-5: Diagram depicting five possible indenter designs (A) As received design coming with the servo motor, (B) Pad design inspired by [53], (C) Hemisphere design, (D) Pyramids design, (E) Double-indenter design. This modality feedback method directly translates the pressure detected by the capacitive sensor to a specific location in the residual arm of the user. This implicates that the sense of touch 125 is still present and not replaced, possibly requiring less training than sensory substitution. Through pilot testing of the device, we found it possible to easily differentiate between small and large pressure sensations. However, as mentioned earlier, the bulkiness of this type of device might be a hindrance to prosthesis users. The volume of the armband will need to be even bigger when implementing shear feedback. This can be reduced by using a smaller servo at the expense of reducing the torque needed or higher cost. Another solution would be to develop a system small enough to fit within the socket, such as a micro-actuator or flat circular pneumatic system, increasing in volume to push against the residual arm of the prosthesis user. However, this might affect the fit of the socket on the user’s stump, as the prosthesis is held in place using a suction mechanism. Shear feedback can be investigated in future work by modifying the indenter pushing on the skin or designing a different actuator that could convey both pressure and shear. Initial literature search, including wearable device for force and slip display [83] or custom controllers for active gaming relaying force and shear feedback [89], has shown that shear can be conveyed to the user in a modality-match manner. One limitation of this technique is its bulkiness. This can be minimized to potentially fit in the socket or in a light weight armband. Another feedback alternative is suggested in this thesis, specifically to propose a feedback system that can be fully integrated into the prosthesis. Instead of modality-match feedback (e.g. using touch or pressure to feedback touch or pressure stimulus), this method uses vibration as a substitution. 126 4.4.1.2.2 Vibrotactile Feedback Chapter 1 states how vibrations can be used to efficiently convey tactile information from a sensor. Yamada et al. researched how vibrotactile sensory feedback affects cognitive load when grasping an object. Deep sensory feedback (i.e. high intensities) can result in the decrease of cognitive strain during grasping, and having one specific location stimulated at different intensities can be more efficient than three spatially separated ones to convey changes of intensity of the input stimulus [90]. To explore this approach, we decided to use a 3to 5 V vibration coin motor from Teknikio of dimensions 9 mm x 4 mm for its small size and ability to vibrate at high frequency. The PWM value can be modified on the Arduino code to provide a certain voltage to the motor, regulating its frequency, and hence its vibration intensity, as shown in Table 4-2. This allows the stimulation to be calibrated and to convey the feedback to the chosen location. Compared to the mechanotactile stimulation, where the indenter can continue to push on the arm when the pressure on the fingertip is constant (i.e. keeping the same angle of the lever arm), the vibration stimulation cannot last for a very long time. This would become irritating for the user and might immune the stimulated area as the skin would get used to it. Thus, we decided to implement three thresholds in the stimulation as shown in green, yellow, and red, in Table 4-2. The stimulation starts when the first threshold is reached, and stops at about 3 seconds if the pressure stays constant. Any increase or decrease of the sensed capacitance from the sensor activates the stimulation again. This time can be adjusted by users depending on their comfort and needs. 127 Table 4-2: Thresholds relating touch and pressure intensities (PWM) to the vibration motor specifications (voltage, frequency, and RPM) Unlike a servo motor needed to be outside of the prosthesis, the vibration motor can be fully integrated into the artificial hand. Indeed, prostheses include a free space located in the forearm, between the wrist and the socket (where the battery fits). This can be used to host a vibration motor and hence vibrate the whole socket, distributing the stimulation across the entire stump. This space is big enough to host a whole battery, a vibration motor, and a Bluetooth controller. The motor can be taped inside the forearm, closer or farther away from the socket depending on the user’s comfort. Shear feedback can also be implemented using vibration motors. This will be investigated in future work. Four vibration motors can be placed on the stump to form a cross-shape (two horizontal motors representing the x-direction, and two vertical motors representing the y-direction. As the sensor detects shear in the positive x-direction, the horizontal left motor is first activated then the horizontal right motor is activated. This process is the same for all four 128 directions. In the case of a pressure stimulus, all four motors vibrate. If both shear and pressure are sensed, the one motor in the direction of shear will vibrate at a higher intensity (e.g. for the positive x-direction, the right horizontal will vibrate more). 4.4.2 Calibration A calibration system is provided with the stimulation device for pressure feedback. Prosthetists can connect the microcontroller to a computer to calibrate the stimulation according to the user’s sensations. An Arduino program was written to gradually increase either the vibration frequency or the servo angle. The user indicates the lowest stimulation that can be felt. This is called the just-noticeable difference (JND), and determines the first touch threshold. Then, the user indicates the highest vibration frequency or servo angle that is bearable. This is the highest threshold representing a high pressure that might damage the sensor. One problem that may arise is the change of JND depending on contextual factors. For example, if the user is moving, he or she might not feel the stimulation in the same way. The user can also determine the vibration frequency or servo angle to change linearly or in steps depending on the stimulation. For the vibrotactile stimulation, the user can indicate how long the vibration lasts when the sensor pressure data stays constant. 4.4.3 Challenges Both stimulation system can right now send pressure feedback and be calibrated. By adjusting the vibrotactile and mechanotactile stimulation devices, it is possible to differentiate several pressure stimulations with both techniques. Shear feedback using vibration motors and Bluetooth connections are currently being revised. Many challenges remain for both stimulation techniques. 129 As mentioned earlier, users might get “used to” the stimulation and not feel it after a certain time. This is an important consideration suggesting that the device should be easily re-calibrated. In the proposed system, only prosthetists can adjust the stimulation (as the microcontroller is hidden inside the hand and under the prosthetic glove). Future work includes a remote Bluetooth or WIFI connection including a user interface. This would help the system to meet users’ needs and give appropriate feedback depending on their skin condition, sensations, and daily desires (turn off the stimulation when needed). When it comes to adding normal force and shear feedback, users might feel overwhelmed by the amount of information they are receiving. Training will be necessary to get used to the system. Unfortunately, the testing of the proposed system was not done, made difficult by the current pandemic conditions. Instead, the proposed testing approach is outlined, which can hopefully be implemented in future work. Proposal: Testing the Device We have developed a set of procedures to test the device with prosthesis users and evaluate its efficiency and usefulness. Due to the small number of expected participants, we focus here on a qualitative study to get a sense of how the system could help amputees in their everyday lives (i.e. decrease visual attention, reduce phantom pain, reduce task achievement time, and improve grip force accuracy). The goal is to get general opinions on the apparatus, find different ways to improve the system, and determine user preferences and performance depending on the factors described above (i.e. vibration vs. pressure feedback, calibration preferences). First, we will describe who can participate in the study and how participants can be recruited. Then, we will 130 outline the four different steps of the process, including a pre-experiment survey, a calibration step, a testing section, and a post-experiment interview. Lastly, we will detail the plan for analysis and how this would impact the device. 4.5.1 Procedures 4.5.1.1 Recruitment and Conditions Participants can be recruited through word-of-mouth and snowballing. All participants must be upper limb amputees, available for a requested time commitment of one to three hours. This is a minimal risk study, and the only concern is fatigue to complete a task. Participants can take a break whenever they would like. The process of this study can be divided into four phases described below. 4.5.1.2 Pre-Experiment Survey It is first crucial to collect anthropometric data and relate it with the sensory feedback performance. To do so, a survey will be handed out to the participants to complete before any testing. This survey can be found in Appendix C.2. Age, sex, level and cause of amputation, type of prosthesis used, time lapse from amputation and from fitting an artificial hand, and information on the prosthesis use, can have an impact on the use and efficiency of the sensory feedback. Amputees are also asked if they experience any phantom pain in their missing hand, and if they feel like the prosthesis is part of their body. These additional questions are used to compare with a post-experiment interview and see how these feelings might have changed while using the sensory feedback. 131 4.5.1.3 Calibration Before using the system, it has to be calibrated to satisfy the participants’ feelings and needs. This will be done for both the servo and vibration motor. First, we will find the just-noticeable difference for both feedback methods. Then, we will determine sensitive areas on the arm of the participants that can be used to convey the feedback, and if this feedback should be linear or transmitted in steps. Finally, for the mechanotactile method, participants will be asked to choose the type of indenter they like the most. For the vibrotactile method, participants will say how long they would like the stimulation to last for in the case of a steady grip force. 4.5.1.4 Task Selection Several tasks were designed by the author with the help of Barber Prosthetics Clinic to determine how sensory feedback can impact task time completion and performance, evaluate a potential decrease in the need for visual attention while completing a task, and get an idea of the importance of touchscreen interaction ability. The first task (task 1) is a Box and Blocks Test, a common exercise in the prosthetics field [14], [36]. This consists of lifting and transporting ten blocks (one-inch cubes) from one place to another as fast as possible. This task will be completed first without sensory feedback, and then with each of vibrotactile or mechanotactile feedback. The entire set will be repeated twice to counterbalance for potential influence of training. The time to complete the task will be calculated for each type of feedback. This aims to only assess how sensory feedback influences task time completion with and without sensory feedback. 132 The second task (task 2) is the same as task 1 but using ten fragile blocks (paper blocks including a fragile mechanical fuse inside). This task is inspired by the “virtual egg test” described by Clemente et al. [32]. This is to test the grip strength control while utilizing sensory feedback. Participants will be asked to move blocks from one place to another. Again, this will be repeated six times (twice without sensory feedback and twice with each feedback method). The time needed to complete each set will be recorded. The goal of this task is to determine the potential benefits of sensory feedback while manipulating fragile objects (i.e. number of unbroken blocks/grip force accuracy as a function of time). The third task (task 3) demands participants to pick up a foam cube inside a black box (where they can’t see their prosthetic hand). After correctly placing the object between the thumb and index fingers, they will be asked to lift the object, hold it up for 10 seconds, and put it back down, with as little force as possible. If participants are passive hand users, they will be asked to press as gently as possible on a scale, also placed inside a black box. This test will be repeated twice, with and without the sensory feedback. A camera will be placed inside the black box to videotape and assess how much participants are squeezing the cube. The fourth task (task 4) consists of participants having their hand inside a black box where they cannot see it. Their prosthesis will be positioned so that it will barely touch a stage that can move in the x (left to right), y (back to front), and z (up and down) directions. While the stage moves in different directions, participants will be asked to explain what they feel. This aims to see if shear feedback can be understood by users. 133 Finally, task 5 will ask participants to input numbers on a mobile phone with their prosthetic hand. This is just to give participants the possibility of experiencing the ability to activate touchscreen technologies with their artificial hands. 4.5.1.5 Post-Experiment Interview After completing the tasks, participants will be interviewed to answer questions about sensory feedback preferences, grip control sensations with and without sensory feedback, and embodiment feelings while completing the tasks without and without sensory feedback. Participants will be asked to give their opinions and share any suggestions on how the device could be improved. The interview template is provided in C.3. 4.5.2 Plan for Analysis The tasks were carefully designed to answer specific questions regarding speed and performance while utilizing sensory feedback. Each task needs to be correlated to anthropometric data to explain potential result differences between participants. Task 1 determines if sensory feedback helps amputees to achieve a task faster. Each set of data (i.e. time to complete the task) is compared to the others and to the repeated set to evaluate how training might impact results. Task 2 was designed to come up with an adjusted time accounting for speed and accuracy simultaneously. In this experiment, each broken block (fail), is reported as a time penalty using the inverse-efficiency score (IES) [91]. To do so, the time taken to complete task 2 (Reaction Time RT) is divided by (1 – PE), PE being the proportion of errors – in our case PE= Number of broken 134 blocks / Total number of blocks. If all blocks are broken (i.e. PE=1), the participant will be asked to redo the task. The IES corresponds then to the time in seconds, accounting for the broken blocks. Examples displayed in Table 4-3 (using made-up values) show that a task completed faster with the same errors leads to a smaller IES (row 2 vs. row 5). Similarly, a task completed with less error but at the same speed, leads to a smaller IES as well (row 3 vs. row 6). This method is very useful to compare the results when using or not using sensory feedback. Table 4-3: Examples of results that can be obtained during the Box and Blocks test using fragile blocks. The time corresponds to the time needed to move 10 fragile blocks from one place to another. The proportion of errors is the number of broken blocks divided by the total number of blocks. The IES is the adjusted time accounting for the broken blocks. The number of broken blocks alone tells us about the grip force accuracy independently of the speed. Tasks 1 and 2 can then give a good sense of the benefits of sensory feedback for speed and performance (i.e. grip force accuracy) while achieving a task. Task 3 aims to see if sensory feedback is helpful when visual cues are missing. During this task, participants only have sensory feedback to sense and apply the force to lift, hold, and place 135 back down a foam cube. The video of the hand tells us how much participants are squeezing the cube with and without sensory feedback, demonstrating if they are “feeling” how much grip force they are applying on the soft object. Tasks 1, 2, and 3 are used to compare results with and without sensory feedback, but also to compare mechanotactile and vibrotactile stimulations. This will be done using the same analysis methods. Tasks 4 and 5 are fully qualitative. These experiments are designed to make participants experience shear detection and touchscreen interaction, and give their opinion during the post-experiment interview. The post-experiment interview aims to qualitatively understand how participants felt during the different tasks, and if they noticed any differences in performance. This would then be studied to evaluate how feelings relate to actual quantitative results. As mentioned earlier in this thesis, one big problem from prosthetic hands is that users don’t have a good sense of ownership over the hand. To investigate how sensory feedback impacts that factor (especially for long time myoelectric users), several embodiment questions using a Likert scale are asked to participants. One question relates to phantom limb perception, if any. Further questions on the interview target future improvement, gathering opinions on the device to eventually add features, and make it more effective for prosthesis users. 136 Conclusion After getting prosthesis users’ opinions on sensory feedback, we decided to implement vibrotactile and mechanotactile pressure sensory feedback to prosthetic hands. Each of these two is actuated to convey the perceived stimulus from the capacitive sensor on the fingertip described in Chapter 2. Mechanotactile feedback is a modality-matched stimulation, meaning that it directly translates the input stimulus to the chosen location (e.g. residual arm of the user). The advantage of this technique is that the feedback is the same as the stimulus, implying a hypothetically faster training and understanding of the feedback while using the prosthesis. However, the stimulated area can become inflamed or habituated to the feedback. We designed several indenters to overcome this issue. Shear feedback can be implemented in the future using an updated armband system. However, this feedback method remains bulky and adds extra volume to the artificial hand. Vibrotactile feedback is an alternate method that translates the fingertip “sensations” to the user by vibrating a chosen location on the body (e.g. residual arm of the user) at different frequencies. This technique might also be used to send shear information in the future, using several vibration motors placed at strategic locations. This approach allows the complete integration of the system inside the arm. It is expected to be a user-friendly technique as users do not have to take care of it. Both methods include a calibration process to suit users’ needs and comfort. We created a set of experiments to evaluate the efficacy of the device for each method and gather users’ suggestions on how to upgrade the system to better satisfy sensory feedback requirements. This can be done in future work. 137 Ultimately the sensors described in Chapters 2 and 3 are combined with the feedback methods of Chapter 4 to create a full sensory feedback system, from a sensitized fingertip including a highly capacitive sensor (Chapter 2 and 3), to an accurate stimulation feedback conveying touch, pressure, and shear information from that fingertip to the user (Chapter 4). We hope this will help those with prostheses function more effectively and connect better with their devices. Chapters 2, 3, and 4 describe in details the development of a sensory feedback system, from a sensitized fingertip including a highly capacitive sensor (Chapter 2 and 3), to an accurate stimulation feedback conveying touch, pressure, and shear information from that fingertip to the user (Chapter 4). 138 139 Chapter 5: Conclusions This thesis presents an innovative low-cost touch and shear sensory feedback system for myoelectric hands – specifically sized for and partially implemented on the SensorHand Speed from Ottobock – that can be further developed in the future to fit many types of prostheses, including more affordable commercially available arms fabricated using 3D printing techniques (e.g. Hero Arm). This could give a chance to all amputees to experience the next level of performance in prostheses, including tactile feedback and touchscreen interaction, while limiting costs. Summary After describing how prostheses evolved throughout the years and how they assist amputees in their everyday lives, Chapter 1 establishes the potential benefits of sensory feedback for prosthesis users. Many studies found such a feature can ameliorate prosthetic use by helping users to perform a task faster, through the reduced need for visual attention and better information on grip force. Only the Luke arm from Open Bionics provides a tactile sensory feedback but at a significant cost. To our knowledge, no prosthetic hand available in the market or researched provides both normal and shear force feedback. Users still lack affordable normal force and shear sensory feedback simultaneously, that can be completely integrated under the prosthetic glove to minimize care and ensure good aesthetics. Additionally, there is still a need for touchscreen interaction for artificial hands. In this thesis, we present a fingertip system including a capacitive soft sensor, that enables touchscreen interaction for prosthesis users, and detects touch and shear 140 stimuli to send tactile feedback. This system including the normal force feedback has been tested by the author and will be tested on prosthesis users in the future. Chapter 2 explains the fingertip system that uses a simple touch and pressure capacitive sensor. The device comprises a 3D printed fingertip, hosting a printed circuit board that connects to a capacitive sensor. This sensor is designed with specific materials, electrode architecture, and a highly compressible dielectric including a pyramidal structure. It can sense pressures of 25 Pa, allowing the sensing of pressures of 1 kPa (interpreted as light touches) up to at least 100 kPa when fitted under the thick rubber glove that covers the Ottobock system’s hand. The fingertip is covered by a conductive fabric to shield proximity effects and enable touchscreen interaction. The key novelty of this system is that it is inexpensive, touchscreen-friendly, and is completely integrated to the artificial hand, allowing the glove to protect it from any damage, and reducing the care-burden for users. Another key point is the possibility of integrating shear detection to the sensor, through the sensor electrode architecture. Chapter 3 presents the implementation of shear detection to the fingertip system. This includes a new sensor electrode architecture, inspired by the work of Sarwar et al. [62], that contains five electrodes (instead of two) for the sensing of normal force and shear in all directions simultaneously. A thorough mechanical characterization of the fabricated sensor shows that shear and pressure can be detected simultaneously but also estimated independently. Results display a pressure sensitivity of ∼4 % per kPa, and a shear sensitivity of ∼3 % per kPa, tested up to 6 kPa normal pressure and up to 0.4 mm shear displacement. It has a pressure resolution of 0.2 kPa and a shear resolution of 0.4 kPa. These results are without the glove, which will reduce sensitivity but also increase the force range. The implementation of such a sensor in the fingertip system is 141 accomplished with the use of a FLEX PCB directly incorporating four of the electrodes as pads. The fingertip device was mounted onto a gripper system to experience in real-time the sensing of pressure and shear when holding a cup and filling it up with water. This experiment shows that the system can be used in real-life application to detect touch, pressure, and shear stimuli, to send a correlated feedback to the prosthesis user. Chapter 4 describes the sensory feedback device utilized to translate the detected stimuli from the fingertip system to the user. After getting users’ opinions with a survey, two types of feedback were implemented into the system. First, mechanotactile feedback directly translates the pressure stimulus sensed by the fingertip system to the user’s arm, using a servo motor rotating at different angles to differentiate pressure intensities. This method is intended for touch and pressure only due to the bulkiness of the armband. Future work will focus on the implementation of shear feedback. Secondly, vibrotactile feedback uses vibration motors, included in the socket, activated at different frequencies to differentiate pressure intensities. This method does not directly translate the stimulus to the user and might require more training than mechanotactile feedback to accurately comprehend the system. However, unlike servo motors requiring an armband system, vibration motors are very small and can be fully integrated within the hand, limiting the bulkiness and weight of the sensory feedback device. Moreover, the integration of four vibration motors at the vicinity of the residual arm permits the feedback of shear in all directions by modulating localized intensities. This implementation is ongoing. 142 Future Work and Limitations The developed device can be further improved and characterized. Table 5-1 summarizes the fingertip and sensory feedback features, outlining what has been done and what needs to be done in the future. The proposed fingertip described in Chapter 3 needs to be reduced in size to fit under the prosthetic glove, and the stimulation device needs to be activated via Bluetooth. After getting the complete characterization of the system (sensor on a curved surface and under the glove), the device can be tested with prosthetic users. Chapter 4 details possible experimentations that can be conducted with users to understand the potential benefits of touch, and shear feedback. A small group of participants can give general ideas on the improvement of the system. More participants and longer experiments – including taking the device at home, will be needed to quantify the benefits of our sensory feedback device. Table 5-1: System features summary 143 Future work also includes the design of micro-actuators to send touch, pressure, and shear mechanotactile feedback. This would permit to investigate pressure stimulation within the socket, hence suppressing the bulky armband. Also, a phone interface needs to be installed to allow users to re-calibrate the system or turn it off on demand. This is essential for users to avoid the area stimulated by the feedback to become desensitized, as discussed in Chapter 4. Finally, while the combination of touch, pressure, and shear stimulation is a key innovation of this system, it needs to be explored with prosthesis users over a long period. The system can potentially give too much information to users, hence imposing higher training, or fatigue and abandonment. To overcome this issue, users need to be able to turn off some stimulus detection depending on the task (e.g. only detect touch and pressure but not shear) or turn the system off completely whenever they wish. Others might benefit from this work to extend the system to multiple fingers, implement it to different types of prosthetic hand (e.g. the Hero Arm), or, ultimately, explore invasive stimulation to directly target the nerves. 144 145 Bibliography [1] R. S. Johansson and G. Westling, “Signals in tactile afferents from the fingers eliciting adaptive motor responses during precision grip,” Exp. Brain Res., vol. 66, no. 1, pp. 141–154, 1987. [2] H. Culbertson, S. B. Schorr, and A. M. Okamura, “Haptics: The Present and Future of Artificial Touch Sensation,” Annu. Rev. Control. Robot. Auton. Syst., vol. 1, no. 1, pp. 385–409, 2018. [3] A. P. Gerratt, H. O. Michaud, and S. P. Lacour, “Elastomeric electronic skin for prosthetic tactile sensation,” Adv. Funct. Mater., vol. 25, no. 15, pp. 2287–2295, 2015. [4] K. Ziegler-Graham, E. J. MacKenzie, P. L. Ephraim, T. G. Travison, and R. Brookmeyer, “Estimating the Prevalence of Limb Loss in the United States: 2005 to 2050,” Arch. Phys. Med. Rehabil., vol. 89, no. 3, pp. 422–429, 2008. [5] K. J. Zuo and J. L. Olson, “The evolution of functional hand replacement: From iron prostheses to hand transplantation,” Can. J. Plast. Surg., vol. 22, no. 1, pp. 44–51, 2014. [6] J. Kirkup, “Introduction and Sources,” in A History of Limb Amputation, Springer, 2007, pp. 1–13. [7] C. Lawrence, “Historical keyword Amputation Lunch with The Lancet Vince and Vance Moss,” Lancet, vol. 371, p. 1065, 2008. [8] A. Bennett Wilson, “History of Amputation Surgery and Prosthetics,” in Atlas of Limb Prosthetics: Surgical, Prosthetic, and Rehabilitation Principles, 2nd ed., American Academy or Orthopedic Surgeons. [9] T. P. Sheehan, Rehabilitation and Prosthetic Restoration in Upper Limb Amputation, Fifth Edit. Elsevier Inc., 2011. 146 [10] D. J. Atkins and R. H. Meier III, “Amputation Levels and Surgical Techniques,” in Comprehensive Management of the Upper-Limb Amputee, Springer-Verlag New York Inc., 1989, pp. 1–15. [11] C. Dietrich et al., “Sensory feedback prosthesis reduces phantom limb pain: Proof of a principle,” Neurosci. Lett., vol. 507, no. 2, pp. 97–100, 2012. [12] L. Resnik et al., “Advanced upper limb prosthetic devices: Implications for upper limb prosthetic rehabilitation,” Arch. Phys. Med. Rehabil., vol. 93, no. 4, pp. 710–717, 2012. [13] F. Cook, “Superhero arm gives bionic boy huge lift,” pp. 1–3, 2019. [14] Ottobock, “MyoFacil prosthetic hand— Ottobock,” 2015. [Online]. Available: https://www.ottobock.com.co/prosthetics/upper-limb/solution-overview/myofacil-prosthesis-system/. [15] “Goetz eiserne hand.” [Online]. Available: https://commons.wikimedia.org/wiki/File:Goetz-eiserne-hand.jpg. [16] McLeod, “Ambroise Paré (1510?-1590): Les oeuvres d’Ambroise Paré, 1585.,” Historical Medical Books at the Claude Moore Health Sciences Library, University of Virginia, 2008. [Online]. Available: https://commons.wikimedia.org/wiki/File:Pare_artificial_hand.jpg. [17] Bartenstein, “Sauerbrucharm mit Hüfnerhand.” [Online]. Available: https://commons.wikimedia.org/wiki/File:Sauerbrucharm_mit_Hüfnerhand.JPG. [18] R. P. Petri and E. Aguila, “The military upper extremity amputee,” Phys. Med. Rehabil. Clin. N. Am., vol. 13, no. 1, pp. 17–43, 2002. [19] Ottobock, “Milestones in our history.” [Online]. Available: https://www.ottobockus.com/about/history/milestones.html%0D. 147 [20] Ottobock, “Body-powered upper limb prostheses.” [Online]. Available: https://www.ottobock-export.com/en/prosthetics/upper-limb/solution-overview/arm-prostheses-body-powered/. [21] Open Bionics, “Hero Arm.” [Online]. Available: https://openbionics.com/hero-arm/. [22] Ottobock, “Dynamic Arm,” 2020. [Online]. Available: https://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solution-overview/above-elbow-prosthesis-featuring-dynamicarm/. [23] Ottobock, “Bebionic Hand,” 2020. [Online]. Available: https://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solution-overview/bebionic-hand/. [24] M. Bionics, “LUKE Arm Details,” 2019. [Online]. Available: https://www.mobiusbionics.com/luke-arm/. [25] P. Svensson, U. Wijk, A. Björkman, and C. Antfolk, “A review of invasive and non-invasive sensory feedback in upper limb prostheses,” Expert Rev. Med. Devices, vol. 14, no. 6, pp. 439–447, 2017. [26] E. Biddiss and T. Chau, “Upper limb prosthesis use and abandonment: A survey of the last 25 years,” Prosthet. Orthot. Int., vol. 31, no. 3, pp. 236–257, 2007. [27] B. Stephens-Fripp, M. Jean Walker, E. Goddard, and G. Alici, “A survey on what Australians with upper limb difference want in a prosthesis: justification for using soft robotics and additive manufacturing for customized prosthetic hands,” Disabil. Rehabil. Assist. Technol., vol. 0, no. 0, pp. 1–8, 2019. [28] M. L. Hammock, A. Chortos, B. C. K. Tee, J. B. H. Tok, and Z. Bao, “25th anniversary article: The evolution of electronic skin (E-Skin): A brief history, design considerations, 148 and recent progress,” Adv. Mater., vol. 25, no. 42, pp. 5997–6038, 2013. [29] D. J. Lipomi et al., “Skin-like pressure and strain sensors based on transparent elastic films of carbon nanotubes,” Nat. Nanotechnol., vol. 6, no. 12, pp. 788–792, 2011. [30] B. Li, A. K. Fontecchio, and Y. Visell, “Mutual capacitance of liquid conductors in deformable tactile sensing arrays,” Appl. Phys. Lett., vol. 108, no. 1, pp. 1–4, 2016. [31] S. C. B. Mannsfeld et al., “Highly sensitive flexible pressure sensors with microstructured rubber dielectric layers,” Nat. Mater., vol. 9, no. 10, pp. 859–864, 2010. [32] F. Clemente, M. D’Alonzo, M. Controzzi, B. B. Edin, and C. Cipriani, “Non-Invasive, Temporally Discrete Feedback of Object Contact and Release Improves Grasp Control of Closed-Loop Myoelectric Transradial Prostheses,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 12, pp. 1314–1322, Dec. 2016. [33] B. T. Nghiem et al., “Providing a Sense of Touch to Prosthetic Hands,” Plast. Reconstr. Surg., vol. 135, no. 6, pp. 1652–1663, 2015. [34] T. Boretius et al., “A transverse intrafascicular multichannel electrode (TIME) to interface with the peripheral nerve,” Biosens. Bioelectron., vol. 26, no. 1, pp. 62–69, 2010. [35] A. E. Schultz and T. A. Kuiken, “Neural Interfaces for Control of Upper Limb Prostheses: The State of the Art and Future Possibilities,” PM R, vol. 3, no. 1, pp. 55–67, 2011. [36] M. Schiefer, D. Tan, S. M. Sidek, and D. J. Tyler, “Sensory feedback by peripheral nerve stimulation improves task performance in individuals with upper limb loss using a myoelectric prosthesis,” J. Neural Eng., vol. 13, no. 1, 2015. [37] T. A. Kuiken, P. D. Marasco, B. A. Lock, R. N. Harden, and J. P. A. Dewald, “Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation,” Proc. Natl. Acad. Sci. U. S. A., vol. 104, no. 50, pp. 20061–149 20066, 2007. [38] E. L. Graczyk, L. Resnik, M. A. Schiefer, M. S. Schmitt, and D. J. Tyler, “Home use of a neural-connected sensory prosthesis provides the functional and psychosocial experience of having a hand again,” Sci. Rep., vol. 8, no. 1, pp. 1–17, 2018. [39] B. Stephens-Fripp, G. Alici, and R. Mutlu, “A review of non-invasive sensory feedback methods for transradial prosthetic hands,” IEEE Access, vol. 6, pp. 6878–6899, 2018. [40] B. Stephens-Fripp, R. Mutlu, and G. Alici, “Using Vibration Motors to Create Tactile Apparent Movement for Transradial Prosthetic Sensory Feedback,” Proc. IEEE RAS EMBS Int. Conf. Biomed. Robot. Biomechatronics, vol. 2018-Augus, pp. 213–218, 2018. [41] H. J. B. Witteveen, H. S. Rietman, and P. H. Veltink, “Vibrotactile grasping force and hand aperture feedback for myoelectric forearm prosthesis users.,” Prosthet. Orthot. Int., vol. 39, no. 3, pp. 204–12, 2015. [42] A. E. Pena, L. Rincon-gonzalez, J. J. A. Id, and R. J. Id, “Effects of vibrotactile feedback and grasp interface compliance on perception and control of a sensorized myoelectric hand,” pp. 1–21, 2019. [43] K. Bark, J. Wheeler, P. Shull, J. Savall, and M. Cutkosky, “Rotational skin stretch feedback: A wearable haptic display for motion,” IEEE Trans. Haptics, vol. 3, no. 3, pp. 166–176, 2010. [44] M. D’Alonzo, F. Clemente, and C. Cipriani, “Vibrotactile stimulation promotes embodiment of an Alien hand in amputees with phantom sensations,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 23, no. 3, pp. 450–457, 2015. [45] Juan Pena, “Vibration Motor,” Sparkfun. [Online]. Available: https://www.sparkfun.com/products/8449 . 150 [46] G. Chai, X. Sui, S. Li, L. He, and N. Lan, “Characterization of evoked tactile sensation in forearm amputees with transcutaneous electrical nerve stimulation,” J. Neural Eng., vol. 12, no. 6, p. 066002, Dec. 2015. [47] N. V. Thakor et al., “Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain,” Sci. Robot., vol. 3, no. 19, p. eaat3818, 2018. [48] D. D. Damian, A. H. Arita, H. Martinez, and R. Pfeifer, “Slip speed feedback for grip force control,” IEEE Trans. Biomed. Eng., vol. 59, no. 8, pp. 2200–2210, 2012. [49] A. Gibson and P. Artemiadis, “Object discrimination using optimized multi-frequency auditory cross-modal haptic feedback,” 2014 36th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBC 2014, no. October 2015, pp. 6505–6508, 2014. [50] J. Gonzalez, H. Suzuki, N. Natsumi, M. Sekine, and W. Yu, “Auditory display as a prosthetic hand sensory feedback for reaching and grasping tasks,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, pp. 1789–1792, 2012. [51] S. Dosen, M. Markovic, D. Farina, H. Karnal, and B. Graimann, “GLIMPSE: Google Glass interface for sensory feedback in myoelectric hand prostheses,” J. Neural Eng., vol. 14, no. 3, p. 036007, 2017. [52] S. Casini, M. Morvidoni, M. Bianchi, M. Catalano, G. Grioli, and A. Bicchi, “Design and realization of the CUFF - Clenching upper-limb force feedback wearable device for distributed mechano-tactile stimulation of normal and tangential skin forces,” IEEE Int. Conf. Intell. Robot. Syst., vol. 2015-Decem, pp. 1186–1193, 2015. [53] C. Antfolk, C. Balkenius, G. Lundborg, B. Rosén, and F. Sebelius, “Design and technical construction of a tactile display for sensory feedback in a hand prosthesis system,” Biomed. Eng. Online, vol. 9, pp. 1–9, 2010. 151 [54] C. Tawk, G. Spinks, M. in het Panhuis, and G. Alici, “3D Printable Linear Soft Vacuum Actuators (LSOVA): their modeling, performance quantification and application in soft robotic systems,” IEEE/ASME Trans. Mechatronics, vol. 4435, no. c, pp. 1–1, 2019. [55] C. Antfolk, A. Björkman, S. O. Frank, F. Sebelius, G. Lundborg, and B. Rosen, “Sensory feedback from a prosthetic hand based on airmediate d pressure from the hand to the forearm skin,” J. Rehabil. Med., vol. 44, no. 8, pp. 702–707, 2012. [56] M. Rossi, M. Bianchi, E. Battaglia, M. G. Catalano, and A. Bicchi, “HapPro: A Wearable Haptic Device for Proprioceptive Feedback,” IEEE Trans. Biomed. Eng., vol. 66, no. 1, pp. 138–149, 2019. [57] B. Stephens-Fripp, R. Mutlu, and G. Alici, “Applying Mechanical Pressure and Skin Stretch Simultaneously for Sensory Feedback in Prosthetic Hands,” Proc. IEEE RAS EMBS Int. Conf. Biomed. Robot. Biomechatronics, vol. 2018-Augus, pp. 230–235, 2018. [58] K. Kim, J. E. Colgate, J. J. Santos-Munné, A. Makhlin, and M. A. Peshkin, “On the design of miniature haptic devices for upper extremity prosthetics,” IEEE/ASME Trans. Mechatronics, vol. 15, no. 1, pp. 27–39, 2010. [59] E. Biddiss, D. Beaton, and T. Chau, “Consumer design priorities for upper limb prosthetics,” Disabil. Rehabil. Assist. Technol., vol. 2, no. 6, pp. 346–357, 2007. [60] S. Ebrahimi Takalloo, H. Seifi, and J. D. W. Madden, “Design of ultra-thin high frequency trilayer conducting polymer micro-actuators for tactile feedback interfaces,” Electroact. Polym. Actuators Devices 2017, vol. 10163, no. April 2017, p. 1016312, 2017. [61] C. E. Stepp, Q. An, and Y. Matsuoka, “Repeated training with augmentative vibrotactile feedback increases object manipulation performance,” PLoS One, vol. 7, no. 2, 2012. [62] M. S. Sarwar, “Soft Capacitive Sensors for Proximity, Touch, Pressure, and Shear 152 Measurements,” University of British Columbia, 2019. [63] S. Meanley, “Different approaches and cultural considerations in third world prosthetics,” Prosthet. Orthot. Int., vol. 19, no. 3, pp. 176–180, 1995. [64] B. Peerdeman, “Myoelectric forearm prostheses:State of the art from a user-centered perspective,” J. Rehabil. Res. Dev., vol. 48, no. 6, 2011. [65] K. R. Schoepp, M. R. Dawson, J. S. Schofield, J. P. Carey, and J. S. Hebert, “Design and Integration of an Inexpensive Wearable Mechanotactile Feedback System for Myoelectric Prostheses,” IEEE J. Transl. Eng. Heal. Med., vol. 6, no. April, pp. 1–11, 2018. [66] T. Rosenbaum-Chou, W. Daly, R. Austin, P. Chaubey, and D. A. Boone, “Development and real world use of a vibratory haptic feedback system for upper-limb prosthetic users,” J. Prosthetics Orthot., vol. 28, no. 4, pp. 136–144, 2016. [67] J. Segil, R. Patel, J. Klingner, R. F. ff Weir, and N. Correll, “Multi-modal prosthetic fingertip sensor with proximity, contact, and force localization capabilities,” Adv. Mech. Eng., vol. 11, no. 4, pp. 1–9, 2019. [68] W. Navaraj and R. Dahiya, “Fingerprint‐Enhanced Capacitive‐Piezoelectric Flexible Sensing Skin to Discriminate Static and Dynamic Tactile Stimuli,” Adv. Intell. Syst., vol. 1, no. 7, p. 1900051, 2019. [69] W. Deng et al., “Microstructure-Based Interfacial Tuning Mechanism of Capacitive Pressure Sensors for Electronic Skin,” J. Sensors, vol. 2016, 2016. [70] B. C. K. Tee, A. Chortos, R. R. Dunn, G. Schwartz, E. Eason, and Z. Bao, “Tunable flexible pressure sensors using microstructured elastomer geometries for intuitive electronics,” Adv. Funct. Mater., vol. 24, no. 34, pp. 5427–5434, 2014. [71] Frenoy, “Getting Started - Piksey Pico, Nano,” 2020. [Online]. Available: 153 https://bitsnblobs.com/getting-started-piksey-pico-nano/. [72] S. Eom and S. Lim, “Stretchable complementary split ring resonator (CSRR)-based radio frequency (RF) sensor for strain direction and level detection,” Sensors (Switzerland), vol. 16, no. 10, pp. 1–12, 2016. [73] L. G. Hector and H. L. Schultz, “The dielectric constant of air at radiofrequencies,” J. Appl. Phys., vol. 7, no. 4, pp. 133–136, 1936. [74] D. Watzenig and G. Steiner, “Offset Compensation for Capacitive Angular Position Sensors by Evaluating the Kalman Filter Innovation Sequence,” pp. 1457–1462, 2004. [75] H. Feng, M. Pettinari, and H. Stang, “Study of normal and shear material properties for viscoelastic model of asphalt mixture by discrete element method,” Constr. Build. Mater., vol. 98, pp. 366–375, 2015. [76] “FormLabs.” [Online]. Available: https://formlabs.com/. [77] J. Shintake, H. Sonar, E. Piskarev, J. Paik, and D. Floreano, “Soft pneumatic gelatin actuator for edible robotics,” IEEE Int. Conf. Intell. Robot. Syst., vol. 2017-Septe, pp. 6221–6226, 2017. [78] A. S. Augurelle, A. M. Smith, T. Lejeune, and J. L. Thonnard, “Importance of cutaneous feedback in maintaining a secure grip during manipulation of hand-held objects,” J. Neurophysiol., vol. 89, no. 2, pp. 665–671, 2003. [79] M. Sensorics and I. A. M. Hand, “THE GRIP-STABILISING-SENSOR ’ AN EXAMPLE FOR INTEGRATING,” pp. 44–45. [80] Ottobock, “Myoelectric Speed hands,” 2020. [Online]. Available: https://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solution-overview/myoelectric-devices-speedhands/. 154 [81] H. K. Lee, J. Chung, S. Il Chang, and E. Yoon, “Normal and shear force measurement using a flexible polymer tactile sensor with embedded multiple capacitors,” J. Microelectromechanical Syst., vol. 17, no. 4, pp. 934–942, 2008. [82] M. Y. Cheng, C. L. Lin, Y. T. Lai, and Y. J. Yang, “A polymer-based capacitive sensing array for normal and shear force measurement,” Sensors (Switzerland), vol. 10, no. 11, pp. 10211–10225, 2010. [83] D. D. Damian, M. Ludersdorfer, Y. Kim, A. Hernandez Arieta, R. Pfeifer, and A. M. Okamura, “Wearable haptic device for cutaneous force and slip speed display,” Proc. - IEEE Int. Conf. Robot. Autom., pp. 1038–1043, 2012. [84] H. K. Lee, J. Chung, S. Il Chang, and E. Yoon, “Real-time measurement of the three-axis contact force distribution using a flexible capacitive polymer tactile sensor,” J. Micromechanics Microengineering, vol. 21, no. 3, 2011. [85] Ottobock, “Myoelectric Speed Hands.” [Online]. Available: https://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solution-overview/myoelectric-devices-speedhands/. [86] A. Charalambides and S. Bergbreiter, “A novel all-elastomer MEMS tactile sensor for high dynamic range shear and normal force sensing,” J. Micromechanics Microengineering, vol. 25, no. 9, 2015. [87] U. Wijk and I. Carlsson, “Forearm amputees’ views of prosthesis use and sensory feedback,” J. Hand Ther., vol. 28, no. 3, pp. 269–278, 2015. [88] E. Biddiss and T. Chau, “Upper-limb prosthetics: Critical factors in device abandonment,” Am. J. Phys. Med. Rehabil., vol. 86, no. 12, pp. 977–987, 2007. [89] Tactical Haptics, “Products.” [Online]. Available: https://tacticalhaptics.com/products/. 155 [90] H. Yamada, Y. Yamanoi, K. Wakita, and R. Kato, “Investigation of a cognitive strain on hand grasping induced by sensory feedback for myoelectric hand,” Proc. - IEEE Int. Conf. Robot. Autom., vol. 2016-June, pp. 3549–3554, 2016. [91] R. Bruyer and M. Brysbaert, “Combining Speed and Accuracy in Cognitive,” Psychol. Belg., vol. 51, pp. 5–13, 2011. 156 157 Appendices 158 Appendix A - Chapter 2 A.1 Stiffness Calculations We know that: • 𝜎 =𝐹𝐴 where 𝜎 is the stress, F is the force and A is the area. In this case, 𝐴 = 𝑤2. • 𝜎 = 𝐸𝜖 where 𝜖 =Δ𝐿𝐿 is the strain, and E is the elastic modulus. • 𝐹 = 𝑘Δ𝑦0 where k is the spring constant, in our case, the stiffness. Combining these equations, we find: (1): 𝑘 =𝐹Δ𝑦0=𝜎𝐴Δ𝑦0=𝐸𝜖𝐴Δ𝑦0=𝐸𝐴Δ𝑦0Δ𝑦0(𝑦0)=𝐸𝐴𝑦0 We will now integrate this equation: Using similar triangles: 𝑦0𝑤=𝑦0 − 𝑦𝑥 (2): 𝑥 =𝑤(𝑦0 − 𝑦)𝑦0= 𝑤(1 −𝑦𝑦0) (2) in (1): 𝑘 =𝐸𝑤2𝑦0 → 𝑑𝑘 =𝐸𝑥2𝑑𝑦=𝐸(𝑤(1 −𝑦𝑦0))2𝑑𝑦 159 Modeling each portion as springs in series, we use k𝑡𝑜𝑡𝑎𝑙 = (∑1𝑘𝑖)−1𝑦0𝑖=0 1k𝑡𝑜𝑡𝑎𝑙= ∫1𝐸 (𝑤 (1 −𝑦𝑦0))2 𝑑𝑦 =1𝐸ℎ0∫ (𝑤 (1 −𝑦𝑦0))−2 𝑑𝑦ℎ0 1k𝑡𝑜𝑡𝑎𝑙= 1𝑤2𝐸×(1 −𝑦𝑦0)−1(−1𝑦0) (−1)|𝑦00= 1𝑤2𝐸𝑦0(1 −𝑦𝑦0)|𝑦00 1k𝑡𝑜𝑡𝑎𝑙=1𝑤2𝐸𝑦0(1 −𝑦𝑦0)−1𝑤2𝐸𝑦0(1 − 0)=1 − (1 −𝑦𝑦0)𝑤2𝐸𝑦0(1 −𝑦𝑦0)=(𝑦𝑦0)𝑤2𝐸𝑦0(1 −𝑦𝑦0) k𝑡𝑜𝑡𝑎𝑙 =𝑤2𝐸𝑦0(1 −𝑦𝑦0) (𝑦0𝑦) =𝑤2𝐸𝑦(1 −𝑦𝑦0) = 𝑤2𝐸 (1𝑦−1𝑦0) 160 A.2 Additional Sensor Data Figure A.2- 1: 25 kPa stress experiment on the sensor (A) % Change in Capacitance corresponding to the pressure applied, (B) Change in force and displacement from the BOSE mechanical analyzer, (C) % Change in capacitance corresponding to the displacement applied 161 Figure A.2- 2: Step experiment of 25 kPa stresses followed by 1.2 kPa stresses on the sensor laid on a flat surface (A) % Change in capacitance and pressure data, (B) Force and displacement data (C) Displacement and % change in capacitance data 162 A.3 Additional fingertip data Figure A.3- 1: Percent change in capacitance of the fingertip system vs. pressure for ten load cycles of 2 N. 163 Appendix B - Chapter 3 B.1 Setup Specifications The specifications of the ATI Nano 17 sensor are as follows: Calibration Fx,Fy (N) Fz (N) Tx,Ty (N.mm) Tz (N.mm) Fx,Fy (N) Fz (N) Tx,Ty (N.mm) Tz (N.mm) SI-12-0.12 12 17 120 120 1/320 1/320 1/64 1/64 SENSING RANGES RESOLUTION The specifications of the ThorLabs NanoMax stage is as follows: Description Magnitude Maximum displacement 4 mm Minimum travel 20 m Maximum velocity 5 mm/s Maximum acceleration 5 mm/s2 164 B.2 Base Sensor Data Figure B.2- 1: Base Sensor Pressure Characterization (A) The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the pressure applied by the load cell in black, (B) Percent change in capacitance for each displacement in the z-direction, (C) Percent change in capacitance for each pressure in the z-direction 165 Figure B.2- 2: Shear data at different normal loads of 0.4 N, 0.8 N, 1.2 N, and 1.6 N (A) Percent change in capacitance for the x-direction as a function of displacement and shear force, (B) -x-direction data, (C) y-direction direction, (D) -y-direction data. 166 B.3 Mathematics Calculating shear displacement only (with normal strain applied): d: distance between the top and bottom electrodes 𝜆: shear strain 𝜂: normal strain 𝐴 = 𝑊𝐿, where A is the area of the electrode, W the width and L the length. C: Capacitance Cp: parasitic capacitance 𝜖: absolute permittivity 167 Calculating normal strain only (when shear applied): 168 B.4 Additional Sensor Data Figure B.4- 1: Shear data in the -x-direction. The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the shear force in black experienced by the load cell in the -x-direction. 169 Figure B.4- 2: Shear data in the y-direction. The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the shear force in black experienced by the load cell in the y-direction. 170 Figure B.4- 3: Shear data in the -y-direction. The two top graphs represent the load cell data, the third graph is the percent change in capacitance of each electrode, and the last graph is the percent change in capacitance of each electrode and the shear force in black experienced by the load cell in the -y-direction. 171 Figure B.4- 4: Shear displacement characterization curve displaying the calculated shear, based on the capacitance value of each electrode, as a function of shear force. This shows that shear can be independently estimated and predicted from each electrode capacitance (A) x-axis data, (B) y-axis data 172 Figure B.4- 5: Normal strain characterization curves of both x and y directions displaying the calculated normal strain, based on the capacitance value of each electrode, as a function of the normal applied force. This shows that the normal strain can be estimated and predicted independently from shear. 173 Appendix C - Chapter 4 C.1 Survey 174 175 176 177 178 179 C.2 Pre-Experiment Survey 1. How old are you? o 19-29 o 30-49 o 50+ o Prefer not to answer 2. What is your level of amputation? o Partial hand/finger o Wrist disarticulation (through your wrist) o Transradial (through your forearm) o Elbow disarticulation (through your elbow) o Transhumeral (through your upper arm) o Shoulder disarticulation (through your shoulder) 3. What is the cause of your amputation? o Congenital o Trauma o Infection o Vascular o Cancer o Other (please specify): __________________________ 4. What prosthesis do you currently use? o Passive (the hand does not move) o Body powered o Battery powered 5. How long has it been since you have lost your arm? o It’s been: _____years o I was born with a missing arm 6. How long have you been using your prosthesis? ________________________ 7. How often do you use your prosthesis at work? o I don’t use a prosthesis at work o 1 to 2 hours o 2 to 4 hours o 4 to 8 hours o More than 8 hours 180 8. How often do you use your prosthesis during your recreation time? o I don’t use a prosthesis during my recreation time o 1 to 2 hours o 2 to 4 hours o 4 to 8 hours o More than 8 hours 9. Do you feel any phantom pain? o Definitely yes o Probably yes o Might or might not o Probably not o Definitely not 10. Do you feel like your prosthesis is part of your body? o Definitely yes o Probably yes o Might or might not o Probably not o Definitely not 181 C.3 Template for Interview Post-Experiment 1. How would you compare the differences between completing the tasks with and without the sensory feedback technology? __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ 2. Embodiment questions 182 3. Did the sensory feedback make any differences in your phantom limb perception? o Yes o Maybe o No o I don’t perceive any phantom limb o If yes or maybe, please describe: _________________________________________________________ 4. Is there a specific task you think this device could help you with? Is there a specific task in which shear feedback could help you? ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ 5. Do you have any suggestions for improvement? ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ 6. What additional functions would you like to perceive? o Temperature o Proximity 183 7. Do you think this technology might be beneficial for people using a myoelectric hand for the first time? Please describe. o Yes o Maybe o No o Please describe: ______________________________________________________________________ 8. What is the most important feature in your prosthesis? Please rank from most to least important. __ - Comfort __ - Appearance __- Light weight __ - Quiet / not noisy __ - Ease of maintenance __ - Grip strength __ - Grip control __ - Overall function __ - Cost 9. If this product was commercially available, how would you like it to be different? ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Thesis/Dissertation
eng
Biomedical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Touchscreen-friendly fingertip system for tactile feedback in prosthetic hands
Text