Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Jalilvand, Iman
Doctor of Philosophy - PhD
University of British Columbia
Extended reality (XR) and machine learning (ML) are becoming pivotal enabling technologies in smart manufacturing, particularly for optimizing complex/multi-step processes. XR platforms provide immersive, interactive training environments for operators, while ML enables precise control and optimization of manufacturing processes. However, integrating these technologies into dynamic, real-time environments still presents challenges, particularly when faced with process variability and uncertainty. This dissertation addresses this technological trend in the thermoforming manufacturing process by developing a set of new XR systems, including virtual reality (VR) and mixed reality (MR) platforms integrated with real-time heat transfer simulations, for operator training. These immersive systems offer hands-on interaction with critical process variables such as temperature distribution, allowing operators to gain practical, real-world experience in a simulated environment. It is shown that the enhanced user engagement and precision achieved through the developed XR platforms significantly improve training performance metrics, as opposed to traditional training methods which often lack interactivity and real-time feedback. In the second part of this research, deep reinforcement learning (DRL) models are trained, using the same simulation models, and utilized to optimize the thermoforming process. Algorithms such as Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN) are employed and compared to control temperature distribution across thermoplastic sheet, resulting in better energy efficiency and product quality. The study also explores a multi-agent reinforcement learning (MARL) to optimize multi-heater systems in the process, providing a decentralized control and improving the model training process efficiency. Finally, a novel transfer learning (TL) application is introduced to improve the adaptability of the above RL models, ensuring a robust performance despite practicable uncertainties like diverse material properties and varying environmental conditions that are faced in practice.
EXTENDED REALITY-BASED TRAINING AND ADAPTIVE MACHINE LEARNING-BASED OPTIMIZATION FOR THERMOFORMING PROCESS by Iman Jalilvand B.Sc., Science and Research Branch of Azad University, 2015 M.Sc., Amirkabir University of Technology, 2018 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in The College of Graduate Studies (Mechanical Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Okanagan) January 2025 © Iman Jalilvand, 2025 ii The following individuals certify that they have read, and recommend to the College of Graduate Studies for acceptance, the dissertation entitled: EXTENDED REALITY-BASED TRAINING AND ADAPTIVE MACHINE LEARNING-BASED OPTIMIZATION FOR THERMOFORMING PROCESS submitted by Iman Jalilvand in partial fulfilment of the requirements for the degree of Doctor of Philosophy Dr. Abbas S. Milani, School of Engineering Supervisor Dr. Bhushan Gopaluni, University of British Columbia Co-supervisor Dr. Rudolf Seethaler, School of Engineering Supervisory Committee Member Dr. Apurva Narayan, Irving K. Barber Faculty of Science Supervisory Committee Member Dr. Mohammad Khalad Hasan, Irving K. Barber Faculty of Science University Examiner Dr. Jinfeng Liu, University of Alberta External Examiner iii Abstract Extended reality (XR) and machine learning (ML) are becoming pivotal enabling technologies in smart manufacturing, particularly for optimizing complex/multi-step processes. XR platforms provide immersive, interactive training environments for operators, while ML enables precise control and optimization of manufacturing processes. However, integrating these technologies into dynamic, real-time environments still presents challenges, particularly when faced with process variability and uncertainty. This dissertation addresses this technological trend in the thermoforming manufacturing process by developing a set of new XR systems, including virtual reality (VR) and mixed reality (MR) platforms integrated with real-time heat transfer simulations, for operator training. These immersive systems offer hands-on interaction with critical process variables such as temperature distribution, allowing operators to gain practical, real-world experience in a simulated environment. It is shown that the enhanced user engagement and precision achieved through the developed XR platforms significantly improve training performance metrics, as opposed to traditional training methods which often lack interactivity and real-time feedback. In the second part of this research, deep reinforcement learning (DRL) models are trained, using the same simulation models, and utilized to optimize the thermoforming process. Algorithms such as Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN) are employed and compared to control temperature distribution across thermoplastic sheet, resulting in better energy efficiency and product quality. The study also explores a multi-agent reinforcement learning (MARL) to optimize multi-heater systems in the process, providing a decentralized control and improving the model training process efficiency. Finally, a novel transfer learning (TL) application is introduced to improve the adaptability of the above RL models, iv ensuring a robust performance despite practicable uncertainties like diverse material properties and varying environmental conditions that are faced in practice. v Lay Summary This dissertation investigates the application of emerging artificial intelligence tools to enhance thermoforming process, with a particular focus on operator training and process optimization. Thermoforming is a complex process used to shape plastics and composites into intricate 3D forms, yet it poses significant challenges for new operators to master. By leveraging immersive technologies such as virtual reality (VR) and mixed reality (MR), this research introduces innovative training methods in a simulated, cost-effective, and safe environment. In parallel, machine learning (ML) techniques are employed to optimize the heating phase of the process, improving energy efficiency and product quality. Furthermore, adaptive ML models are applied to manage uncertainties, such as variations in the forming sheet’s material properties. Ultimately, this research contributes to advancing both operator training and the overall efficiency of thermoforming processes, aligning with the goals of Industry 5.0 by promoting greater automation, sustainability, and waste reduction in smart manufacturing systems. vi Preface This dissertation was developed under the co-supervision of Dr. Bhushan Gopaluni and Dr. Abbas S. Milani at the University of British Columbia, within the Composites Research Network (CRN) in Canada. My contributions involved the creation of the solution framework, establishing and developing parts of a thermoforming setup in the laboratory environment, the development of simulation-based immersive platforms for operator training, the design and implementation of multiple user studies to assess the user behaviour and evaluate the designed applications, the design of Artificial Intelligence (AI)-based control strategies for process optimization. No generative AI was involved in any part of this research. The original research conducted for this dissertation has resulted in multiple journal and conference publications, as summarized below. 1. Part of Chapter 3 has been published in the Journal of Manufacturing Systems. I was responsible for developing the modeling framework, conducting the user studies, analyzing the results, and drafting the manuscript. Jalilvand, I., Jang, J., Gopaluni, B., & Milani, A. S. (2024). VR/MR systems integrated with heat transfer simulation for training of thermoforming: A multicriteria decision-making user study. Journal of Manufacturing Systems, 72, 338-359, https://doi.org/10.1016/j.jmsy.2023.11.007. 2. A part of Chapter 3 was published in the proceedings of the 25th International Conference on Human-Computer Interaction (HCII-2023). The paper received the Silver Award in the student design competition of the conference. I was responsible for developing the immersive application design and simulation framework, conducting the user study, analyzing the results, and drafting the manuscript. vii Jalilvand, I., Jiyoung, J., Hosseinionari, H., Seethaler, R., Gopaluni, B., Milani, A.S. (2023). An Interactive Digital Twin of a Composite Manufacturing Process for Training Operators via Immersive Technology. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2023 Posters. HCII 2023. Communications in Computer and Information Science, vol 1836. Springer, Cham. https://doi.org/10.1007/978-3-031-36004-6_28. 3. A version of Chapter 4 has been published in the proceedings of the International Conference on Industrial and Systems Engineering (IISE-2024). I was responsible for developing and comparing two different AI control models, analyzing the results, and drafting the manuscript. Jalilvand, I., Gopaluni, B., & Milani, A. S. Towards a Robust Deep Reinforcement Learning for Optimization of Heating Setup in Thermoforming Process, In: A. Brown Greer, C. Contardo, J.-M. Frayret, eds., Proceedings of the IISE Annual Conference & Expo, 2024. https://doi.org/10.21872/2024IISE_7645. 4. A version of Chapter 5 has been submitted to the Journal of Advanced Engineering Informatics. I developed the machine learning methods, conducted the experiments, analyzed the results, and written the manuscript. Jalilvand, I., Soufi, A., Hosseinionari, H., Seethaler, R., Narayan, A., Gopaluni, B., Milani, A. S. Towards a Robust Multi-Agent Transfer Reinforcement Learning for Optimization of Heating Setup in Thermoforming, (2024). (under review) viii Throughout my PhD studies, I have participated in several side projects in collaboration with other research labs at UBC, leading to the following journal and conference publications: 5. Hosseinionari, H., Seethaler, R., Krishnamurthy, R. J., Jalilvand, I., & Milani, A. S. (2024). A scalable multi-agent deep reinforcement learning in thermoforming: An experimental evaluation of thermal control by infrared camera-based feedback. Journal of Manufacturing Processes, 131, 312-326. https://doi.org/10.1016/j.jmapro.2024.09.019. 6. Krishnamurthy, R.J., Lo, V., Jang, J., Jalilvand, I., Date, A., Ambrosetti, D., Khayyam, H., Hasan, M.K., Milani, A. S. A Healthcare Education Exploratory Case Study with SDL: Extended Reality for Intravenous (IV) Infusion Training (2024). Journal of Computers in Human Behaviour Reports. (Under review). ix Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ......................................................................................................................... ix List of Tables .............................................................................................................................. xiv List of Figures ...............................................................................................................................xv List of Symbols ........................................................................................................................... xxi List of Abbreviations .............................................................................................................. xxvii Acknowledgements .................................................................................................................. xxxi Dedication ............................................................................................................................... xxxiii Chapter 1: Background and thesis organization ........................................................................1 1.1 Introduction ..................................................................................................................... 1 1.2 Objectives ....................................................................................................................... 2 1.3 Key engineering contributions expected ......................................................................... 4 1.4 Thesis organization and methodology overview ............................................................ 5 Chapter 2: Literature Review .......................................................................................................9 2.1 Industry 5.0 ..................................................................................................................... 9 2.2 Immersive platforms in manufacturing and workforce training ..................................... 9 2.2.1 Physics-based process simulations within XR .......................................................... 14 2.2.2 UI, UX and usability in XR applications .................................................................. 16 2.3 Artificial intelligence and machine learning in advanced manufacturing .................... 17 2.3.1 Application of DRL in manufacturing process optimization .................................... 18 x 2.3.1.1 DRL in thermoplastic composites manufacturing ............................................ 20 2.3.2 Application of TL in manufacturing ......................................................................... 22 2.4 Integration of TL and RL in industrial applications ..................................................... 24 2.5 Summary: research gaps ............................................................................................... 25 Chapter 3: Developing Extended Reality Platforms for Thermoforming Training: A Multicriteria Decision Making User Study................................................................................28 3.1 Introduction ................................................................................................................... 28 3.2 Methodology ................................................................................................................. 29 3.2.1 Thermoforming set-up in the case study ................................................................... 30 3.2.2 Transient (time dependent) heat transfer modeling of the process ............................ 31 3.2.3 XR development ....................................................................................................... 35 3.2.3.1 VR application development ............................................................................. 36 3.2.3.2 MR application development ............................................................................ 37 3.2.4 User study design ...................................................................................................... 38 3.2.4.1 Hypotheses ........................................................................................................ 40 3.2.4.2 Experimental procedure and questionnaires ..................................................... 40 3.2.4.3 Questionnaire design ......................................................................................... 42 3.2.4.4 Participants diversity ......................................................................................... 43 3.2.5 Statistical analysis with blocks ................................................................................. 45 3.2.6 Multi-Criterial Decision Making (MCDM) .............................................................. 48 3.3 Results and discussion .................................................................................................. 49 3.3.1 VR application .......................................................................................................... 49 3.3.2 MR application.......................................................................................................... 51 xi 3.3.3 User study ................................................................................................................. 53 3.3.3.1 ANOVA with blocks and post-hoc tests ........................................................... 56 3.3.3.2 MCDM results .................................................................................................. 59 3.3.4 A qualitative interpretation of user study outcomes ................................................. 61 3.4 Summary ....................................................................................................................... 64 Chapter 4: Single-Agent Reinforcement Learning Algorithms for Optimizing the Heating Setup in Thermoforming .............................................................................................................66 4.1 Introduction ................................................................................................................... 66 4.2 The role of AI in manufacturing and thermoforming optimization .............................. 66 4.3 Methodology ................................................................................................................. 67 4.3.1 Deep reinforcement learning model .......................................................................... 69 4.3.1.1 State space:........................................................................................................ 72 4.3.1.2 Action Space ..................................................................................................... 72 4.3.1.3 Reward function ................................................................................................ 73 4.3.2 Deep-Q Networks (DQN) ......................................................................................... 74 4.3.3 Proximal Policy Optimization (PPO) ........................................................................ 77 4.4 Results and discussion .................................................................................................. 81 4.4.1 Single-heater optimization with single-agent RL ..................................................... 81 4.4.2 Multi-heater optimization with single-agent RL: column arrangement ................... 83 4.4.3 Multi-heater optimization with single-agent RL: scatter arrangement ..................... 85 4.4.4 All-heater optimization with single-agent RL: full sheet ......................................... 87 4.5 Summary ....................................................................................................................... 88 xii Chapter 5: Integrating Transfer Learning with Multi-Agent Reinforcement Learning: How to Simultaneously Increase Modeling Efficiency and Energy Saving in the process? ..........90 5.1 Introduction ................................................................................................................... 90 5.2 Methodology ................................................................................................................. 91 5.2.1 A modified reward function proposed ...................................................................... 91 5.2.2 Overall reward function ............................................................................................ 94 5.2.3 Multi-agent RL.......................................................................................................... 95 5.2.3.1 Decentralized multi-agent PPO model .............................................................. 98 5.2.3.2 Centralized multi-agent PPO model ............................................................... 100 5.2.3.3 Proposed multi-agent transfer reinforcement learning (MATRL) .................. 104 5.3 Results and discussion ................................................................................................ 108 5.3.1 Single RL agent ....................................................................................................... 109 5.3.2 Centralized MATRL with direct parameters transfer ............................................. 110 5.3.3 Testing MATRL’s adaptability for other manufacturing scenarios/variations: with sample transfer mode .......................................................................................................... 113 5.3.3.1 Example 1: Higher convective heat transfer coefficients (HTCs) .................. 114 5.3.3.2 Example 2: Sheets with higher heat conduction properties ............................ 118 5.3.3.3 Example 3: Higher initial ambient temperatures ............................................ 121 5.3.4 Overall comparison of the trained models .............................................................. 125 5.4 Summary ..................................................................................................................... 127 Chapter 6: Conclusion and Future Work ................................................................................130 6.1 Summary of the main findings .................................................................................... 130 6.2 Contributions to knowledge ........................................................................................ 133 xiii 6.3 Future work and recommendations ............................................................................. 134 Bibliography ...............................................................................................................................138 Appendices ..................................................................................................................................158 Appendix A : User interface design for MR application in Microsoft HoloLens 2: initial case study ........................................................................................................................................ 158 A.1 Methodology ........................................................................................................... 158 A.2 UI and UX design ................................................................................................... 159 A.3 User study and data analysis ................................................................................... 161 A.4 MR application........................................................................................................ 161 A.5 User study ............................................................................................................... 162 A.6 Remarks .................................................................................................................. 164 Appendix B : User survey used in Chapter 3 .......................................................................... 166 Appendix C : Additional ANOVA Data and Checking its Assumptions ............................... 169 xiv List of Tables Table 3.1:ANOVA with blocks results for the usefulness criterion. ............................................ 57 Table 3.2: Means of the three training methods (decision alternatives) per each criterion along with the criteria weights assigned. ........................................................................................................ 60 Table 3.3: TOPSIS Rankings of the training methods (alternatives) based on various sets of criteria weights. ......................................................................................................................................... 61 Table 4.1: DQN algorithm and Pseudocode. ................................................................................ 76 Table 4.2 - DQN hyperparameters. ............................................................................................... 76 Table 4.3 - PPO hyperparameters. ................................................................................................ 80 Table 5.1 – FCNN-based MAPPO algorithm and pseudocode. ................................................. 103 Table 5.2 – Summary of the tuned MAPPO hyperparameters. .................................................. 104 Table 5.3 - Comparison between single and multi-agent models, in terms of effectiveness and energy usage................................................................................................................................ 127 Table A.1. The SUS score for different users of the MR application. ........................................ 163 Table C.2. ANOVA with blocks results for ease of use. ............................................................ 171 Table C.3. ANOVA with blocks results for satisfaction. ........................................................... 171 Table C.4. ANOVA with blocks results for trust........................................................................ 171 Table C.5. ANOVA with blocks results for reliability. .............................................................. 171 Table C.6. ANOVA with blocks results for training. ................................................................. 171 Table C.7. ANOVA with blocks results for preference. ............................................................. 171 xv List of Figures Figure 1.1: Organization of the thesis. Starting with introduction and literature review, and moving on to operator training with immersive platforms, and their comparison in terms of effectiveness. Next, thermoforming process optimization using ML methods. Lastly fine-tuning the optimization using a combination of TL and RL while sharing experience, to optimize the process and reduce energy consumption. ....................................................................................................................... 8 Figure 3.1: Overview of the methods employed in the present study. ......................................... 30 Figure 3.2: A typical thermoforming process and example of potential defects in the final part [38]........................................................................................................................................................ 31 Figure 3.3: Heat transfer between thermoplastic sheets, heaters, and the environment. .............. 32 Figure 3.4: Finite difference modelling: (a) conduction heat transfer, and (b) including all modes of heat transfer. ............................................................................................................................. 32 Figure 3.5: XR design (left) versus the actual setup (right). ......................................................... 36 Figure 3.6: The Unity state machine and animator employed. ..................................................... 37 Figure 3.7: VR input system definition......................................................................................... 37 Figure 3.8: The MR input system and hand gestures for near and remote interactions. a) Neutral position, Hover, and touch. b) Grab, move, and rotate. ................................................................ 38 Figure 3.9: Flowchart of the experimental procedure in the user study. ...................................... 42 Figure 3.10: Participants' demographic information. a) gender, b) educational level and c) experience level. ........................................................................................................................... 45 Figure 3.11: Participants' age distribution. ................................................................................... 45 Figure 3.12: Illustration of the VR app steps followed by a user in the experiments. .................. 50 xvi Figure 3.13: Steps of the developed MR application for training operators. a) Placing the thermoplastic sheet on the sheet holder. b) Prompted control panel to allow user interactions. c) Activating the heaters using the buttons on control panel. d) Adjusting the heater's power range using the slider. e) Deactivating all the heaters at once using the push button. f) Placing the sheet holder on the mold. g) Activating the vacuum pump, using the push button. .............................. 52 Figure 3.14: Features of the developed MR application: control panel on the left, the virtual setup on the right and the actual setup between them. In the shown example, the first row of heaters (in red) in the virtual set-up are working at full power (here 500W), and a heater in the middle of the second row (in green) is working at 150W power (the screen on top of the control panel depicts the info about the latter heater). Additionally, the sheet is heated, and the colour gradient in various regions reflects its surface temperature, as it is also shown on the operator’s hand once remotely selected via hand raycast. .............................................................................................................. 53 Figure 3.15: The distribution of data obtained for each training method with respect to the performance criteria. ..................................................................................................................... 54 Figure 3.16: Stacked mean (left) and median (right) performance scores, among different training platforms. ...................................................................................................................................... 55 Figure 3.17: Comparison of XR novices and XR experienced with respect to the measured criteria, for each of the three training methods. ......................................................................................... 55 Figure 3.18: Normality check via Q-Q plot of the usefulness criterion data (left). Constant variance check for the criterion (right). ....................................................................................................... 56 Figure 3.19: Means of the three training methods (decision alternatives) per each criterion along with the criteria weights assigned. ................................................................................................ 60 Figure 4.1 – Overview of the methods employed in the present chapter. .................................... 68 xvii Figure 4.2 – Overview of the thermoforming setup employed and its components. .................... 69 Figure 4.3: MATLAB Simulink Block Diagram designed for the DRL model. .......................... 81 Figure 4.4: DQN’s reward against episode number. a) 9 actions space, b) 3 action space, and c) random target temperature space. ................................................................................................. 82 Figure 4.5: PPO’s reward against episode numbers. a) 9 actions space, b) 3 action space, and c) random target temperature space. ................................................................................................. 82 Figure 4.6: a) Low temperature profile, and b) high temperature profile selected as desired distribution. ................................................................................................................................... 83 Figure 4.7: a) Error, b) actions in the low desired temperature profile, and c) cumulative reward........................................................................................................................................................ 84 Figure 4.8: a) Cumulative reward, b) error, and c) actions in the high desired temperature profile........................................................................................................................................................ 85 Figure 4.9: Desired temperature profile: scattered arrangement. ................................................. 86 Figure 4.10: a) Error, b) Actions in the high desired temperature profile, and c) Cumulative reward........................................................................................................................................................ 87 Figure 4.11: Retrained single agent to optimize the heating for all zones (full sheet). a) Cumulative reward, b) Error. ............................................................................................................................ 88 Figure 5.1 - Block diagram of the multi-agent RL optimization model. ...................................... 96 Figure 5.2 – Schematic of a decentralized multi-agent PPO model. .......................................... 100 Figure 5.3 – Schematic of the developed centralized multi-agent PPO model. ......................... 101 Figure 5.4 - Direct parameter transfer involves layer-wise transfer from source to target network. Sample transfer uses experiences from multiple agents in a shared replay memory, resulting in more stable training and generalized policies. ............................................................................ 107 xviii Figure 5.5 - Retrained single agent (via hierarchical RL), to concurrently minimize the error and energy usage for 5 zones (left side of the sheet). a) Actions, b) Error, and c) Cumulative reward...................................................................................................................................................... 110 Figure 5.6 – Assumed desired temperature profile on the thermoplastic sheet. ......................... 112 Figure 5.7 - Trained multi-agent transfer reinforcement learning (MATRL), optimizing error and energy usage, as well as stability, for all zones (full sheet). a) Cumulative rewards of the multiple agents, b) Actions, and c). Error distributions. ........................................................................... 113 Figure 5.8 - Low (h1) and high (h2) convective heat transfer coefficient between heaters and sheet...................................................................................................................................................... 114 Figure 5.9 – Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 1: h1 = 7.5Wm2. K. .............................................................................................. 115 Figure 5.10 – Case 1 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 1: h1 = 7.5Wm2. K. .................................................................... 116 Figure 5.11 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 2: h2 = 15Wm2. K. ............................................................................................... 117 Figure 5.12 - Case 2 - a) Total number of activations for each heater. b) Its distribution throughout the process - case 2: h2 = 15Wm2. K. ....................................................................................... 117 Figure 5.13 - Low (k3) and high (k4) conductivity in the thermoplastic materials. .................. 118 Figure 5.14 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 3: k3 = 0.4Wm.K . ............................................................................................... 119 Figure 5.15 - Case 3 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 3: k3 = 0.4Wm.K. ...................................................................... 120 xix Figure 5.16 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 4: k4 = 0.04Wm. K. .............................................................................................. 120 Figure 5.17 - Case 4 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 4: k4 = 0.04Wm. K. .................................................................... 121 Figure 5.18 - Low (Ta5) and high (Ta6) initial ambient temperature between the heaters and the sheet. ........................................................................................................................................... 122 Figure 5.19 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 5: Ta5 = 50℃. ...................................................................................................... 123 Figure 5.20 - Case 5 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 5: Ta5 = 50℃. ............................................................................ 123 Figure 5.21 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 6: Ta6 = 100℃. .................................................................................................... 124 Figure 5.22 - Case 6 - a) Total number of activations for each heater b) Its distribution throughout the process - case 6: Ta6 = 100℃. ............................................................................................ 125 Figure A.1. The present MR application design framework…………………………………….159 Figure A.2. Screenshots from the “Main Menu” and “Practice Mode” scenes of the MR app. a) shows the available options once the application starts. b) Practice scene options, which allow the user to explore hand interactions (touch, pinch, grab, move, and scale) before testing the Training application. c) Near-Hand menu in Practice Mode. d) Near-Hand menu in Training Application……………………………………………………………………………………...161 Figure A.3. Features of the developed digital twin for training thermoforming operators. a) Control panel provides the info screen, showing the surface temperature distribution and the power consumption of heaters. b) Visualizing the heater’s surface temperature by hovering the hand over xx a selected heater. Once the heater is pinched, a visual heat bar guide is also shown on the side. c) Adjusting the heater's power range using the slider. d) Text instruction and the guiding arrow are facing the operator………………………………………………………………………………162 Figure A.4. Users’ responses distribution on the System Usability Scale (SUS), ranging from strongly disagree to strongly agree. Notice that some of the SUS questions are deliberately (by the standard) set in a negative form (i.e. in these cases ’strongly disagree’ is the top rate) ……….163 Figure C.1. Normality check via Q-Q plots: ease of use (left) and satisfaction (right)…………169 Figure C.2. Normality check via Q-Q plots: trust (left) and reliability (right)………………….169 Figure C.3. Normality check via Q-Q plots: training (left) and preference (right)……………...169 Figure C.4. Constant variance check: ease of use (left) and satisfaction (right)………………...170 Figure C.5. Constant variance check: trust (left) and reliability (right)…………………………170 Figure C.6. Constant variance check: training (left) and preference (right)…………………….170 xxi List of Symbols Symbols Description Chapter 3: qz+ Heat transmission in the positive vertical direction qz- Heat transmission in the negative vertical direction qx+ Heat transfer in the positive longitudinal direction qx- Heat transfer in the negative longitudinal direction qy+ Heat transfer in the positive transverse direction qy- Heat transfer in the negative transverse direction m Index of element in 𝑥 direction n Index of element in 𝑦 direction ∆𝑥 Length of the element ∆𝑦 Width of the element ∆𝑧 Height of the element 𝑄𝑖𝑐𝑜𝑛𝑑 Conduction heat transfer in 𝑖 direction 𝑄𝑥𝑐𝑜𝑛𝑑. Conduction heat transfer in 𝑥 direction 𝑄𝑦𝑐𝑜𝑛𝑑. Conduction heat transfer in 𝑦 direction 𝑘 Material's conductivity 𝐴 Cross-sectional surface area of the element ∆𝑇 Temperature difference between the elements 𝐿 Distance between the elements 𝑇𝑚,𝑛 Temperature of the element 𝑚, 𝑛 𝑄𝑖𝑐𝑜𝑛𝑣 Convection heat transfer in 𝑖 direction xxii 𝑄𝑧+𝑐𝑜𝑛𝑣. Convection heat transfer in 𝑧+ direction 𝑄𝑧−𝑐𝑜𝑛𝑣. Convection heat transfer in 𝑧− direction ℎ1 Convective heat transfer coefficient on top of thermoplastic sheet ℎ2 Convective heat transfer coefficient in the bottom of thermoplastic sheet 𝑇𝑎𝑚𝑏𝑖𝑒𝑛𝑡 Temperature of the ambient 𝐴𝑖 Surface area of 𝑖𝑡ℎ node where the heat transfer takes place 𝑄𝑖𝑟𝑎𝑑. Radiation heat transfer in 𝑖 direction 𝐴𝑗 Surface area of emitter 𝜀𝑒 Effective emissivity 𝜎 Stefan-Boltzmann constant 𝐹𝑗,𝑖 View factor between the 𝑗𝑡ℎ heater and the 𝑖𝑡ℎ elements 𝑇𝑗 Absolute temperature of the 𝑗𝑡ℎ emitter 𝑇𝑖 Absolute temperature of the 𝑖𝑡ℎ element on the sheet 𝑁 Total number of the heaters 𝑑𝑈 Change in internal energy 𝑞𝑖 Heat transfer from all surfaces to each element on the thermoplastic sheet 𝑇𝑡 New temperature on the element at time-step 𝑡 𝑇𝑡−∆𝑡 Temperature from the preceding time-step ∆𝑡 Difference between two time-steps 𝜌 sheet density 𝑉 element volume 𝐶 sheet material’s heat capacity xxiii 𝑑𝑇𝑖𝑑𝑡 time-dependent temperature variations of an element due to radiation 𝑊𝑖 Normalized Likert scale weights 𝑥𝑖 Likert scale raw weights 𝑍𝑖𝑗𝑘 weighted response for each question in Likert scale 𝑆𝑗𝑘 Normalized weighted response for each question in Likert scale 𝑀𝑘 Average of user response in Likert scale Chapter 4: ℳ Infinite-horizon discounted Markov decision process 𝒮 Set of states within the control system 𝒜 Set of feasible actions that can be taken to control the system 𝒫 State transition probability function ℛ Reward function 𝛾 Discount factor for future rewards 𝜋 Control policy 𝜂(𝜋) Optimal policy 𝐸𝜏 Expectation over the state–action sequence 𝜏 State–action sequence 𝑠𝑖 State at time-step 𝑖 𝑎0 Action at time-step 𝑖 𝑝 Penalty gain in reward function 𝑄(𝑠, 𝑎) Action-value function: expected reward for an action in a state 𝛼 Learning rate xxiv 𝑠′ Future state 𝑎′ Future action 𝑟 Reward at state s 𝑄∗ Target action-value function estimation 𝒟 Replay memory 𝜖𝐷𝑄𝑁 Probability of selecting a random action (exploration parameter) 𝜙𝑡 Preprocessed state representation 𝜃 Neural network parameters ∇𝜃𝑖𝐿𝑖(𝜃𝑖) Gradient of loss function 𝑦𝑖 Target q-value 𝑚𝑎𝑥𝑎′ 𝑄(𝜙𝑗+1, 𝑎′; 𝜃) Maximum predicted q-value for the next state 𝜙𝑗+1 across actions 𝜋𝜃(𝑎|𝑠) Probability of action 𝑎 from state 𝑠 under policy with parameters 𝜃. 𝐴𝜋 Advantage function 𝜖 Clipping parameter 𝑉𝜋 Value function 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡 Target state value 𝑉𝜔𝑜𝑙𝑑 Estimated state value ?̂?𝑡 Reward-to-go 𝜃𝑘 Policy parameters at iteration 𝑘 ∅𝑘 Parameters of the value function approximator 𝑔 (𝜖, 𝐴𝜋𝜃𝑘(𝑠, 𝑎)) Clipped advantage function 𝒟𝑘 Set of trajectories 𝜏𝑖 Individual trajectory xxv Chapter 5: 𝐿𝑆(𝑥) Linear scalarization function 𝑤𝑜 Objective functions’ weights 𝑓𝑜(𝑥) Objective function 𝑁𝑎𝑐𝑡 Number of activations per heater 𝑡𝑠 Timestep 𝑡ℎ Total heater usage time 𝑅1 Energy minimization reward term 𝑅2 Energy conservation reward term 𝑅3 Stability reward term 𝜎 Standard deviation of error 𝜇 Average error ℋ Joint policy in multi-agent RL 𝜙𝑘+1 Updated critic parameters 𝜆 Generalized Advantage Estimate (GAE) weight 𝐿(𝜃) Policy loss function 𝐵 Batch Size 𝑛 Number of agents in the multi-agent setup 𝑉𝜙 Critic value estimate 𝜎𝑒 Entropy coefficient 𝑉𝜙𝑜𝑙𝑑 Old critic value estimate 𝑉𝜙 Value function for state 𝒯 Time horizon xxvi ?̃?𝑡 Augmented state 𝐿 Chunk length: length of trajectory segments for mini-batch updates 𝐷 Data buffer 𝜒𝑆 State mapping function 𝜒𝐴 Action mapping function 𝜅𝑠 Source knowledge 𝜅𝑡 Target knowledge 𝐷𝑠 Source domain 𝐷𝑡 Target domain 𝜋∗ Optimal policy: maximizing action-value function ℳ𝑆 Source task model ℳ𝑇 Target task model 𝑀 Mean error 𝑆𝐷 Standard deviation of error ℎ𝑖 Convection constant for surface 𝑖 𝑘𝑖 Sheet thermal conductivity 𝑑𝑖 Vertical distance between sheet and heaters 𝑇𝑎𝑖 Ambient temperature xxvii List of Abbreviations Abbreviation Definition IoT IIoT Internet of Things Industrial Internet of Things ML Machine Learning DT Digital Twin AI Artificial Intelligence SVM Support Vector Machine RL Reinforcement Learning DRL Deep Reinforcement Learning TL Transfer Learning UI User Interface UX User Experience MR Mixed Reality VR Virtual Reality PPO Proximal Policy Optimization DQN Deep Q-Networks MARL Multi-Agent Reinforcement Learning HTC Heat Transfer Coefficient XR Extended Reality AR Augmented Reality VRTS Virtual Reality Training Systems xxviii HMD Head-Mounted Displays ITS Intelligent Tutoring System FOV Field of View NDE Non-Destructive Evaluation SVM Support Vector Machine RL-MPC Reinforcement Learning Model Predictive Control MPC Model Predictive Control CFD Computational Fluid Dynamic DDPG Deep Deterministic Policy Gradient ATL Automated Tape Laying MA Memory-Augmented CFRP Carbon Fibre Reinforced Plastic GNN Graph Neural Networks RTM Resin Transfer Molding CNN Convolutional Neural Networks RF Random Forest AE Acoustic Emission AutoBTL Automated Broad Transfer Learning DTL Deep Transfer Learning HPO Hyper-Parameter Optimization C-LSTMN Convolutional Long Short-Term Memory Network MK-JMMD Multi-Kernel Joint Maximum Mean Discrepancy xxix HASAC Hyper-Actor Soft Actor–Critic SAC Soft Actor-Critic CPS Cyber-Physical Systems AMS Autonomous Manufacturing Systems LPTN Lumped-Parameter Thermal Network TD3 Twin Delayed Deep Deterministic Policy Gradient DDPG Deterministic Policy Gradient ResNet Residual Network I4.0 Fourth Industrial Revolution I5.0 Fifth Industrial Revolution MCDM Multi-Criteria Decision Making FD Finite Difference MRTK Mixed Reality Toolkit VRT Virtual Reality Training MRT Mixed Reality Training DVT Demo Video Training SUS System Usability Scale ANOVA Analysis of Variance WSM Weighted Sum Method WPM Weighted Product Method TOPSIS Technique for Order Preference by Similarity to Ideal Solution PIS Positive Ideal Solution xxx NIS Negative Ideal Solution CRNO Composite Research Network-Okanagan MDP Markov Decision Process TRPO Trust Region Policy Optimization GAE Generalized Advantage Estimation MOO Multi-Objective Optimization LS Linear Scalarization CTDE Centralized Training and Decentralized Execution MAPPO Multi-Agent Proximal Policy Optimization IPPO Independent Proximal Policy Optimization FCNN Fully Connected Neural Network MATRL Multi-Agent Transfer Reinforcement Learning KPI Key Performance Indicator FPS Frames Per Second xxxi Acknowledgements I would first like to express my profound gratitude to Dr. Abbas S. Milani for his belief in my potential and for welcoming me as a Ph.D. student in his lab at UBC, as part of the Composites Research Network (CRN). His trust, encouragement, and constant support gave me the freedom to pursue my ideas and push the boundaries of my research. I am equally greatly thankful to my co-supervisor, Dr. Bhushan Gopaluni, whose insightful guidance, expert advice, and committed support were crucial in completing this dissertation. I am deeply indebted to my thesis committee members, Dr. Rudolf Seethaler and Dr. Apurva Narayan, for their patience, constructive feedback, and invaluable insights throughout my Ph.D. journey. Special thanks also go to the CRN team at the Vancouver campus and the researchers at Boeing for the enriching discussions on machine learning applications in manufacturing, which greatly expanded my understanding. I would like to extend my appreciation to Mr. Diego Faiguenbaum from Kohler Canada for providing vital materials and industry expertise that significantly shaped this research. My heartfelt thanks go to my colleagues at the Composites Research Network at the Okanagan campus. A special mention goes to Dr. Bryn Crawford and Dr. Milad Ramezankhani, whose expertise and generosity in sharing their knowledge of thermoforming composites were indispensable. I would also like to thank Dr. Tina Olfatbakhsh, Haoyang Liu, Dr. Farzan Gholamreza, and Rohith J. K. for their collaborative spirit and continuous support throughout this journey. To my off-campus friends, I am deeply grateful for your encouragement and companionship. I especially thank Dr. Hamed Mirzaei and Dr. Sajjad Mohammad Nejad for the wonderful memories we created playing football together. To my best friend, Alidad Ghasser, your support during critical moments has meant the world to me. xxxii Lastly, I am forever thankful to my family for their unconditional love and support. To my parents, thank you for giving me the opportunity to pursue my academic dreams and for always being there for me. To my brothers, Soroosh and Ehsan, our long-distance chats and video calls, especially watching my nephews and niece (Kian, Ryan, Aiden, and Ava) grow, brought much-needed joy during this process. And to my wonderful wife, Niloofar, your constant support, encouragement, and presence through every step of this challenging yet fulfilling journey have been my greatest source of strength. Your love and belief in me are my deepest motivations for success on daily basis. xxxiii Dedication …To my dearest wife Niloofar and my beloved parents Ebrahim and Zohreh… 1 Chapter 1: Background and thesis organization 1.1 Introduction The fourth and fifth industrial revolutions (known as Industry 4.0 and 5.0) have already set the stage for the digital transformation of manufacturing, driving the integration of advanced technologies like the Internet of Things (IoT), Machine Learning (ML), and immersive technologies to enhance production efficiency and reduce costs. Artificial Intelligence (AI) is broadly defined as the capability of machines to mimic human intelligence, including tasks such as learning, reasoning, and problem-solving, while ML is a subset of AI that focuses on enabling systems to learn and improve from data without being explicitly programmed [1]. Recent developments have shifted the focus toward combining human expertise with intelligent systems, a hallmark of Industry 5.0 as opposed to Industry 4.0, where human operators collaborate with semi-automated machines in a more empowered and informed manner, particularly towards environmentally more friendly solutions [1]. Central to these transformations is the use of Artificial Intelligence (AI) and ML, which have become pivotal in optimizing manufacturing processes, improving production efficiency, and enhancing decision-making capability by the human operators alongside the robots [2]. Among key areas of interest in ML and advanced manufacturing is the application of Reinforcement Learning (RL) and Transfer Learning (TL), which can offer robust solutions for optimizing complex, dynamic processes in nearly real-time [3]. While RL has been widely applied in areas such as process planning, quality control, and robotics, its use in advanced manufacturing, particularly in complex sectors like composites, remains underexplored. The potential of RL to handle sequential decision-making tasks offers significant promise for improving production efficiency, particularly when integrated with deep learning frameworks [4]. This is particularly 2 relevant to the field of composites manufacturing, where parts are produced through a series of complex processes to achieve customizable mechanical properties and low structural weight, essential for meeting the growing demands of industries like aerospace, automotive, and healthcare. [5]. In addition to AI-driven optimization, immersive technologies are gaining momentum as valuable tools for workforce training. The introduction of Extended Reality (XR), including Virtual Reality (VR) and Mixed Reality (MR), has opened new possibilities for operator training in manufacturing environments. These technologies provide enhanced, simulation-based learning experiences, allowing operators to gain a deeper understanding of complex processes while ensuring precise adherence to manufacturing steps [6]. The motivation of this research is to address current gaps in the integration of AI and immersive technology in advanced composites manufacturing. By focusing on the application of RL and TL to optimize such processes, and exploring the potential of XR for operator training, this work is expected to contribute to the human-centered digitalization goals of Industry 5.0 (more detailed gaps have been identified and presented as part of the literature review in Chapter 2). 1.2 Objectives The overarching goal of this thesis is to advance the immersive technology component of I5.0 for workforce training, and secondly implement ML-based process optimization models to tackle practical challenges (such as energy consumption minimization), during thermoforming manufacturing process. To achieve this, the thesis has been carried out based on the following specific (sub)objectives: Immersive technology part (Part I): 3 1. Develop and evaluate a spatial user interface (UI) and user experience (UX) for a MR application on the Microsoft HoloLens 2 for a lab-scale thermoforming process. This application seamlessly integrates virtual content by simulating heat transfer behavior in the thermoforming setup, enhancing operator training through immersive learning and ensuring precise adherence to manufacturing steps. 2. Develop and compare the performance of other immersive training tools including VR via Meta Quest 2, against the above MR via Microsoft HoloLens 2 as a 3D platform, as well as the conventional 2D video training tool in the same thermoforming process. In this context, Multiple Criterial Decision Making (MCDM) techniques have also been used for the first time to assess various training factors including usability, user-friendliness, trust, reliability, training effectiveness, and task completion time of each training method. ML-based process optimization part (Part II): 3. Utilize a set of single-agent Deep Reinforcement Learning (DRL) algorithms, namely Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN) and compare their performances in optimizing the heating parameters of the thermoforming via simulations. 4. Optimize the training performance of the DRL model, while considering practical considerations such as energy consumption during the process, as well as potential material or operational uncertainties. ❖ Conduct a detailed comparison of single-agent and multi-agent DRL methods, as well as decentralized versus centralized multi-agent DRL (MARL), within the same thermoforming. Identify the most efficient DRL approach for optimizing the training process, while ensuring precise temperature distribution on the sheet. 4 ❖ Improve the adaptability and robustness of MARL in the thermoforming process by incorporating TL (Transfer Learning) to handle potential manufacturing process uncertainties. This is to maintain consistent product quality and reduce waste despite variable Heat Transfer Coefficient (HTC), material properties, and initial sheet temperature; showcase the application of this innovative integration of MARL with TL for energy saving purposes during the process. 1.3 Key engineering contributions expected This research partly addresses the thermoforming operator training and process optimization challenges faced in practice, by developing and using XR technologies and ML approaches. XR provides a transformative training solution, overcoming the constraints of conventional training methods including physics-based simulations like Finite Element Analysis (FEA). While FEA models are useful for training new operators and designers for understanding thermoforming dynamics, their reliance on simplified assumptions, high computational demands, and the need for expert interpretation limit their practicality for on-site and real-time training [7], [8]. In contrast, XR technologies can offer immersive, interactive simulations that enhance skill development in a safe, controlled environment while providing real-time feedback and customizable scenarios, including hazardous or rare conditions [9]. These features reduce dependency on physical equipment, lower costs, and deliver an engaging training solution critical for operator-intensive processes like thermoforming until full automation becomes feasible [9]. Similarly, next to the training phase, for subsequent process optimization tasks by operators, Reinforcement Learning (RL) can address the limitations of current traditional control methods like Proportional-Integral-Derivative (PID) controllers and Model Predictive Control (MPC). RL’s 5 adaptability allows for effective handling of dynamic and non-linear thermoforming conditions, where more conventional methods often require extensive tuning and precise system models [10], [11]. RL optimizes control policies directly through interaction with the environment, bypassing the need for explicit models. While RL requires longer training times, its resulting policies are rapidly implementable and could adapt effectively to changing conditions with some transfer learning trials, making it possibly a robust and scalable solution for optimizing thermoforming processes [11]. By tackling both XR and RL, this research overall is expected to provide innovative contributions to thermoforming technology, enhancing both operators training and process optimization efficiency. 1.4 Thesis organization and methodology overview Based on the above objectives, this thesis is structured into two principal phases. Part I addresses the initial two objectives, while Part II is dedicated to the subsequent objectives. The dissertation is composed of six chapters, with Chapters 3, 4, and 5 constituting the core of the research on the proposed models. Chapter 3 embarks on an independent line of research, addressing the challenges of the thermoforming process through the lens of training. Conversely, Chapters 4 and 5 focus on optimizing the thermoforming process, providing an in-depth analysis from an optimization perspective. Figure 1.1 provides an overview of the organization of this work. Chapter 2 offers a comprehensive literature review on various aspects of XR and its applications in manufacturing settings. This includes an in-depth analysis of UI and UX design guidelines specific to XR applications, and the usability considerations for these immersive technologies. The chapter also explores different types of immersive platforms such as VR and MR for workforce training in industrial environments and discusses methods for evaluating their effectiveness. Physics-based 6 process simulations within XR and their practical applications are also considered. Additionally, Chapter 2 reviews the literature on ML approaches for optimizing manufacturing processes. It provides a detailed examination of the application of RL techniques, including both single-agent and multi-agent scenarios. The chapter explores multi-agent reinforcement learning (MARL) by differentiating between centralized and decentralized training methods, offering a thorough evaluation of their effectiveness and performance in various conditions. Chapter 3 (fulfilling objectives 1 and 2) focuses on developing immersive applications using both MR and VR platforms to simulate the steps that operators follow in thermoforming industrial processes. Given the critical role of the heating phase in determining the uniformity of the final product's thickness, the XR applications are designed to incorporate heat transfer simulations. This feature allows operators to monitor and adjust the temperature in real-time, optimizing the heating levels. Furthermore, the chapter evaluates the immersive training methods by examining various UX design aspects through a comprehensive, custom-designed user study. The comparison emphasizes the advantages of MR over VR and traditional video training methods, positioning it as a more advanced and efficient training approach. Chapter 4 (fulfilling objective 3) approaches the thermoforming process from an optimization perspective. Using simulation data, two selected RL algorithms are trained to control the temperature of a specific zone on the thermoplastic sheet during the heating process, under various state conditions. These conditions include large and small discrete action spaces, fixed target temperatures (both high and low), and the introduction of random noise in the target temperature to assess the robustness of each RL agent. Based on this comparison, the RL agent that demonstrates satisfactory results is chosen for further optimization, with the goal of controlling the temperature across a larger area of the sheet. 7 Chapter 5 (fulfilling objective 4) builds on the concepts introduced in Chapter 4, extending the optimization of temperature distribution to the entire thermoplastic sheet. Given the limitations of a single RL agent and the inherent complexities of the process, which require hardware with substantial computational capabilities even with minimized action spaces, a multi-RL agent approach is necessary to overcome these challenges. This shift opens up new possibilities by transitioning from a single-agent model to a centralized multi-agent model, enabling collaboration and experience sharing between agents during training. This approach significantly reduces training time and enhances efficiency. Furthermore, by transferring parameters from the previously trained single agent, the expanded model can quickly adapt to new conditions and uncertainties that may arise during the training process. Finally, Chapter 6 provides a comprehensive summary of the results and findings of this dissertation and outlines a few promising directions for future research. 8 Figure 1.1: Organization of the thesis. Starting with introduction and literature review, and moving on to operator training with immersive platforms, and their comparison in terms of effectiveness. Next, thermoforming process optimization using ML methods. Lastly fine-tuning the optimization using a combination of TL and RL while sharing experience, to optimize the process and reduce energy consumption. 9 Chapter 2: Literature Review 2.1 Industry 5.0 The emerging Industry 5.0 (I5.0) paradigm emphasizes on the importance of collaboration between machines and human operators in smart factories, thereby preventing full automation and elimination of the skilled workforce [12]. One technology that has been proposed recently as a means of achieving this collaboration, is the XR, particularly for operators training and upskilling [9], [13]. Other technological themes under I5.0 are blockchain and IIoT, digital twin (DT) and human-cyber-physical fusion, and data/knowledge-aided intelligent decision-making support [12]. Given the focus of the present thesis, a detailed literature review of applications of XR and ML technologies under I5.0 in emerging smart manufacturing processes is presented next. 2.2 Immersive platforms in manufacturing and workforce training Over the past decade, the use of XR systems has increased in a range of applications, including training, education, and safety [9]. The XR platforms often include augmented reality (AR) as a perspective of the physical world with added computer-generated elements, VR as a dynamic, computer-generated component within an entirely virtual environment, and MR as a combination of the real-world and virtual components in an interactive representation. The manufacturing sector is not an exception to this overarching pattern. In particular, unprecedented interests are emerging for the applications of immersive technologies for complex machine operators/workforce training and skill advancement. Among different XR platforms, the use of VR in industrial trainings has been most popular to date [13], [14]. However, it remains as a continuing scientific endeavour to link this platform to, e.g., numerical simulation tools. Prattico et al. [15] argued that using VR training simulators as self-learning tools is still up for debate, even though they have been advantageous as alternatives 10 to the traditional learning materials used by trainers. They performed a user study to determine the effectiveness of VR assistance systems in training KUKA industrial robot operators for routine maintenance duties. The study's generalizability was constrained by a modest sample size. It highlighted the need for further research to enhance skill transfer efficacy in Virtual Reality Training Systems (VRTSs), particularly for interactions with small and delicate objects. Leder et al. [16] examined VR and PowerPoint as means for providing safety training, using variables such as risk perception, learning, and dangerous decisions. Their research discovered an influence of training strategy on risk perception in terms of probability judgements and hazardous choices, but not on learning. The study was conducted among a sample of high-school students, where the participants were restricted to the role of observers, even in the immersive VR mode. Ragan et al. [17] evaluated an XR system that was created to teach individuals a visual scanning technique for identifying targets (armed individuals) in a virtual environment of an urban setting. In that complex scenario, those trained in realistic environments performed the technique more effectively. The study, however, did not quantify the impact of the VR training system fidelity on the training effectiveness itself. In the study conducted by Sportillo et al. [18], it was observed that using Head-Mounted Displays (HMDs) improved performance of a virtual assembly task. Although the research showed promising preliminary results, no user study was conducted to objectively evaluate the efficacy of training system. López et al. [19] performed multiple case studies to determine the efficiency of VR systems in various training contexts. The study employed a relatively small sample size. Additionally, the complexity of the training scenarios resulted in a lower-than-expected retention performance two weeks after training. In another investigation on assembly tasks conducted in an artificially created environment, Vélaz et al. [20] studied the usefulness of VR as a teaching aid. Their evaluation relied solely on training time as a sole measure 11 of XR efficiency assessment. Pan et al. [21] concentrated on a different aspect of VR, attempting to understand the impact of various forms of avatars on the learning and memory of the users/participants. However, the learning effect was inevitable through the sequence of experiments. Salah et al. [22] introduced a technique that utilized the predominant VR-based visualization method in product manufacturing and addressed the lack of innovation in training to keep pace with industry progress. The study was restricted to the task completion time as the primary evaluation criterion. Next to the VR-based user studies above, several AR applications have also been developed and tested for training, maintenance, and monitoring purposes. Lai et al. [23] integrated multi-modal AR instructions with a deep learning framework for tool detection, improving assembly quality and efficiency. The system provided on-site instructions using visual renderings generated by a fine-tuned neural network. This integrated system enhanced the operator’s performance by addressing assembly requirements. Although their research proposed a novel approach for integrating Deep Learning and AR, it did not provide detailed suggestions for generalizing the method in a commercialized setup. Moreover, Westerfield et al. [24] aimed at an AR with intelligent tutoring system (ITS), to enhance the training of manual assembly processes. Compared to conventional AR training, their strategy, which comprised a modular software architecture and prototype, increased task performance by 30% and test scores by 25%. It was concluded that an intelligent AR instructor considerably enhances learning. The study employed a small participant pool, necessitating repetitions with a larger student population in future work. Some participants had limited experience with motherboard assembly, potentially impacting the results. Several other studies have explored the application of AR in assembly processes. For instance, Chu et al. [25] examined AR user interface design for human-robot collaborative 12 assembly. Visual and haptic cues were tested to convey robot intent, which reduced visual focus on the robot, with proximity cues being the most effective. The results had restrictions in assessing the influence of human trust on robot performance, as their correlation was weak in the experiment. Additionally, users found that recognizing complex information through haptic system vibration on different parts of the human hand is non-intuitive. Chu et al. [26] in another study developed AR functions to assist manual assembly in visually occluded scenarios. They evaluated the effectiveness of various AR assistive information and found that displaying operator hand movement in AR can reduce assembly time. Limitations encompassed the AR hand model's precision and a focus on occluded components. Further validating the effectiveness of additional AR information, like contextual clues, may be necessary. Mura et al. [27] addressed the challenge of achieving precise alignment between car panels in automotive assembly. They proposed an AR prototype system to guide operators during panel fitting operations, specifically for correcting alignment errors in terms of gap and flushness. A real case study was presented, focusing on the fine alignment of car body panels with the front light projector, to demonstrate the effectiveness of the AR system. Their study highlighted the potential of AR as a highly promising tool for supporting personnel in production processes. However, the assembly model developed was not evaluated through a user study, which is a critical step for XR implementations. Webel et al. [28] developed a multimodal AR-based framework for teaching maintenance and assembly skills, including subskill training and system assessment. In their evaluation, they determined that although the training period using the AR model was longer than the standard approach, the technicians’ performance in a real-world maintenance scenario was quicker; after testing the system with two users and assessing their performance. The study employed a small sample size. Schwald et al. [29] proposed an AR system for training and assisting in industrial 13 equipment maintenance. The system comprised crucial hardware components such as an optical see-through HMD, a tracking system, and a stand for installation. The research explored using an infrared optical tracking technology and associated calibration methods for ideal virtual overlay outcomes, as well as a scenario-based approach for guiding users through training or maintenance activities. Limitations included the need to reduce equipment weight, improve tracking system accuracy and latency, and enhance information access and internal communication for improved usability. Werrlich et al. [30] investigated the usefulness of AR-enabled HMDs for training transfer tasks for assembly activities. They demonstrated that the AR approach outperformed conventional training approaches, such as video training and learning through printed text guidelines, in terms of immediate and long-term memory after training. Heinz et al. [31] created an AR system to aid users in comprehending complicated automated systems, despite their rising complexity and diminishing availability of experts. The study solely developed an AR platform and did not present a user study. Within the domain of industrial applications leveraging MR, a select number of scholarly endeavors have been undertaken. Hoover et al. [32] utilized the Microsoft HoloLens HMD to instruct employees on how to assemble a wing. The study indicated that apart from assembly duties, MR may also be a valuable aid in maintenance chores. Some users noted issues with the HoloLens device's comfort, persistent graphics interference, and tracking errors as the limitations of the study. Muñoz et al. [33] introduced a novel MR-based user interface for quality control inspection, with the aim of improving ergonomics, reducing work-related stress, and enhancing productivity. They presented an experimental prototype and compared it with existing factory interfaces, like those at Mercedes-Benz, through usability tests. The study did not utilize a standard questionnaire like NASA Task Load to assess the effect of workload on each user. Baroroh et al. 14 [34] proposed the use of MR simulation for the design of production systems with manual operations. They presented a MR-based planning tool that enabled collaboration between a manager and an operator wearing HoloLens. The planning results demonstrated the superiority of the MR tool over traditional simulation software in terms of planning quality and flexibility. The work validated the feasibility of MR as an effective approach for designing human-centric production systems. Murauer et al. [35] created language-independent attribute selection instructions using MR. The inconvenience induced by the HoloLens was highlighted as a system shortcoming. In terms of training considerations, Sirakaya et al. [36] used AR to boost efficacy and enthusiasm in vocational education and training. From a strategic point of view, according to the critical study by Kaplan et al. [37] on multiple XR platforms, one of the principal advantages of immersive tools in training is that they can be incorporated in environments/applications that are extremely difficult, time-consuming, or dangerous for traditional methods of teaching. For instance, Farra et al. [38] devised a new reality simulation for disaster training to bolster the preceding point. XR has also proven advantageous in other application areas such as safety, education and military training, rehabilitation, and medical training. To exemplify a few, Pedram et al. [39] employed XR to assist mine rescuers in safety. Gonzalez et al. [40] outlined the procedures involved in developing an immersive simulation to hone naval abilities. 2.2.1 Physics-based process simulations within XR Hernández et al. [41] integrated classical simulations with real-time physics engines using deep learning, resulting in a real-time AR simulator for interacting with virtual deformable solids. They reduced online computation time for XR HMDs, but faced challenges in scaling for larger meshes and required extensive offline training for their deep learning model. In a different study, 15 Badías et al. [42] used a model order reduction to create a real-time AR interaction method, blending machine learning, computer vision, and computer graphics. They addressed occlusion with pixel-depth analysis and GLSL shaders; yet limitations in the depth camera hindered distant object detection. Weistroffer et al. [43] developed the SEEROB framework for assessing safety and ergonomics in robotic workstations. It allowed direct communication with robot controllers, employed a precise physics engine for evaluations and workstation adjustments, used XR for user comprehension, and included motion capture for ergonomics assessments within the DT. Limitations included the need for hard-to-obtain DT parameters, custom adaptations for real-time robot data, and a lack of physical workstation feedback control. Jalilvand et al. [44] conducted a case study on MR effectiveness for training thermoforming process operators, with a particular focus on UI, UX, and usability. They introduced a spatial UI for interacting with virtual objects and incorporated enhanced UX via real-time heat transfer simulations. The usability of their approach was validated only using Microsoft HoloLens 2. However, the study was limited to a small group of eight users. Li et al. [45] introduced an integrated AR framework that combined real-time multi-material simulation with efficient hand gesture interaction. They used an RGBD camera for 3D data acquisition, enabling real-time gestures in simulations and expanding AR applications. However, the user study was restricted to the PC use due to its reliance on the RGBD camera and Leap Motion for gesture recognition and scene acquisition. Koduri et al. [46] introduced AUREATE, an Augmented Reality Test Environment that enhanced automated driving testing by augmenting real-world sensor data with simulated data. It was effective for object detection algorithm validation. However, AUREATE faced limitations due to hardware constraints and resource-intensive LiDAR augmentation. Achieving sensor data realism for perception 16 algorithm training presented challenges that required complex dynamics models beyond basic physics. 2.2.2 UI, UX and usability in XR applications MR applications feature distinct UIs compared to standard digital training systems, since they use wearable technology to superimpose virtual content on the physical surroundings. MR may not only alter a user's perspective of the actual world, but it also enriches the UX in the actual world instead of replacing it [47]. A significant benefit of adopting MR headsets is that they enable users to move freely inside the MR world [48]. Moreover, vision-based wearable devices allow users to engage with the MR world utilizing hand gesture controls [49]. Since their debut, Microsoft HoloLens and HoloLens 2 have been regularly evaluated by developers and academics from many perspectives [47], [48]. While HoloLens 2 offers several advantageous features that distinguish it from other MR HMDs, it also has some technical limitations. For instance, the restricted field of view (FOV) of HoloLens has a negative influence on the system's usability and UX. Thus, building a personalized UI for HoloLens devices’ applications is a potential way to improve UX [50]. When compared to its predecessor (Microsoft HoloLens 1), Pose-Dez-de-la-Lastra et al. [51] studied the use of Microsoft HoloLens 2 in orthopedic procedures and discovered that it had a nearly 25% increase in AR projection accuracy. Nguyen et al. [52] Developed a MR system for nondestructive evaluation (NDE) training that lets students interact with virtual objects and execute NDE procedures without causing them injury. Their user study favored the MR-based NDE training method over conventional training methods. Under the framework of Industry 4.0, Pusch et al. [53] considered the use of MR for operator training. The authors assess the efficacy of three prototypes, including a haptic tablet, a large 17 screen, and a Microsoft HoloLens-based MR version. Feedback indicated that, despite ergonomic limitations, the MR version has the potential to replace existing training techniques and even enable novices to complete the training independently. However, further tests were recommended to generalize these findings. Wu et al. [54] proposed a visual warning system that combines digital twins, deep learning technology, and MR using HoloLens to enhance the safety of construction workers. Testing revealed that the system operates effectively, but more improvement is necessary to solve lens distortion, image-based prediction function, and manual alignment. The applicability of a MR environment for instructing construction engineering students about sensing technologies was also investigated by Ogunseiju et al. [55]. Using eye tracking, usability questionnaires, and verbal feedback, their research assessed the MR environment and discovered that the accuracy of characterized construction activities, the quality of animations, and the ease of access to information and resources are crucial for building effective MR learning environments. Unfortunately, there is still a scarcity of literature on customized UI design for mixed-reality apps employing headsets. 2.3 Artificial intelligence and machine learning in advanced manufacturing Integrating AI into real-world problems has become common practice across various application fields, from manufacturing to healthcare and education, to name a few. The manufacturing sector, in particular, has begun benefiting significantly from the AI, namely to develop more automated, intelligent, and optimized processes that can save production costs, energy, and time [2], [3]. In the latter sector, a range of AI techniques are employed today, including decision trees, neural networks, support vector machine (SVM), and RL, among others. Conventional supervised/unsupervised ML approaches are used e.g., for quality analysis, early defect detection, 18 and production scheduling, where they leverage extensive amount of historical data and established rules. However, RL is increasingly favored due to its adaptability in model-free environments and its ability to handle delayed consequences through agent-environment interactions. RL is particularly known to be effective for sequential decision-making tasks, such as job dispatching, maintenance scheduling, and resource allocation, proving to be more responsive and efficient in dynamic manufacturing settings compared to conventional ML methods [4]. Particularly, when using deep neural network to shape DRL, its adoption in manufacturing continues to grow, offering sizable improvements in productivity, quality, and system performance [4]. In general, DRL applications in industry may be categorized into four product lifecycle phases: Design, Manufacturing, Logistics, and Maintenance. Specifically, among all, DRL is mainly utilized for the manufacturing stage (approximately 80%), addressing tasks such as order selection, process planning, scheduling, process control, optimization, quality control, and robotics [5]. 2.3.1 Application of DRL in manufacturing process optimization Optimizing manufacturing process plans according to order demand is crucial for enhancing the quality, production efficiency, and cost-effectiveness of parts [56]. As such, Ghorbel et al. [57] and Wu et al. [58] utilized the DRL-based approaches to refine manufacturing and assembly processes under dynamic resource constraints, while He et al. [59] combined DQN with MARL frameworks to optimize multiple objectives, including quality, productivity, and cost. DRL has also been applied for sustainable manufacturing and equipment-level control systems. Sustainable manufacturing aims e.g. at energy saving and emissions reduction, with actor-critic methods proposed by Kohne et al. [60], Huang et al. [61], Schwung et al. [62], and Zhu et al. [63] 19 for optimal energy reduction scheduling. MARL-based energy management strategies were developed by Bakakeu et al. [64], Roesch et al. [65], and Lu et al. [66] to ensure energy efficiency and load management. Arroyo et al. [67] introduced Reinforced Model Predictive Control (RL-MPC), enhancing the adaptability of Model Predictive Control (MPC) for dynamic energy control problems. Brandi et al. [68] compared online and offline DRL with MPC for energy management, emphasizing the efficiency of online-trained DRL agents. Gupta et al. [69] applied DRL to heating control in smart buildings, improving thermal comfort and reducing energy costs. Wang et al. [70] used DRL for forced convection control, achieving a lower temperature with a novel value-based DQN. Hachem et al. [71] utilized PPO in a Computational Fluid Dynamic (CFD) environment for conjugate heat transfer systems. Römer et al. [72] employed a DRL method called Deep Deterministic Policy Gradient (DDPG) for temperature control in Automated Tape Laying (ATL) processes. Zhao et al. [73] introduced a memory-augmented (MA) DRL algorithm for energy management in commercial buildings with dueling networks to mitigate time delays. Szarski et al. [74] optimized Carbon Fibre Reinforced Plastic (CFRP) manufacturing cycle time using DRL, reducing cycle time in aerospace parts. Xiao et al. [75] used meta-RL to adaptively determine optimal control parameters in machining, while Huang et al. [76] combined Graph Neural Networks (GNN) and MARL for collaborative control of machining process parameters. Dornheim et al. [77] applied DRL for optimal structure formation, reducing reliance on expert knowledge. Li et al. [78] proposed a DRL-based trajectory smoothing method, and Zhang et al. [79] and Samsonov et al. [80] optimized tool orientation planning and clamping positions. Schoop et al. [81] used PPO to maximize cutting tool life, and Gulde et al. [82] applied DRL for vibration compensation, achieving higher machining 20 precision and longer component lifetime. Jiang et al. [83] modeled internal CNC data with an Long Short-Term Memory (LSTM) network, proposing a control strategy to reduce contour prediction error and enhance machining compensation. 2.3.1.1 DRL in thermoplastic composites manufacturing In advanced manufacturing sector, such as aerospace and automotive application, composites are highly valued for their customizable mechanical properties and low structural weights. Thermoforming is a frequently-used processing technique in composites manufacturing due to its ability to efficiently shape thermoplastic materials into intricate 3D forms. This method is also beneficial for rapid prototyping, as it allows for swift design iterations [84]. Among other processing techniques, a novel optimization method for Carbon Fiber Reinforced Polymer (CFRP) was developed for a curing process by Szarski et al. [74], using DRL to create an adaptive temperature controller that efficiently manages the heating and cooling to reduce cure cycle time and prevent thermal runaway. A Bayesian Optimization loop refined tooling geometry, achieving a 27-40% reduction in ramp-to-cure time for aerospace parts. Zimmerling et al. [85] introduced a RL approach to optimize manufacturing parameters for varying component geometries in composite fabric forming. The RL method trained a neural network to estimate optimal process parameters based on component geometry. It provided accurate parameter estimations for both trained and new geometries, demonstrating its ability to generalize and reuse process knowledge. Lastly, the study by Stieber et al. [86] tackled the challenge of an even liquid polymer distribution in Resin Transfer Molding (RTM) by using DRL for online control. The RL system adjusted input pressure at different inlets, reducing failure rates from 54% to 27%. Unlike MPC, their method used a flexible reward function and demonstrated a superior performance over static parameterization. 21 Of note, most of the above ML-based optimizations operate under the assumption that the model and process remain static, disregarding adaptability to variations in processes conditions, such as change in the input material properties, etc. [87]. Real-world manufacturing environments, however, specifically composites manufacturing, are very sensitive to the process changes and frequently deviate from these assumptions, leading to decreased performance and applicability of ML models [88]. Recent advancements in integrating RL with established process control methods have shown promising results for addressing challenges in thermoplastic composite manufacturing. Among these, the hybrid approach of combining Model Predictive Control (MPC) with RL has gained attention for its ability to balance predictive accuracy with adaptability in dynamic and uncertain environments [89]. MPC is widely recognized for its ability to handle system constraints effectively while optimizing control actions based on future state predictions. However, its computational demands, particularly for high-dimensional systems, often limit its real-time applicability in industrial scenarios [89]. In contrast, a standard RL offers a model-free approach that adapts to complex, non-linear dynamics but typically requires significant training time and data exploration. A notable advancement is the MPC-guided RL framework, which leverages the predictive capabilities of MPC to accelerate RL training. By reducing the exploration burden, this approach enables faster convergence and achieves robust performance with lower computational requirements compared to standalone MPC systems. Such hybrid strategies have demonstrated effectiveness in thermoforming processes, particularly in maintaining energy efficiency and product quality under dynamic process conditions. Hosseinionari et al. [89] provided a strong foundation for the application of RL in thermoforming, showcasing the potential of hybrid approaches to address the limitations of traditional control methods. However, the latter approach can be extended for the 22 thermoforming RL models to be more scalable and adaptive to new manufacturing conditions in real-world, using e.g. a combination of Multi-Agent RL (MARL) and Transfer Learning (TL) as proposed here. These hybrid methods offer opportunities to further enhance the acceptability and reliability of RL by industries in handling varying complex and dynamic manufacturing environments. 2.3.2 Application of TL in manufacturing Farahani et al. [90] conducted a detailed review of time-series pattern recognition in Smart Manufacturing Systems within the context of Industry 4.0. The authors focused on the application of TL to address the diverse manufacturing landscape, enabling the effective use of AI and ML. The study categorized and defined various concepts using an ontology, identifying prevalent techniques and algorithms such as Convolutional Neural Networks (CNN) and Random Forest (RF). Emphasis was placed on the importance of preprocessing in TL tasks and the need for objective algorithm classification to mitigate subjective biases. Potential research areas like Acoustic Emission (AE) were also highlighted. Liu et al. [91] developed an automated broad transfer learning algorithm (AutoBTL) to enhance cross-domain fault diagnosis in smart manufacturing systems. The study addressed the limitations of traditional deep transfer learning (DTL) algorithms, which are often domain-specific and require extensive hyperparameter optimization (HPO). AutoBTL integrates a broad classifier, an active estimator, and an HPO module to automate the modeling process, improving predictive accuracy and efficiency. The algorithm was validated across three benchmark datasets, demonstrating superior performance in cross-domain tasks compared to existing predictive modeling techniques. The study emphasized the significance of automated HPO and the potential of broad learning frameworks to adapt to diverse industrial data environments. Ma et al. [92] introduced an adaptive deep transfer learning 23 method for thermal error modeling of electric spindles under variable conditions. The study addresses the limitations of traditional deep learning techniques, which often struggle with data distribution discrepancies across different working conditions. The proposed method integrates multi-sensory information, including temperature, current, and power data, through a convolutional long short-term memory network (C-LSTMN) and employs a multi-kernel joint maximum mean discrepancy (MK-JMMD) measure for domain adaptation. The method was validated through multiple transfer tasks, demonstrating superior prediction accuracy and robustness compared to existing models, particularly in scenarios where labeled thermal error samples are unavailable. Xu Li et al. [93] proposed a federated transfer learning method for fault diagnosis, emphasizing data privacy and target self-adaptation. The study tackled the challenge of domain shift and data privacy in industrial settings by using federated learning to allow multiple clients to collaboratively train fault diagnosis models without sharing raw data. The method involved generating fake data classes to enhance decision boundary suppression and leveraging a prediction alignment scheme for the target client to adapt independently. The proposed approach demonstrated superior performance in handling partial transfer learning problems, addressing discrepancies in label sets between source and target domains. Validation on real-world machinery datasets confirmed the effectiveness and practicality of the method for industrial applications. Zhang et al. [94] proposed an intelligent tool wear monitoring method that employs multi-channel hybrid information and deep transfer learning to improve prediction accuracy under variable working conditions. The study utilized multidimensional sensor data and process information, integrating them through a deep residual network (ResNet). Transfer learning was applied to fine-tune the model with a small number of samples when conditions changed, such as different tools or workpieces. Validation using local experimental and NASA datasets showed 24 significant improvements in prediction accuracy, with multi-channel hybrid information outperforming single-channel sensor data. Zhang et al. [95] provided a comprehensive review of the top ten intelligent algorithms in smart manufacturing, focusing on their application, performance, and challenges. The study emphasized TL as a critical technique for enhancing the adaptability and efficiency of these algorithms in diverse manufacturing environments. The authors highlighted the importance of TL in mitigating issues related to data scarcity and domain discrepancies, enabling the effective reuse of pre-trained models across different manufacturing tasks. The review also categorized the algorithms into shallow learning, deep learning, optimization, and logic-based reasoning, discussing their respective strengths and suitable applications in smart manufacturing. Challenges such as data quality, algorithm interpretability, and the need for real-time processing were identified, suggesting future research directions to improve the integration and performance of intelligent algorithms in manufacturing systems. 2.4 Integration of TL and RL in industrial applications There have been limited studies on integrating TL and RL in industrial applications. Raziei et al. [96] introduced the Hyper-Actor Soft Actor–Critic (HASAC) framework, combining TL with DRL to enhance robot adaptability in automation. HASAC utilizes task modularization and policy transfer, allowing RL agents to reuse knowledge from previous tasks. Tested on robotic manipulation tasks, it showed improved success rates over other RL approaches. Zheng et al. [97] combined RL with TL for dispatching in dynamic manufacturing, using a 2D matrix for shop floor state representation and policy transfer to generalize across production lines, leading to better performance in simulations. Alam et al. [98] aimed to improve sample efficiency in autonomous manufacturing systems (AMS) by integrating TL with RL. Their two-step approach, involving a Gaussian process for transferring knowledge, minimized the number of iterations required, 25 demonstrated in crystal manufacturing. Lastly, Klar et al. [99] used RL and TL for multi-objective factory layout planning, incorporating a discrete event simulation (DES) to simulate dynamic objectives. This hybrid approach showed improved efficiency and scalability over traditional methods in layout planning tasks. 2.5 Summary: research gaps In the domain of VR, AR, and MR training systems for industrial applications, most of the reviewed studies in this chapter exhibited a few common limitations. Notably, most of the user studies suffered from small sample sizes, diminishing the extent to which the findings may be generalized. Additionally, certain studies limited participants selection to specific user groups, thereby potentially restricting the broader applicability of the training system for others. Furthermore, a prevalent limitation was deemed to be the adoption of a unidimensional (single criteria) evaluation approach, primarily centered on task completion time. Notably, user studies were omitted at times, consequently impeding the ability to evaluate practical usability of the developed systems. Moreover, recurrently unaddressed are technical concerns linked to system setup, equipment weight, tracking precision, and user comfort, which can potentially impact both the user experience and the interpretability of outcomes. Finally, to the best of our knowledge, no study directly compared different XR training platforms within the context of workforce training in a same manufacturing operation. Typically, the studies have focused on singular XR platforms for adaptation and evaluation. Among immersive platforms, the utilization of AR applications for industrial training purposes may be deemed less interactive, given that AR applications primarily serve to augment virtual objects within the user's field of vision, with limited user interaction with the physical world. In contrast, 26 MR systems emerge as a more viable platform facilitating practical interactions between users and augmented virtual objects. In addition to the immersive platforms, it is deemed essential to consider and compare a traditional base training approach (e.g., 2D videos) as commonly employed in manufacturing settings. Finally, most of the established XR applications for training purposes, have generally followed sequential steps without involving physics-based process simulations (e.g., finite element models), which are often the backbone of design and decision-making in advanced manufacturing engineering. In view of the above gaps, one of the main tasks of this thesis (see also the objectives section 1.2) is to develop, implement, and compare different immersive training tools in a same complex manufacturing process (Chapter 3). Regarding the ML applications, also based on the literature review in this chapter, with exception of few studies [96] - [99] that primarily focus on the benefits of TL in single-agent RL models for manufacturing processes in general, to the best of author’s knowledge there has been no related research exploring the potential of TL in MARL settings to optimize thermoforming process. The present research seeks to address this gap; specifically, Chapters 4 and 5 will make contributions to advancing the optimization of thermoforming processes through the application of RL and TL. Chapter 4 focuses on the development and evaluation of single-agent RL algorithms, namely Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN), to control heating parameters across various target temperature profiles. This approach addresses the limitations of traditional process control methods, which often rely on explicit models and require extensive manual tuning, by demonstrating the adaptability of RL in managing diverse manufacturing conditions. Building upon these findings, Chapter 5 extends the investigation to 27 dynamic and adaptive process conditions, introducing a MARL framework enhanced with TL (called MATRL). This chapter evaluates the robustness of MATRL under scenarios involving variable environmental factors, highlighting its novelty in optimizing energy efficiency and product quality. Together, Chapters 4 and 5 address the critical gap in applying RL and TL to thermoforming optimization, establishing a foundation for dynamic, adaptive, and data-driven process control strategies in advanced manufacturing. 28 Chapter 3: Developing Extended Reality Platforms for Thermoforming Training: A Multicriteria Decision Making User Study 3.1 Introduction Thermoplastic materials and their composites have garnered significant attention in aerospace, marine, renewable energy, and automotive industries due to their superior thermo-mechanical properties and lightweight nature. However, several challenges, such as high manufacturing costs, time-consuming design tools, and low product quality repeatability, continue to impede mass production of advanced thermoplastic structural parts in some sectors. To address these challenges, the advent of the Fourth and, more recently, the Fifth Industrial Revolution (I4.0 and I5.0) has spurred digital transformations in manufacturing processes aimed at enhancing efficiency, reducing costs, and accelerating production speeds [1]. Enabling technologies, including the IoT, Simulation, ML, and XR technologies, are being explored to develop Digital Twins (DTs) that facilitate efficient operator training, process monitoring, and control in collaboration with automated and semi-automated machines, a hallmark of I5.0 that emphasizes human empowerment on the shop floor. This chapter primarily focuses on the XR component of I5.0 for workforce training in thermoforming. Earlier during this PhD, an initial (pre)case study (proof of concept), detailed in Appendix A (fulfilling Objective 1, per section 1.2), assessed the feasibility of developing an MR application for training operators in the thermoforming industry, using Microsoft HoloLens 2. This preliminary assessment emphasized on UI design, UX, and usability, demonstrated through spatial UI and real-time heat transfer simulations. Building on these findings, this chapter expands the scope to a comprehensive analysis and further development of XR applications, specifically comparing MR and VR environments for operator training in thermoforming processes. The study 29 in this chapter aims to integrate both MR (via Microsoft HoloLens 2) as well as VR (via Meta Quest 2) platforms, along with traditional video-based training methods, to evaluate their effectiveness in enhancing training efficiency, user satisfaction, and overall performance (fulfilling Objective 2). 3.2 Methodology As the case study, the thermoforming manufacturing process involving multiple human operations is considered, where heat transfer phenomenon has a significant effect in the quality of the final formed product. As illustrated in Figure 3.1, the first stage of the proposed methodology is creating a DT of the process, where the heat transfer between different components of the manufacturing system can be accounted for. Following the modelling of the heat transfer simulation tool (here in C#), and the development of its visual components using 3D model software, object behaviours are defined using Unity and C# and are assigned to the components inside the XR (here VR and MR) applications. In the next steps, the user input methods and profiles for each VR/AR environment are established, based on the respective devices that are utilized (here Meta Quest 2 (VR headset) and Microsoft HoloLens 2 (MR headset)). Interactive elements of the programmes are adapted to the user's input system, such as joystick controls for Quest 2 and hand gestures for HoloLens 2. Finally, a user survey is conducted to gather data on how well the developed VR and MR applications would fit specific performance criteria. The obtained user study data is analyzed statistically and used in a MCDM model to rank and determine which platform is most suitable for training the operators. 30 Figure 3.1: Overview of the methods employed in the present study. 3.2.1 Thermoforming set-up in the case study In a typical thermoforming process, thermoplastic sheet is heated up and placed into a mould by vacuum pressure. The aim is to heat the sheet to reach the desired temperature distribution before it forms. During this heating stage, maintaining the temperature at a certain level in each region of the sheet is crucial (to ensure the subsequent mechanical forming effectiveness and yield a defect-free final product with a uniform thickness). Figure 3.2 illustrates a typical procedure followed during the thermoforming process and the resulting sample regions with a typical thickness variation deficiency [100]. Unfortunately, the current industrial thermoforming methods have not all achieved the required efficiency to avoid such manufacturing defects through automation of the process. As a result, some of the final products are being discarded (the production failure rate has been seen to be high as 30% if the operator does not carefully optimize the process). 31 Figure 3.2: A typical thermoforming process and example of potential defects in the final part [38]. 3.2.2 Transient (time dependent) heat transfer modeling of the process The numerical model consisted of fifteen heating elements above the thermoplastic sheet. It incorporates all the three conductive, radiative, and convective heat transfer modes, as illustrated in Figure 3.3, and Figure 3.4 indicates the thermoplastic sheet's constituent parts. Given the small sheet thickness in relation to its length and width, only one layer of elements is considered. The four sides (±X, ±Y) of each element are considered for heat conduction between elements. Radiation heat transmission from the heating coils occurs through the sheet's top face (+Z). Every surface of the element in contact with its neighbors and the environment is subject to heat conduction. qz+ and qz- represent heat transmission in the vertical direction, while (qx+,qx-) and (qy+,qy-) represent heat transfer in the longitudinal and transverse directions, respectively. Equation (3.1) represents the heat transfer rate by conduction in a given direction [101]. 32 Figure 3.3: Heat transfer between thermoplastic sheets, heaters, and the environment. Figure 3.4: Finite difference modelling: (a) conduction heat transfer, and (b) including all modes of heat transfer. 𝑄𝑖𝑐𝑜𝑛𝑑. = −𝑘𝐴∆𝑇𝐿 (3.1) Where 𝑘 [𝑊𝑚.𝐾] is the material's conductivity, A [𝑚2] is the cross-sectional surface area, ∆𝑇 [𝐾] is the temperature difference between the ends, and L [𝑚] is the distance between the ends. To calculate the conduction heat transfer, only ±𝑥 , ±𝑦 directions are considered. Therefore, the corresponding in-plane conduction heat transfer equations are depicted as Equations (3.2), (3.3): 𝑄𝑥𝑐𝑜𝑛𝑑. = 𝑘∆𝑦∆𝑧𝑇𝑚+1,𝑛 − 2𝑇𝑚,𝑛 + 𝑇𝑚−1,𝑛∆𝑥 , 𝑖𝑛 ± 𝑥 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 (3.2) 33 𝑄𝑦𝑐𝑜𝑛𝑑. = 𝑘∆𝑥∆𝑧𝑇𝑚,𝑛+1 − 2𝑇𝑚,𝑛 + 𝑇𝑚,𝑛−1∆𝑦 , 𝑖𝑛 ± 𝑦 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 (3.3) At the air-contacting surfaces of the element, convective heat transfer occurs between the element and its surroundings. This includes the ±𝑧 sides of the thermoplastic sheet. Equation (3.4) demonstrates the rate of heat convection from the 𝑖𝑡ℎ node to the surrounding environment [101]. 𝑄𝑖𝑐𝑜𝑛𝑣. = ℎ𝐴𝑖(𝑇𝑖 − 𝑇𝑎𝑚𝑏𝑖𝑒𝑛𝑡) (3.4) Where ℎ [𝑊𝑚2.𝐾] is the convective heat transfer coefficient, 𝐴𝑖 [𝑚2] is the surface area of 𝑖𝑡ℎ node where the heat transfer takes place, 𝑇𝑖 [𝐾] is the temperature of the 𝑖𝑡ℎ node of sheet and 𝑇𝑎𝑚𝑏𝑖𝑒𝑛𝑡 [𝐾] is the temperature of the ambient. When a cold environment is formed around a hot, horizontal sheet, free convection occurs. However, the coefficient of heat transfer will be variable between the plate's top and bottom surfaces as ℎ1 and ℎ2. Equations (3.5), (3.6) explain the free convection equation for elements positioned on various sides of the thermoplastic sheet. 𝑄𝑧+𝑐𝑜𝑛𝑣. = ℎ1∆𝑥∆𝑦(𝑇𝑚,𝑛 − 𝑇𝑎𝑚𝑏𝑖𝑒𝑛𝑡) , 𝑖𝑛 + 𝑧 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 (3.5) 𝑄𝑧−𝑐𝑜𝑛𝑣. = ℎ2∆𝑥∆𝑦(𝑇𝑚,𝑛 − 𝑇𝑎𝑚𝑏𝑖𝑒𝑛𝑡) , 𝑖𝑛 − 𝑧 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 (3.6) Finally, as the radiation heaters are situated above the sheet, the input energy to elements can also enter via face +Z through radiation mode. Similarly, the heat may be lost from thermoplastic sheet by radiation from both ±𝑍 sides to the environment. Equation (3.7) expresses the net rate of radiant heat transmission from the heaters to the 𝑖𝑡ℎ element on the plastic sheet [102]. 𝑄𝑖𝑟𝑎𝑑. = 𝐴𝑗𝜀𝑒𝜎 [∑𝐹𝑖,𝑗(𝑇𝑗4 − 𝑇𝑖4)𝑁𝑗=1] (3.7) 34 Where 𝐴𝑗 [𝑚2] is the surface area of emitter, 𝜀𝑒 is the effective emissivity, 𝜎 is the Stefan-Boltzmann constant: 5.67 × 10−8 [𝑊𝑚2𝐾4], 𝐹𝑖,𝑗 is the view factor between the 𝑗𝑡ℎ heater and the 𝑖𝑡ℎ node, 𝑇𝑗 [𝐾] is the absolute temperature of the 𝑗𝑡ℎ emitter, 𝑇𝑖 [𝐾] is the absolute temperature of the 𝑖𝑡ℎ node on the sheet and 𝑁 is the total number of the heaters. The net heat flow, which is equal to the change in internal energy, may be computed using the heat flow in all directions outward from the element, given in Equation (3.8). The temperature change may be calculated from the corresponding change in internal energy, where 𝑞𝑖 represents the heat transfer from all surfaces to each element on the thermoplastic sheet. By combining the temperature shift caused by the change in internal energy and the temperature from the preceding time step, the new temperature (𝑇𝑡) is obtained in Equation (3.9). Where 𝑡 is the time step, 𝜌 is the sheet density, 𝑉 is the element volume and 𝐶 is the sheet material’s heat capacity [103]. 𝑑𝑈 = ∑𝑞𝑖6𝑖=1= 𝑄𝑖𝑟𝑎𝑑 + 𝑄𝑖𝑐𝑜𝑛𝑣 + 𝑄𝑖𝑐𝑜𝑛𝑑 (3.8) 𝑇𝑡 = 𝑇𝑡−∆𝑡 + (∆𝑡𝜌𝑉𝐶)𝑑𝑈 (3.9) The view factor is the ratio of energy emitted by the first surface to that absorbed by the second surface. Mathematical calculations for view factors are known for the relative locations of the surfaces [104]. Based on these equations for view factor, models of the heating phase of the thermoforming process have been developed. In this instance, it is assumed that the heating source is a flat, temperature-maintaining plate. Equation (3.10) depicts the time-dependent temperature variations of the sheet's elements due to radiation heat transfer using a single heater. The view factor between the components of the sheet and the heater may be calculated at the outset of the heating process using Equation (3.11) 35 (given the thermoplastic's low thermal conductivity, which is due to its molecular structure and relatively weak intermolecular bonding). 𝑑𝑇𝑖𝑑𝑡= (1𝜌𝑉𝐶)𝐴ℎ𝜀𝑒𝜎𝐹ℎ,𝑖(𝑇ℎ4 − 𝑇𝑖4) (3.10) 𝐹ℎ,𝑖 =𝜌𝑉𝐶𝐴ℎ𝜀𝑒𝜎(𝑇ℎ4 − 𝑇𝑖4)×𝑑𝑇𝑖𝑑𝑡 (3.11) Following the above equations, the temperature variation can be obtained on each element of the thermoplastic sheet based on the applied power on each heater. Once combined for fifteen heaters, it is possible to simulate the whole process in real-time and measure the temperature distribution on the whole surface of the sheet. It is worth noting that the validation of this Finite Difference (FD) transient heat transfer simulation was performed through the established commercialized COMSOL software outlined in [105]. 3.2.3 XR development To create interactive DTs of the process (i.e. VR/AR tools integrated with the heat transfer simulations; Figure 3.1) for the operator training applications, 3D components of the system were designed in Unity® ProBuilder and SolidWorks®. Once all the components were designed, their STL files were converted into OBJ formats to be recognized as prefabs inside the Unity environment. These were used for object programming in C# language through Unity Engine. A demo video was recorded using this physical set-up and used as a baseline training method for the operators. The video training will be compared with the XR application tools developed in Section 3. The XR design and the actual lab-scale setup are demonstrated in Figure 3.5. 36 Figure 3.5: XR design (left) versus the actual setup (right). 3.2.3.1 VR application development The VR application incorporates two distinct scenes: the "Setup Outline" and the "Tutorial." The "Setup Outline" serves as a central repository, where all components of the DT within the XR design are consolidated, as depicted in Figure 3.5. Its primary function is to provide users with a comprehensive perspective of the entire process, allowing them to acquaint themselves with the setup components prior to undergoing formal training. Conversely, the "Tutorial" scene is exclusively dedicated to instructional purposes. It encompasses all training modules designed for amateur users who require practice with the DT before engaging with the actual setup. The "Tutorial" scene primarily functions as the training environment for users to acquire proficiency in the thermoforming process. Essentially, tutorial scene uses the Unity animation state machine in which each step is triggered once a specific set of actions (such as pressing a button or displacing an object) are completed, as demonstrated in Figure 3.6. This was done by incrementing the step numbers in the animator interface as soon as the condition for the completion of the previous tasks is met. These two scenes are interchangeable, and the application restarts as soon as switched. 37 Figure 3.6: The Unity state machine and animator employed. Meta Quest 2 was selected as the target device for developing the VR application. To define the input system, three buttons were assigned to the controllers to allow the user to interact with the DT. As illustrated in Figure 3.7, the “Axis1D.PrimaryIndexTrigger” buttons were defined as the “Tap/Select” option and “Axis1D.PrimaryHandTrigger” buttons were defined as the “Grab and rotate” option. Moreover, the “Start” buttons were selected as the “Menu” option. Figure 3.7: VR input system definition. 3.2.3.2 MR application development The MR application concept development was very similar to what was implemented in the VR application (Section 2.3.1). However, the target device, user input system definition, profiles, and some of the interactive component designs were different. Namely, the training DT in MR case was developed using Microsoft’s HoloLens 2, based on the input system provided by 38 the Mixed Reality Toolkit (MRTK), to allow the user to interact with the environment. Similarly, the UI consisted of two scenes, the Setup outline and the Tutorial. The Setup outline scene includes all the components of the DT without any assistance presented to the operator. In contrast, in the tutorial scene, the user should follow six steps based on text instructions and visual guides, which are provided at each step. To define the input system in the MR application, MRTK profiles were utilized to define various controls for the operator. Since HoloLens 2 does not come with separate controllers similar to Quest 2, hand gestures are specifically designed to define various controls. As illustrated in Figure 3.8, two modes of input systems were defined for close and remote interactions. In Figure 3.8a neutral position, hovering, and touching an object is demonstrated, where the palm is completely open, and for the remote selection, the ray cast was used, which is shown as a dashed line. Figure 3.8b exhibits grabbing, moving, and rotating the objects by pinching (closing the thumb and index fingers). The activated remote ray cast is shown as a solid line. Figure 3.8: The MR input system and hand gestures for near and remote interactions. a) Neutral position, Hover, and touch. b) Grab, move, and rotate. 3.2.4 User study design Overall, three modes of training scenarios were defined for the user study: 1. Virtual Reality Training (VRT) 2. Mixed Reality Training (MRT) 39 3. Demo Video Training (DVT) (as the base training method) The objective of the first two training methods in practice would be to offer an immersive training tool for the novice operator to learn the essential steps of the thermoforming process considerably more quickly than the conventional video training method. Additionally, the operator can be provided with some valuable insights and real-time visual information about the physical status of the process (here the temperature distribution on the heaters and sheet). The immersive training environment designed for each user was based on six sequential tasks: 1. Placing the thermoplastic sheet onto the fixture. 2. Turning on a specific set of heaters. 3. Adjusting each heater’s power using the button/slider. 4. Checking the surface temperature on various heaters; once all the heaters reach their desired surface temperature, turning off all the heaters. 5. Releasing the sheet frame and moving it down onto the mould. 6. Turning on the vacuum pump. All the above steps, except Step 4, incorporate human direct actions that can be broken down into moving, grabbing, hovering and pushing/pulling objects, which are all specified based on the colliding boundaries and physics-based relations defined for the target objects. Step 4, fueled by the numerical simulation model in real-time, requires a pre-condition to be satisfied, which is to check the sheet surface temperature on all the heaters by selecting them separately. Each VR and MR environment was designed based on their input systems to allow the operator to choose each heater as they touch/hover over them. The screen information on the control panel is updated instantly based on the governing heat transfer laws. A video demonstration of the developed application can be found at the Supplementary Material. 40 3.2.4.1 Hypotheses The overarching hypothesis considered in the user study was: MR would be the most suitable platform for training purposes, compared to VR and video training. To better outline the features of this hypothesis, it was divided into sub-hypotheses as follows: Sub-H1: Interacting with the designed DT in MR environment would be more effective for training purposes compared to learning via a VR environment and training video. Sub-H2: Operators would feel more convenient being trained by an MR environment compared to VR and the training video. Sub-H3: Among all the provided training options, operators are satisfied with their performance during and their training via MR environment. Sub-H4: MR would be the most trustable environment among the three platforms. Sub-H5: MR would be the most reliable source for the trainees to function properly compared to the VR environment and training video. 3.2.4.2 Experimental procedure and questionnaires Prior to the commencement of the tests, participants were assigned to workstations equipped with either the DVT device (laptop), MRT device (HoloLens 2), or VRT device (Quest 2), following a Latin square sequence. The use of the Latin square design aimed to balance the order in which the training tools were introduced across participants. While it did not eliminate potential experience bias entirely, it reduced the influence of the order of tool exposure, ensuring that no single device consistently benefited or suffered from a systematic learning or fatigue effect. Participants signed a consent form, answered demographic questions, and provided details about their prior VR/AR experience before starting the experiment. Participants were organized into three distinct groups, each assigned to a different sequence of training methods, as illustrated in 41 Figure 3.9. Each group initiated their experimental procedure with a different tool, following the predetermined Latin square sequence. Specifically, Group One's sequence was: 1. DVT, 2. VRT, and 3. MRT. Group Two followed: 1. VRT, 2. MRT, and 3. DVT. Finally, Group Three used the sequence: 1. MRT, 2. DVT, and 3. VRT. In the VRT phase, the Meta Quest 2 was utilized as the VR device, providing participants with a fully immersive virtual environment. They were instructed to follow the six steps (tasks) outlined in Section 3.2.4. For safety, the application marked a designated working area, allowing participants to focus on their training without concerns about physical collisions. In the DVT phase, participants were introduced to the physical lab-scale test setup and provided a concise demonstration video showcasing the experimental tasks. This video guided participants through the thermoforming process, similar to the structure of the other two training modes. During the MRT phase, participants donned the HoloLens 2 headset to engage in the training experiment. They followed visual cues and audio instructions at each stage. The MR setup allowed participants to maintain awareness of their physical surroundings while interacting with virtual digital twin components as they progressed through each task. The user study employed a “within-subject” design, allowing each participant to experience all three training methods. This design facilitated a comprehensive evaluation of individual responses across the different conditions. After completing all three trials, participants assessed their experiences and provided qualitative feedback on the effectiveness of the immersive training methods compared to conventional video training, as well as on the overall design of the immersive applications. To minimize fatigue and ensure consistency, each user study session was kept concise, with the entire process taking approximately 45 minutes per participant. 42 Figure 3.9: Flowchart of the experimental procedure in the user study. 3.2.4.3 Questionnaire design As mentioned in section 3.1, an initial case study had previously been conducted (see Appendix A) on thermoforming operator training exclusively using the MR system, while examining UI, UX, and usability of Microsoft HoloLens 2 [44]. The latter study involved eight participants and utilized the System Usability Scale (SUS) questionnaire [106], resulting in an approximate usability score of 85/100. Building upon those findings (Appendix A), this chapter expanded the user study to encompass a larger sample size (26 users) along with comparing both a VR and a modified MR application, in addition to the traditional video-based training. Also to address the SUS survey limitations in the earlier case study [44], a revised questionnaire was devised based on users feedback. Namely, the new questionnaire was tailored to specifically assess criteria related to XR design and operator training, areas not traditionally addressed by SUS or 43 other conventional UX questionnaires. Unlike the SUS questionnaire, which primarily measures usability and ease of use, the new survey evaluated the trust, reliability, user satisfaction, training effectiveness, as well as usability and ease of use, and lastly overall preference, as reflected in Appendix B. In order to statistically evaluate the reliability of the designed questionnaire, Cronbach's alpha [107] was used, which is a widely known measure for assessing the internal consistency of a given questionnaire or survey. In other words, it is a tool that helps to determine whether the survey questions are measuring the same underlying concept. The possible values for Cronbach's alpha range from 0 to 1, where a score of 0 implies no consistency, whereas a score of 1 indicates perfect consistency. Typically, a threshold value of 0.7 or higher is considered acceptable for scientific studies [36]. In addition to the users' responses to the designed questionnaire (subjective feedback), the completion time for each of the training methods was also employed as an objective metric to assess the training effectiveness of DVT, VRT, and MRT. 3.2.4.4 Participants diversity 26 participants (13 males and 13 females) were recruited from the local university community, as shown in Figure 3.10a. The number of participants in this study, aligned with sample sizes commonly employed in similar studies evaluating XR technologies. For instance, prior research on VR-based operator training often involved 8-20 participants [31], [44]. Similarly, studies evaluating AR or MR applications frequently utilized sample sizes between 10-30 participants [24], [32]. While the inclusion of 26 participants may still be considered modest compared to large-scale clinical or social studies, it was sufficient for identifying meaningful 44 patterns and trends within the targeted domain. The diversity in participant demographics and their exposure to the technology further enhanced the reliability of the conclusions. The educational levels of the participants ranged from high school diploma to doctorate, details are shown in Figure 3.10b (53.85% had a master's degree, 19.23% had a bachelor's degree, 15.38% had a doctorate, 7.69% had a high school diploma, and 3.85% held other credentials). Half of the participants had prior experience with immersive applications in VR and MR environments, while the other half had no prior experience (Figure 3.10c). This was primarily due to the recruitment of a large group of participants for a more comprehensive assessment of the training methods. Additionally, limited access to experienced XR users resulted in half of the participants being novices. Therefore, the presence or absence of XR technology experience was treated as an independent variable in the user study, enabling the analysis of differences between XR novices and experts for all the criteria introduced in the survey. The distribution of all participants with prior (relative) XR experience could be classified into three groups: "little experience" with two participants (less than one month), "moderate experience" with four participants (one to six months), and "extensive experience" with seven participants (six months and more). The majority of experiences were gained via gaming (13 participants). Moreover, participants’ ages ranged from 18 to 36, with a mean of 27.54 and a standard deviation of 4.23, as indicated in Figure 3.11, with normal distribution. Lastly, everyone in the study had normal or corrected-to-normal eyesight. 45 Figure 3.10: Participants' demographic information. a) gender, b) educational level and c) experience level. Figure 3.11: Participants' age distribution. 3.2.5 Statistical analysis with blocks Although there are several methods for testing user study hypotheses, the analysis of variance (ANOVA) is perhaps the most widely used method in similar experimentations [108], [109], [110]. ANOVA is essentially used to determine whether there are significant differences between the means of three or more distinct (unrelated) study groups (often with a significance level of 5%; [25], [26]). When implementing ANOVA, data blocks can also be employed to control potential sources of data variation other than the treatments or primary independent variables themselves. In other words, blocks are used to eliminate known causes of variance in the interpretation of final results. In this research, because the participants were completing the same questionnaire against 46 the three training methods (VRT, MRT, DVT) and their individual perspective was common between the questionnaires, each user was considered as a block during ANOVA. Post-ANOVA analysis methods such as Fisher LSD, Tukey, and Bonferroni can also be employed to determine the statistically significant difference among pairs of the study groups, and in this study Bonferroni correction is employed. To utilize the standard ANOVA for experimental data analysis, it is necessary to verify (meet) the following assumptions [110]: • The populations from which the samples are taken, have a normal distribution. • The samples/data points are independent. • Data populations have the same variance, i.e., the error distribution variances are homogeneous across populations. According to the study by Mircioiu et al. [111], comparing parametric and non-parametric methods for analyzing Likert scale data, both methods consistently yielded identical results in inter-subgroup comparisons, regardless of whether they indicated significance or non-significance. This consistency held true when working with "large" (>15) response datasets and similar, though not necessarily normal, distributions among various subgroups. Given the large number of participants (26 persons) in the current study, ANOVA method was used. Regardless, the underlying assumptions of ANOVA were also checked. To analyze the results obtained from the questionnaire, the Likert scales were quantified to enable mathematical processing and statistical comparisons. Assigning numerical values to the scale allowed for the transformation of qualitative responses (e.g., "strongly disagree" to "strongly agree") into numerical data, facilitating the calculation of means and their subsequent comparison using analysis of variance (ANOVA). Values ranging from 1 (strongly disagree) to 5 (strongly agree) were assigned, denoted as 𝑖 = {1,2,3,4,5}, with 𝑖 representing the scales index. These values 47 were then normalized to a range between 0 and 1, represented as weights (𝑊𝑖) to ensure a uniform representation of responses for further analysis, as shown in Equation (3.12). 𝑊𝑖 =𝑥𝑖∑ 𝑥𝑖𝑖 (3.12) Given that the survey consisted of multiple questions, denoted by 𝑗 = {1,2, … , 𝑁} where N represents the total number of questions for a criterion, the weighted response (𝑍𝑖𝑗𝑘) for each question was calculated using Equation (3.13). This computation relied on the normalized weights and binary responses (𝑌𝑗𝑘) where 𝑌𝑗𝑘 ∈ {0,1}, with 1 indicating the selected response and 0 for all other options: 𝑊𝑖 × 𝑌𝑗𝑘 = 𝑍𝑖𝑗𝑘 (3.13) Subsequently, the normalized weighted response (𝑆𝑗𝑘) was determined using Equation (3.14), where 𝐶𝑗 represents the total weighted response across all questions for a given criterion, defined as ∑ ∑ 𝑍𝑖𝑗𝑘𝑘𝑖 : 𝑆𝑗𝑘 =∑ 𝑍𝑖𝑗𝑘𝑖𝐶𝑗 (3.14) Finally, the overall score (𝑀𝑘) of each training method relative to a criterion was calculated by averaging the user responses across all questions, as expressed in Equation (3.15): 𝑀𝑘 =∑ 𝑆𝑗𝑘𝑗𝑁 (3.15) The y-axis in Figures 3.15 to 3.17 represents these computed scores (𝑀𝑘), derived using the above equations. These scores reflect the normalized and averaged results of the questionnaire responses, enabling objective and comparable evaluations of the training methods. The transformation of the original 5-point Likert scale into normalized values was essential for 48 mitigating scale bias, standardizing the data, and enabling further statistical comparisons, particularly ANOVA, across training methods and criteria. 3.2.6 Multi-Criterial Decision Making (MCDM) The study employed five common MCDM techniques for final ranking of the three alterative training methods under the multiple performance criteria listed in Section 3.2.4.3. These methods included TOPSIS, Weighted Sum Method (WSM), Weighted Product Method (WPM), Multi MOORA, and VIKOR, as briefly reviewed below. The technique for order preference by similarity to the ideal solution (TOPSIS) was introduced in [112]. The TOPSIS MCDM technique ranks alternatives by emphasizing those that are "closest" to the positive ideal solution (PIS) and "furthest" from the Negative Ideal Solution (NIS) [113]. The NIS is used to maximize cost-type criteria and minimize benefit-type criteria, whereas the PIS is used to accomplish the reverse. Due to its theoretical precision, TOPSIS has been widely used in different MCDM applications [114]. In WSM, objectives are scaled (aggregated) into a single objective by multiplying each criterion by a decision maker-defined weight. Typically, the weight allocated to a criterion is proportionate to the criterion's importance relevance in the given application. The weights are normalized to guarantee that none of the criteria are over-scaled. On the other hand, the WPM is closely linked to WSM, with the key distinction being that WPM involves a product of criterial values, rather than a sum [115]. MULTIMOORA, similar to other popular MCDM methods, employs vector normalization to provide uniform ratings and uses three supplementary ranking techniques: The Ratio System, the Reference Point Approach, and the Full Multiplicative Form. Given that each of the three 49 ranking approaches has advantages and disadvantages, MULTIMOORA employs a hybrid version of these strategies [116]. As another viable ranking technique for MCDM, VIKOR provides a compromise ranking strategy [117], [118]. The VIKOR technique was developed to handle discrete choice problems and to improve complicated systems with inconsistent and non-comparable characteristics on a multi-criteria basis [117], [118]. When the decision-maker cannot convey their choice to find compromise solutions, VIKOR becomes particularly a helpful method. The reached compromise solutions minimize individual regret while maximizing community benefit, which is why the decision-makers can accept them. This method focuses on ranking and selecting among possibilities when there are conflicting requirements. To aid decision-makers in minimising the trade-offs necessary to attain the best choice, ranking by VIKOR may be undertaken with varied values of criteria weights. This analysis examines the impact of criterion weights on the recommended compromise solution [117], [118]. According to [119], fuzzy VIKOR and VIKOR are currently among the most practical methods for ranking and selection problems in large-scale settings. 3.3 Results and discussion 3.3.1 VR application As described in Section 3.2.3, a practical six-step guidance was presented to each operator/user to follow and finish the manufacturing procedure via the developed VR application. Figure 3.12 illustrates a typical scene recorded from one of the user’s tests on each of these steps. The first step requires the operator to lay the thermoplastic sheet on the moving fixture. To simplify the visualization and minimize confusion, the remaining components of the setup are not superimposed on the fixture in this stage (Figure 3.12a). Next in Figure 3.12b, the operator is 50 instructed to complete the next three steps in a single scene in which all the setup components, including the main frame, heaters, adjustable frame, sheet, and mold, are presented. Step 2 requires the operator to switch on the heaters by pressing the button on the left side of the setup. During the third step of the procedure, the user may modify the power for each heater by choosing the desired heater and then rotating the nob. In the fourth step, after the heating procedure is finished, the operator may switch off all heaters simultaneously by pressing the prompted button on the right. After completing the preceding stages, the user should release the moving frame from the side hangers and lower it to the mold, which is indicated as step 5 in Figure 3.12c. After placing the sheet on top of the mold and securing the moving frame, it is time to begin the vacuum process by activating the vacuum pump. This is demonstrated in Figure 3.12d as the last step of the thermoforming process. Figure 3.12: Illustration of the VR app steps followed by a user in the experiments. 51 3.3.2 MR application Figure 3.13 shows typical scenes recorded from one of the study user’s tests using MR application, to complete the same six tasks identified in Section 3.2.4. In the first step as depicted in Figure 3.13a, the user is asked to grab the thermoplastic sheet and position it on top of the moving fixture. Then, to assist the user in navigating between the panels if they are not at the proper height or if the items are far away, the operator can select and rearrange the objects. Only the relevant interactive elements are shown in each step to make the MRT more intuitive. In the second step on visualization, a control panel is added to the screen to activate the heating elements (Figure 3.13b). There are 15 buttons on the panel, one for each heating element. Each button consists of a red push button for switching the heaters on and off and a slider for adjusting the power usage for each heater (Figure 3.13c and Figure 3.13d). In addition, a screen above the control panel presents real-time information regarding each heater's surface temperature and power consumption, which was initially accessible in VRT mode. As soon as the heating process is finished, a push button is activated to allow user to deactivate all the heaters at once (Figure 3.13e). Also, once the operator's palm hovers above a heating element, the screen's information is updated immediately. Figure 3.13f illustrates step 5, in which the user will move down the sheet holder and place it on the mould. Lastly, by activating the vacuum pump thermoplastic sheet is formed as indicated in Figure 3.13g. Figure 3.14 demonstrates the MR application’s features, which are accessible to the operator who is working beside the actual developed thermoforming setup in the laboratory environment. 52 Figure 3.13: Steps of the developed MR application for training operators. a) The main menu. b) Hand menu prompted next to the operator’s hand to facilitate the navigation between application windows. c) Placing the thermoplastic sheet on the sheet holder. d) monitoring the heater’s temperature in real-time. e) Placing the sheet holder on the mold. 53 Figure 3.14: Features of the developed MR application: control panel on the left, the virtual setup on the right and the actual setup between them. In the shown example, the first row of heaters (in red) in the virtual set-up are working at full power (here 500W), and a heater in the middle of the second row (in green) is working at 150W power (the screen on top of the control panel depicts the info about the latter heater). Additionally, the sheet is heated, and the colour gradient in various regions reflects its surface temperature, as it is also shown on the operator’s hand once remotely selected via hand raycast. Within both VR and MR apps as soon as the heating process begins in step 3, the color gradient (temperature of the heaters) is updated graphically in real-time depending on the power consumption of each heater; with dark blue representing the lowest temperature and red representing the highest. 3.3.3 User study After going through all the training methods (DVT, MRT and VRT) in their Latin square orders, the participants were asked to take a survey and provide feedback on the three training platforms experienced, considering usefulness, ease-of-use, satisfaction, trust, reliability, training effectiveness and overall preference (as outlined in section 3.2.4). In order to compare all the 54 obtained data from the user’s feedback, the procedure defined in Section 3.2.5 was employed. Figure 3.15 illustrates the normalized mean and standard deviation for each training method, considering each performance criterion. As seen, in most criteria, MRT has had the highest mean compared to VRT and DVT, except for the ‘ease of use and trust’ criterion. Moreover, VRT was never ranked as the top priority in any of the mean measurements, while being considered the least trustable, convenient, and reliable training platform. Figure 3.15: The distribution of data obtained for each training method with respect to the performance criteria. Additionally, a comparison of cumulative means among the three training methods is shown in Figure 3.16. The latter also clearly indicates that MRT has the highest cumulative mean with respect to all the criteria combined. Finally, to compare the responses between novice and experienced XR users, the means of all the criteria for each training method were compared, as illustrated in Figure 3.17 (discussion of results to follow in subsequent sections). 55 Figure 3.16: Stacked mean (left) and median (right) performance scores, among different training platforms. Figure 3.17: Comparison of XR novices and XR experienced with respect to the measured criteria, for each of the three training methods. To assess the reliability of the designed questionnaire, the Cronbach's alpha coefficient was calculated as a measure of overall internal consistency of the survey. The obtained mean value of 56 Cronbach's alpha was 0.922, indicating a very high level of consistency between the designed survey questions and responses. 3.3.3.1 ANOVA with blocks and post-hoc tests One-way ANOVA with blocks was implemented on the collected data to specify whether the described criteria means in Figure 3.15 are statistically different under each criterion. To further analyze the mean differences, post hoc tests were conducted, to specifically identify which groups are different. Bonferroni correction was used as the post hoc method to conclude the results. Before conducting the ANOVA test on each criterion, the normality and the variance of the data were also checked. Figure 3.18a demonstrates a sample normality check as a Q-Q plot with a 95% confidence interval for the usefulness criterion. Moreover, the data population variance for the same criterion is illustrated in Figure 3.18b. The same procedure was employed for all the other criteria to ensure the data is suitable for conducting ANOVA tests (see the Appendix C). The ANOVA with block measurements (F(2, 77) = 15.086, p-value = 7.48E-06) determined a statistically significant difference in usefulness criterion across the training platforms, as indicated in Table 3.1 (see the Appendix C for results of other criteria). Figure 3.18: Normality check via Q-Q plot of the usefulness criterion data (left). Constant variance check for the criterion (right). 57 Table 3.1:ANOVA with blocks results for the usefulness criterion. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Training Platforms 0.361 2 0.180 15.086 7.48E-06 3.182 Users 0.682 25 0.027 2.277 0.006 1.727 Error 0.599 50 0.012 Total 1.643 77 The post-hoc analysis using the Bonferroni correction revealed that the VRT - MRT as well as DVT – MRT pairs have statistically significant different means, and in both cases, MRT is more useful compared to the others (MRT (M = 0.394, SD = 0.123); VRT (M = 0.278, SD = 0.133); DVT (M = 0.233, SD = 0.134)). There was no significant difference in usefulness between DVT – VRT pair. Same procedure for the rest of the criteria was followed to identify the significant factors affecting the training methods. Participants’ answers for the ease-of-use criterion (F(2, 77) = 1.903, p-value = 0.159) reflected that in terms of convenience, there is no statistically significant difference between the alternatives and therefore, no pairwise comparison is necessary. Moreover, the outcome of the one-way ANOVA with block test for the satisfaction (F(2, 77) = 9.247, p-value = 3.82E-4), suggested that MRT is statistically more significantly satisfactory compared to VRT and DVT. Additionally, a pairwise Bonferroni correction test indicated that there is statistically significant difference between VRT (M = 0.261, SD = 0.134) and MRT (M = 0.36, SD = 0.135); as well as DVT (M = 0.237, SD = 0.119) and MRT (M = 0.36, SD = 0.135). There was no significant difference found in usefulness between DVT – VRT pair. In terms of user’s trust on the training platforms, participants provided a statistically significantly higher rating for the DVT (M = 0.305, SD = 0.152) over the MRT (M = 0.274, SD = 0.115) and VRT (M = 0.206, SD = 0.127) methods, where one-way ANOVA with block test indicated F(2, 77) = 6.101, p-value = 0.004. A post hoc pairwise test via Bonferroni method showed that DVT – VRT is the only pair with statistically significant mean difference, where DVT 58 is preferred over VRT in terms of user’s trust. On the other hand, the obtained results of one-way ANOVA with block for reliability measurements, proved that there is no statistically significant difference between the training platforms (F(2, 77) = 0.796, p-value = 0.456). Statistical analysis demonstrated that the training (F(2, 77) = 12.52, p-value = 3.90E-5) with MRT (M = 0.376, SD = 0.132) is the most efficient method in comparison to VRT (M = 0.297, SD = 0.140) and DVT (M = 0.202, SD = 0.122). Bonferroni pairwise comparison suggested that the DVT – MRT and DVT – VRT pairs are the ones with statistically significant mean differences, which is not the case for the VRT – MRT pair. Finally, the outcome of the one-way ANOVA with block test for the Preference criterion (F(2, 77) = 13.35, p-value = 2.25E-5), suggested that MRT is statistically significantly preferred the most, compared to VRT and DVT. Furthermore, the pairwise Bonferroni test showed that there is statistically significant difference between the pair of VRT (M = 0.223, SD = 0.147) and MRT (M = 0.334, SD = 0.171); as well as the pair of DVT (M = 0.134, SD = 0.048) and MRT (M = 0.334, SD = 0.171). There was no significant difference found in usefulness between DVT – VRT pair. Moreover, an evaluation was conducted to assess the impact of participants' prior exposure to XR technology, as depicted in Figure 3.17. The results of the statistical analysis indicated that no statistically significant differences between the two groups are observed with respect to any of the criteria, including Usefulness (F(1, 77) = 0.136, p-value = 0.713), Ease of use (F(1, 77) = 0.608, p-value = 0.438), User Satisfaction (F(1, 77) = 3.459, p-value = 0.067), Trust (F(1, 77) = 0.039, p-value = 0.844), Reliability (F(1, 77) = 0.086, p-value = 0.770), Training (F(1, 77) = 0.943, p-value = 0.335), and Overall preference (F(1, 77) = 0.713, p-value = 0.401). In terms of task completion time, the initiation and conclusion of participants' training sessions for each training method were recorded and subjected to analysis. A statistically significant difference in Completion Time across the training platforms was determined using ANOVA with block 59 measurements (F(2, 77) = 4.302, p-value = 0.017). Subsequent post-hoc analysis, employing the Bonferroni correction, revealed that the DVT and VRT exhibited statistically significant differences in means. Specifically, VRT (M = 220.1, SD = 28.62) emerged as the fastest training method, while DVT (M = 244.1, SD = 33.1) was the slowest. In contrast, MRT (M = 236.2, SD = 28.2) demonstrated intermediate performance. 3.3.3.2 MCDM results As outlined in Section 3.2.6, five conventional approaches of MCDM were implemented to find the best alternative among the three training methods. Based on the above results from the statistical analysis and the ANOVA tests, out of all the seven subjective criteria defined to assess the training efficiency of the platforms, Ease-of-use and Reliability were eliminated as there was no statistically significant mean differences, and hence they cannot be part of final decision-making in this case study. Furthermore, Completion Time was found to be statistically significant as an objective metric. This left the MCDM problem with six criteria, which are shown in Table 3.2; including the criterion types indicated by min or max (cost-type or benefit-type) and their weights, as well as the calculated means of the alternatives. As can be seen, the best mean values (shown in bold) under different criteria pertain to different training methods, and therefore to decide the best alternative, MCDM techniques are needed. Given the study's focus on comparing three distinct training methods, the highest weight was assigned to training effectiveness, reflecting its paramount importance. Additionally, usability holds significant weight due to its crucial role. User satisfaction and trust, while substantial, occupy an intermediate level of importance compared to the preceding criteria. Finally, overall preference and completion time bear the lowest weights, as factors such as preference and training speed were comparatively less critical when compared to training effectiveness and usability. Lastly, in 60 preparation for the MCDM analysis, normalization of the mean measurements for each criterion was undertaken (not included in Table 3.2). Table 3.2: Means of the three training methods (decision alternatives) per each criterion along with the criteria weights assigned. CRIT. ALT. Usefulness (max) W. 0.25 Satisfaction (max) W. 0.2 Trust (max) W. 0.15 Training (max) W. 0.3 Preference (max) W. 0.05 Time (min) W.0.05 MRT 0.3949 0.3606 0.2744 0.3769 0.3346 236.2 VRT 0.2782 0.2615 0.2064 0.2974 0.2231 220.1 DVT 0.2333 0.2375 0.3051 0.2026 0.1346 244.1 Figure 3.19 demonstrates the final rankings of alternatives based on the five MCDM approaches of TOPSIS, WSM, WPM, Multi MOORA, and VIKOR. MRT is considered the first rank based on the results obtained from all the MCDM methods, whereas VRT and DVT share the second and third ranks. Figure 3.19: Means of the three training methods (decision alternatives) per each criterion along with the criteria weights assigned. Finally, to assess the effect of user/expert opinion criteria weights on the MCDM outcome, a sensitivity analysis was conducted, considering seven different weighting scenarios/cases. Case 1 provides equal weights among all the criteria, whereas the rest of the cases considered five 61 criteria with the minimum weights, to emphasize on the sixth criterion with the highest weight (Table 3.3). The TOPSIS method was then implemented for each case. Based on Table 3.3, the rankings of the alternatives based on various criteria weighting cases consistently indicated that MRT is the preferred alternative as compared to the other training methods. In particular, notice that the score of MRT (out of the maximum 1.0) is consistently higher than 0.8, except for cases 2 and 5. Table 3.3: TOPSIS Rankings of the training methods (alternatives) based on various sets of criteria weights. Criteria Base case Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7 Usefulness 0.25 0.166 0.1 0.1 0.1 0.1 0.1 0.5 Satisfaction 0.2 0.166 0.1 0.1 0.1 0.1 0.5 0.1 Trust 0.15 0.166 0.1 0.1 0.1 0.5 0.1 0.1 Training Effectiveness 0.3 0.166 0.1 0.1 0.5 0.1 0.1 0.1 Overall Preference 0.05 0.166 0.1 0.5 0.1 0.1 0.1 0.1 Completion Time 0.05 0.166 0.5 0.1 0.1 0.1 0.1 0.1 Alternatives Ranking (scores) MRT 1 (0.917) 1(0.887) 1 (0.771) 1 (0.965) 1 (0.951) 1 (0.677) 1 (0.935) 1 (0.948) VRT 2 (0.416) 2 (0.399) 2 (0.477) 2 (0.438) 2 (0.529) 3 (0.237) 2 (0.273) 2 (0.296) DVT 3 (0.165) 3 (0.201) 3 (0.189) 3 (0.065) 3 (0.091) 2 (0.558) 3 (0.121) 3 (0.098) 3.3.4 A qualitative interpretation of user study outcomes The quantitative results derived from the above user study collectively revealed a significant enhancement in users' training experiences upon utilizing the MRT system. Remarkably, even among participants who had no prior exposure to either the MR system or the thermoforming process, the consensus favored the MRT interface over both the DVT and VRT alternatives. It was interesting to note that participants' experience levels had no discernible impact 62 on the subjective criteria, which could be attributed to the intuitive UI design of the present immersive applications. Despite MRT consistently ranking the highest according to the MCDM, trust emerged as a pivotal factor influencing users' preferences as some users favored DVT over VRT. Particularly, users highly valued MRT in terms of training effectiveness and overall preference, while DVT garnered the least favor in both criteria. Users' post-experimental qualitative feedback revealed a desire to maintain awareness of their surroundings during training, offering a rationale for the preference for MRT. Additionally, VRT received the highest score in terms of training effectiveness, with users indicating a preference for the input method within the VRT system over MRT. The joystick input, as opposed to the hand gestures utilized in the mixed reality HMD, was cited as being more user-friendly. However, despite this input method preference, the results decisively established MRT as the superior environment when compared to VRT, affirming its suitability for training purposes in the present manufacturing application. In essence, while the input method within VRT may have been preferred due to its familiarity, the broader context of user experience, effectiveness, and situational awareness within MRT ultimately established it as the superior choice for training purposes. The users' preference for input method alone did not overshadow the holistic advantages offered by the MRT system. Based on the statistical analysis and quantitative evaluations conducted through the user study and MCDM, all the sub-hypotheses (Sub-H1 to Sub-H5) were validated, confirming that the Mixed Reality Training (MRT) environment is likely the most suitable platform for training purposes in the current application. Each hypothesis was supported by the findings, demonstrating MRT's superiority across multiple dimensions, including training effectiveness, user comfort, satisfaction, trustworthiness, and reliability. 63 Sub-H1 posited that interacting with the designed DT in the MRT environment would be more effective for training purposes compared to the VR environment and video training. The statistical analysis confirmed this, showing that participants trained in the MRT environment performed better and demonstrated higher learning outcomes compared to the other platforms. Sub-H2 suggested that operators would feel more comfortable being trained in an MRT environment compared to VRT and DVT. This hypothesis was validated, as participants consistently rated MRT as the most comfortable platform for their training experience. Sub-H3 examined user satisfaction with their performance during and after training. The findings supported this hypothesis, indicating that participants felt more confident and satisfied with their performance after training in the MRT environment compared to VRT and DVT. Sub-H4 proposed that MRT would be the most trustworthy environment among the three platforms. The study's results confirmed this, with MRT receiving the highest ratings for trustworthiness compared to VRT and DVT. Sub-H5 suggested that MRT would be the most reliable platform for trainees to function properly during training. This hypothesis was also validated, as MRT was consistently rated higher in terms of reliability compared to the other two platforms. Limitations: Although this chapter made efforts to include a diverse range of target users, it still faced limitations in terms of accessing participants with specific manufacturing skills, particularly thermoforming. The cognitive skills, notably procedural memory, as perceived by thermoforming experts, may have had an impact on the study's outcomes. Furthermore, it is worth noting that HMDs have limitations in computational capacity, leading to the necessity of reducing simulation resolution to enable real-time interaction for operators. There is also room for improvement in the UI design to enhance the overall user experience, acknowledging that UI design enhancements can continually refine the user's interaction with the system. Another 64 constraint of the study was its reliance solely on completion time as the objective metric. Additional factors, such as the frequency of errors during task completion for each training method, or an examination of training transfer encompassing both immediate and long-term recall, can possibly provide a more comprehensive perspective. Future MCDM evaluation framework may also include complex human factors such as emotions or e.g. cognitive resistance to change [120] toward a non-conventional training method, etc. 3.4 Summary With recent breakthroughs reviewed in Chapter 2 in immersive technologies, new prospects await industrial processes that rely on smart interactions between machines and human operators. Advanced polymers and composites manufacturing sector is a seamless beneficiary of such technologies, with a promising future in joining I5.0, as it involves many manual or minimally automated procedures during different material processing techniques. Manual methods for staff training in such manufacturing processes using experimental trials can be very costly and time-consuming, which brings about a critical need for the adaptation of immersive technologies. In this chapter, we developed a novel adaptive and interactive DT interface in both VR and MR environments to demonstrate how a complex thermoforming process used in thermoplastic manufacturing can be simulated in real-time and used for training novice operators under immersive experience. Meta Quest 2 and Microsoft HoloLens 2 were employed as the target devices to validate the study hypothesis. Aligned with the earlier research, several evaluation criteria for VRT and MRT were employed, including Usefulness, Ease of use, Satisfaction, Trust, Reliability, Training effectiveness, and overall user Preference in addition to completion time. To provide a comprehensive evaluation of the immersive platforms, a DVT was also employed as the benchmark training method. Regarding the aforementioned criteria and training methods, a user 65 study (𝑛 = 26) was carried out and the acquired data was analyzed using one-way ANOVA with blocks. Only five of the seven criteria, including Usefulness, Satisfaction, Training Efficacy, Trust, and Overall User Preference, exhibited statistically significant mean differences across the training platforms. Accordingly, MCDM techniques were employed to rank the training platforms. Except for the 'Ease of use' and 'Trust' criterion, the MCDM findings and sensitivity analysis of criteria weights demonstrated a strong preference of the users towards MRT (score range: 0.677 to 0.965) compared to VRT and DVT. Although it was the fastest training method, the VRT consistently scored second, with ratings ranging from 0.237 to 0.529, despite being the least reliable, convenient, and trustable training platform. Except for one scenario in the sensitivity analysis (case 5), DVT was overall the least favored training technique. In the base case with weights determined by expert opinion, the MCDM results revealed that the preference score for MRT was 2.2 times higher than that of VRT; and 5.56 times higher than that of DVT. According to the sensitivity analysis, overall, ‘Trust’ was the most influential criterion affecting the rating order of the training platforms. 66 Chapter 4: Single-Agent Reinforcement Learning Algorithms for Optimizing the Heating Setup in Thermoforming 4.1 Introduction In the previous chapter, we explored the advantages of using immersive technologies to enhance operator training in thermoforming. These technologies provide a standardized training method that replaces traditional approaches, significantly improving the education and skill levels required for manual tasks. However, while immersive technologies address the needs of manual labor, they do not extend to optimizing machinery and hardware, which is crucial for further advancing manufacturing processes. Fulfilling objective 3 (see Section 1.2), here we aim to develop a set of Deep Reinforcement Learning (DRL) algorithms, specifically Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN), and test them for optimizing heating parameters in thermoforming. We compare single-agent DQN with single-agent PPO under conditions of fixed and variable target temperatures, different action spaces, and further evaluate single-agent PPO's performance in multi-heater optimization and full-sheet heating scenarios. 4.2 The role of AI in manufacturing and thermoforming optimization The integration of AI into real-world applications has become a standard practice across various sectors, including manufacturing. AI has been particularly beneficial in creating intelligent, optimized processes that result in cost, energy, and time savings. This trend has led to significant advancements, especially in manufacturing [2]. As mentioned in Chapter 2, in the context of composites manufacturing, particularly thermoforming, AI’s potential has not been fully realized. Although the benefits of thermoforming, such as its versatility and effectiveness in shaping complex 3D forms, were discussed in detail in the previous chapter, there remains a gap in the application of AI to optimize this process. Given 67 the comprehensive review of AI and ML techniques discussed in Chapter 2:, DRL emerges as a strong candidate for optimizing the thermoforming process in composites manufacturing, particularly because of its adaptability and dynamic learning capabilities. Traditional machine learning models like decision trees and SVM, while effective in static, data-rich environments, often fall short in manufacturing scenarios that involve high levels of uncertainty, non-linearity, and dynamic conditions, which are common in thermoforming [87], [88]. This chapter aims to bridge that gap by employing DRL to fine-tune the heating parameters of a lab-scale thermoforming setup. The DRL task is to achieve the desired temperature distribution on the thermoplastic sheet, thereby enhancing the overall efficiency and effectiveness of the thermoforming process. 4.3 Methodology In common thermoforming processes, a thermoplastic sheet is subjected to heat and then shaped within a mold through vacuum pressure. The primary objective is to establish an even temperature distribution across the sheet, ensuring optimal formability and final product quality. It is essential to exercise precise control over temperature in all regions of the sheet to guarantee the production of defect-free parts. Nevertheless, the temperature across the sheet undergoes fluctuations during the process, introducing complexities into the control dynamics [121]. Figure 4.1 illustrates potential defects that may arise at different stages of the process. Non-uniform thickness distribution, attributed to inconsistent heating/cooling or excessive/low forming pressure, can lead to cracks, shear, or tears in the final product. Additionally, defects such as webbing or bridging may result from material overstretching before heating or temperature variations during the process. Distortion, another common defect in thermoforming, is primarily caused by overheating the thermoplastic material or uneven cooling. Consequently, it becomes 68 evident that optimizing the heat transfer between heaters and the thermoplastic sheet is a pivotal factor for the overall process performance in thermoforming [121]. To achieve the most effective heating configuration, given a part nominal shape and material, a series of sequential decisions must be made (e.g. which heather(s) to power on first and at which level, etc.), introducing a certain level of complexity to the problem. This is where leveraging RL methods can be highly efficient in this process optimization. Figure 4.1 – Overview of the methods employed in the present chapter. Depicted in Figure 4.2 is a laboratory-scale thermoforming setup, providing a visual representation of the process intricacies and its individual components. To replicate laboratory performance on a small scale, the setup integrates fifteen ceramic heaters. Furthermore, the gap between the thermoplastic sheet and the heaters can be precisely adjusted using a motor-driven fixture equipped with a linear ball-screw and sliding guide on each side of the setup. This ensures accurate 69 positioning and flat orientation of the sheet beneath the heaters. These settings, along with heater controls, are managed through a custom-developed control box equipped with a microcontroller. This setup was developed by the author and the team at the Composite Research Network – Okanagan (CRNO) and is used as a reference for a simulation model validation (to generate synthetic data) and the subsequent data-driven optimizations studies to follow. Figure 4.2 – Overview of the thermoforming setup employed and its components. 4.3.1 Deep reinforcement learning model DRL has been highly successful in solving complex control problems with high-dimensional inputs; e.g., see [122]. These problems are notoriously difficult to solve due to the complexity of formulating control policies. As reviewed in Chapter 2, traditional control methods, such as Model Predictive Control (MPC) and Proportional-Integral-Derivative (PID) controllers, have proven effective in several industrial process controls, but also face limitations in dynamic and non-linear systems like thermoforming. The MPC, while adept at handling constraints and delivering optimal actions, requires highly accurate system models and substantial computational resources, making real-time application challenging [123]. PID controllers, though simple and easy to implement, rely on manual tuning and fixed parameters, limiting their adaptability to changing heat transfer conditions or varying material properties [123]. 70 In contrast, DRL provides a model-free framework that learns adaptive control policies through interactions with the environment. This approach eliminates the need for precise system models, making it particularly suitable for thermoforming processes characterized by significant uncertainty, non-linearity, and dynamic conditions [123]. By directly optimizing control strategies, DRL offers a compelling alternative to traditional controllers, enabling robust and scalable solutions for complex thermoforming environments [123]. In this context, the thermoforming process is formulated as an infinite-horizon discounted Markov Decision Process (MDP), denoted by ℳ. This formulation involves a tuple ℳ = ⟨𝒮,𝒜,𝒫, ℛ, 𝛾⟩, where each component plays a crucial role: • 𝒮: Represents the set of states within the system. For instance, it includes variables such as the temperature of the thermoplastic sheet and the current power levels of each heater. • 𝒜: Denotes the set of feasible actions that can be taken to control the system. These actions could involve adjusting the power level of each heater or manipulating other variables related to the thermoforming process. • 𝒫 ∶ 𝒮 × 𝒜 × 𝒮 → [0,1]: This function determines the state transition probabilities. It specifies the likelihood of transitioning from one state to another based on the action taken. For example, it calculates the probability of reaching a certain temperature in the thermoplastic sheet after adjusting the power levels of the heaters. • ℛ: 𝒮 × 𝒜 × 𝒮 → This function represents the reward structure of the system. It assigns numerical rewards to state-action pairs, indicating the desirability of performing certain actions in specific states. For instance, it assigns higher rewards to actions that lead to 71 achieving desired temperature ranges or improving the efficiency of the thermoforming process. • 𝛾 ∈ [0,1): This parameter, known as the discount factor, determines the importance of future rewards compared to immediate rewards. A lower value of γ implies more weight on immediate rewards, while a higher value indicates greater consideration for future rewards in the decision-making process. To control thermoforming process represented as an MDP, the primary objective is to determine the control policy π ∶ 𝒮 × 𝒜 → [0,1] that maximizes the reward function (expected return) over time, crafted to minimize the difference between the achieved temperature distribution and the desired temperature distribution on the thermoplastic sheet, all within the confines of defined constraints, considering both immediate and future rewards. This policy dictates the actions to be taken in each state to achieve optimal performance in controlling the thermoforming process. Mathematically, this is expressed as Equation (4.1): 𝜂(𝜋) = 𝐸𝜏 [∑𝛾𝑖ℛ(𝑠𝑖 , 𝑎𝑖 , 𝑠𝑖+1)∞𝑖=0] (4.1) where 𝐸𝜏 represents the expectation over the state–action sequence 𝜏 = (𝑠0, 𝑎0, 𝑠1, 𝑎1, … ) [74]. To optimize temperature distribution on the thermoplastic sheet, first the environment should be structured to interact with the DRL agent. This is implemented in accordance with the MATLAB Simulink interface, with 𝑟𝑒𝑠𝑒𝑡() and 𝑠𝑡𝑒𝑝() methods. These methods serve to reset the simulation or advance it by one timestep, respectively. Upon initialization, the simulator described in section 3.2.2 is created as the RL environment. The process simulator incorporates a 𝑠𝑡𝑒𝑝() 72 method, which allows for the adjustment of air temperature boundary conditions and advances the heat transfer simulation by one timestep. Given the thermal dynamism of the heating control problem, it does not exhibit Markovian property [74]; i.e. relying solely on a current state of sheet temperature distribution. Therefore, a memory term is introduced by considering the dependency of heat transfer between heaters and the sheet temperature at a given timestep 𝑡, in which the sheet temperature relies on the integral of heating and cooling rates from 𝑡 = 0 to 𝑡 = 𝑡. To approximate the future state, the current sheet and heater temperature, one timestep of historical observations (for gradient capture), and the last action taken are appended in the observation. The RL environment representing the thermoforming process simulation is implemented in the following sub-sections. 4.3.1.1 State space: The state space consists of the following four parameters to be used as the observations for the RL agents: • Time: Current time of the process [𝑠𝑒𝑐]. • Heater surface temperature: The current temperature on each of the fifteen heaters [℃]. • Current error: The current difference between the target temperature profile and the current temperature profile in all fifteen zones [℃]. 4.3.1.2 Action Space The action space consists of the two options (i.e. power level of each of the 15 heaters in Figure 4.2) for RL agent: • Large action space (9 possible actions): {0, 150, 200, 250, 300, 350, 400, 450, 500} [𝑊] • Small action space (3 possible actions): {0, 150, 500} [𝑊] 73 where the agent selects either of these options at every timestep based on the current state, as a continuous control problem. 4.3.1.3 Reward function The core of the reward function is to minimize the error in the thermoforming setting, which is the difference between the actual current temperature and the desired temperature. The reward function for this thermoforming optimization problem is designed to be time-sensitive, highlighting the importance of the agent minimizing error within a shorter duration. Thus, a penalty coefficient is introduced, gradually increasing over time. This coefficient begins at 0.5 to mitigate initial penalty impact, rises to 0.75 midway through the process, and reaches 2 towards the end. This accounts for the initial lag in temperature alignment between the heater and the sheet, which diminishes as the process progresses. This is demonstrated through Equations (4.2), (4.3): 𝐸𝑟𝑟𝑜𝑟 = 𝐷𝑒𝑠𝑖𝑟𝑒𝑑 𝑇𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒 − 𝐶𝑢𝑟𝑟𝑒𝑛𝑡 𝑇𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒 (4.2) ℛ = 𝑝 ×(−∑(𝐸𝑟𝑟𝑜𝑟2) − 𝑚𝑎𝑥|𝐸𝑟𝑟𝑜𝑟|2)6 (4.3) This reward function type prioritizes error magnitude through the sum of squares, which is then negated to incentivize error minimization. Given that the agent oversees five heaters arranged in a column format, the equation is divided by 6 (the sum of the five heaters plus the maximum among them) to ensure uniform error consideration. The selection of the DRL algorithms in this study was guided by their demonstrated ability to address key challenges in thermoforming process optimization. The algorithms were chosen based on their suitability for discrete action spaces and their robustness in dynamic environments with complex state-action relationships. Deep Q-Networks (DQN) [124] and Proximal Policy Optimization (PPO) [125] were selected due to their complementary strengths in handling discrete control tasks and iterative learning stability, respectively. 74 DQN has proven effective in discrete action domains, leveraging techniques like experience replay and target networks to ensure stable and efficient learning. Its value-based approach offers computational simplicity, making it an appropriate choice for initial evaluations in this study. PPO, in contrast, employs a policy-gradient method with a clipped objective function, ensuring stable and controlled updates during training. While traditionally applied in continuous action spaces, PPO’s robustness and adaptability make it equally suitable for discrete settings, as utilized in this research. The comparative evaluation of these algorithms under various heating scenarios serves as a foundation for identifying the most effective approach for extending to multi-agent frameworks in Chapter 5. 4.3.2 Deep-Q Networks (DQN) Q-learning stands as a foundational algorithm in RL, strategically crafted to empower agents in learning optimal policies within environments characterized by discrete state and action spaces. The "Q" in Q-learning symbolizes the quality of an action in each state. The algorithm's objective is to iteratively update Q-values based on received rewards and the maximum expected future rewards. The underlying concept revolves around learning a Q-function, estimating cumulative future rewards for each state-action pair. Initially, the agent uses the epsilon-greedy exploration strategy to choose an action, striking a balance between exploiting the best-known action and exploring a random one. Upon making this choice, the agent moves to the next state and obtains a reward from the environment. Subsequently, the Q-value corresponding to the state-action pair is revised according to the Bellman equation, which incorporates both the immediate reward and the anticipated future rewards. This revised Q-value is then recorded in the Q-table. The process is repeated iteratively until the Q-values converge or the target performance level is reached, as reflected in Equation (4.4). 75 𝑄(𝑠, 𝑎) = (1 − 𝛼)𝑄(𝑠, 𝑎) + 𝛼(𝑟 + 𝛾 max𝑎(𝑠′, 𝑎′)) (4.4) In this context, the current state is denoted by 𝑠, the selected action by 𝑎, the resulting next state by 𝑠′, and the action that yields the highest Q-value in the subsequent state by 𝑎′. The learning rate is represented by 𝛼, and the discount factor by 𝛾. By continuously iterating these steps and updating the Q-table, the Q-values eventually converge, ultimately representing the expected reward for selecting an action based on the current state. Deep Q learning represents a major advancement in traditional Q learning, enabling the approximation of Q values through the use of a neural network. Rather than relying on a Q-table to store all possible state-action pairs and their corresponding Q values, a Q network is employed to estimate these values. This is achieved by encoding the state into a feature vector, which allows the network to learn the mapping between the feature vector and the Q value. The Q function is structured to take the state as input, with the output dimensions corresponding to the number of possible actions. The maximum value among these output nodes is then selected as the Q value for the next state-action pair. The DQN Pseudocode [124] can be found in Table 4.1. 76 Table 4.1: DQN algorithm and Pseudocode. Algorithm 1 - DQN with experience replay Initialize replay memory 𝒟 to capacity 𝑐. Initialize action-value function 𝑄 with random weights. for episode = 1, 𝑙 do initialize sequence 𝑠1 = {𝑥1} and preprocessed sequenced 𝜙1 = 𝜙(𝑠1) for 𝑡 = 1, 𝑟 do with probability 𝜖𝐷𝑄𝑁 select a random action 𝑎𝑡 otherwise select 𝑎𝑡 = 𝑚𝑎𝑥𝑎 𝑄∗(𝜙(𝑠𝑡), 𝑎; 𝜃) execute action 𝑎𝑡 in emulator and observe reward 𝑟𝑡 and image 𝑥𝑡+1 set 𝑠𝑡+1 = 𝑠𝑡 , 𝑎𝑡 , 𝑥𝑡+1 and preprocess 𝜙𝑡+1 = 𝜙(𝑠𝑡+1) store transition (𝜙𝑡 , 𝑎𝑡 , 𝑟𝑡 , 𝜙𝑡+1) in 𝒟 sample random mini-batch of transitions (𝜙𝑗 , 𝑎𝑗 , 𝑟𝑗 , 𝜙𝑗+1) from 𝒟 set 𝑦𝑖 = {𝑟𝑗 for terminal 𝜙𝑗+1 𝑟𝑗 + 𝛾 𝑚𝑎𝑥𝑎′ 𝑄(𝜙𝑗+1, 𝑎′; 𝜃) for non − terminal 𝜙𝑗+1 perform a gradient decent step on (𝑦𝑖 − 𝑄(𝜙𝑗 , 𝑎𝑗; 𝜃))2following below equation: ∇𝜃𝑖𝐿𝑖(𝜃𝑖) = 𝔼𝑠,𝑎 [(𝑟 + 𝛾 max𝑎′𝑄 (𝑠′, 𝑎′; 𝜃𝑖−1) − 𝑄(𝑠, 𝑎; 𝜃𝑖)) ∇𝜃𝑖𝑄(𝑠, 𝑎; 𝜃𝑖)] end for end for Moreover, the hyperparameters used in training the DQN agent are listed in Table 4.2. Table 4.2 - DQN hyperparameters. Hyperparameters Values Network structure Activation functions Learning rate Batch Size Discount Factor Experience Buffer Length Target Smooth Factor Epsilon Epsilon Decay Rate 6; 64; 31; 1 ReLU; ReLU; Linear 1𝑒 − 3 64 0.98 1𝑒4 1𝑒 − 3 0.3 0.005 77 4.3.3 Proximal Policy Optimization (PPO) Introduced in 2017 by [125], PPO is still regarded as a reliable and effective reinforcement learning algorithm. Derived from Trust Region Policy Optimization (TRPO), PPO ensures that updates to the policy network remain within acceptable limits. This prevents significant changes in the probability distribution generated by the policy after each update, addressing the issue of irrelevant states encountered during learning. However, TRPO's computational complexity renders it impractical for many real-world applications. PPO simplifies the algorithm while maintaining performance. The PPO algorithm comprises two key processes: a) Policy Optimization, where data is iteratively collected by executing the current policy in the environment to enhance it while ensuring stability within a "trust region," and b) Value Function Estimation, which assesses action and state quality through estimating the value function, crucial for calculating Advantage during policy optimization. The initialization involves setting parameters for both policy and value function, aiming to optimize them to achieve the desired outcome. There are two primary variants of policy optimization in PPO: 1. PPO-Penalty: In contrast to TRPO's strict constraint on updating parameters in case of large policy updates, PPO-Penalty treats this constraint as a penalty, making it a soft constraint. Here, a substantial difference in distributions penalizes the objective function, offering flexibility while maintaining control. 2. PPO-Clip: This algorithm ensures stability by "clipping" the ratio between the old and new policy distributions, preventing it from straying too far from 1. This method, which is also utilized in this chapter, has previously demonstrated superior performance compared to PPO-Penalty. The clipped objective function is defined as Equation (4.5) [125]. 78 𝐿(𝑠, 𝑎, 𝜃𝑘, 𝜃) = 𝑚𝑖𝑛 (𝜋𝜃(𝑎|𝑠)𝜋𝜃𝑘(𝑎|𝑠)𝐴𝜋𝜃𝑘(𝑠, 𝑎), 𝑐𝑙𝑖𝑝 (𝜋𝜃(𝑎|𝑠)𝜋𝜃𝑘(𝑎|𝑠), 1 − 𝜖, 1 + 𝜖)𝐴𝜋𝜃𝑘(𝑠, 𝑎)) (4.5) In the above equation, the current policy is denoted as 𝜋𝜃, and the old policy as 𝜋𝜃𝑘. Consequently, 𝜋𝜃 (𝑎|𝑠) represents the probability of taking action 𝒂 from state 𝒔 according to the current policy. Additionally, 𝐴𝜋𝜃 denotes the Advantage, which is the estimated increase in reward observed when taking action 𝒂 from state 𝒔 using policy 𝜋𝜃. The clipping ratio is represented by 𝜀. In general, the Value Function provides an estimate of the expected return from the current state, as reflected in Equation (4.6). The training process for the value function follows a straightforward approach: an error is computed by contrasting the actual return observed at the end of an experience with the estimate, frequently employing a function such as the Mean-Squared Error. 𝑉𝜋(𝑠𝑖) = 𝔼𝑎𝑖,𝑠𝑖+1,…[∑𝛾𝑙ℛ(𝑠𝑖+𝑙 , 𝑎𝑖+𝑙 , 𝑠𝑖+𝑙+1)∞𝑙=0] (4.6) In the context of estimating the policy and value function, a PPO agent utilizes two distinct function approximators. Firstly, the Actor function maps the current state space 𝒔 to a probability distribution across the action space. Secondly, the Critic function estimates the value function, which represents the anticipated cumulative reward achievable by an agent from a specific state 𝒔 while adhering to a prescribed policy. This value function considers the long-term rewards attainable by following the current policy. In other words, the actor learns which action to take based on environmental observations (i.e., the control policy). When the actor’s chosen action is executed, the agent receives a reward that is processed by the critic, whose role is to evaluate whether the action improved or worsened the environment’s state. The critic’s feedback informs optimization of the actor’s decision-making. Specifically, the critic produces a rating for the action 79 taken in the previous state. By comparing this rating, the agent evaluates its current policy against a new policy and determines how to enhance decision-making [126]. The Advantage Estimate 𝐴𝜋 measures the comparative effectiveness of selecting a specific action in a given state, in relation to the state's value. This comparison is based on the state's target value 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡, as outlined in Equation (4.7). 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡= 𝑟𝑡 + 𝛾𝑅𝑡+1 + 𝛾2𝑅𝑡+2 + ⋯+ 𝛾𝑛−1𝑅𝑡+𝑛−1 + 𝛾𝑛𝑉𝜔𝑜𝑙𝑑(𝑆𝑡+𝑛) (4.7) Where 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡 represents the target state value for state 𝑠𝑡, 𝑅𝑡 is the reward for action 𝑎𝑡 in state 𝑠𝑡, and 𝛾 is the discount factor. The term “𝑡 + 𝑛” indicates the time step when the trajectory ends at state 𝑠𝑡+𝑛. If the trajectory ends due to reaching the maximum length 𝑻, 𝑉𝜔𝑜𝑙𝑑(𝑠𝑡+𝑛) signifies the predicted state value for 𝑠𝑡+𝑛. Otherwise, If the trajectory terminates earlier, 𝑉𝜔𝑜𝑙𝑑(𝑠𝑡+𝑛) is set to 0, indicating that the agent has reached a terminal state and can no longer accumulate rewards. Thus, the calculation of advantage estimates builds upon the determination of target state values. Equation (4.8) presents the formula for computing the advantage estimate 𝐴𝜋, which evaluates the impact of taking action 𝒂𝒕 in state 𝒔𝒕 over a trajectory of length 𝑻. 𝐴𝜋 = 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡− 𝑉𝜔𝑜𝑙𝑑(𝑠𝑡) (4.8) Here, 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡 is derived following the method described in Equation (4.7), and 𝑉𝜔𝑜𝑙𝑑(𝑠) represents the state value predicted by the state value network during data generation. Essentially, Equation (4.8) assesses the observed return 𝑉𝑡𝑡𝑎𝑟𝑔𝑒𝑡 against the current estimate 𝑉𝜔𝑜𝑙𝑑(𝑠) of the state's value, providing insight into the action's effectiveness in the given state. The pseudocode for the PPO-Clip algorithm is provided as follows: 1. The policy and value function parameters are initialized. 80 2. In each iteration, a set of experiences (states, actions, rewards) is collected using the current policy. Then, the advantage is estimated using the value function. Subsequently, the policy parameters are updated using the objective function, and the value function parameters are updated using gradient descent. PPO, while effective in optimization tasks, faces challenges with hyperparameter sensitivity and maintaining exploration over time. This can lead to suboptimal solutions in complex environments, requiring extensive tuning of parameters like the clipping ratio (ϵ) to optimize performance. So, the hyperparameters are meticulously defined to ensure the optimum performance of the PPO model, as outlined in Table 4.3. Table 4.3 - PPO hyperparameters. Hyperparameters Values Actor network: Network structure Activation functions Learning rate Critic network: Network structure Activation functions Learning rate Batch Size Discount Factor Clipping Factor Experience Horizon Generalized Advantage Estimation (GAE) factor Entropy loss weight 35; 45; 90; 45; 25 ReLU; ReLU; ReLU; SoftMax 1𝑒 − 4 35; 45; 90; 45; 1 ReLU; ReLU; ReLU; Linear 8𝑒 − 3 64 0.95 0.02 256 0.95 0.01 Figure 4.3 represents the designed block diagram in MATLAB synchronized with the above-mentioned heat transfer simulation tool as the environment, used for both RL models. 81 Figure 4.3: MATLAB Simulink Block Diagram designed for the DRL model. 4.4 Results and discussion 4.4.1 Single-heater optimization with single-agent RL To assess the efficiency of PPO and DQN algorithms in the simulated thermoforming control problem, a processing scenario was configured with a single heater as the base case. The algorithms were compared under the following distinct conditions: • Fixed target temperature (50 ℃) o Large action space (9 possible actions) o Small action space (3 possible actions) • Random variable target temperature via small action space (50 ± 5℃) o Uniformly distributed and constantly changing in every episode. Given the practical operating power range of the ceramic heaters in the experimental setup (150 W to 500 W), two action (search) spaces/scenarios were assumed: small and large spaces. The small space included three modes: 0 W (heater off), 150 W (minimum power), and 500 W 82 (maximum power). The large space had nine modes, represented as {0, 150, 200, 250, 300, 350, 400, 450, 500}. Figure 4.4a and b illustrate DQN's performance in optimizing heater settings, with cumulative rewards over 5000 training episodes for 9 and 3 action spaces. Similarly, Figure 4.5a and b show results for the PPO agent, with the 3-action space configuration proving more robust, consistently converging within around 500 episodes compared to DQN. Testing a random variable target temperature (50±5℃) with smaller action space (comparing Figure 4.4c and Figure 4.5c) demonstrate that PPO was consistently outperforming DQN in terms of convergence and stability. Q0 in Figure 4.4 and Figure 4.5 represents the expected cumulative reward at each episode based on prior state-action pairs and policies. a) b) c) Figure 4.4: DQN’s reward against episode number. a) 9 actions space, b) 3 action space, and c) random target temperature space. a) b) c) Figure 4.5: PPO’s reward against episode numbers. a) 9 actions space, b) 3 action space, and c) random target temperature space. 83 4.4.2 Multi-heater optimization with single-agent RL: column arrangement PPO proved to be, in the above case study, the more optimal choice for training the agent, based on the achieved performance on a single heater scenario. Therefore, the model was then opted to extend to control temperature distribution in an expanded region, namely the full left side of the thermoplastic sheet, now using five heaters. Figure 4.6a shows a hypothetical low-temperature desired profile (57 to 62℃, 𝜇 = 59.2, 𝜎 = 2.2), while Figure 4.6b displays a high-temperature desired profile (158 to 175℃, 𝜇 = 164.6, 𝜎 = 6.6); but assumed to be without randomness. To enhance the agent's performance, the refined action space included a complete deactivation (0 W) and a complete activation at maximum power (500 W). The PPO agent was trained for 2500 episodes, with each episode simulating the complete thermoforming process lasting 1000 seconds. As illustrated in Figure 4.7a the agent achieved a minimal error band (difference between the desired temperature distribution and the actual temperature distribution) of ±2℃ after 650 seconds into the process. Figure 4.7b provides a visual representation of the actions taken by the agent at each time step throughout the entire process. Finally, as depicted Figure 4.7c, the agent reached the peak cumulative reward after nearly 1200 episodes and. a) b) Figure 4.6: a) Low temperature profile, and b) high temperature profile selected as desired distribution. 84 a) b) c) Figure 4.7: a) Error, b) actions in the low desired temperature profile, and c) cumulative reward. Leveraging the success of the preceding training phase, the satisfactory results encouraged a subsequent trial involving a higher temperature profile with an increased standard deviation (Figure 4.6b). This was intentionally aimed to evaluate the agent's performance in a more challenging scenario. The latter training outcomes are depicted in Figure 4.8. In this case, the agent maintained an error margin of ±5℃ (𝑀 = 0.39℃ , 𝑆𝐷 = 3.02) across all five zones within 400 seconds of the process, as indicated in Figure 4.8b. Given the elevated temperature profile, the heaters exhibited a higher activation frequency than that observed in the low-temperature profile 85 (compare Figure 4.7b and Figure 4.8b). The trained agent was also able to achieve the maximum cumulative reward after approximately 950 episodes, reflected in Figure 4.8c. a) b) c) Figure 4.8: a) Cumulative reward, b) error, and c) actions in the high desired temperature profile. 4.4.3 Multi-heater optimization with single-agent RL: scatter arrangement To further check the performance of the RL agent, we then opt for a randomized scattered configuration, introducing a new target temperature profile with even higher standard deviation, that deals with zones 1, 6, 7, 13, 15, as indicated in Figure 4.9. 86 Figure 4.9: Desired temperature profile: scattered arrangement. After 500 episodes of training, the model achieved the error band of (−4 𝑡𝑜 + 7℃), almost same as the previous case, as reflected in Figure 4.10. Moreover, unlike the previous case, where the heaters were enabled mostly after 550 seconds into the process, here the activations are scattered throughout the entire process (Figure 4.10b), which is reasonable given the scattered arrangement of the zones, and that the radiation heat transfer for each zone is not significantly affecting the others, as opposed to the previous case. Lastly, the agent achieved the maximum cumulative reward after nearly 1300 episodes of training, somewhat higher than the previous two cases (Figure 4.10c). a) b) 87 c) Figure 4.10: a) Error, b) Actions in the high desired temperature profile, and c) Cumulative reward. 4.4.4 All-heater optimization with single-agent RL: full sheet The extended version of the previous model incorporates all fifteen heaters to control all thermoplastic sheet's zones. Unlike the earlier models, where promising results were achieved, this version struggled with stability. Although the reward plot shows convergence, as seen in Figure 4.11a, the extended model's results, depicted in Figure 4.11b indicate a substantial error range across all zones. In other words, the agent was unable to stabilize the temperature throughout the process and did not achieve an acceptable error margin (−18 to +27℃). Additionally, the training process was computationally intensive, taking nearly two days on a high-performance computer (10-core Xeon CPU, 3.70 GHz, 32.0 GB memory). 88 a) b) Figure 4.11: Retrained single agent to optimize the heating for all zones (full sheet). a) Cumulative reward, b) Error. This unsatisfactory outcome prompts an exploration of further optimization enhancements by extending the existing single-agent model to a multi-agent RL approach, aiming to attain a reduced error range, temperature stability, and reduce training time, which is the topic of the next chapter. 4.5 Summary This chapter explored the application of RL in thermoforming. The DQN and PPO algorithms were first employed and compared for optimizing a single heater power setting. PPO exhibited superior performance under varying test conditions, including different action spaces and desired temperature profiles. The model was then extended to control the full column of five heaters (across one side of the sheet), with PPO retrained for simultaneous optimization. Results showcased the DRL model achieving a ±2℃ error band within 600 seconds under a low-temperature profile, and −5 to +7℃ in a high-temperature profile, maintained after only 400 seconds. Training on a high-performance PC (10-core Xeon CPU, 3.70 GHz, 32.0 GB memory) 89 for the multiple heater scenario lasted almost five hours, four times longer than the single-heater scenario, indicating significant computational cost. In the scattered heater configuration, the RL agent achieved an error band of (-4 to +7℃) after 500 episodes, similar to previous cases. Heater activations were spread throughout the process, reflecting minimal interaction between zones. The agent reached maximum cumulative reward after 1300 episodes, slightly exceeding prior configurations. This emphasizes the need for future exploration into multi-agent DRL models, considering computational cost improvement and refined reward functions to reduce errors further. The current model could merely optimize the left side of the sheet’s temperature distribution (as proof of concept). Next chapter involves developing multi-agent RL models to cover the full sheet area, integrating techniques to also concurrently reduce energy consumption in the process. 90 Chapter 5: Integrating Transfer Learning with Multi-Agent Reinforcement Learning: How to Simultaneously Increase Modeling Efficiency and Energy Saving in the process? 5.1 Introduction In the results presented in Chapter 4, it was evident that a single RL agent struggled to optimize temperature distributions across all fifteen zones of the sheet during the thermoforming process. The large and unstable error range observed highlighted the limitations of this approach. To address these shortcomings, this chapter (fulfilling objective 4, per section 1.2) introduces a solution leveraging MARL (Multi-Agent Reinforcement Learning) techniques to overcome the challenges inherent in the single-agent RL framework. Focusing on PPO strategy which was found to be working more efficient (see section 4.4.1/Figure 4.4 and Figure 4.5) than DQN in Chapter 4, here its multi-agent extension will include modifications to the state and action spaces and significant alterations to the reward function. The revised reward function now incorporates terms for error minimization, stability, and energy consumption efficiency, providing a more practicable approach to optimizing the thermoforming process for real-world applications. Namely, a comparative analysis between single-agent RL and multi-agent RL, as well as between decentralized and centralized MARL systems, is first provided in this chapter (fulfilling Objective 4.1). These comparisons are conducted within the context of the same thermoforming process employed in the thesis. Next, the chapter will focus on strategies for managing potential uncertainties in the thermoforming process, by integrating the TL (Transfer Learning) within the MARL framework (called MATRL). This integration is designed to enhance training time as well as achieve a critical design goal concurrently: minimizing the energy consumption in the heating process. It is shown how the MATRL, using both ‘parameter-transfer’ and ‘sample-transfer’ TL 91 strategies, can become adaptable to variabilities inherent to real-world manufacturing practice. More specifically, the fluctuations in airflow, material properties, and initial temperature variations. The implementation of this approach ensures consistent product quality, minimizes energy consumption during the process, and eventually contributes to more efficient and reliable manufacturing processes (fulfilling Objective 4.2). Finally, some practical guidelines from the performed adaptivity examples are driven. 5.2 Methodology In the single-agent RL models discussed in Chapter 4, a smaller action space proved to be more effective than a larger one. Consequently, in this chapter, the action space has been further reduced to include only the minimum and maximum power levels {0W, 500W}. Additionally, the agents' observations have been updated to incorporate the previous actions that led to their current state. This added memory component enables the agents to consider the history of actions, allowing them to better correlate past decisions with their current state. 5.2.1 A modified reward function proposed The core of the reward function is to minimize the error in the thermoforming setting, which is the difference between the actual current temperature and the desired temperature. In the current study, the optimization problem is viewed as multi-objective, aiming for an optimal action in the next timestep that benefits all objectives. Multi-objective MDPs, or MO-MDPs, form the core framework in multi-objective optimization (MOO) [122]. Unlike traditional MDPs, MO-MDPs replace the single reward signal with a vector of reward signals, denoted as ?⃗? (𝑠𝑖 , 𝑎𝑖) = (𝑅1(𝑠𝑖 , 𝑎𝑖), … , 𝑅𝑚(𝑠𝑖 , 𝑎𝑖)), where 𝑚 represents the number of objectives and 𝑖 is the index of a particular state-action pair within the MDP. This setup allows for the consideration of multiple objectives simultaneously. However, the 92 presence of multiple components in the reward vector, each representing different objectives, often leads to conflicts during optimization. Consequently, there arises the need to learn trade-offs between these objectives, resulting in a diverse set of policies. This set of optimal policies, considering either individual objectives or combinations thereof, is known as the Pareto optimal set [127]. An integral aspect of multi-objective optimization involves the selection of actions, considering diverse objectives that may be complementary, conflicting, or independent. A frequent method is to utilize scalarization functions to score action selection strategies. Essentially, these functions streamline a multi-objective problem into a single objective by applying a function across the objectives, thereby generating a unified score for an action 𝑎 across various objectives 𝑜. This singular score can then be employed to assess the suitability of a specific action 𝑎. With these scores at hand, conventional action selection strategies from single-objective reinforcement learning, such as 𝜖 − 𝑔𝑟𝑒𝑒𝑑𝑦 and 𝐵𝑜𝑙𝑡𝑧𝑚𝑎𝑛𝑛, can be employed to determine the optimal action to choose. Many scalarization functions involve associating each objective (𝑜) with a weighted coefficient, allowing users to adjust the policy's nature by assigning varying emphasis to individual objectives. Within a multi-objective setting, this balance is parameterized by 𝑤𝑜 ∈ [0,1] for objective (𝑜), with ∑ 𝑤𝑜 = 1𝑚𝑜=1 . Among the most common functions is the linear scalarization function (LS) due to its simplicity and directness. Specifically, for a multi-objective solution, LS calculates a weighted sum across each objective function (i.e., 𝑓𝑜 with 𝑜 = 1,… ,𝑚) and their respective weights to derive the score of that solution, expressed as Equation (5.1) [127]: 𝐿𝑆(𝑥) = ∑𝑤𝑜. 𝑓𝑜(𝑥)𝑚𝑜=1 (5.1) The reward function for this thermoforming optimization problem, consists of the following three distinct objectives: 93 • Error minimization: The reward function, thoroughly explored in the previous chapter, emphasizes time-sensitive error minimization. A gradually increasing penalty coefficient starts at 0.5 and reaches 2, addressing the initial temperature misalignment between the heater and the sheet, which decreases as the process advances. The function prioritizes error magnitude, using the sum of squares and normalization by the number of heaters plus the maximum error, to ensure uniform error consideration, as reflected in Equation (5.2). 𝑅1 = 𝑝 ×(−∑(𝐸𝑟𝑟𝑜𝑟2) − 𝑚𝑎𝑥|𝐸𝑟𝑟𝑜𝑟|2)6 (5.2) • Energy saving: To enrich the reward function, energy conservation was incorporated as an additional consideration. Equation (5.3) details the calculation of this aspect, where 𝑁𝑎𝑐𝑡 denotes the number of heater activations per timestep, 𝑡𝑠 represents the timestep, and 𝑇ℎ indicates the total duration of heater activity. Aligning with the previous rationale, the objective is to minimize the total heater usage time by reducing the frequency of activations, thereby promoting energy efficiency, as expressed in Equation (5.4). 𝑇ℎ = ∑ (𝑁𝑎𝑐𝑡)𝑡1000𝑡=1 𝑠𝑒𝑐× 𝑡𝑠 (5.3) 𝑅2 = −(𝑡ℎ)2 (5.4) • Maintaining temperature stability: As the third objective, the focus is on preserving the stability of achieved results while simultaneously minimizing the standard deviation across all five error signals. This entails ensuring that once the error signals converge close to zero after a certain time into the process, 94 they remain stable and consistently minimized until the process concludes. This is achieved through Equation (5.5), where 𝜎 is the standard deviation and 𝜇 is the average error. 𝑅3 = −(𝜎(𝐸𝑟𝑟𝑜𝑟))2= −(∑ (𝐸𝑟𝑟𝑜𝑟𝑖 − 𝜇)215𝑖=115) (5.5) 5.2.2 Overall reward function Though adapting MDP formalization for multi-objective settings usually entails adjustments, here, competing objectives are consolidated into a single reward function, simplifying the problem. At each timestep, the total reward is the sum of the three local rewards mentioned earlier, as defined in Equation (5.6). Given its multi-objective nature, solving it can involve either a Pareto optimal solution or hierarchical RL optimization. 𝑅𝑇𝑜𝑡𝑎𝑙 = 𝑅1 + 𝑅2 + 𝑅3 (5.6) Given the characteristics of Pareto optimal solutions, wherein no single solution dominates others across all objectives, hierarchical RL is selected as the approach to address this challenge. This method involves decomposing the objectives of the current optimal problem into three distinct sub-problems, wherein the agent endeavors to address them based on their priority. Initially, the agent resolves the primary problem to achieve the minimum error margin across all zones of the sheet. Subsequently, the trained agent undergoes further training, with a focus on the second objective, energy conservation. Finally, the same agent receives additional training regarding the third objective, prioritizing the stability of the solution throughout the entire process while retaining and applying all prior experience to this final optimization problem. The advantage of the hierarchical RL approach lies in the ability to monitor the agent's progress at every step. 95 5.2.3 Multi-agent RL As discussed before, the heat transfer simulation tool forms the cornerstone for generating input data essential for training the reinforcement learning algorithms. The model was trained using two primary RL methods: PPO and DQN. In Chapter 4, the performances of these methods were compared under various conditions based on the cumulative rewards generated, where PPO was selected as the superior method in this case study [121]. Since PPO is extensively discussed in previous chapter, in this chapter its extension as a multi-agent RL model will be explored in detail. The structure of multi-agent reinforcement learning (as indicated in Figure 5.1) constitutes a stochastic game rooted in the MDP, denoted by the tuple ℳ < 𝒮,𝒜1, …𝒜𝑛, ℛ1, … , ℛ𝑛, 𝒫 >, where ′𝑛′ denotes the count of agents. Here, 𝒜 = 𝒜1 × …× 𝒜𝑛 represents the collective action space encompassing all agents, ℛ𝑛 ∶ 𝒮 × 𝒜 × 𝒮 → ℛ signifies the reward function specific to each agent, and 𝒫 ∶ 𝒮 × 𝒜 × 𝒮 → [0, 1] denotes the state transition function and it is assumed that the reward function is constrained within a bounded range [128], [129]. In multi-agent scenarios, state transitions occur as a consequence of the collective actions of all agents. Consequently, the rewards received by agents are influenced by the joint policy, denoted as ℋ ∶ 𝒮 × 𝒜 → [0, 1], with each agent receiving its respective reward, as indicated in Equation (5.7) [128]. ℛ𝑖𝐻 = 𝔼[ℛ𝑡+1|𝒮𝑡 = 𝑠,𝒜𝑡,𝑖 = 𝑎,ℋ] (5.7) 96 Figure 5.1 - Block diagram of the multi-agent RL optimization model. 97 So, the optimal action-value function, which follows the Bellman optimality equation is as Equation (5.8): 𝑄𝑖𝐻(𝑠, 𝑎) = 𝔼𝑖ℋ[ℛ𝑡+1 + 𝛾𝑄𝑖ℋ(𝒮𝑡+1, 𝒜𝑡+1)|𝒮𝑡 = 𝑠,𝒜𝑡 = 𝑎] (5.8) Stochastic games in particular, encompass fully cooperative, fully competitive, and mixed games. In fully cooperative settings, agents share an identical reward function (ℛ1 = ℛ2 = ⋯ =ℛ𝑛), aiming to maximize collective common rewards. Conversely, in fully competitive scenarios such as when 𝑛 = 2 and ℛ1 = −ℛ2, agents pursue opposing objectives. Mixed games feature agents with distinct yet correlated reward structures. Establishing effective general objectives for agents poses a significant challenge. Literature on learning objectives highlights two key aspects: stability, where agents converge to a consistent policy, and adaptability, ensuring sustained performance amid policy changes by other agents [128]. The distributed nature of multi-agent solutions, such as the speedup from parallel computation, brings inherent benefits to MARL. Additionally, MARL can gain further advantages through experience sharing mechanisms such as communication, teaching, and imitation. By sharing experiences, RL agents engaged in similar tasks can accelerate their learning process and enhance performance. For instance, agents can share information via communication [130], serve as teachers for other agents [129], or enable learners to observe and imitate their actions [131]. The ability of MARL to leverage parallel computation allows agents to efficiently utilize the decentralized nature of tasks, as explored in these studies [132], [133]. In scenarios where one or more agents fail, the remaining agents can take over the tasks, demonstrating MARL's robustness. Furthermore, the design of most multi-agent systems supports the seamless integration of new agents, which enhances scalability. 98 On the other hand, MARL presents unique challenges beyond those found in single-agent RL, such as the curse of dimensionality [129] and the exploration-exploitation trade-off. Additional challenges specific to MARL include defining a learning goal, dealing with the non-stationarity of the learning problem, and ensuring coordination among agents. The curse of dimensionality stems from the exponential increase in the discrete state-action space with the addition of state and action variables (dimensions). This issue is exacerbated in MARL due to the inclusion of each agent's variables in the joint state-action space, making it crucial to limit the number of agents. Agents need to explore to understand both the environment and the behaviors of other agents. However, excessive exploration can destabilize other agents, complicating the learning process for the exploring agent. Coordination is essential because the impact of an agent’s actions depends on the actions taken by other agents [129]. TL offers potential solutions to some of these MARL-specific challenges by enabling the transfer of knowledge across agents and tasks. This approach can help address issues like the curse of dimensionality, the exploration-exploitation trade-off, and the need for effective coordination. 5.2.3.1 Decentralized multi-agent PPO model The implementation of PPO in multi-agent settings closely resembles the structure of PPO in single-agent settings, whereby a policy and a value function 𝑉𝜙(𝑠) are learned; these functions are represented as two separate neural networks. 𝑉𝜙(𝑠) is utilized solely for variance reduction and is only employed during training; thus, it can incorporate additional global information not present in the agent’s local observation, enabling PPO in multi-agent domains to adhere to the Centralized Training and Decentralized Execution (CTDE) structure. For clarity, PPO with centralized value 99 function inputs is denoted as Multi-Agent PPO (MAPPO), while PPO with local inputs for both the policy and value function (decentralized) is referred to as Independent PPO (IPPO) [134]. IPPO presents a simplified adaptation of PPO tailored for multi-agent reinforcement learning scenarios, illustrated in Figure 5.2. Every agent adheres to identical PPO sampling and training procedures, rendering it a flexible benchmark across diverse MARL tasks. In contrast to alternative MARL methods, IPPO operates autonomously without necessitating information exchange among agents. The mathematical formulation of IPPO bears similarity to that of PPO. However, in MARL, agents commonly lack access to the global state, particularly in settings with partial observability. Consequently, 𝒐 is used to denote local observations, while 𝒔 represents the global state. This results in a rewriting of the mathematical formulation of PPO. Critic learning ensures that with each iteration, a better value function is obtained as reflected in Equation (5.9) [134]: 𝜙𝑘+1 = arg𝑚𝑖𝑛𝜙1|𝒟𝑘|𝑇∑ ∑(𝑉𝜙(𝑜𝑡) − ?̂?𝑡)2𝑇𝑡=0𝜏 ∈ 𝒟𝑘 (5.9) General Advantage Estimation (GAE), which assesses the effectiveness of current actions relative to the baseline critic value, is obtained through Equation (5.10): 𝐴𝑡 = ∑(𝛾𝜆)𝑙𝛿𝑡+𝑙𝑉∞𝑡=0 (5.10) The discount value, denoted as 𝛾, determines the extent to which future rewards are considered in RL algorithms. It ranges between 0 and 1, where a value closer to 1 implies more consideration for future rewards. The weight value of GAE, denoted as 𝜆, controls the balance between bias and variance in estimating advantages. It also ranges between 0 and 1, with a value 100 of 1 indicating higher emphasis on reducing variance but potentially increasing bias, and a value of 0 indicating the opposite. Moreover, policy learning involves computing the policy gradient using the estimated advantage to update the policy function via Equations (5.11) and (5.12): 𝐿(𝜃) = [1𝐵𝑛∑ ∑ 𝑚𝑖𝑛(𝑟𝜃,𝑖𝑘 𝒜𝑖𝑘, clip(𝑟𝜃,𝑖𝑘 , 1 − 𝜖, 1 + 𝜖) 𝒜𝑖𝑘)𝑛𝑘=1𝐵𝑖=1] (5.11) 𝑟𝜃,𝑖𝑘 =𝜋𝜃(𝑎𝑖𝑘|𝑜𝑖𝑘)𝜋𝜃𝑜𝑙𝑑(𝑎𝑖𝑘|𝑜𝑖𝑘) (5.12) Figure 5.2 – Schematic of a decentralized multi-agent PPO model. 5.2.3.2 Centralized multi-agent PPO model In the centralized approach the experience is shared among all the agents, as reflected in Figure 5.3. MAPPO employs two distinct neural networks: one for the actor, characterized by parameters 𝜃, and another for the value function, known as the critic, governed by parameters 𝜙. In scenarios where agents exhibit homogeneity, these networks can be shared among them. However, each agent also has the option to possess its individual set of actor and critic networks. For the sake of notation simplicity, we assume a shared setup for both actor and critic networks among all agents. The critic network, labeled as 𝑉𝜙, undertakes the mapping from state 𝒮 → ℝ, where the global state might be either agent-specific or agent-agnostic. Meanwhile, the actor network, denoted as 𝜋𝜃, translates agent observations 𝑜𝑡𝑎 into a categorical distribution across 101 actions within discrete action spaces, or into mean and standard deviation vectors within a Multivariate Gaussian Distribution, from which actions are subsequently sampled, in continuous action spaces. Actor network is trained aiming to maximize the following objective (Equation (5.13)): 𝐿(𝜃) = [1𝐵𝑛∑∑𝑚𝑖𝑛(𝑟𝜃,𝑖𝑘 𝒜𝑖𝑘, 𝑐𝑙𝑖𝑝(𝑟𝜃,𝑖𝑘 , 1 − 𝜖, 1 + 𝜖) 𝒜𝑖𝑘)𝑛𝑘=1𝐵𝑖=1+𝜎𝐵𝑛∑∑𝒮[𝜋𝜃(𝑜𝑖𝑘)]𝑛𝑘=1𝐵𝑖=1] (5.13) Where 𝐴𝑖𝑘 is calculated using the GAE approach, with 𝒮 representing the policy entropy, and 𝜎 standing for the entropy coefficient as hyperparameter. Moreover, as for the critic network, the loss function should be minimized as in Equation (5.14): 𝐿(𝜙) =1𝐵𝑛∑∑(𝑚𝑎𝑥 [(𝑉𝜙(𝑠𝑖𝑘) − ?̂?𝑖)2, (𝑐𝑙𝑖𝑝(𝑉𝜙(𝑠𝑖𝑘), 𝑉𝜙𝑜𝑙𝑑(𝑠𝑖𝑘) − 𝜖, 𝑉𝜙𝑜𝑙𝑑(𝑠𝑖𝑘) + 𝜖) − ?̂?𝑖)2])𝑛𝑘=1𝐵𝑖=1 (5.14) In the provided loss functions, ?̂?𝑖 represents the discounted reward-to-go. Here, 𝐵 denotes the batch size, while 𝑛 signifies the number of agents [129]. Figure 5.3 – Schematic of the developed centralized multi-agent PPO model. A more comprehensive summary and the pseudocode of the FCNN-based MAPPO is provided in the Table 5.1. In the following pseudocode, the augmented state ?̃?𝑡 is constructed to include not just the current observation 𝑜𝑡 and current state 𝑠𝑡 but also relevant historical 102 information such as previous states 𝑠𝑡−1 and previous actions 𝑎𝑡−1. This effectively compensates for the absence of a hidden state in FCNNs. 103 Table 5.1 – FCNN-based MAPPO algorithm and pseudocode. Algorithm FCNN-based Multi-Agent PPO (MAPPO) 1. Initialization Initialize 𝜽, the parameters for policy 𝝅, and 𝝓, the parameters for Critic 𝑽, using orthogonal initializations. Set the learning rate 𝜶. 2. Training Loop While 𝑠𝑡𝑒𝑝 ≤ 𝑠𝑡𝑒𝑝𝑚𝑎𝑥 , do: Initialize an empty data buffer 𝐷 = {}. for 𝑖 = 1 to 𝑏𝑎𝑡𝑐ℎ_𝑠𝑖𝑧𝑒, do: Initialize an empty list 𝜏 = [] to store trajectory data. for 𝑡 = 1 to 𝒯, do: for each agent 𝑎, do: Compute action probabilities 𝑝𝑡 using current observation 𝑜𝑡 and augmented state ?̃?𝑡, which includes relevant historical information: 𝑝𝑡 = 𝜋(𝑜𝑡, ?̃?𝑡; 𝜃) Sample an action 𝑎𝑡 from 𝑝𝑡. Compute value estimate 𝑣𝑡 using current augmented state ?̃?𝑡: 𝑣𝑡 = 𝑉(?̃?𝑡;𝜙) end for Execute actions 𝑎𝑡 = {𝑎𝑡1, 𝑎𝑡2, … , 𝑎𝑡𝑛} for all agents, observe reward 𝑟𝑡, next state 𝑠𝑡+1, and next observation 𝑜𝑡+1. Augment next state ?̃?𝑡+1 to include current state and historical information. Append the transition data to 𝜏: 𝜏+ = [𝑠𝑡 , 𝑜𝑡, ?̃?𝑡 , 𝑎𝑡 , 𝑟𝑡 , 𝑠𝑡+1, 𝑜𝑡+1, ?̃?𝑡+1] end for Compute advantage estimate 𝒜𝑡 via GAE over trajectory 𝜏. Compute reward-to-go 𝑅𝑡 for trajectory 𝜏 and normalize the rewards. Split trajectory 𝜏 into chunks of length 𝐿. for 𝑙 = 0, 1, …, 𝒯 /L do Add each chunk to the data buffer 𝐷: 𝐷 = 𝐷 ∪ {(𝜏[𝑙: 𝑙 + 𝑇],𝒜𝑡[𝑙: 𝑙 + 𝐿], 𝑅𝑡[𝑙: 𝑙 + 𝐿])} end for end for for 𝑚𝑖𝑛𝑖 − 𝑏𝑎𝑡𝑐ℎ 𝑘 = 1, . . . , 𝐾 do: Sample a random mini-batch 𝑏 from data buffer 𝐷 that includes all agent data. for each data chunk 𝑐 in the mini-batch 𝑏, do: ?̃?𝑡 to include both current and past information (i.e., 𝑎𝑡−1, 𝑠𝑡−1). Adam update on 𝜃 using loss function 𝐿(𝜃) from data in mini-batch 𝑏. Adam update on 𝜙 using loss function 𝐿(𝜙) from data in the mini-batch 𝑏. end for end for end while 104 As mentioned before, despite its acceptable performance in many optimization problems, MAPPO also has some challenges such as hyperparameter sensitivity and achieving the correct balance between exploration and exploitation. Additionally, MAPPO tends to become less exploratory over time, potentially leading to suboptimal solutions and difficulty navigating complex environments. Therefore, selecting appropriate hyperparameters such as the clipping ratio (𝜖) requires extensive tuning due to their significant impact on model performance. So, the hyperparameters are defined to ensure the optimum performance of the MAPPO model, as outlined in Table 5.2. Table 5.2 – Summary of the tuned MAPPO hyperparameters. Hyperparameters Values Actor network: Network structure Activation functions Learning rate Critic network: Network structure Activation functions Learning rate Batch Size Discount Factor Clipping Factor Experience Horizon Generalized Advantage Estimation (GAE) factor Entropy loss weight 35; 45; 90; 45; 25 ReLU; ReLU; ReLU; SoftMax 1𝑒 − 4 35; 45; 90; 45; 1 ReLU; ReLU; ReLU; Linear 8𝑒 − 3 64 0.97 0.07 100 0.95 0.01 5.2.3.3 Proposed multi-agent transfer reinforcement learning (MATRL) The fundamental concept underpinning TL is the extension of previously acquired knowledge from one task to enhance learning in another, typically related, task. Numerous considerations arise in TL methods: a) the distinctions between tasks, b) the necessity of task 105 mappings to establish connections between the source and target tasks, and c) the determination of what knowledge should be transferred [135]. When integrated with RL, tasks can exhibit varying state spaces, from fixed variables [136] to completely distinct state variables [137]. More sophisticated approaches extend this variability to include differences in both state and action spaces, as well as reward and transition functions [138]. These methods often depend on inter-task mappings to link source and target tasks, typically defined by functions 𝜒𝑆(𝑠): 𝑆𝑇𝑎𝑟𝑔𝑒𝑡 → 𝑆𝑠𝑜𝑢𝑟𝑐𝑒 and 𝜒𝐴(𝛼): 𝐴𝑇𝑎𝑟𝑔𝑒𝑡 → 𝐴𝑠𝑜𝑢𝑟𝑐𝑒 [138]. Such mappings can also be autonomously learned [139], [140]. Transferred knowledge can range from low-level tuples such as 〈𝑠, 𝑎, 𝑟, 𝑠′〉 [141], value functions [138], or policies, to more complex forms like rules [136], action subsets, or reward shaping [137]. Despite this, TL in MARL remains largely unexplored. A notable example from single-agent TL is the method incorporated by Madden and Howley [136], which uses a symbolic learner to extract rules from previously learned action value functions, subsequently guiding action selection in the target task. The basic concept of TL, are well-described in [142]. Here, the source domain (𝐷𝑠) is defined to contain prior knowledge (𝜅𝑠), while the target domain (𝐷𝑡) encompasses target knowledge (𝜅𝑡). The objective of TL is to generate an optimal policy 𝜋∗ for the agent by leveraging both expert knowledge 𝜅𝑠 from 𝐷𝑠 and target knowledge 𝜅𝑡 from 𝐷𝑡. This is expressed as Equation (5.15): 𝜋∗ = arg𝑚𝑎𝑥𝜋∈Π̂𝔼𝑠~𝑑,𝑎∼𝜋[𝑄𝐷𝜋(𝑠, 𝑎)] (5.15) Here, 𝜋 ∈ ?̂?(𝜅𝑠 ∼ 𝐷𝑠, 𝜅𝑡 ∼ 𝐷𝑡): 𝑆 → 𝐴 represents a function mapping states to actions for the target domain 𝐷𝑡, while 𝑄𝐷𝜋(𝑠, 𝑎) denotes the action-value function. Information from both 𝜅𝑡 and 𝜅𝑠 is assimilated by the agent, thereby enhancing its learning process in the target domain 𝐷𝑡 compared to scenarios where 𝜅𝑠 is not utilized [143]. Diverse approaches and methods have been 106 introduced to facilitate the learning task through knowledge transfer, depending on variations in the source and target learning structures and the nature of the knowledge being transferred. In this chapter, two distinct approaches for TL were employed: 1) Direct Parameter Transfer where parameters from a trained single agent are transferred to three agents for further fine-tuning, and 2) Sample Transfer, where parameters are shared (i.e. individual variations not allowed) between agents during retraining, as illustrated in Figure 5.4. Direct parameter transfer involves the scenario where the network parameters (or weights) learned from a previous task are directly transferred to initialize the training of a new task. Ideally, the agent can then fine-tune the pre-trained network, facilitating quick adaptation to the new task. As illustrated in Figure 5.4, weight transfer can encompass all network weights 𝜃𝑆 → 𝜃𝑇 or a subset of selected layers 𝜃𝑆𝑙 → 𝜃𝑇𝑙 , typically the initial layers 𝑙 of the network. While deep learning (DL) networks progressively learn abstract representations of states across layers, in DRL the network learns the policy within the same network, complicating the straightforward transfer of knowledge compared to supervised learning tasks. Weights from a previous task can be fine-tuned during training of the target task, either fully or only in upper layers, with lower layers remaining fixed. This aims to improve the initialization of network weights for the new task. Alternatively, training the network on sequential tasks allows observing knowledge transfer and retention, including the impact of prolonged periods without training [144]. A significant challenge in learning within real physical systems is the combination of high sampling costs and the sample inefficiency of RL [145]. To address this, a straightforward yet effective method involves reusing samples of interactions from a source task in a target task ({ℳ𝑆1 , . . . ,ℳ𝑆𝑖} → ℳ𝑇, see Figure 5.4). The objective is to determine how to leverage experiences < 𝑠, 𝑎, 𝑟, 𝑠′ > from the source task to expedite learning in the target task. One approach is to treat 107 the source task experiences as if they originated from the target task, integrating them into the learning process without differentiation or proportional adjustment. Employing sample transfer in this manner has the potential to drastically reduce the cost of generating new experiences and speed up the training process. However, careful processing of these samples is necessary to avoid negative transfer. Furthermore, reusing samples in this manner may compel the agent to glean insights from actions it might not explore when learning from scratch, potentially leading to the acquisition of more generalized policies and a smoother learning trajectory [144]. Figure 5.4 - Direct parameter transfer involves layer-wise transfer from source to target network. Sample transfer uses experiences from multiple agents in a shared replay memory, resulting in more stable training and generalized policies. The direct parameter transfer method was selected as the primary approach in this study due to several reasons rooted in the specific characteristics of the thermoforming process and the 108 nature of the MARL framework: 1) Efficiency and stability in multi-agent systems, direct parameter transfer utilizes pre-trained parameters from a single-agent RL model optimized for temperature control, enabling the MARL to start from a well-established baseline. This approach accelerates convergence by reducing the need for each agent to learn from scratch, thereby maintaining stability across all agents during initial training. 2) Reduction in training time, the primary objective is to achieve a substantial reduction in training time without compromising the quality of the learned policies. 3) Minimization of negative transfer, when using TL in MARL settings, there's a risk of negative transfer, where knowledge from a source task might not suit the target task, leading to poor performance. Direct parameter transfer was chosen over methods like instance-based transfer to minimize this risk by ensuring that the transferred parameters are directly relevant to the control tasks in the neighboring zones. On the other hand, sample transfer involves sharing experiences (samples) among multiple agents during retraining, allowing the system to benefit from shared knowledge across different agents. This method helps in reducing the overall training burden by reusing valuable experiences from related tasks [146]. 5.3 Results and discussion The results presented in this section are to evaluate various RL strategies in the thermoforming process, specifically examining adaptability in dynamic manufacturing settings. Chapter 4: established a baseline by assessing the performance of a single RL agent. In this chapter, we explore centralized MATRL framework using ‘direct parameter transfer’ strategy and will evaluate the adaptability of the MATRL model through ‘sample transfer’, focusing on the model’s resilience to uncertainties such as airflow fluctuations, material diversity, and temperature variations. 109 5.3.1 Single RL agent In Chapter 4, a single PPO agent was trained to control temperature distribution on the full left side of the thermoplastic sheet, using a given single column of five heaters. The target temperature distribution for these selected five zones were {160 ℃, 166 ℃, 172 ℃, 164 ℃, 158 ℃}, respectively (shown in left side of Figure 5.4). The PPO agent was trained for 2500 episodes, each simulating a complete thermoforming process of 1000 seconds (approximately 16 minutes). The actions (heater’s activation and deactivation) are indicated in Figure 4.10b. The agent reached the maximum cumulative reward after about 950 episodes (Figure 4.10c). It maintained an error margin of ±5℃ (𝑀 = 0.39℃ , 𝑆𝐷 = 3.02) across all five zones within the first 400 seconds of the process, as depicted in Figure 4.10a. Next, following section 5.2.1, in pursuit of further modeling applicability, adjustments were made to the base reward function (Equation (5.2)), now incorporating an energy-saving as well as stability terms (Equations (5.3), (5.5)). This was implemented through a hierarchical reinforcement learning approach, whereby the previously designed system, initially trained for error minimization, was retrained to effectively encompass the new optimization terms in the reward function. Upon comparing Figure 5.5a with the previous iteration of Figure 4.10b, it is evident that the overall number of activations has decreased for all heaters by 17.8% (from 1010 to 830 times). Furthermore, Figure 5.5b emphasizes the attained error range between -9℃ to +4℃ (𝑀 = −3℃ ,𝑆𝐷 = 4.3), reflecting almost similar outcomes to the preceding optimization case. In line with the earlier scenario, Figure 5.5c illustrates the cumulative reward, which was retrained based on the previous case, attaining the maximum possible reward almost after a few episodes into the training 110 process, i.e., the agent could adapt promptly to include the energy term and optimize the problem accordingly. a) b) c) Figure 5.5 - Retrained single agent (via hierarchical RL), to concurrently minimize the error and energy usage for 5 zones (left side of the sheet). a) Actions, b) Error, and c) Cumulative reward. 5.3.2 Centralized MATRL with direct parameters transfer Next, to enhance the efficiency of the training process and ensure convergence of the results indicated in Figure 4.11, the parameters obtained from the trained single agent (Figure 5.5c) were transferred to multiple agents (here, three per Figure 5.3). Illustrated in Figure 5.1, the proposed 111 block diagram reflects the collaborative training process involving the MARL with a centralized training policy. This configuration enabled the agents to share learned experiences in each episode, ultimately leading to a reduction in training time (from 2500 down to 300 episodes). To optimize the problem, per section 5.2.1 the reward function was formulated with consideration for the following sub-objectives: • Error minimization • Energy Saving • Temperature stabilization Additionally, for TL implementor, the training process entailed transferring hyperparameters of a pre-trained single agent, originally designed for a column of five heaters (left side), to the two neighboring columns with somewhat analogous learning tasks but different boundary conditions. This transfer occurred through a parallel training, allowing the sharing of experiences via this centralized approach, as explained in the methodology section 5.2.3.3. The following conditions were considered for the agents as the reference case: • Desired temperature profile (as in Figure 5.6): 𝑀 = 164.73, 𝑆𝐷 = 8.0042 • Upper Surface Convection Constant: ℎ0 = 5 𝑊𝑚2.𝐾 • Ambient Temperature: 𝑇𝑎0 = 25 ℃ • Sheet Thermal Conductivity: 𝑘0 = 0.18 𝑊𝑚.𝐾 112 Figure 5.6 – Assumed desired temperature profile on the thermoplastic sheet. Figure 5.7a illustrates the convergence of all three agents to an optimal policy in almost 100 episodes of MATRL training, with a drastic reduction in training time (under 300 episodes) as opposed to 2500 episodes for the single agent case (Figure 4.8). Figure 5.7b shows the selected actions throughout the entire process for each of the heaters separately. The total number of activations at full power (500W) across all heaters amounted to 199 times, each lasting for 10 seconds, equivalent to 276.4 Wh. This optimization leads to a maximum error range of 𝑀 =−0.73 ℃, 𝑆𝐷 = 7.73 , as presented in Figure 5.7c. a) b) 160166172164158168174180173167154159166159151 113 c) Figure 5.7 - Trained multi-agent transfer reinforcement learning (MATRL), optimizing error and energy usage, as well as stability, for all zones (full sheet). a) Cumulative rewards of the multiple agents, b) Actions, and c). Error distributions. It is essential to note that further reduction in this error range was unattainable due to the inherent physics of the heat transfer problem and the predefined desired temperature profile across all zones of the sheet. 5.3.3 Testing MATRL’s adaptability for other manufacturing scenarios/variations: with sample transfer mode The initial approach in section 5.3.2 for ‘parameter transfer’ TL involved training on a column of heaters and transferring the hyperparameters as initial guesses for learning other columns (zones), in order to configure the multi-agent system covering all the fifteen heaters/zones simultaneously. Upon the encouraging results of this approach (section 5.3.2) it was perceived that there may be further potential for application of the Multi-Agent Transfer Reinforcement Learning (MATRL) framework. Specifically, TL was utilized in transferring knowledge from the initially 114 trained MARL on the previous case (full sample), to possibly swiftly learn variations in the process (i.e. to adapt to the real-world varying test conditions), without retraining the entire model for each and every new manufacturing case. Such variations in practice would include e.g., ‘change in air flow rate’, affecting the convection heat transfer coefficient on top of the thermoplastic sheet [147]. Additionally, they may include ‘different sheet materials’ with diverse conductive heat coefficients, as well as different shop floor conditions of ‘high or low ambient (initial) temperature’ around the sheet. In the subsequent sections, each of these scenarios/examples are examined, by implementing sample transfer TL strategy from the reference case (pre-trained sample) of section 5.3.2 for each new case. 5.3.3.1 Example 1: Higher convective heat transfer coefficients (HTCs) Convection heat transfer occurs between any moving fluid/air and a solid surface (here, thermoplastic sheet). A high HTC can be considered when there is a high velocity flow of air around the sheet (e.g., in presence of fan) as opposed to low HTC, which typically occurs due to stagnant air. Normally the free convection rates (for the stagnant air) are ranging from 2.5 −25 𝑾𝒎𝟐×𝑲 [147]. As the first adaptability assessment, two relatively higher convective HTCs were considered (ℎ1 = 7.5𝑊𝑚2.𝐾 and ℎ2 = 15𝑊𝑚2.𝐾 respectively), as if there is a flow of air on top of the thermoplastic sheet with various rates (a typical varying condition in thermoforming), resulting in elevated HTC values compared to the reference case (ℎ = 5𝑊𝑚2.𝐾), shown in Figure 5.8. Figure 5.8 - Low (𝒉𝟏) and high (𝒉𝟐) convective heat transfer coefficient between heaters and sheet. 115 When comparing the reference case (ℎ = 5𝑊𝑚2.𝐾) with case 1 (where ℎ1 = 7.5𝑊𝑚2.𝐾), it is evident that fluctuations appear in the MATRL error plot (Figure 5.9), as one would anticipate. These fluctuations stem from the higher HTC, leading to sudden temperature variations. Based on Figure 5.9 the agents effectively minimized the overall error within about 580 seconds into the process, maintaining an almost-stable state until the process concluded. The average error range achieved was 𝑀 = 1.67℃ , 𝑆𝐷 = 5.44. Figure 5.9 – Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 1: 𝒉𝟏 = 𝟕. 𝟓𝑾𝒎𝟐.𝑲. The total number of activations across all heaters amounted to 274 times, equivalent to 380.56 Wh. This represents an increase in energy consumption compared to the base case (276.4 Wh), a reasonable observation considering the heightened instability of the overall temperature due to the higher HTC. Figure 5.10a and b respectively illustrate the total number of activations for each heater throughout the process and the activation distribution at specific times during the process. 116 a) b) Figure 5.10 – Case 1 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 1: 𝒉𝟏 = 𝟕. 𝟓𝑾𝒎𝟐.𝑲. In case 2 under this example, where ℎ2 = 15𝑊𝑚2.𝐾 is twice as large as ℎ₁, the temperature fluctuations intensified, resulting in a slower settling time of approximately 700 seconds, as depicted in Figure 5.11. However, it is evident that case 2 exhibited the fastest rise time among all cases, taking only about 280 seconds. This suggests that a higher HTC at the onset of the process could prove beneficial if achieving the desired temperature in the shortest possible time is paramount. However, to mitigate significant temperature fluctuations, the HTC should be reduced to a level where air movement is minimal, thus enabling the maintenance of the desired temperature without unnecessary energy consumption. This optimization case achieved an error margin of 𝑀 = 4.74℃ , 𝑆𝐷 = 4.71. 117 Figure 5.11 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 2: 𝒉𝟐 = 𝟏𝟓𝑾𝒎𝟐.𝑲. Figure 5.12a shows that, in case 2, the total number of activations increased by 64.9% compared to case 1 and 127% compared to the reference, totaling 452 activations (627.78 Wh). This rise reflects significant energy use to maintain the same temperature across all zones. Additionally, Figure 5.12b reveals a sharp increase in activation frequency toward the end of the process (after 700 seconds) compared to case 1, likely due to the challenges in stabilizing the temperature gradient with higher HTC. a) b) Figure 5.12 - Case 2 - a) Total number of activations for each heater. b) Its distribution throughout the process - case 2: 𝒉𝟐 = 𝟏𝟓𝑾𝒎𝟐.𝑲. 118 Case 2, the extreme scenario in example 1, was tested with and without TL to compare training time and achieved error distribution. MATRL reduced training time to 5.3 hours, a 29.3% improvement over the 7.5 hours needed for standard MARL (no TL involved). Additionally, the error distribution was significantly narrower with MATRL (𝑀: 4.74°𝐶, 𝜎 = 4.71) compared to MARL (𝑀 = 11.76℃ , 𝜎 = 10.81), demonstrating TL’s effectiveness in enhancing efficiency. 5.3.3.2 Example 2: Sheets with higher heat conduction properties The heat conductivity of a sheet refers to its capacity to conduct heat, here during the thermoforming process. Different materials vary in their ability to conduct heat, with some transferring it through conduction slowly across the sheet, while others possessing high thermal conductivities, as illustrated in Figure 5.13. The thermal conductivity can be studied using the Equation (5.16): 𝑞 = −𝑘∇𝑇 (5.16) Where, q is the heat flux, k is the thermal conductivity, and ∇𝑇 is the temperature gradient. Figure 5.13 - Low (𝒌𝟑) and high (𝒌𝟒) conductivity in the thermoplastic materials. General thermoplastic polymers typically exhibit a thermal conductivity ranging from approximately 0.04𝑊𝑚.𝐾 to 0.41𝑊𝑚.𝐾 [148]. Therefore, to encompass the extreme low and high levels of conductivity in the present study, the minimum and maximum levels were selected as 119 cases 3 and 4, respectively. These cases will be compared to the reference case of section 5.3.2, which featured a thermal conductivity of 𝑘0 = 0.18𝑊𝑚.𝐾. Figure 5.14 illustrates the error distribution across all heaters over time, with the average error and standard deviation depicted in the inner diagram. In case 3, where the sheet thermal conductivity is 𝑘3 = 0.4 𝑊𝑚.𝐾, the error range reaches to an average of 𝑀 = −0.29℃ with the standard deviation of 𝑆𝐷 = 7.26 towards the end of the process, with a settling time of approximately 480 seconds. Contrary to expectations, case 3 does not exhibit a faster settling time compared to the base case with 𝑘0 = 0.18 𝑊𝑚.𝐾. This unexpected result may be attributed to the overall low scale of thermal conductivities; even with a threefold increase in k, it may not significantly impact the temperature distribution across the sheet. Figure 5.14 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 3: 𝒌𝟑 = 𝟎. 𝟒𝑾𝒎.𝑲 . Figure 5.15a and b illustrate the actions of the RL agents, showing a total of 207 activations across all heaters, equivalent to 287.5 Wh. Although it was expected that higher thermal 120 conductivity would reduce energy consumption compared to the reference case, the results show similar consumption levels, likely due to the relatively low thermal conductivity scale. a) b) Figure 5.15 - Case 3 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 3: 𝒌𝟑 = 𝟎. 𝟒𝑾𝒎.𝑲. Case 4 with 𝑘4 = 0.04 𝑊𝑚.𝐾, shows relatively similar results to the case 3 in terms of both minimized error range obtained according to Figure 5.16 (𝑀 = 0.5℃ , 𝑆𝐷 = 6.81) and the energy consumption of 290.28 Wh. Also, the settling time is almost at 450 seconds. Figure 5.16 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 4: 𝒌𝟒 = 𝟎. 𝟎𝟒𝑾𝒎.𝑲. 121 Based on Figure 5.17a, the number of activations among the same heaters is nearly identical to that of case 3. However, as shown in Figure 5.17b, the heating pattern differs somewhat in case 4. Unlike the scattered activation seen across 1000 seconds in case 3, in case 4, heaters are predominantly active at the beginning of the process (before 220 seconds) and towards the end (after 600 seconds). This suggests that the sheet's low thermal conductivity causes the trapped heat in the sheet maintain a high temperature gradient during the 220 – 600 second period, requiring minimal additional heating. Conversely, towards the end of the process, when the sheet’s temperature is dropping, more heat is needed to sustain the temperature. Upon comparing cases 3 and 4, it becomes evident that the thermal conductivity level of the thermoplastic sheet does not exert a significant influence on the optimization problem. This observation aligns with the understanding that the scale of thermal conductivity in thermoplastic polymers is relatively small, typically less than 1𝑊𝑚.𝐾, especially when compared to that of other materials in the present study. a) b) Figure 5.17 - Case 4 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 4: 𝒌𝟒 = 𝟎. 𝟎𝟒𝑾𝒎.𝑲. 5.3.3.3 Example 3: Higher initial ambient temperatures As the last adaptability scenario exemplified, the ambient temperature is considered as potential varying factor, which could be different at the beginning of the process e.g. due to 122 seasonal changes in a shop floor. As another potential cause for such variation, considering the industrial practice in thermoforming at mass scale, usually, each process can be repeated with a short transition time from producing one unit to another. Hence, to avoid downtime the heaters are often kept active between the forming events, at somewhat lower rates to maintain the temperature and expedite the productions. Therefore, this will cause the ambient temperature to be higher at the beginning of a subsequent process. To test the adaptability of the MATRL trained model for such variations, two scenarios are considered (Figure 5.18) with initial ambient temperature levels of 50℃ (case 5) and 100℃ (case 6), which are compared to the base case of section 5.3.2 with 𝑇𝑎 = 25℃. Figure 5.18 - Low (𝑻𝒂𝟓) and high (𝑻𝒂𝟔) initial ambient temperature between the heaters and the sheet. In case 5, the ambient temperature is twice the room temperature, 𝑇𝑎5 = 50℃. As depicted in Figure 5.19, settling time is 450 seconds, and the average error and standard deviation achieved 𝑀 = 0 ℃ , 𝑆𝐷 = 5.91. The MATRL agents were able to adapt to the new conditions, achieving comparable results to those of the reference case in terms of settling time and the minimum error band. 123 Figure 5.19 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 5: 𝑻𝒂𝟓 = 𝟓𝟎℃. Furthermore, Figure 5.20a illustrates a total of 164 activations across all heaters (227.78 Wh), signaling reduced energy consumption compared to previous cases, as anticipated. Additionally, Figure 5.20b displays the activation pattern for each heater throughout the process, resembling that of the reference case, although with differing total activations. a) b) Figure 5.20 - Case 5 - a) Total number of activations for each heater and b) Its distribution throughout the process - case 5: 𝑻𝒂𝟓 = 𝟓𝟎℃. 124 Similar to case 5 but with a higher ambient temperature of 𝑇𝑎6 = 100℃, the MATRL agents in case 6 achieved the narrowest error range (𝑀 = 1.99, 𝑆𝑇𝐷 = 3.04), with a settling time of 410 seconds, as shown in Figure 5.21. Figure 5.21 - Errors (average & standard deviation within internal diagram) of adaptive MATRL model - case 6: 𝑻𝒂𝟔 = 𝟏𝟎𝟎℃. Interestingly, the energy usage is significantly reduced to 113.89 Wh (Figure 5.22a), all occurring within the initial 120 seconds of the process, as shown in Figure 5.22b. This underscores the crucial role of energy conservation in conducting consecutive experiments. 125 a) b) Figure 5.22 - Case 6 - a) Total number of activations for each heater b) Its distribution throughout the process - case 6: 𝑻𝒂𝟔 = 𝟏𝟎𝟎℃. 5.3.4 Overall comparison of the trained models Table 5.3 provides the summary comparison of different agent configurations trained in terms of the number of RL agents, the usage of TL, number of controlled zones on the thermoplastic sheet, computational effort required for training the agents, the achieved average error and standard deviation, settling time, and energy usage. This table clearly highlights the performance and efficiency of single agents versus MARL and its adaptive variants under various process conditions. Based on the achieved results in MARL cases with adaptability analysis in section 5.3.3, it is evident that TL has boosted the performance of MARL by reducing the computational effort, achieved errors, and settling time in all the cases discussed under section 5.3.3 as expected, where the modified HTC was the most affected case. The inclusion of TL in the MARL framework resulted in shorter settling time and reduced energy consumption in the thermoforming process, primarily due to its ability to accelerate learning and optimize control strategies. By initializing agents with pre-trained parameters from a previously learned related task 126 (e.g., the optimal heating profile in the left zone on the sheet), TL enabled agents to bypass the extensive exploration phase that is typically required when re-training from scratch. This head start facilitated faster convergence to optimal control policies across the entire process, minimizing instances of temperature overshooting or oscillations. Additionally, the shared replay memory via sample transfer in the MATRL framework facilitated collaborative learning among agents, ensuring that heaters operated synergistically and achieved the target thermal profile more efficiently. These combined factors significantly reduced the settling time required for the system to stabilize. Energy consumption was also reduced as a result of TL’s ability to transfer knowledge from the single-agent scenario in the left zone, incorporating energy-efficient policies into the MARL framework. This inherited expertise ensured that agents maintained synchronized heating, avoiding unnecessary power usage during the initial learning phases. Furthermore, the MATRL framework enhanced agent coordination, effectively mitigating inefficiencies such as excessive compensation by individual heaters. By reducing oscillatory behavior, TL contributed to smoother control dynamics and minimized energy waste, ensuring higher thermal efficiency throughout the process. Therefore, the MATRL not only expedited system stabilization but also optimized energy usage, offering significant advantages over MARL without TL. 127 Table 5.3 - Comparison between single and multi-agent models, in terms of effectiveness and energy usage. RL Type Control Scenario Transfer Learning Computational Efficiency (Hours) Average Error & Standard Deviation Settling Time (S) to error range ±𝟏𝟎℃ Energy (Wh) Single Agent 5 zones* No ~ 5 𝑀 = 0.39℃ , 𝜎 = 3.02 375 140.28 5 zones No ~ 5 𝑀 = −3℃ , 𝜎 = 4.3 420 115.28 Full sheet No > 48 𝑀 = 12.31℃ , 𝜎 = 14.6 - 505.5 Multi-Agent Full sheet Yes ~ 7.5 𝑀 = −0.73℃ , 𝜎 = 7.73 420 276.4 𝐻𝑇𝐶 = 15𝑊𝑚2. 𝐾 Yes ~ 5.3 𝑀 = 4.74℃ , 𝜎 = 4.71 700 627. 8 No ~ 7.5 𝑀 = 6.76℃ , 𝜎 = 7.21 850 872.2 𝑘 = 0.4 𝑊𝑚.𝐾 Yes ~ 5.1 𝑀 = −0.29℃ , 𝜎 = 7.26 480 287.5 No ~ 7.5 𝑀 = 3.64℃ , 𝜎 = 7.44 620 409.7 𝑇𝑎 = 100℃ Yes ~ 5.2 𝑀 = 1.99℃ , 𝜎 = 3.4 350 113.9 No ~ 7.5 𝑀 = 3.84℃ , 𝜎 = 8.1 600 198.6 * Optimized for error minimization. Energy saving was not considered in this specific case. 5.4 Summary This chapter primarily aimed to develop and compare various RL models, focusing on single-agent versus multi-agent RL and decentralized versus centralized MARL in a smart thermoforming. It specifically explored how integrating TL with RL may improve training time, and adaptability of the process optimization model under varying manufacturing conditions like airflow fluctuations, material properties, and temperature variations. The conducted simulations and analyses yielded several significant findings in the domain of temperature control using RL agents, as follows. Single-Agent RL: Initially, a single RL agent trained with PPO showed an ability to control temperature distribution with an error margin of ±5℃ (M=0.39℃, SD=3.02) across pre-selected five zones of the sheet, i.e., using a column of heaters. Adjusting the reward function 128 reduced energy saving by 17.8%, while maintaining similar performance to the original five-zone case, where the reward function focused solely on error minimization. Extended Single-Agent Model: When expanded to the fifteen heaters, the single-agent model faced sizable stability issues, resulting in a substantial error range and a computationally intensive training process, lasting nearly two days to converge (using a PC equipped with parallel computing capabilities, 32 GB of RAM, and a 10-core Intel Xeon CPU). Despite achieving the reward function convergence, the temperature stabilization across zones was not acceptable. Multi-Agent Transfer Reinforcement Learning: By transferring hyperparameters from the single-agent to a MATRL, significant improvements were observed for the full (fifteen)-zone model. The training time was drastically reduced from 2500 to 300 episodes. The multi-agent system maintained a maximum error range of 𝑀 = −0.73℃ , 𝑆𝐷 = 7.73 while reducing energy consumption and achieving temperature stability more efficiently. This model (with three agents, corresponding to the three columns of the sheet zones) proved efficient for optimizing the heating of thermoforming process with all the heaters. Though, in the case of actual industrial thermoforming often with higher number of heaters (typically ~100), more RL agents will be required, depending on the setup configuration. Nonetheless, the proposed MATRL approach would still be expected to significantly reduce the training time. Adaptability Analysis: Varying manufacturing conditions were tested to assess the ‘sample-transfer’ based MATRL's adaptivity: - Convective Heat Transfer Coefficient (HTC): Higher heat transfer coefficients led to increased temperature fluctuations, with the lower coefficient resulting in an average error of 1.67°C and energy consumption of 380.56 Wh. Doubling the coefficient caused a greater error of 4.74°C and higher energy usage of 627.78 Wh. This 129 demonstrates a trade-off, where more precise temperature control can compromise energy efficiency. - Thermal Conductivity: Different levels of thermal conductivity showed minimal effects on energy consumption and temperature error. With the higher conductivity, the error was -0.29°C and energy use was 287.5 Wh, while reducing conductivity slightly increased the error to 0.5°C with 290.28 Wh consumed. This suggested that variations within the lower range of thermal conductivities typical for thermoplastics may not significantly impact the process optimization efforts. - Initial Ambient Temperature: Higher starting ambient temperatures resulted in quicker stabilization and reduced energy consumption. With an initial temperature of 50°C, the average error was around 0°C and energy consumption was 227.78 Wh, whereas at 100°C, the error increased to 1.99°C but with significantly lower energy use of 113.89 Wh, demonstrating considerable energy savings. 130 Chapter 6: Conclusion and Future Work This chapter provides a collective summary of the key findings and conclusions drawn from the thesis. It highlights the significance of the research outcomes and their contribution to the existing body of knowledge within the field. Additionally, the limitations inherent in the research are discussed, offering a transparent evaluation of the study’s scope. Finally, this chapter outlines a number of promising directions for potential future research in the field. 6.1 Summary of the main findings Chapter 3 explored the integration of XR technologies, specifically VR and MR, into the training of operators in thermoforming manufacturing processes. The study aimed to compare the effectiveness of MR and VR environments in enhancing training outcomes. A DT of the thermoforming process was created, incorporating real-time heat transfer simulations. Using Meta Quest 2 for VR and Microsoft HoloLens 2 for MR, along with a traditional video training method (DVT) for comparison, a user study with 26 participants evaluated the training tools based on criteria such as usefulness, trust, reliability, user satisfaction, and training effectiveness. The results indicated a strong preference for MR over VR and DVT, with MR providing greater user satisfaction, higher reliability, and training effectiveness. Specifically, MR was rated 2.2 times higher than VR and 5.56 times higher than DVT in overall user preference. Statistical analyses, including ANOVA and MCDM techniques, supported the conclusion that MR is the superior training tool in this context. The study also found that participants favored MR due to its enhanced presence experience and user-friendliness. Despite VR being the fastest training method, MR was consistently rated higher in overall performance, with a Cronbach's alpha of 0.922 indicating high internal consistency of the survey. The chapter highlighted the potential of MR in improving operator training through interactive capabilities and real-time feedback, while 131 acknowledging limitations such as the small sample size and the need for further research to enhance XR applications in manufacturing. Future research should focus on improving the user interface design, incorporating additional objective metrics, and exploring cognitive factors influencing training outcomes. Chapter 4 focused on the integration of AI, specifically DRL, into the thermoforming process of composites manufacturing. The study leveraged DRL to optimize heating parameters, aiming to achieve uniform temperature distribution on thermoplastic sheets. Two primary algorithms, DQN and PPO, were employed and compared. The results demonstrated that PPO consistently outperformed DQN in terms of convergence and stability, with PPO achieving optimal performance in fewer episodes. A heat transfer simulation tool was developed to serve as the environment for training the RL agents. The study first focused on a single heater scenario, then extended to multiple heaters controlling the full left side of the thermoplastic sheet. In the single heater scenario, the PPO algorithm proved to be more robust, consistently converging within approximately 500 episodes and handling a variable target temperature effectively. When extended to control five heaters, the PPO agent reached the maximum cumulative reward after about 1200 episodes and maintained a minimal error band of ±2℃ after 650 seconds in the low-temperature profile scenario. In a more challenging high-temperature profile scenario, the agent achieved an error band of ±7℃ within 400 seconds. Statistical analyses highlighted that PPO was the superior choice for optimizing the thermoforming process, demonstrating significant improvement over traditional methods. The chapter concludes by emphasizing the potential of DRL in enhancing manufacturing processes, noting the need for further research into multi-agent DRL models to optimize the entire 132 sheet area and reduce computational costs. The chapter's findings underscore the feasibility of using advanced AI techniques to improve the efficiency and quality of composites manufacturing. Chapter 5 expanded on the outcomes from chapter 4 and tried to explore the efficiency and energy conservation in thermoforming by employing TL in a hierarchical MARL framework. The integration of DRL techniques in manufacturing, particularly in thermoforming, aims to optimize heating parameters and achieve uniform temperature distribution on thermoplastic sheets. The study employed multi-agent RL approach combined with TL, comparing their performances and highlighting the adaptability and robustness of these methods under varying manufacturing conditions. Expanding on the idea from previous chapter, which was focused on training a single PPO agent to control the temperature distribution across five zones using a column of heaters, in this chapter incorporating energy-saving terms into the reward function led to a 17.8% reduction in heater activations without compromising performance. Extending the model to control fifteen heaters, however, posed stability issues, prompting the transition to a multi-agent system. By transferring parameters from the single-agent model to a centralized multi-agent framework, the training time was drastically reduced from 2500 to 300 episodes. The multi-agent system achieved a maximum error range of 𝑴 = −𝟎. 𝟕𝟑℃, 𝑺𝑫 =𝟕. 𝟕𝟑, and reduced energy consumption while maintaining temperature stability more efficiently. The study further explored the adaptability of the MATRL model under different conditions, including variations in convective heat transfer coefficient, thermal conductivity, and initial ambient temperature. The results underscored the adaptability of the MATRL model, in optimizing thermoforming processes under varying conditions. This study highlights the 133 potential for significant enhancements in industrial thermoforming processes, improving efficiency, adaptability, and energy conservation. Future research will investigate the combined effects of multiple variables and validate the simulation results in actual thermoforming setups to further enhance the applicability of the findings. 6.2 Contributions to knowledge • Development of XR Training Systems: Implemented spatial UI design according to UX guidelines for XR applications for MR and VR platforms, validated by Microsoft HoloLens 2 and Meta Quest 2 headsets, respectively, for enhancing the training of thermoforming operators. • Heat Transfer Simulation Integration in XR Platforms: Integrated real-time heat transfer simulations into the XR applications, allowing operators to monitor and control the manufacturing process interactively. • User Study and MCDM Evaluation: Validated the developed XR applications by conducting a user study involving 26 participants to assess the effectiveness of VR and MR training tools. • Integration of DRL for Thermoforming Optimization: Developed and implemented common DRL algorithms, specifically PPO and DQN, to optimize the temperature control in thermoforming processes. These algorithms were shown to effectively manage heating settings, resulting in significant energy savings and improved product quality. • Multi-Agent Reinforcement Learning (MARL) Implementation: Applied MARL frameworks to control multiple heaters in the thermoforming process. This approach improved the efficiency and scalability of the temperature control system, achieving better 134 energy conservation and consistent temperature distribution across the thermoplastic sheets. • Hybrid DRL and TL for Enhanced Adaptability: Leveraged TL to reduce the training time and computational resources required for the DRL models in the field. This approach enabled the adaptation of trained models to variable manufacturing conditions. • Energy Efficiency Improvements: Demonstrated the potential for energy savings in the thermoforming process by optimizing the heating setup through MATRL. The research highlighted the trade-offs between temperature stabilization and process duration, providing insights for further optimization. 6.3 Future work and recommendations While the current study demonstrated the effectiveness of XR for enhancing operator training and process understanding in the lab environment, one promising direction for future work may be expanding the application of present XR technologies, such as the Microsoft HoloLens and Oculus Quest, in smart manufacturing and training environments on-site. Future research could also explore the development of collaborative XR environments where multiple operators can interact with and learn from the manufacturing process simultaneously. This could include the creation of virtual workshops or training sessions where operators can practice and refine their skills in a controlled, yet interactive, environment. Additionally, incorporating gamification elements into these collaborative XR environments could further enhance engagement and learning outcomes. For instance, operators could participate in simulations that reward efficient problem-solving and process optimization, fostering a more engaging and effective training experience. 135 Building on the integration of XR technologies, future research could investigate the use of XR for predictive maintenance and real-time process monitoring. By leveraging XR in combination with ML algorithms, it would be possible to create predictive models that alert operators to potential issues before they occur, allowing for prognostic maintenance and reducing downtime. This approach could also enable real-time monitoring of key performance indicators (KPIs) within the manufacturing process, providing operators with immediate insights and enabling rapid adjustments to optimize performance. Exploring the integration of haptic feedback within XR environments could further enhance the realism and effectiveness of training and monitoring applications, providing tactile responses to virtual interactions. Additionally, the combination of XR technologies with DRL models offers an exciting frontier for future research. One feasible avenue could be the integration of XR-based real-time monitoring with DRL-driven adaptive control systems in manufacturing. For example, operators equipped with XR headsets could receive real-time visualizations of process data and predictive insights generated by DRL models. This setup would allow operators to make immediate, informed adjustments to the manufacturing process, improving responsiveness and efficiency. Additionally, implementing a feedback loop where DRL models continuously learn from the adjustments made by operators in the XR environment could further enhance process optimization and operator training. This holistic approach, combining XR, DRL, and TL, could revolutionize smart manufacturing, driving significant improvements in efficiency, productivity, and product quality. Future research should also consider the development of uncertainty-aware models for DRL in thermoforming. By integrating uncertainty quantification methods, such as Bayesian neural networks, these models could better handle variations in process conditions and material properties. This integration would enable more robust optimization strategies capable of adapting 136 to real-time changes and uncertainties in the manufacturing environment. For example, employing a Bayesian optimization framework to fine-tune process settings based on desired outputs could significantly enhance the reliability and performance of the thermoforming process, particularly for parts with complex geometries or unconventional specifications. This approach could provide a more comprehensive understanding of the trade-offs between different process parameters, leading to more informed decision-making. In the optimization phase with DRL frameworks, this research employed a discrete action space with two fixed energy input levels, 0 and 500 watts, to simplify the computational demands of training the RL model. This strategy, while computationally efficient, inherently limited the resolution of the input system and constrained the ability of the model to perform fine-grained energy adjustments within the range of 0 to 500 watts. In real-world manufacturing settings, this limitation could be easily mitigated through the application of Pulse Width Modulation (PWM) signals, which effectively simulate intermediate energy levels and allow the RL model to maintain its adaptability and dynamic response capabilities. However, the reliance on such a discrete action space represents an area for potential refinement. Future research could explore the integration of continuous action spaces to enhance the resolution of energy inputs while maintaining computational feasibility. By leveraging advancements in computational efficiency or hybrid approaches, this refinement could further optimize the RL model's performance and expand its applicability to more precise manufacturing scenarios, particularly those requiring higher resolution and nuanced control. Another exciting avenue for future research would be the application of lifelong learning techniques to enhance the adaptability and efficiency of the models. While this study has demonstrated the effectiveness of MARL with TL, future work could investigate lifelong learning 137 frameworks that allow models to continually learn and adapt from new data over time without forgetting previously acquired knowledge. This approach could be particularly beneficial in dynamic manufacturing environments where process conditions and material properties frequently change. For instance, developing a lifelong learning framework that enables the DRL models to update their knowledge base incrementally as new data becomes available could significantly improve their adaptability and efficiency. This continuous learning capability would ensure that the models remain relevant and effective in optimizing the manufacturing process among evolving conditions. In the area of DRL, future work could expand beyond the optimization of temperature to encompass multiple critical aspects of the thermoforming process. While this study has successfully optimized temperature control, the next iteration of DRL or MARL models could integrate the optimization of additional factors such as vacuum pressure, material feed rate, and cooling rates. By creating a comprehensive optimization framework, these advanced models could simultaneously manage and optimize various interdependent parameters, leading to a more holistic and efficient thermoforming process. This multi-faceted approach would not only enhance product quality and consistency but also improve overall process efficiency, reduce energy consumption, and minimize material waste. By leveraging the strengths of DRL and MARL to handle the complexities and interdependencies of multiple process variables, future research could drive significant advancements in smart manufacturing and thermoforming technologies. 138 Bibliography [1] A. Ustundag and E. Cevikcan, Industry 4.0: Managing The Digital Transformation, no. January. 2018. [2] Y. Wang and F. Zhang, Eds., Trends in Control and Decision-Making for Human–Robot Collaboration Systems. Cham: Springer International Publishing, 2017. [3] D. Hein et al., “A benchmark environment motivated by industrial control problems,” 2017 IEEE Symp. Ser. Comput. Intell. SSCI 2017 - Proc., vol. 2018-Janua, pp. 1–8, 2018, doi: 10.1109/SSCI.2017.8280935. [4] J. F. Arinez, Q. Chang, R. X. Gao, C. Xu, and J. Zhang, “Artificial Intelligence in Advanced Manufacturing: Current Status and Future Outlook,” J. Manuf. Sci. Eng. Trans. ASME, vol. 142, no. 11, pp. 1–16, 2020, doi: 10.1115/1.4047855. [5] C. Li, P. Zheng, Y. Yin, B. Wang, and L. Wang, “Deep reinforcement learning in smart manufacturing: A review and prospects,” CIRP J. Manuf. Sci. Technol., vol. 40, pp. 75–101, 2023, doi: 10.1016/j.cirpj.2022.11.003. [6] T. Rachman, “Towards Intelligent Industrial Co-robots,” Angewandte Chemie International Edition, 6(11), 951–952., 2018. https://bair.berkeley.edu/blog/2017/12/12/corobots/. [7] J. D. Reid, “Admissible modeling errors or modeling simplifications?,” Finite Elem. Anal. Des., vol. 29, no. 1, pp. 49–63, 1998, doi: 10.1016/S0168-874X(98)00014-6. [8] X. Xie, L. Ma, Q. Zhang, J. Zhang, W. Dong, and R. Chen, “Real-time Finite Element Analysis based on surrogate model,” J. Phys. Conf. Ser., vol. 2761, no. 1, 2024, doi: 10.1088/1742-6596/2761/1/012036. [9] S. Doolani et al., “A Review of Extended Reality (XR) Technologies for Manufacturing 139 Training,” Technologies, vol. 8, no. 4, p. 77, 2020, doi: 10.3390/technologies8040077. [10] N. Nievas, A. Pagès-Bernaus, F. Bonada, L. Echeverria, and X. Domingo, “Reinforcement Learning for Autonomous Process Control in Industry 4.0: Advantages and Challenges,” Appl. Artif. Intell., vol. 38, no. 1, pp. 0–53, 2024, doi: 10.1080/08839514.2024.2383101. [11] V. Abad-Alcaraz, M. Castilla, and J. D. Álvarez, “A Comparison of Classical and Reinforcement Learning-based Tuning Techniques for PI controllers,” IFAC-PapersOnLine, vol. 58, no. 7, pp. 180–185, 2024, doi: 10.1016/j.ifacol.2024.08.031. [12] C. Zhang et al., “Towards new-generation human-centric smart manufacturing in Industry 5.0: A systematic review,” Adv. Eng. Informatics, vol. 57, no. July, p. 102121, 2023, doi: 10.1016/j.aei.2023.102121. [13] J. E. Naranjo, D. G. Sanchez, A. Robalino-Lopez, P. Robalino-Lopez, A. Alarcon-Ortiz, and M. V. Garcia, “A scoping review on virtual reality-based industrial training,” Appl. Sci., vol. 10, no. 22, pp. 1–31, 2020, doi: 10.3390/app10228224. [14] A. de Giorgio, M. Romero, M. Onori, and L. Wang, “Human-machine Collaboration in Virtual Reality for Adaptive Production Engineering,” Procedia Manuf., vol. 11, no. June, pp. 1279–1287, 2017, doi: 10.1016/j.promfg.2017.07.255. [15] F. G. Pratticò and F. Lamberti, “Towards the adoption of virtual reality training systems for the self-tuition of industrial robot operators: A case study at KUKA,” Comput. Ind., vol. 129, 2021, doi: 10.1016/j.compind.2021.103446. [16] J. Leder, T. Horlitz, P. Puschmann, V. Wittstock, and A. Schütz, “Comparing immersive virtual reality and powerpoint as methods for delivering safety training: Impacts on risk perception, learning, and decision making,” Saf. Sci., vol. 111, no. May 2018, pp. 271–286, 2019, doi: 10.1016/j.ssci.2018.07.021. 140 [17] E. D. Ragan, D. A. Bowman, R. Kopper, C. Stinson, S. Scerbo, and R. P. McMahan, “Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task,” IEEE Trans. Vis. Comput. Graph., vol. 21, no. 7, pp. 794–807, 2015, doi: 10.1109/TVCG.2015.2403312. [18] D. Sportillo, G. Avveduto, F. Tecchia, and M. Carrozzino, “Training in VR: A Preliminary Study on Learning Assembly/Disassembly Sequences,” 2015, pp. 332–343. [19] M. Murcia López, “The effectiveness of training in virtual environments,” Dr. thesis, UCL (University Coll. London)., p. 157, 2018, [Online]. Available: https://discovery.ucl.ac.uk/id/eprint/10063411/. [20] Y. Vélaz, J. R. Arce, T. Gutiérrez, A. Lozano-Rodero, and A. Suescun, “The influence of interaction technology on the learning of assembly tasks using virtual reality,” J. Comput. Inf. Sci. Eng., vol. 14, no. 4, pp. 1–9, 2014, doi: 10.1115/1.4028588. [21] Y. Pan and A. Steed, “Avatar type affects performance of cognitive tasks in virtual reality,” Proc. ACM Symp. Virtual Real. Softw. Technol. VRST, 2019, doi: 10.1145/3359996.3364270. [22] B. Salah, M. H. Abidi, S. H. Mian, M. Krid, H. Alkhalefah, and A. Abdo, “Virtual reality-based engineering education to enhance manufacturing sustainability in industry 4.0,” Sustain., vol. 11, no. 5, pp. 1–19, 2019, doi: 10.3390/su11051477. [23] Z. H. Lai, W. Tao, M. C. Leu, and Z. Yin, “Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing,” J. Manuf. Syst., vol. 55, no. July 2019, pp. 69–81, 2020, doi: 10.1016/j.jmsy.2020.02.010. [24] G. Westerfield, A. Mitrovic, and M. Billinghurst, “Intelligent augmented reality training for motherboard assembly,” Int. J. Artif. Intell. Educ., vol. 25, no. 1, pp. 157–172, 2015, 141 doi: 10.1007/s40593-014-0032-x. [25] C. H. Chu and Y. L. Liu, “Augmented reality user interface design and experimental evaluation for human-robot collaborative assembly,” J. Manuf. Syst., vol. 68, no. April, pp. 313–324, 2023, doi: 10.1016/j.jmsy.2023.04.007. [26] C. H. Chu and C. H. Ko, “An experimental study on augmented reality assisted manual assembly with occluded components,” J. Manuf. Syst., vol. 61, no. April, pp. 685–695, 2021, doi: 10.1016/j.jmsy.2021.04.003. [27] M. Dalle Mura and G. Dini, “An augmented reality approach for supporting panel alignment in car body assembly,” J. Manuf. Syst., vol. 59, no. February, pp. 251–260, 2021, doi: 10.1016/j.jmsy.2021.03.004. [28] S. Webel, U. Bockholt, T. Engelke, N. Gavish, M. Olbrich, and C. Preusche, “An augmented reality training platform for assembly and maintenance skills,” Rob. Auton. Syst., vol. 61, no. 4, pp. 398–403, 2013, doi: 10.1016/j.robot.2012.09.013. [29] B. Schwald and B. DeLaval, “An Augmented Reality System for Training and Assistance to Maintenance in the Industrial Context,” 11th Int. Conf. Cent. Eur. Comput. Graph. Vis. Comput. Vis., pp. 425–432, 2003. [30] S. Werrlich, E. Eichstetter, K. Nitsche, and G. Notni, “An Overview of Evaluations Using Augmented Reality for Assembly Training Tasks,” Int. J. Comput. Inf. Eng., vol. 11, no. 10, pp. 1129–1135, 2017, [Online]. Available: https://www.researchgate.net/profile/Stefan_Werrlich/publication/324438098_An_Overview_of_Evaluations_Using_Augmented_Reality_for_Assembly_Training_Tasks/data/5acdb18e4585154f3f40e904/DOK-20170812-An-Overview-of-Evaluations-Using-Augmented-Reality-for-Ass. 142 [31] M. Heinz, S. Büttner, and C. Röcker, “Exploring training modes for industrial augmented reality learning,” ACM Int. Conf. Proceeding Ser., pp. 398–401, 2019, doi: 10.1145/3316782.3322753. [32] M. Hoover, “An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task,” Iowa State University, Digital Repository, Ames, 2018. [33] A. Muñoz, X. Mahiques, J. E. Solanes, A. Martí, L. Gracia, and J. Tornero, “Mixed reality-based user interface for quality control inspection of car body surfaces,” J. Manuf. Syst., vol. 53, no. August, pp. 75–92, 2019, doi: 10.1016/j.jmsy.2019.08.004. [34] D. K. Baroroh and C. H. Chu, “Human-centric production system simulation in mixed reality: An exemplary case of logistic facility design,” J. Manuf. Syst., vol. 65, no. March, pp. 146–157, 2022, doi: 10.1016/j.jmsy.2022.09.005. [35] N. Murauer, D. Schön, F. Müller, N. Pflanz, S. Günther, and M. Funk, “An analysis of language impact on augmented reality order picking training,” ACM Int. Conf. Proceeding Ser., pp. 351–357, 2018, doi: 10.1145/3197768.3201570. [36] M. Sirakaya and E. K. Cakmak, “Effects of augmented reality on student achievement and self-efficacy in vocational education and training,” Int. J. Res. Vocat. Educ. Train., vol. 5, no. 1, pp. 1–18, 2018, doi: 10.13152/IJRVET.5.1.1. [37] A. D. Kaplan, J. Cruit, M. Endsley, S. M. Beers, B. D. Sawyer, and P. A. Hancock, “The Effects of Virtual Reality, Augmented Reality, and Mixed Reality as Training Enhancement Methods: A Meta-Analysis,” Hum. Factors, vol. 63, no. 4, pp. 706–726, 2021, doi: 10.1177/0018720820904229. [38] S. Farra, E. Miller, N. Timm, and J. Schafer, “Improved Training for Disasters Using 3-D Virtual Reality Simulation,” West. J. Nurs. Res., vol. 35, no. 5, pp. 655–671, 2013, doi: 143 10.1177/0193945912471735. [39] S. Pedram, S. Palmisano, R. Skarbez, P. Perez, and M. Farrelly, “Investigating the process of mine rescuers’ safety training with immersive virtual reality: A structural equation modelling approach,” Comput. Educ., vol. 153, no. March, p. 103891, 2020, doi: 10.1016/j.compedu.2020.103891. [40] M. Gonzalez-Franco et al., “Immersive mixed reality for manufacturing training,” Front. Robot. AI, vol. 4, no. FEB, pp. 1–8, 2017, doi: 10.3389/frobt.2017.00003. [41] Q. Hernández, A. Badías, F. Chinesta, and E. Cueto, “Thermodynamics-informed neural networks for physically realistic mixed reality,” Comput. Methods Appl. Mech. Eng., vol. 407, p. 115912, 2023, doi: 10.1016/j.cma.2023.115912. [42] A. Badías, D. González, I. Alfaro, F. Chinesta, and E. Cueto, “Real-time interaction of virtual and physical objects in mixed reality applications,” Int. J. Numer. Methods Eng., vol. 121, no. 17, pp. 3849–3868, 2020, doi: 10.1002/nme.6385. [43] V. Weistroffer, F. Keith, A. Bisiaux, C. Andriot, and A. Lasnier, “Using Physics-Based Digital Twins and Extended Reality for the Safety and Ergonomics Evaluation of Cobotic Workstations,” Front. Virtual Real., vol. 3, no. February, pp. 1–18, 2022, doi: 10.3389/frvir.2022.781830. [44] I. Jalilvand, J. Jiyoung, H. Hosseinionari, R. Seethaler, B. Gopaluni, and A. S. Milani, “An Interactive Digital Twin of a Composite Manufacturing Process for Training Operators via Immersive Technology,” Commun. Comput. Inf. Sci., vol. 1836 CCIS, pp. 207–214, 2023, doi: 10.1007/978-3-031-36004-6_28. [45] J. Li, H. Deng, Y. Gao, A. Chen, Z. Song, and A. Hao, “Real-time Physics-based Interaction in Augmented Reality,” Proc. - 2023 IEEE Conf. Virtual Real. 3D User 144 Interfaces Abstr. Work. VRW 2023, pp. 573–574, 2023, doi: 10.1109/VRW58643.2023.00129. [46] T. Koduri, D. Bogdoll, S. Paudel, and G. Sholingar, “AUREATE: An Augmented Reality Test Environment for Realistic Simulations,” SAE Tech. Pap., vol. 2018-April, pp. 1–9, 2018, doi: 10.4271/2018-01-1080. [47] S. H.-W. Chuah, “Why and Who Will Adopt Extended Reality Technology? Literature Review, Synthesis, and Future Research Agenda,” SSRN Electron. J., no. 2018, 2019, doi: 10.2139/ssrn.3300469. [48] G. Evans, J. Miller, M. Iglesias Pena, A. MacAllister, and E. Winer, “Evaluating the Microsoft HoloLens through an augmented reality assembly application,” Degrad. Environ. Sensing, Process. Disp. 2017, vol. 10197, no. May 2017, p. 101970V, 2017, doi: 10.1117/12.2262626. [49] Z. Lv et al., “PreprintTouch-less Interactive Augmented Reality Game on Vision Based Wearable Device,” Springer, 2015, [Online]. Available: https://arxiv.org/pdf/1504.06359. [50] R. Hammady, M. Ma, C. Strathern, and M. Mohamad, “Design and development of a spatial mixed reality touring guide to the Egyptian museum,” Multimed. Tools Appl., vol. 79, no. 5–6, pp. 3465–3494, 2020, doi: 10.1007/s11042-019-08026-w. [51] A. Pose-Díez-De-la-lastra et al., “HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions,” Sensors, vol. 22, no. 13, 2022, doi: 10.3390/s22134915. [52] T. V. Nguyen, S. Kamma, V. Adari, T. Lesthaeghe, T. Boehnlein, and V. Kramb, “Mixed reality system for nondestructive evaluation training,” Virtual Real., vol. 25, no. 3, pp. 709–718, 2021, doi: 10.1007/s10055-020-00483-1. 145 [53] A. Pusch and F. Noël, Augmented Reality for Operator Training on Industrial Workplaces – Comparing the Microsoft HoloLens vs. Small and Big Screen Tactile Devices, vol. 565 IFIP. 2019. [54] S. Wu, L. Hou, G. (Kevin) Zhang, and H. Chen, “Real-time mixed reality-based visual warning for construction workforce safety,” Autom. Constr., vol. 139, no. December 2021, p. 104252, 2022, doi: 10.1016/j.autcon.2022.104252. [55] O. R. Ogunseiju, N. Gonsalves, A. A. Akanmu, D. Bairaktarova, D. A. Bowman, and F. Jazizadeh, “Mixed reality environment for learning sensing technology applications in Construction: A usability study,” Adv. Eng. Informatics, vol. 53, no. June 2021, p. 101637, 2022, doi: 10.1016/j.aei.2022.101637. [56] A. Vatankhah Barenji, X. Liu, H. Guo, and Z. Li, “A digital twin-driven approach towards smart manufacturing: reduced energy consumption for a robotic cell,” Int. J. Comput. Integr. Manuf., vol. 34, no. 7–8, pp. 844–859, 2021, doi: 10.1080/0951192X.2020.1775297. [57] H. Ghorbel et al., “SOON: Social Network of Machines to Optimize Task Scheduling in Smart Manufacturing,” IEEE Int. Symp. Pers. Indoor Mob. Radio Commun. PIMRC, vol. 2021-Septe, pp. 0–5, 2021, doi: 10.1109/PIMRC50174.2021.9569644. [58] W. Wu, Z. Huang, J. Zeng, and K. Fan, “A decision-making method for assembly sequence planning with dynamic resources,” Int. J. Prod. Res., vol. 60, no. 15, pp. 4797–4816, 2022, doi: 10.1080/00207543.2021.1937748. [59] Z. He, K. P. Tran, S. Thomassey, X. Zeng, J. Xu, and C. Yi, “Multi-objective optimization of the textile manufacturing process using deep-Q-network based multi-agent reinforcement learning,” J. Manuf. Syst., vol. 62, no. November 2020, pp. 939–949, 2022, 146 doi: 10.1016/j.jmsy.2021.03.017. [60] T. Kohne, H. Ranzau, N. Panten, and M. Weigold, “Comparative study of algorithms for optimized control of industrial energy supply systems,” Energy Informatics, vol. 3, no. Suppl 1, pp. 1–19, 2020, doi: 10.1186/s42162-020-00115-7. [61] X. Huang, S. H. Hong, M. Yu, Y. DIng, and J. Jiang, “Demand Response Management for Industrial Facilities: A Deep Reinforcement Learning Approach,” IEEE Access, vol. 7, pp. 82194–82205, 2019, doi: 10.1109/ACCESS.2019.2924030. [62] D. Schwung, A. Schwung, and S. X. Ding, “On-line Energy Optimization of Hybrid Production Systems Using Actor-Critic Reinforcement Learning,” 9th Int. Conf. Intell. Syst. 2018 Theory, Res. Innov. Appl. IS 2018 - Proc., pp. 147–154, 2018, doi: 10.1109/IS.2018.8710466. [63] D. Zhu, B. Yang, Y. Liu, Z. Wang, K. Ma, and X. Guan, “Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park,” Appl. Energy, vol. 311, no. 61731012, p. 118636, 2022, doi: 10.1016/j.apenergy.2022.118636. [64] A. Pfeiffer and M. Bugeja, Introducing the Concept of “Digital-Agent Signatures”: How SSI Can Be Expanded for the Needs of Industry 4.0., vol. 928. 2021. [65] M. Roesch, C. Linder, C. Bruckdorfer, A. Hohmann, and G. Reinhart, “Industrial load management using multi-agent reinforcement learning for rescheduling,” Proc. - 2019 2nd Int. Conf. Artif. Intell. Ind. AI4I 2019, pp. 99–102, 2019, doi: 10.1109/AI4I46381.2019.00033. [66] R. Lu, Y. C. Li, Y. Li, J. Jiang, and Y. Ding, “Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management,” Appl. Energy, vol. 276, no. July, p. 115473, 2020, doi: 10.1016/j.apenergy.2020.115473. 147 [67] J. Arroyo, C. Manna, F. Spiessens, and L. Helsen, “Reinforced model predictive control (RL-MPC) for building energy management,” Appl. Energy, vol. 309, no. November 2021, p. 118346, 2022, doi: 10.1016/j.apenergy.2021.118346. [68] S. Brandi, M. Fiorentini, and A. Capozzoli, “Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management,” Autom. Constr., vol. 135, no. April 2021, p. 104128, 2022, doi: 10.1016/j.autcon.2022.104128. [69] A. Gupta, Y. Badr, A. Negahban, and R. G. Qiu, “Energy-efficient heating control for smart buildings with deep reinforcement learning,” J. Build. Eng., vol. 34, no. March 2020, p. 101739, 2021, doi: 10.1016/j.jobe.2020.101739. [70] Y. Z. Wang, X. J. He, Y. Hua, Z. H. Chen, W. T. Wu, and Z. F. Zhou, “Closed-loop forced heat convection control using deep reinforcement learning,” Int. J. Heat Mass Transf., vol. 202, p. 123655, 2023, doi: 10.1016/j.ijheatmasstransfer.2022.123655. [71] E. Hachem, H. Ghraieb, J. Viquerat, A. Larcher, and P. Meliga, “Deep reinforcement learning for the control of conjugate heat transfer,” J. Comput. Phys., vol. 436, p. 110317, 2021, doi: 10.1016/j.jcp.2021.110317. [72] M. Römer, J. Bergers, F. Gabriel, and K. Dröder, “Temperature Control for Automated Tape Laying with Infrared Heaters Based on Reinforcement Learning,” Machines, vol. 10, no. 3, 2022, doi: 10.3390/machines10030164. [73] H. Zhao, B. Wang, H. Liu, H. Sun, Z. Pan, and Q. Guo, “Exploiting the Flexibility Inside Park-Level Commercial Buildings Considering Heat Transfer Time Delay: A Memory-Augmented Deep Reinforcement Learning Approach,” IEEE Trans. Sustain. Energy, vol. 13, no. 1, pp. 207–219, 2022, doi: 10.1109/TSTE.2021.3107439. 148 [74] M. Szarski and S. Chauhan, “Composite temperature profile and tooling optimization via Deep Reinforcement Learning,” Compos. Part A Appl. Sci. Manuf., vol. 142, no. December 2020, p. 106235, 2021, doi: 10.1016/j.compositesa.2020.106235. [75] Q. Xiao, C. Li, Y. Tang, and L. Li, “Meta-Reinforcement Learning of Machining Parameters for Energy-Efficient Process Control of Flexible Turning Operations,” IEEE Trans. Autom. Sci. Eng., vol. 18, no. 1, pp. 5–18, 2021, doi: 10.1109/TASE.2019.2924444. [76] J. Huang, J. Zhang, Q. Chang, and R. X. Gao, “Integrated process-system modelling and control through graph neural network and reinforcement learning,” CIRP Ann., vol. 70, no. 1, pp. 377–380, 2021, doi: 10.1016/j.cirp.2021.04.056. [77] J. Dornheim, L. Morand, S. Zeitvogel, T. Iraki, N. Link, and D. Helm, “Deep reinforcement learning methods for structure-guided processing path optimization,” J. Intell. Manuf., vol. 33, no. 1, pp. 333–352, 2022, doi: 10.1007/s10845-021-01805-z. [78] B. Li, H. Zhang, P. Ye, and J. Wang, “Trajectory smoothing method using reinforcement learning for computer numerical control machine tools,” Robot. Comput. Integr. Manuf., vol. 61, no. May 2019, 2020, doi: 10.1016/j.rcim.2019.101847. [79] Y. Zhang, Y. Li, and K. Xu, “Reinforcement learning–based tool orientation optimization for five-axis machining,” Int. J. Adv. Manuf. Technol., vol. 119, no. 11–12, pp. 7311–7326, 2022, doi: 10.1007/s00170-022-08668-5. [80] V. Samsonov, C. Enslin, H. G. Köpken, S. Baer, and D. Lütticke, “Using reinforcement learning for optimization of a workpiece clamping position in a machine tool,” ICEIS 2020 - Proc. 22nd Int. Conf. Enterp. Inf. Syst., vol. 1, no. Iceis, pp. 506–514, 2020, doi: 10.5220/0009354105060514. 149 [81] J. Schoop, H. A. Poonawala, D. Adeniji, and B. Clark, “AI-enabled dynamic finish machining optimization for sustained surface integrity,” Manuf. Lett., vol. 29, pp. 42–46, 2021, doi: 10.1016/j.mfglet.2021.04.002. [82] R. Gulde, M. Tuscher, A. Csiszar, O. Riedel, and A. Verl, “Reinforcement learning approach to vibration compensation for dynamic feed drive systems,” Proc. - 2019 2nd Int. Conf. Artif. Intell. Ind. AI4I 2019, pp. 26–29, 2019, doi: 10.1109/AI4I46381.2019.00015. [83] Y. Jiang, J. Chen, H. Zhou, J. Yang, P. Hu, and J. Wang, “Contour error modeling and compensation of CNC machining based on deep learning and reinforcement learning,” Int. J. Adv. Manuf. Technol., vol. 118, no. 1–2, pp. 551–570, 2022, doi: 10.1007/s00170-021-07895-6. [84] J. L. Throne, “Understanding Thermoforming,” Underst. Thermoforming, pp. I–XIII, 2008, doi: 10.3139/9783446418554.fm. [85] C. Zimmerling, C. Poppe, O. Stein, and L. Kärger, “Optimisation of manufacturing process parameters for variable component geometries using reinforcement learning,” Mater. Des., vol. 214, p. 110423, 2022, doi: 10.1016/j.matdes.2022.110423. [86] S. Stieber, L. Heber, C. Obertscheider, and W. Reif, “Control of Composite Manufacturing Processes Through Deep Reinforcement Learning,” Proc. - 22nd IEEE Int. Conf. Mach. Learn. Appl. ICMLA 2023, pp. 17–22, 2023, doi: 10.1109/ICMLA58977.2023.00011. [87] M. Priyadharshini, D. Balaji, V. Bhuvaneswari, L. Rajeshkumar, M. R. Sanjay, and S. Siengchin, “Fiber Reinforced Composite Manufacturing With the Aid of Artificial Intelligence – A State-of-the-Art Review,” Arch. Comput. Methods Eng., vol. 29, no. 7, 150 pp. 5511–5524, 2022, doi: 10.1007/s11831-022-09775-y. [88] A. Kusiak, “Smart manufacturing,” Int. J. Prod. Res., vol. 56, no. 1–2, pp. 508–517, 2018, doi: 10.1080/00207543.2017.1351644. [89] H. Hosseinionari and R. Seethaler, “The integration of Model Predictive Control and deep Reinforcement Learning for efficient thermal control in thermoforming processes,” J. Manuf. Process., vol. 115, no. September 2023, pp. 82–93, 2024, doi: 10.1016/j.jmapro.2024.01.085. [90] M. A. Farahani et al., “Time-series pattern recognition in Smart Manufacturing Systems: A literature review and ontology,” J. Manuf. Syst., vol. 69, no. May, pp. 208–241, 2023, doi: 10.1016/j.jmsy.2023.05.025. [91] G. Liu, W. Shen, L. Gao, and A. Kusiak, “Automated broad transfer learning for cross-domain fault diagnosis,” J. Manuf. Syst., vol. 66, no. July 2022, pp. 27–41, 2023, doi: 10.1016/j.jmsy.2022.11.003. [92] S. Ma et al., “A novel adaptive deep transfer learning method towards thermal error modeling of electric spindles under variable conditions,” J. Manuf. Syst., vol. 74, no. July 2023, pp. 112–128, 2024, doi: 10.1016/j.jmsy.2024.02.012. [93] X. Li, C. Zhang, X. Li, and W. Zhang, “Federated transfer learning in fault diagnosis under data privacy with target self-adaptation,” J. Manuf. Syst., vol. 68, no. May, pp. 523–535, 2023, doi: 10.1016/j.jmsy.2023.05.006. [94] P. Zhang, D. Gao, D. Hong, Y. Lu, Z. Wang, and Z. Liao, “Intelligent tool wear monitoring based on multi-channel hybrid information and deep transfer learning,” J. Manuf. Syst., vol. 69, no. May, pp. 31–47, 2023, doi: 10.1016/j.jmsy.2023.06.004. [95] M. Zhang, F. Tao, Y. Zuo, F. Xiang, L. Wang, and A. Y. C. Nee, “Top ten intelligent 151 algorithms towards smart manufacturing,” J. Manuf. Syst., vol. 71, no. August, pp. 158–171, 2023, doi: 10.1016/j.jmsy.2023.09.006. [96] Z. Raziei and M. Moghaddam, “Adaptable automation with modular deep reinforcement learning and policy transfer,” Eng. Appl. Artif. Intell., vol. 103, no. November 2020, p. 104296, 2021, doi: 10.1016/j.engappai.2021.104296. [97] S. Zheng, C. Gupta, and S. Serita, “Manufacturing Dispatching Using Reinforcement and Transfer Learning,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 11908 LNAI, pp. 655–671, 2020, doi: 10.1007/978-3-030-46133-1_39. [98] M. F. Alam, M. Shtein, K. Barton, and D. Hoelzle, “Reinforcement Learning Enabled Autonomous Manufacturing Using Transfer Learning and Probabilistic Reward Modeling,” IEEE Control Syst. Lett., vol. 7, pp. 508–513, 2023, doi: 10.1109/LCSYS.2022.3188014. [99] M. Klar et al., “Transferable multi-objective factory layout planning using simulation-based deep reinforcement learning,” J. Manuf. Syst., vol. 74, no. March 2023, pp. 487–511, 2024, doi: 10.1016/j.jmsy.2024.04.007. [100] W. de O. Leite, J. C. C. Rubio, F. M. Cabrera, A. Carrasco, and I. Hanafi, “Vacuum thermoforming process: An approach to modeling and optimization using artificial neural networks,” Polymers (Basel)., vol. 10, no. 2, 2018, doi: 10.3390/polym10020143. [101] J. P. Holman, Heat transfer, New York, 10th edition. McGraw-Hill, 2009. [102] M. I. Chy and P. O. November, “Estimation and Control of Plastic Temperature in Heating Phase of Thermoforming Process,” 2013. [103] M. M. I. Chy and B. Boulet, “A new method for estimation and control of temperature 152 profile over a sheet in thermoforming process,” Conf. Rec. - IAS Annu. Meet. (IEEE Ind. Appl. Soc., pp. 5–12, 2010, doi: 10.1109/IAS.2010.5615714. [104] F. Erchiqui and G. D. Ngoma, “Analyse comparative des méthodes de calcul des facteurs de formes pour des surfaces à contours rectilignes,” Int. J. Therm. Sci., vol. 46, no. 3, pp. 284–293, 2007, doi: 10.1016/j.ijthermalsci.2006.06.001. [105] H. Hosseinionari, M. Ramezankhani, R. Seethaler, and A. S. Milani, “Development of a Computationally Efficient Model of the Heating Phase in Thermoforming Process Based on the Experimental Radiation Pattern of Heaters,” 2023. [106] J. Brooke, “SUS: A ‘Quick and Dirty’ Usability Scale,” Usability Eval. Ind., no. July, pp. 207–212, 1995, doi: 10.1201/9781498710411-35. [107] L. J. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika, vol. 16, no. 3, pp. 297–334, 1951, doi: 10.1007/BF02310555. [108] K. Bilisik, “Pull-out properties of polyester woven fabrics: Effects of softening agent and interlacement on single and multiple yarn pull-out forces and analysis by statistical model,” Fibers Polym., vol. 12, no. 8, pp. 1106–1118, 2011, doi: 10.1007/s12221-011-1106-0. [109] B. IşIk and E. Ekici, “Experimental investigations of damage analysis in drilling of woven glass fiber-reinforced plastic composites,” Int. J. Adv. Manuf. Technol., vol. 49, no. 9–12, pp. 861–869, 2010, doi: 10.1007/s00170-009-2440-x. [110] D. C. A. S. U. Montgomery, D esign and Analysis of Experiments Ninth Edition. 2017. [111] C. Mircioiu and J. Atkinson, “A Comparison of Parametric and Non-Parametric Methods Applied to a Likert Scale,” Pharmacy, vol. 5, no. 4, p. 26, 2017, doi: 10.3390/pharmacy5020026. 153 [112] J. J. Tzeng, G. H., & Huang, Multiple attribute decision making: methods and applications. CRC press., 2011. [113] M. Rabbani, N. A. Saravi, H. Farrokhi-Asl, S. F. W. T. Lim, and Z. Tahaei, “Developing a sustainable supply chain optimization model for switchgrass-based bioenergy production: A case study,” J. Clean. Prod., vol. 200, pp. 827–843, Nov. 2018, doi: 10.1016/j.jclepro.2018.07.226. [114] J.-J. H. Gwo-Hshiung Tzeng, Multiple Attribute Decision Making: Methods and applications, vol. 4, no. 1. 2011. [115] O. Onajite and S. A. Oke, “The Application of WSM, WPM and WASPAS Multicriteria Methods for Optimum Operating Conditions Selection in Machining Operations,” J. Rekayasa Sist. Ind., vol. 10, no. 1, pp. 1–14, 2021, doi: 10.26593/jrsi.v10i1.4271.1-14. [116] E. K. Zavadskas, J. Antucheviciene, J. Saparauskas, and Z. Turskis, “MCDM methods WASPAS and MULTIMOORA: Verification of robustness of methods when assessing alternative solutions,” Econ. Comput. Econ. Cybern. Stud. Res., vol. 47, no. 2, 2013. [117] S. Opricovic and G. H. Tzeng, “Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS,” Eur. J. Oper. Res., vol. 156, no. 2, pp. 445–455, 2004, doi: 10.1016/S0377-2217(03)00020-1. [118] S. Opricovic and G. H. Tzeng, “Extended VIKOR method in comparison with outranking methods,” Eur. J. Oper. Res., vol. 178, no. 2, pp. 514–529, 2007, doi: 10.1016/j.ejor.2006.01.020. [119] P. Tučník and V. Bureš, “Experimental Evaluation of Suitability of Selected Multi-Criteria Decision-Making Methods for Large-Scale Agent-Based Simulations,” PLoS One, vol. 11, no. 11, p. e0165171, Nov. 2016, doi: 10.1371/journal.pone.0165171. 154 [120] A. S. Milani, A. Shanian, and C. El-Lahham, “A decision-based approach for measuring human behavioral resistance to organizational change in strategic planning,” Math. Comput. Model., vol. 48, no. 11–12, pp. 1765–1774, 2008, doi: 10.1016/j.mcm.2008.06.018. [121] I. Jalilvand, B. Gopaluni, and A. S. Milani, “Towards a Robust Deep Reinforcement Learning for Optimization of Heating Setup in Thermoforming Process,” 2024, [Online]. Available: https://iise.confex.com/iise/2024/meetingapp.cgi/Paper/7645. [122] M. A. Wiering and E. D. De Jong, “Computing optimal stationary policies for multi-objective Markov decision processes,” Proc. 2007 IEEE Symp. Approx. Dyn. Program. Reinf. Learn. ADPRL 2007, no. Adprl, pp. 158–165, 2007, doi: 10.1109/ADPRL.2007.368183. [123] A. T. Submitted et al., Physics-based Simulation and Reinforcement Learning Control of the Heating Phase in the Thermoforming Process, no. July. 2024. [124] V. Mnih et al., “Playing Atari with Deep Reinforcement Learning,” pp. 1–9, 2013, [Online]. Available: http://arxiv.org/abs/1312.5602. [125] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” pp. 1–12, 2017, [Online]. Available: http://arxiv.org/abs/1707.06347. [126] H. K. Lim, J. B. Kim, J. S. Heo, and Y. H. Han, “Federated reinforcement learning for training control policies on multiple IoT devices,” Sensors (Switzerland), vol. 20, no. 5, pp. 1–14, 2020, doi: 10.3390/s20051359. [127] K. Van Moffaert, M. M. Drugan, and A. Nowe, “Scalarized multi-objective reinforcement learning: Novel design techniques,” IEEE Symp. Adapt. Dyn. Program. Reinf. Learn. 155 ADPRL, pp. 191–199, 2013, doi: 10.1109/ADPRL.2013.6615007. [128] W. Du and S. Ding, “A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications,” Artif. Intell. Rev., vol. 54, no. 5, pp. 3215–3238, 2021, doi: 10.1007/s10462-020-09938-y. [129] L. Busoniu, R. Babuska, and B. De Schutter, “Chapter 7 Multi-agent Reinforcement Learning : An Overview,” Technology, vol. 38, no. 2, pp. 183–221, 2010. [130] M. Tan, “Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents,” Proc. 10th Int. Conf. Mach. Learn. ICML 1993, pp. 330–337, 1993, doi: 10.1016/b978-1-55860-307-3.50049-6. [131] B. Price and C. Boutilier, “Accelerating reinforcement learning through implicit imitation,” J. Artif. Intell. Res., vol. 19, pp. 569–629, 2003, doi: 10.1613/jair.898. [132] J. R. Kok, M. T. J. Spaan, and N. Vlassis, “Non-communicative multi-robot coordination in dynamic environments,” Rob. Auton. Syst., vol. 50, no. 2–3, pp. 99–114, 2005, doi: 10.1016/j.robot.2004.08.003. [133] C. Guestrin, M. Lagoudakis, and R. Parr, “Coordinated reinforcement learning,” Icml, pp. 227–234, 2002, [Online]. Available: http://www.cs.berkeley.edu/~russell/classes/cs294/f05/papers/guestrin+al-2002.pdf. [134] C. Yu et al., “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games,” Adv. Neural Inf. Process. Syst., vol. 35, no. NeurIPS, 2022. [135] G. Boutsioukis, I. Partalas, and I. Vlahavas, “Transfer learning in multi-agent reinforcement learning domains,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 7188 LNAI, no. January 2012, pp. 249–260, 2012, doi: 10.1007/978-3-642-29946-9_25. 156 [136] M. G. Madden and T. Howley, “Transfer of experience between reinforcement learning environments with progressive difficulty,” Artif. Intell. Rev., vol. 21, no. 3–4, pp. 375–398, 2004, doi: 10.1023/B:AIRE.0000036264.95672.64. [137] G. Konidaris and A. Barto, “Autonomous shaping: Knowledge transfer in reinforcement learning,” ACM Int. Conf. Proceeding Ser., vol. 148, pp. 489–496, 2006, doi: 10.1145/1143844.1143906. [138] M. E. Taylor, P. Stone, and Y. Liu, “Transfer learning via inter-task mappings for temporal difference learning,” J. Mach. Learn. Res., vol. 8, pp. 2125–2167, 2007. [139] V. Soni and S. Singh, “Using homomorphisms to transfer options across continuous reinforcement learning domains,” Proc. Natl. Conf. Artif. Intell., vol. 1, pp. 494–499, 2006. [140] M. E. Taylor, G. Kuhlmann, and P. Stone, “Autonomous transfer for reinforcement learning,” Proc. Int. Jt. Conf. Auton. Agents Multiagent Syst. AAMAS, vol. 1, no. May, pp. 278–285, 2008. [141] M. E. Taylor, N. K. Jong, and P. Stone, “Transferring instances for model-based reinforcement learning,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 5212 LNAI, no. PART 2, pp. 488–505, 2008, doi: 10.1007/978-3-540-87481-2_32. [142] H. Chen, H. Luo, B. Huang, B. Jiang, and O. Kaynak, “Transfer Learning-Motivated Intelligent Fault Diagnosis Designs: A Survey, Insights, and Perspectives,” IEEE Trans. Neural Networks Learn. Syst., vol. 35, no. 3, pp. 2969–2983, 2024, doi: 10.1109/TNNLS.2023.3290974. [143] B. Du et al., “Safe Adaptive Policy Transfer Reinforcement Learning for Distributed 157 Multiagent Control,” IEEE Trans. Neural Networks Learn. Syst., vol. PP, pp. 1–8, 2023, doi: 10.1109/TNNLS.2023.3326867. [144] R. Glatt, F. L. da Silva, R. A. da Costa Bianchi, and A. H. R. Costa, “A Study on Efficient Reinforcement Learning Through Knowledge Transfer,” Adapt. Learn. Optim., vol. 27, pp. 329–356, 2023, doi: 10.1007/978-3-031-11748-0_14. [145] G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of Real-World Reinforcement Learning,” 2019, [Online]. Available: http://arxiv.org/abs/1904.12901. [146] S. Niu, Y. Liu, J. Wang, and H. Song, “A Decade Survey of Transfer Learning (2010–2020),” IEEE Trans. Artif. Intell., vol. 1, no. 2, pp. 151–166, 2020, doi: 10.1109/TAI.2021.3054609. [147] W. M. Rohsenow, J. P. Hartnett, E. N. Ganic, and P. D. Richardson, Handbook of Heat Transfer Fundamentals (Second Edition), vol. 53, no. 1. 1986. [148] H. Chen et al., “Thermal conductivity of polymer-based composites: Fundamentals and applications,” Prog. Polym. Sci., vol. 59, pp. 41–85, 2016, doi: 10.1016/j.progpolymsci.2016.03.001. [149] A. Fonnet, N. Alves, N. Sousa, M. Guevara, and L. Magalhães, “Heritage BIM integration with mixed reality for building preventive maintenance,” EPCGI 2017 - 24th Encontro Port. Comput. Graf. e Interacao, vol. 2017-Janua, pp. 1–7, 2017, doi: 10.1109/EPCGI.2017.8124304. [150] J. R. Lewis and J. Sauro, “The factor structure of the system usability scale,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 5619 LNCS, pp. 94–103, 2009, doi: 10.1007/978-3-642-02806-9_12. 158 Appendices Appendix A : User interface design for MR application in Microsoft HoloLens 2: initial case study Here a spatial UI for the HoloLens application as part of a digital learning prototype was designed, enabling user interaction with a combination of virtual content and a real thermoforming set-up. Adhering to UX design principles, the prototype was evaluated via a user study to verify its usability. Thermoforming requires strict adherence to unique sequences of steps to avoid nonuniform thickness distribution (defect) in the final product. Therefore, the present case study is critical to explore MR’s potential in such advanced and high-risk manufacturing settings. To simulate the process, the system’s 3D heat transfer behaviors amongst digital assets such as heaters and sheets has been simulated (via numerical finite difference method; more details can be found in [105]) and integrated into the MR headset (Microsoft HoloLens 2). A.1 Methodology The proposed XR design workflow is demonstrated in Figure A.1. In the first step, 3D asset and UI elements are designed using the Mixed Reality Toolkit (MRTK), SolidWorks, and ProBuilder feature in Unity. In step 2, objects are imported to Unity 3D scene, and additional visual and audio effects are added. Next, in step 3, the XR development process is accrued out, where all the scenes are designed, and all the virtual elements are added. In steps 4 and 5, objects interactive behavior, animations, user input systems, and real-time 3D heat transfer simulation among certain elements (heaters and sheet) are developed in C# and deployed to the scene using Visual Studio and Unity. 159 Figure A.1. The present MR application design framework. A.2 UI and UX design The methodology for the UI design for HoloLens 2 encompassed a range of UX principles. These included cognitive metrics, physical characteristics, the design of ambient elements, and the provision of visual and auditory instructions. To avoid strain on the user’s neck, the spatial UI environment was designed such that the user can grab and move the virtual object to best fit in their comfortable range of height. Since the training procedure involved direct object manipulation, a critical design factor included the size of the object that needed to be manipulated as well as the distance from the viewer’s eyes. In terms of system characteristics (hardware and software) of the HoloLens, scenes that are heavily populated with 3D objects or overloaded with scripts that constantly update the objects render may cause lagging, and a frame rate drop from 300-320 to 20 frames per second (FPS). Higher frame rates produce more detailed visualizations when observing single virtual objects. Therefore, to avoid a low frame rate in the development stage, some modifications were made to reduce the resolution of the heat transfer simulation to 160 compensate for hardware limitations, which resulted in higher frame rates (up to 85 FPS) even in densely populated scenes. Visual and audible instructions are crucial for maintaining user engagement and attention in the application. To ensure that the operator clearly understands the virtual environment and its interactive elements, a 'tag-along' methodology [149] was employed to provide the user with visual prompts to complete specific tasks. Audio prompts and instructions were also used to compensate for the lack of visual information within certain unpopulated areas of the virtual space, which led to a higher quality of the UX. The main features of the UI/UX design in the MR developed app are as follows: • Practice mode in a separate scene to allow the users to get used to HoloLens 2 input system. • Voice and visual text instructions (cues) for each training step. • UI menus constantly follow the user to avoid limited FOV in HoloLens 2 and loss of content in the interface. • Remote object manipulation (resizing, moving, rotating) designed to help the user better access and interact with objects. • Spatial Audio (360) to help the user better navigate and find objects (with sound effects) which might not be visible in the user’s FOV. • Hand menu options allow the user to restart a training step, go to the main menu, or change scenes between practice mode and training. • Various sound/visual behaviours on interactable objects (pinch, touch, grab, hover) for a better immersion experience. 161 A.3 User study and data analysis For assessing the developed MR application’s usability, the System Usability Scale (SUS) [106] was used as a standard evaluation method. Once the user study was conducted, the correlations between pairs of the SUS questions (using the participants’ responses) were statistically analyzed via the Spearman rank-order correlation coefficients. To further analyze the reliability of the SUS and the survey, Cronbach's alpha [107], as a measure of internal (mean) consistency of the data, was also calculated (theoretically, the values can vary between 0 to 1). A.4 MR application Figure A.2 shows the start menu and practice modes of the app., each with various UI features allowing the user to explore and acquaint themselves with the input system of the HoloLens 2. Figure A.3 displays the steps of user training within the MR application. The application was simplified by displaying only important interactive features in each phase, and a control panel, which controls power adjustment and on/off switch functions. A screen also displayed the real-time temperature and power usage of each heater. Figure A.2. Screenshots from the “Main Menu” and “Practice Mode” scenes of the MR app. a) shows the available options once the application starts. b) Practice scene options, which allow the user to explore hand interactions (touch, pinch, grab, move, and scale) before testing the Training application. c) Near-Hand menu in Practice Mode. d) Near-Hand menu in Training Application. 162 Figure A.3. Features of the developed digital twin for training thermoforming operators. a) Control panel provides the info screen, showing the surface temperature distribution and the power consumption of heaters. b) Visualizing the heater’s surface temperature by hovering the hand over a selected heater. Once the heater is pinched, a visual heat bar guide is also shown on the side. c) Adjusting the heater's power range using the slider. d) Text instruction and the guiding arrow are facing the operator. A.5 User study Eight participants with different backgrounds were recruited to test the MR application and provide feedback. Figure A.4 depicts their responses on Likert Scale (from strongly disagree to strongly agree). Table A.1. also demonstrates the total score obtained from the SUS questionnaire. With the overall mean of 84.34 (out of 100) and the median of 86.25, the study suggests that the participants have indicated a high usability rate. 163 Figure A.4. Users’ responses distribution on the System Usability Scale (SUS), ranging from strongly disagree to strongly agree. Notice that some of the SUS questions are deliberately (by the standard) set in a negative form (i.e. in these cases ’strongly disagree’ is the top rate). Table A.1. The SUS score for different users of the MR application. User ID #1 #2 #3 #4 #5 #6 #7 #8 Survey points 40 23 33 31 35 37 37 34 SUS Score 100 57.5 82.5 77.5 87.5 92.5 92.5 85 After performing the statistical analysis of the responses, it was found that some questions within SUS were significantly correlated when using a significance (alpha) level of 0.05. Results of the Spearman analysis indicated that there were some significant positive correlations between the following pairs: questions 1 and 9 (r = 0.717), questions 3 and 9 (r = 0.737). Similar significances were found for some negative-form questions: questions 4 and 8 (r = 0.71), questions 4 and 10 (r = 0.732), and questions 8 and 10 (r = 0.802). Of course, it is possible that the observed correlations might not also be fully logical due to the small sample size. Hence, the Cronbach’s alfa mean 164 metric was next calculated to estimate an ‘overall internal consistency’ measure of the survey, without relying on a particular significance level. The Cronbach’s alfa mean was 0.879, indicating that the SUS questions and responses have been relatively consistent. Of particular note, among the questions, based on Figure A.4, the speed of learning via the MR app (question 7) was highly noted (all participants rated as ‘agree’ or ‘strongly agree’). During the qualitative assessment phase of data collection, participants were also given minimal instructions to operate the HoloLens application. However, some participants required further guidance to use the HoloLens HMD independently. This suggested that developing the necessary skillset through practice is crucial for users to control the virtual environment effectively. About physical aspects and variations in individuals' heights within the test group, potential safety concerns were observed when utilizing the HoloLens HMD. Another issue that emerged during the testing phase concerned the HoloLens2 direct manipulation input model. While general gestures are intuitive, the pinch gesture for grasping augmented objects has been judged non-intuitive because the fingers appear to pass through the objects themselves. A.6 Remarks This study investigated how a MR system could be developed and used to train thermoforming manufacturing operators. The user study (with 8 participants in a university lab environment) emphasized the significance of UI, UX, and usability in MR applications. A spatial user interface was developed for the MR application, enabling users to interact with virtual items in the physical surroundings. In addition, a real-time 3D simulation of the heat transfer phenomenon in the process was developed and included in the application to enhance the UX. The suggested architecture was tested using a Microsoft HoloLens 2 headset. Participants indicated that the app was usable, with an overall usability score of ~85 out of 100, based on the System Usability Scale (SUS) 165 questionnaire. Internal consistency of the survey data was also evaluated as satisfactory, suggesting that the SUS responses have been useable despite the limited sample size. Some limitations were observed during the development of the application, including the need for assistance from users to better utilize the HoloLens app, emphasizing the significance of prior practice with the HMD. Additionally, variations in individuals' heights raised potential safety concerns during testing. Moreover, the pinch motion was considered non-intuitive as the fingers appeared to pass through the augmented objects. However, this is directly related to the HoloLens input system and hand gestures, which cannot be modified in the current version. Future research options may also involve targeting participants with other relevant backgrounds, such as industrial technicians. Additionally, incorporating uncertainties into the thermoforming process during training may be beneficial for simulating real-world manufacturing scenarios. 166 Appendix B : User survey used in Chapter 3 • Trust: Each training technique's trust level was evaluated by participants using a 5-point Likert scale (1 – lowest score, 5 – highest score). Additionally, an open-ended question invited participants to provide insights on how to enhance user trust in each training mode. Q 1. I trust the system completely. Q 2. The system is completely predictable. Q 3. The system is completely safe. (Feels safe while working with) • Reliability: On a 5-point Likert scale (1 – Strongly disagree, 5 – Strongly agree), participants were asked to evaluate their reliance on each of the training techniques. Moreover, one open-ended question was posed to gather user’s comments on potential improvements. Q 4. The system provides the type of help I require to make my decision. Q 5. The system performs reliably. Q 6. I can rely on the system to function properly (While using). • User satisfaction: Participants were asked to rate their satisfaction with the training platforms on a 5-point Likert scale (1 – lowest score, 5 – highest score), mirroring the format of previous criteria questions. Also, an open-ended question requested ways to improve user satisfaction in each training mode. Q 7. I feel comfortable while/after using the system. Q 8. I would recommend others to use the system. Q 9. Feels natural to use the system. 167 • Training effectiveness: Similarly, another short survey comprising three questions asked the user to assess the training efficacy based on their experience with the platforms. Questions were designed on a 5-point scale (1 – Strongly disagree, 5 – Strongly agree). Q 10. I understand high-level details of the training with the system. Q 11. The system is suitable for complex training purposes. Q 12. The system is practical to follow step by step. • Usability and ease of use: Participants were queried to assess the usability and ease of use for each training method with responses on a scale of 1 to 5 (1 - strongly disagree, 5 - strongly agree). A few questions focused on the learnability, followed by a similar approach that was taken by Lewis et al. [150], suggesting that factor analysis of two independent SUS data sets included two inherent factors: "Usable" and "Learnable". Additionally, participants offered qualitative input through two open-ended questions in a post-experiment questionnaire, suggesting potential improvements for each training mode in terms of usability and ease of use. Q 13. The system would help me to be more productive. Q 14. The system would be useful over learning manufacturing-related topics. Q 15. The system would give me more control for learning manufacturing-related topics. Q 16. The system is easy to use. Q 17. The system is user friendly. Q 18. The system is simple to use. 168 • Preference: This part of survey consisted of two questions, one with a maximum of five responses (from “Do not prefer” to “Prefer very much”), to evaluate the overall preference of the user over the training methods, and one open-ended question for user’s comments. Q 19. My level of preference towards using the system. 169 Appendix C : Additional ANOVA Data and Checking its Assumptions Figure C.1. Normality check via Q-Q plots: ease of use (left) and satisfaction (right). Figure C.2. Normality check via Q-Q plots: trust (left) and reliability (right). Figure C.3. Normality check via Q-Q plots: training (left) and preference (right). 170 Figure C.4. Constant variance check: ease of use (left) and satisfaction (right). Figure C.5. Constant variance check: trust (left) and reliability (right). Figure C.6. Constant variance check: training (left) and preference (right). 171 Table C.1. ANOVA with blocks results for ease of use. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Training Platform 0.082 2 0.041 1.904 0.160 3.183 Users 0.299 25 0.012 0.559 0.942 1.727 Error 1.071 50 0.021 Total 1.452 77 Table 6.2. ANOVA with blocks results for satisfaction. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Rows 0.221 2 0.111 9.247 0.0004 3.183 Columns 0.668 25 0.027 2.231 0.0079 1.727 Error 0.598 50 0.012 Total 1.487 77 Table C.3. ANOVA with blocks results for trust. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Rows 0.133 2 0.066 6.102 0.004 3.183 Columns 0.776 25 0.031 2.857 0.001 1.727 Error 0.544 50 0.011 Total 1.453 77 Table 6.4. ANOVA with blocks results for reliability. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Rows 0.024 2 0.012 0.797 0.457 3.183 Columns 0.473 25 0.019 1.250 0.247 1.727 Error 0.757 50 0.015 Total 1.255 77 Table C.5. ANOVA with blocks results for training. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Rows 0.396 2 0.198 12.521 0.00004 3.183 Columns 0.517 25 0.021 1.307 0.207 1.727 Error 0.791 50 0.016 Total 1.704 77 Table 6.6. ANOVA with blocks results for preference. Source of Variation Sum of Squares df Mean Square F-value P-value F critical Rows 0.522 2 0.261 13.356 0.000 3.183 Columns 0.366 25 0.015 0.749 0.781 1.727 Error 0.978 50 0.020 Total 1.866 77
Thesis/Dissertation
eng
Mechanical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Milani, Abbas S.
Gopaluni, Bhushan
Extended reality-based training and adaptive machine learning-based optimization for thermoforming process
Text