Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Bond, Corey
Master of Science - MSc
University of British Columbia
Business process mining is a field that allows businesses to leverage data collected by business systems to gain insight into their processes. While process mining is a broad field, this thesis concentrates on process duration prediction based on information gained by reviewing historical data. Specifically, we investigate a provincial energy regulator's permit application process, and explore methods to predict the duration of the application, along with one of the sub-processes. The primary concern is developing an interpretable model to support the regulatory body in explaining the results of the predictions. Our goals are to solve the problem of how long a permit application might take to process, and support engagement with local First Nations communities by providing accurate estimates of the consultation process. These predictions are complicated by the highly complex processes being predicted, which include parallel execution of process tasks, causing significant data complexity. We solve the challenge of predicting a highly complex process by bridging the gap between traditional process mining and machine learning methods, while focusing on an interpretable and explainable model. We modify a state of the art random forest method called eXplainable Reasonably Randomized Forest (XRRF) by implementing a comprehensive data cleaning and engineering strategy. This modification also extends the feature selection method that provides interpretability of the results, and increases the hyperparameters available to improve the model's accuracy. In this thesis, we present the modified method, called Extended eXplainable Reasonably Randomized Forest (EXRRF). With this method, we were able to identify the factors that have the greatest impact on the duration of the consultation and application processes, while maintaining accuracy within 29-35% of the mean duration for each of these highly complex and variable processes.
Predicting remaining time for anenergy regulator’s highly parallelprocessbyCorey BondB.Sc. Hons., The University of British Columbia, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)June 2023© Corey Bond, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Predicting remaining time for an energy regulator’s highly parallel processsubmitted by Corey Bond in partial fulfilment of the requirements of thedegree of Master of SciencePatricia Lasserre, Irving K. Barber Faculty of ScienceSupervisorYves Lucet, Irving K. Barber Faculty of ScienceCo-SupervisorIfeoma Adaji, Irving K. Barber Faculty of ScienceSupervisory Committee MemberMohamed Shehata, Irving K. Barber Faculty of ScienceSupervisory Committee MemberChen Feng, Faculty of Applied ScienceUniversity ExamineriiAbstractBusiness process mining is a field that allows businesses to leverage datacollected by business systems to gain insight into their processes. Whileprocess mining is a broad field, this thesis concentrates on process durationprediction based on information gained by reviewing historical data. Specif-ically, we investigate a provincial energy regulator’s permit application pro-cess, and explore methods to predict the duration of the application, alongwith one of the sub-processes. The primary concern is developing an inter-pretable model to support the regulatory body in explaining the results ofthe predictions.Our goals are to solve the problem of how long a permit applicationmight take to process, and support engagement with local First Nationscommunities by providing accurate estimates of the consultation process.These predictions are complicated by the highly complex processes beingpredicted, which include parallel execution of process tasks, causing signifi-cant data complexity.We solve the challenge of predicting a highly complex process by bridgingthe gap between traditional process mining and machine learning methods,while focusing on an interpretable and explainable model. We modify astate of the art random forest method called eXplainable Reasonably Ran-domized Forest (XRRF) by implementing a comprehensive data cleaningand engineering strategy. This modification also extends the feature selec-tion method that provides interpretability of the results, and increases thehyperparameters available to improve the model’s accuracy.In this thesis, we present the modified method, called Extended eXplain-able Reasonably Randomized Forest (EXRRF). With this method, we wereable to identify the factors that have the greatest impact on the durationof the consultation and application processes, while maintaining accuracywithin 29-35% of the mean duration for each of these highly complex andvariable processes.iiiLay SummaryBusinesses often search for ways to improve processes such as processingtime. This usually leads to evaluation of available data to determine whereimprovements can be made, using a technique called process mining. Pro-cess mining can answer questions such as how long processes might take tocomplete. However, predictions related to the amount of time that a processwill take are usually difficult to explain.This thesis presents an extended version of an explainable random forestmodel, which we call Extended eXplainable Reasonably Randomized Forest(EXRRF). We use this method to predict how long a provincial energy regu-lator’s permit application process takes, and support engagement with localFirst Nations communities by accurately estimating how long a consultationprocess will take. The importance of the extended model is demonstrated byour ability to explain the predictions and the impact that the modificationshad to the information gained.ivTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . vList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xvDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 The British Columbia Energy Regulator (BC-ER) . . . . . . 11.2 Research goals . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 62.1 Process mining . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Neural networks . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Complications related to highly parallel processes . . . . . . . 122.4 Statistical methods . . . . . . . . . . . . . . . . . . . . . . . . 132.4.1 Feature engineering vs. feature encoding . . . . . . . . 152.4.2 Improvements to statistical methods . . . . . . . . . . 162.5 Prediction methods in oil and gas . . . . . . . . . . . . . . . . 18Chapter 3: Methodology . . . . . . . . . . . . . . . . . . . . . . 193.1 Background work . . . . . . . . . . . . . . . . . . . . . . . . . 19vTABLE OF CONTENTS3.2 eXplainable Reasonably Randomized Forest . . . . . . . . . . 223.2.1 Step 1: Learning performance . . . . . . . . . . . . . . 223.2.2 Step 2: Interpretability . . . . . . . . . . . . . . . . . 253.2.3 Step 3: Model accuracy . . . . . . . . . . . . . . . . . 303.2.4 Step 4: Explainability . . . . . . . . . . . . . . . . . . 313.3 Data cleaning . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.1 Data preparation . . . . . . . . . . . . . . . . . . . . . 333.3.2 Development of data tables and views . . . . . . . . . 363.3.3 Feature engineering and feature encoding . . . . . . . 403.4 Extended eXplainable Reasonably Randomized Forest (EXRRF) 493.4.1 Threshold for feature selection . . . . . . . . . . . . . 493.4.2 Range of features selected . . . . . . . . . . . . . . . . 513.4.3 Threshold for tree inclusion . . . . . . . . . . . . . . . 513.4.4 Implementation of EXRRF method . . . . . . . . . . 52Chapter 4: Predicting process duration . . . . . . . . . . . . . 554.1 Predicting FN Consultation duration . . . . . . . . . . . . . . 554.1.1 FN Consultation prediction results . . . . . . . . . . . 564.2 Predicting application duration . . . . . . . . . . . . . . . . . 604.2.1 Model 1: Predicting overall application duration . . . 614.2.2 Model 2: Predicting application revision duration . . . 644.3 Computational time . . . . . . . . . . . . . . . . . . . . . . . 70Chapter 5: Discussion . . . . . . . . . . . . . . . . . . . . . . . . 725.1 Validity of changes to XRRF method . . . . . . . . . . . . . . 725.1.1 Impact of changes to data cleaning . . . . . . . . . . . 725.1.2 Impact of additional feature selection thresholds . . . 745.1.3 Impact of broadening range of features included inCARTs . . . . . . . . . . . . . . . . . . . . . . . . . . 775.1.4 Validity of setting a minimum R2 threshold for treeinclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 785.2 Interpretability gained from EXRRF method . . . . . . . . . 795.3 Accuracy compared to traditional random forests . . . . . . . 795.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Chapter 6: Conclusion and Future Work . . . . . . . . . . . . . 826.1 Summary contribution . . . . . . . . . . . . . . . . . . . . . . 826.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88viTABLE OF CONTENTSAppendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95Appendix A: Supporting tables . . . . . . . . . . . . . . . . . . . . 96Appendix B: Supporting figures . . . . . . . . . . . . . . . . . . . . 98B.1 Generalizability tests . . . . . . . . . . . . . . . . . . . . . . . 100B.2 Residual distributions . . . . . . . . . . . . . . . . . . . . . . 104viiList of TablesTable 2.1 Simplified example of BC-ER case log, event log, andtask stream log. © C. Bond (2023) . . . . . . . . . . . 8Table 3.1 Results of preliminary predictions for application du-ration. © C. Bond (2023) . . . . . . . . . . . . . . . . 20Table 3.2 Results of preliminary predictions for FN Consultationduration. © C. Bond (2023) . . . . . . . . . . . . . . 21Table 3.3 Summary of the data tables used for this research. ©C. Bond (2023) . . . . . . . . . . . . . . . . . . . . . . 38Table 3.4 List of engineered and encoded features for FN Con-sultation and application data. © C. Bond (2023) . . 41Table 4.1 Count of FN Consultation features in Fn, PFs, DFs,BFs for each feature selection threshold. © C. Bond(2023) . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Table 4.2 Results of EXRRF predictions for FN Consultation.© C. Bond (2023) . . . . . . . . . . . . . . . . . . . . 57Table 4.3 Features classified as CF in the EXRRF FN Consul-tation prediction with Model 2. © C. Bond (2023) . . 59Table 4.4 Results of EXRRF predictions for application duration(Model 1). © C. Bond (2023) . . . . . . . . . . . . . . 63Table 4.5 Results of EXRRF predictions for application duration(Model 2). © C. Bond (2023) . . . . . . . . . . . . . . 64Table 4.6 Features classified as CF in the EXRRF applicationprediction with Model 2. © C. Bond (2023) . . . . . . 68Table 4.7 Count of application features in Fn, PFs, DFs, BFsfor each feature selection threshold. © C. Bond (2023) 69Table 5.1 Results of EXRRF predictions for application dura-tion (Model 2), applications prior to June 2021. © C.Bond (2023) . . . . . . . . . . . . . . . . . . . . . . . . 73viiiLIST OF TABLESTable 5.2 Feature-to-class, average feature-to-feature correlation,and counts of CF presence in tree DPs for numberedFN Consultation features. © C. Bond (2023) . . . . . 75Table 5.3 Comparison of EXRRF and random forest results forapplication duration prediction. © C. Bond (2023) . . 80Table 5.4 Comparison of EXRRF and random forest results forapplication duration prediction. © C. Bond (2023) . . 80Table A.1 Hyperparameters used for the EXRRF prediction ofthe FN Consultation duration. © C. Bond (2023) . . 96Table A.2 Hyperparameters used for the EXRRF prediction ofthe entire application duration (Model 1). © C. Bond(2023) . . . . . . . . . . . . . . . . . . . . . . . . . . . 96Table A.3 Hyperparameters used for the EXRRF prediction ofthe application duration (Model 2). © C. Bond (2023) 97Table A.4 Results of EXRRF predictions for application duration(Model 2), with flag for further revisions. © C. Bond(2023) . . . . . . . . . . . . . . . . . . . . . . . . . . . 97ixList of FiguresFigure 1.1 A simplified example of the BC-ER application re-view process. © C. Bond (2023) . . . . . . . . . . . . 4Figure 2.1 An approximate comparison of accuracy and inter-pretability of common models. © C. Bond (2023) . . 17Figure 3.1 Outline of plan to feed FN Consultation predictioninto application duration prediction. © C. Bond (2023) 21Figure 3.2 The four steps of the XRRF process. Figure 2 from© Jain and Jana [22], adapted with permission frompublisher. . . . . . . . . . . . . . . . . . . . . . . . . . 23Figure 3.3 The benefits gained from the interactions of the foursteps in the XRRF algorithm. Figure 1 from© Jainand Jana [22], adapted with permission from publisher. 24Figure 3.4 ‘MVER-based feature selection on a graph G’ Figure3 from © Jain and Jana [22], used with permissionfrom publisher. . . . . . . . . . . . . . . . . . . . . . . 29Figure 3.5 ‘MVNR-based feature selection on a graph G’ Figure4 from © Jain and Jana [22], used with permissionfrom publisher. . . . . . . . . . . . . . . . . . . . . . . 29Figure 3.6 A Venn diagram showing the intersections of featuresets obtained from the MST, MVER, and MVNRalgorithms. Figure 5 from © Jain and Jana [22],adapted with permission from publisher. . . . . . . . 30Figure 3.7 An example of the RRF algorithm. Figure 6 from© Jain and Jana [22], adapted with permission frompublisher. . . . . . . . . . . . . . . . . . . . . . . . . . 32Figure 3.8 The distribution of FN Consultation duration valuesin the BC-ER data. © C. Bond (2023) . . . . . . . . 34xLIST OF FIGURESFigure 3.9 The distribution of application duration, and idle timepresent during the application process. © C. Bond(2023) . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Figure 3.10 An example of the generalizability test. © C. Bond(2023) . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Figure 4.1 Illustration of FN Consultation feature distributionacross the Rs, Re, and Rn sets. © C. Bond (2023) . 58Figure 4.2 Residual distribution for FN Consultation durationprediction (FTpaper). © C. Bond (2023) . . . . . . . 60Figure 4.3 Explanation of the prediction window for predictingthe application duration during the final revision. ©C. Bond (2023) . . . . . . . . . . . . . . . . . . . . . 62Figure 4.4 Explanation of the prediction windows for predictingthe application duration during any revision. © C.Bond (2023) . . . . . . . . . . . . . . . . . . . . . . . 65Figure 4.5 Illustration of application feature distribution acrossthe Rs, Re, and Rn sets. © C. Bond (2023) . . . . . 67Figure 4.6 Residual distribution for application duration predic-tion (FTpaper). © C. Bond (2023) . . . . . . . . . . . 70Figure 5.1 FN Consultation duration prediction generalizabilitytest (FTpaper) © C. Bond (2023) . . . . . . . . . . . 78Figure 6.1 Application duration piecewise model proposal. ©C. Bond (2023) . . . . . . . . . . . . . . . . . . . . . 84Figure B.1 Tuning results for the EXRRF prediction of applica-tion duration. © C. Bond (2023) . . . . . . . . . . . 98Figure B.2 Results of the EBM model completed during prepara-tory work. © C. Bond (2023) . . . . . . . . . . . . . 99Figure B.3 FN Consultation duration prediction generalizabilitytest (FTpaper) © C. Bond (2023) . . . . . . . . . . . 100Figure B.4 FN Consultation duration prediction generalizabilitytest (FTharmonic) © C. Bond (2023) . . . . . . . . . . 100Figure B.5 FN Consultation duration prediction generalizabilitytest (FTmean−sd) © C. Bond (2023) . . . . . . . . . . 101Figure B.6 FN Consultation duration prediction generalizabilitytest (FTmin2) © C. Bond (2023) . . . . . . . . . . . . 101Figure B.7 Application duration prediction generalizability test(FTpaper) © C. Bond (2023) . . . . . . . . . . . . . . 102xiLIST OF FIGURESFigure B.8 Application duration prediction generalizability test(FTharmonic) © C. Bond (2023) . . . . . . . . . . . . 102Figure B.9 Application duration prediction generalizability test(FTmean−sd) © C. Bond (2023) . . . . . . . . . . . . 103Figure B.10 Application duration prediction generalizability test(FTmin2) © C. Bond (2023) . . . . . . . . . . . . . . 103Figure B.11 Residual distribution for FN Consultation durationprediction (FTharmonic). © C. Bond (2023) . . . . . 104Figure B.12 Residual distribution for FN Consultation durationprediction (FTmean−sd). © C. Bond (2023) . . . . . . 104Figure B.13 Residual distribution for FN Consultation durationprediction (FTmin2). © C. Bond (2023) . . . . . . . 105Figure B.14 Residual distribution for application duration predic-tion (FTharmonic). © C. Bond (2023) . . . . . . . . . 105Figure B.15 Residual distribution for application duration predic-tion (FTmean−sd). © C. Bond (2023) . . . . . . . . . 106Figure B.16 Residual distribution for application duration predic-tion (FTmin2). © C. Bond (2023) . . . . . . . . . . . 106xiiGlossary of NotationBC-ER British Columbia Energy Regulator, formerlyBCOGC. 1BCOGC British Columbia Oil and Gas Commission,renamed BC-ER in February 2023. 1BF Basic feature (set). 30BRFN Blueberry River First Nation. 37CART Classification and regression tree. 13CF Contributing features. 31DF Dominant feature (set). 30EBM Explainable boosting machine. 17EXRRF Extended eXplainable Reasonably Random-ized Forest. 3FCM Feature correlation matrix. 26FIPPA Freedom of Information and Protection of Pri-vacy Act. 33FN First Nation. 2GAM Generalized additive model. 17LSTM Long short term memory, a type of RNN. 10MAE Mean absolute error. 14MLP Multi-layer perceptron. 10MST Minimum spanning tree, also Algorithm 1.26, 27MSUS Modified stochastic universal sampling. 30MVC Minimum vertex cover. 26xiiiGlossary of NotationMVER Max-degree vertex edge remover, Algorithm2. 26, 27MVNR Max-degree vertex neighbour remover, Algo-rithm 3. 28NLP Natural language processing. 10PCA Principal component analysis. 16PF Pivotal feature (set). 30RMSE Root mean squared error. 14RNN Recurrent neural network. 10RRF Reasonably randomized forest, Algorithm 4.30, 31SGFL Statistically graph-based feature learning. 25XRRF eXplainable Reasonably Randomized Forest.3, 22xivAcknowledgementsMy hope is that everyone thanked here knows me well enough to under-stand that words cannot express my gratitude for all of the help that hasbeen provided to me during the course of this research. Having said that,I’ll now do my best to find words that can attempt to acknowledge thisunwavering support from everyone.My deepest thanks go to my supervisor Dr. Patricia Lasserre. Herkindness and patience through this process have been invaluable. Her honestfeedback and genuine support have meant so much to me over the years,and I truly appreciate everything that she’s done to help me grow. Furtherthanks go to my co-supervisor Dr. Yves Lucet, who has provided kindness,support, and the perfect balance of humour and practical advice in responseto my questions over the years.I would also like to express gratitude for the other members of the re-search team: Dr. Homayoun Najjaran, Brandon Graham-Knight, Dr. Rah-numa Nishat, and Zenab Kagdiwala. I appreciate Dr. Najjaran’s supportduring the course of the research. I am grateful for Brandon, who not onlyprovided the preliminary work that helped to inspire this research direction,but also provided valuable insight and help whenever I hit a wall. Thankyou to Nishat for keeping everybody on track, and for sharing her experiencewith me. I greatly appreciate Zenab letting me interrupt her work so that Icould talk out mathematical formulas and coding strategies.Thank you to my supervisory committee members Dr. Ifeoma Adajiand Dr. Mohamed Shehata for reviewing the thesis and providing feedback.Additional thanks go to Ifeoma for her cheerful presence and enthusiasm inthe other research work that I have completed with her.Lastly, I would like to thank Jason Gregg and the rest of the BC-ERteam that supported this research. Your domain knowledge was critical inhelping us to understand the importance of our findings. Additionally, thisresearch could not have been completed without the funding provided viathe NSERC Alliance Grant.xvDedicationThis thesis is dedicated to my family. Thank you for supporting me ineverything that I do, and for your continued motivation and patience. Loveyou all!xviChapter 1IntroductionThe field of process mining, where different aspects of business processesare extracted in order to improve workflow, has gained popularity in recentyears as process-based data collection practices have become more common-place. The demand has been driven by the need for businesses to understandthe processes that make up their day to day activities, and understand howlong those processes might take.Process mining considers many different questions such as what stepsare involved in an unknown process, how closely processes adhere to ex-pectations in practical implementation, or how long processes might take.The primary focus of this thesis is process duration prediction, where theduration of a known process is predicted based on available observationalhistorical data. This predicted duration can be used to schedule employees,resources, or plan delivery of services. In recent years, there has been re-newed interest in applying artificial intelligence (AI) in the form of neuralnetworks (NN) to attempt to solve the prediction problems. However, thisresults in black-box models that cannot explain their decision making logic.Nowadays, the research community seeks to find ways to gain the accuracyavailable from black-box models, while also achieving interpretability of thelogic behind the prediction.1.1 The British Columbia Energy Regulator(BC-ER)The British Columbia Energy Regulator (BC-ER), formerly the BritishColumbia Oil and Gas Commission (BCOGC), is a provincial regulatoryagency that is tasked with regulating energy resource activities in BC through-out their life cycle [6]. This includes everything from permit applicationthrough to restoration at the end of the project. As stated on their website,the goal of the BC-ER is to ‘protect public safety and safeguard the environ-ment through the sound regulation of oil, gas and geothermal activities inB.C., while balancing a broad range of environmental, economic and social11.2. RESEARCH GOALSconsiderations’ [6].As a public commission, the application process is well-documented onthe BC-ER website. In general, ‘[c]ompanies looking to explore, develop,produce, and market oil and gas resources in British Columbia must applyto the Commission for oil and gas activity permits’ [10]. An application canhave one or more activities, such as wells or pipelines, associated with it.Each application received by the BC-ER has a number of process tasks thatmay apply. Generally, the application and review process constitutes thefollowing steps [10]:1. ‘Applicant identifies locations for oil and gas activities, conducts anyon site assessments and conducts pre-application engagement.2. Applicant conducts required consultation and notification and FirstNations engagement.3. Applicant uploads spatial data and application requirements in Appli-cation Management System (AMS).4. Applicant validates application for completeness and accuracy. Sub-mits application and pays application fee.5. Regulator conducts in depth technical reviews and carries out FirstNations consultation.6. Regulator makes a determination on the permit and informs affectedland owners and First Nations. Permit holder adheres to timelinesconditions, laws and regulations.’[10]1.2 Research goalsThe BC-ER is interested in understanding how long applications take toprocess, and which factors affect the timeline. The problem of predictinghow long an application might take to process is complex, as the processrepresents multiple sub-processes that can happen simultaneously. Whilethe expectation is to achieve an accurate prediction, the primary goal is toleverage explainable methods that will help to explain which parts of thishighly complex process have the most influence on the timeline associatedwith the application.One of the most important considerations of the application process isconsultation with any First Nations (FN) groups that may be affected by21.3. CONTRIBUTIONthe proposed activities in an application. According to the BC-ER, ‘Con-sultation consists of information sharing between government and affectedIndigenous Nations and seeks to resolve potential adverse impacts to Abo-riginal interests.’ [7]. By accurately predicting the length of time requiredfor an application, or consultation, we can ensure that First Nations com-munities are given enough time to be consulted, and smooth the process forboth the FN communities and the BC-ER.An example of the BC-ER application process, including the FN Con-sultation, is illustrated in Figure 1.1. A proponent wishing to construct andoperate a new well submits an application to the BC-ER. The applicationmay require technical reviews related to whether or not the site is near an ar-chaeologically valuable site, whether it requires forestry changes, or whetherit might be located on land owned or managed by First Nations (see steps inblue). Each of these reviews may have sub-processes associated with themas well. An example of the FN Consultation sub-process is illustrated ingreen. Information obtained during the course of these reviews may de-termine that an application revision is necessary. In the case of a revision,the steps in blue may be repeated if the revised application requires thesteps, or if the revision changes the conclusion reached during that step inthe previous revision. Once all reviews and consultations are complete, thenthe BC-ER makes a determination on the success of the application.1.3 ContributionThis thesis aims to predict the duration of the FN Consultations dis-cussed in the previous example, based on the needs of each application andthe FN communities associated with it. We then leverage that predictedconsultation duration to supplement the information available in the dataset, in order to predict the duration of the overall application process illus-trated in Figure 1.1. To this end, we apply the state-of-the-art eXplainableReasonably Randomized Forest (XRRF) [22] method to a new domain byapplying it to a process mining problem. We introduce a modified version ofthe XRRF method that we call Extended eXplainable Reasonably Random-ized Forest (EXRRF), and take advantage of the white-box explainabilityavailable from a typically black-box method to provide transparency in theoil and gas industry.The contents of this thesis are as follows. Chapter 2 summarizes therelevant terms related to this topic, along with the related work in the field.Chapter 3 provides an explanation of the XRRF method chosen to solve31.3. CONTRIBUTIONFigure 1.1: A simplified example of the BC-ER application review process,including exit to FN Consultation sub-process.41.3. CONTRIBUTIONour problem, and introduces the modified method used for prediction, whilealso explaining the complex data handling required to prepare for predic-tion. Chapter 4 provides a detailed explanation of the results obtained fromapplication of the method to predict the duration of the FN Consultationprocess, then leveraging that prediction to predict the overall applicationduration. Finally, Chapter 5 explores the results to validate our contribu-tions to the extended XRRF method, and contextualizes the accuracy of theresults. The limitations of the modified approach, as well as future work,are explored in Chapter 6.5Chapter 2BackgroundThis thesis crosses the boundaries between statistical problem solving,business process mining, and machine learning methods. This chapter pro-vides the relevant definitions and contextual information to clarify the ap-proach used, and explain why it was necessary to apply statistical methodsto solve a process mining problem. We will present some of the commonapproaches taken, and the factors that affected their appropriateness forapplication to our prediction problem.2.1 Process miningProcess mining is concerned with using data extracted from process exe-cution, in order to gain information about a process that produces the data[46]. This differs from process mapping, which uses domain knowledge tocreate a process map, or diagram, of the process steps. Process mining canbe used for the purpose of process discovery, conformance checking, or pro-cess enhancement [45]. Each of these categories represents a broad rangeof research topics, driven by the need for businesses to understand everypart of their processes. Process discovery can be used to build a processmodel that represents the course of events taken by the data. Process con-formance checking uses the data and the mined model to perform diagnosticchecks on the process. Process enhancement uses the data and the minedmodel to understand the process better in order to find improvements. Thislast category of process enhancement includes extending the process modelto predict the remaining time that a specific process instance may take tocomplete, and is where our problem of predicting process duration falls.The data used to mine the process is automatically generated by sys-tems [49]. The specific systems and their associated output details vary indifferent industries, but the common thread is that they all produce eventlogs. Each event references a particular step of a process, and the collectionof sequentially recorded events is saved as an event log [45]. A specific caseis an instance of the process, from start to finish, while an event trace is a62.1. PROCESS MININGparticular path through the case. In processes with loops or optional steps,this means that the process may have a variety of possible traces.As an example, we can consider a simplified version of the applicationprocess for the BC-ER that is illustrated in Figure 1.1. The system recordsan event whenever a process step is completed in relation to the applicationsuch as the application being received by the commission (process stepsshown in orange). This would be recorded in the application level caselog shown in Table 2.1(a). The application itself would be a case, whilethe specific steps taken during the application process would be recordedas an event log, representing an event trace (process steps shown in blue).These event traces would be recorded in the application task level event logshown in Table 2.1(b). The application process is complex, and is made oftasks that include a variety of environmental, contextual, and communityreviews, each of which is an entire process by itself. As a result, eventsmay be recorded at a lower level, then rolled up to a higher level in thecase (process steps shown in green). An example of this roll-up procedurewould be the FN Consultation sub-process, shown in the sub-process taskstream log in Table 2.1(c). When an application is received, it is reviewed todetermine which FN communities need to be consulted (if any). Once theconsultation requirements are determined, a consultation task is initiatedfor each community, and the event is logged in the system at the task level.This triggers the consultation sub-process, which has a variety of steps fromassignment through to completion of the consultation. Each of these eventsis logged in the sub-process, and once a determination is made, the sub-process logs a completion event for the task level.Due to the large research space represented by process mining, litera-ture surveys were helpful in understanding the current state of the processmining field. Corallo et al. [14] focused on surveying research related to in-dustrial applications of process mining research, and found many thousandsof results that were only sufficiently narrowed once the results were limitedto open-access. The final case studies considered in their survey coveredmany different industries, including healthcare, infrastructure, manufactur-ing, insurance, finance, software, and retail.Verenich et al. [49] completed a survey of remaining time predictionmethods, and highlighted the relative accuracy of the methods identified.They identified a shift in methods over time, with early methods focusing onregression and clustering approaches, and neural networks being introducedin the research around 2017. The methods were broken down into generative,discriminative, and hybrid methods.The generative methods included traditional process mining techniques72.1. PROCESS MINING(a) Application level case logApplication Revision Revision...CaseIdentifier Identifier Number Feature123456 123 0 ... a123456 124 1 ... a123456 125 2 ... a876543 456 0 ... b...............(b) Application task level event logRevision Task Task...EventIdentifier Identifier Name Feature123 1 Task1 ... 1/1/2016123 2 Task2 ... 1/1/2016123 3 Task3 ... 1/8/2016124 4 Task1 ... 3/4/2018...............(c) Sub-process task streamApplication Task First Nation...Task streamIdentifier Identifier Community Feature123456 3 ABC ... Y123456 3 DEF ... N123456 3 GHI ... N123456 4 DEF ... Y...............Table 2.1: Simplified example of BC-ER case log, event log, and task streamlog, that illustrates how different processes at different levels can roll up tothe case level. Table a) shows an example of the application level case logdata. Table b) shows an example of the application task level event log data.Table c) shows an example of the task stream level data.82.2. NEURAL NETWORKSsuch as Petri nets and transition systems. Petri nets use places, transitions,initial markings, and the flow relations between places and transitions topresent a path between places [41]. Rogge-Solti et al. [41] built on this byadding probabilities, weights, and durations to the Petri net to construct amodel that predicted the remaining time of a process. Van der Aalst et al.[47] presented an approach that built a transition system using the state ofthe process, the event labels, and information transitions between the events,then annotated it with information about completion times to support theprediction of process completion time. Both of these methods hinge uponthe ability to accurately mine a process representation that describes thetransitions between events.The discriminative methods included techniques like clustering and prefix-length bucketing along with encoding techniques that included aggregationand LSTM neural networks. Navarin et al. [31] presented an LSTM ap-proach that moved away from mining a specific process and focused onusing data related to the activities in the process to predict the remainingtime. This was less computationally challenging, and allowed for handlingrepeated activities.The hybrid methods included state bucketing and flow analysis approach-es. The neural networks were highly accurate, and outperformed the tradi-tional process mining techniques in experiments, although they representeda higher computational load [49]. Verenich et al. [50] used XGBoost [12] as alearning algorithm to boost the performance of a flow analysis approach thatmined a model and learned how long each event took, while also predictingthe probability of transitions after each event.An alternative survey completed by Neu et al. [32] focused on deeplearning approaches for process prediction, and found that although processprediction has diverse implementation due to the variety of possible pre-diction goals, LSTM models were highly represented. In their survey, theyidentified that the majority of RNN approaches in the literature used LSTM.Other approaches included convolutional neural networks and feed-forwardneural networks, but these were less common and required a fixed input size.2.2 Neural networksNeural networks have gained significant popularity in recent years, bothfor process mining as well as other prediction purposes, especially on verylarge data sets. In simple terms, a neural network is a nonlinear statisti-cal model that uses back-propagation to ‘learn’ improvements for the node92.2. NEURAL NETWORKSweights that determine the classification or regression result [20]. Thesenetworks are fully black-box models where interpretability is sacrificed inexchange for greater accuracy and learning strength.While neural networks typically thrive on structured data where featurepositionality is important, they can sometimes perform well in ensembleswith tree-based algorithms, when the data set is large enough, or whenthere is a large number of features [53]. However, neural networks requiresignificant computational resources due to the potential of massive numbersof parameters associated with a higher number of features in the data. Thenumber of parameters, and the associated computational load is dependenton the data, and the specific neural network structure being implemented[20]. One way to combat this is to consider the precision-cost trade-off,where a balance is found between the desired precision and the computa-tional cost of the model [11]. This can be achieved by tuning the model tocalculate the minimum expected or required precision, while minimizing thecomplexity of the model and the resources required. As a result, when decid-ing whether to proceed with neural networks rather than ‘typical’ machinelearning methods, it is necessary to evaluate the available computationalresources and determine whether added precision and accuracy justifies thecomputational cost.There are many different types of neural networks, which can be highlycustomizable to the problem being solved. We will focus on Multilayer Per-ceptrons (MLP) as an example of a simple network, and Recurrent NeuralNetworks (RNN), specifically a type of RNN called long short term memory(LSTM), as examples of common networks applied to process mining appli-cations. We will also briefly introduce Natural Language Processing (NLP),as it is the larger umbrella that holds RNN and LSTM models, and canbe considered a way to consider contextual information related to processpredictions.An MLP is a simple ‘vanilla’ neural network, with any basic networkstructure deeper than one layer [20]. As a simple example of a neural net-work, it is a good starting place to determine whether the complexity andlack of interpretability of a neural network is appropriate to solve a prob-lem. These are used less often in state of the art research, but are a strongfoundation to start from.NLP is a neural network approach specific to using language to solvea problem [13]. These models use an understanding of language to an-swer questions or extract information, however they face challenges relatedto their contextual knowledge being limited by the availability of trainingdata. In order to provide enough information to the model to output a102.2. NEURAL NETWORKScomprehensible result, it is necessary to train the model on a large amountof data, which increases the computational cost drastically.While transformer architectures have invaded research fields like NLP,they are not commonly applied for solving process mining problems. Inour research, of the recent surveys on process mining techniques [14, 32,40, 49], just one survey identified a paper where the authors use part of thetransformer mechanism to speed up training for next activity prediction, butquestions remained about the effectiveness of the approach on heterogeneousdata [40].An RNN is an example of a deep learning NLP approach that canuse language-based improvements to neural networks in other applications.RNNs are used to predict upcoming events in sequence, using contextualinformation, where non-linear relationships exist [16]. This allows the RNNto be applied to process mining techniques, where the sequence of eventsis important. Each layer in the network maintains a state by feeding in-formation back to itself, providing information about what came before thecurrent state. In a process mining application, this means that the networkcan base predictions on the previous event in the log.Similarly, LSTMs are a specific application of an RNN that allow forlong-term dependencies [42]. These networks use gate parameters to deter-mine which parts of previous states they should forget when new informationis added [16]. This allows them to identify which event might be most likelybased on the sequence of events leading up to the current state. This func-tions as a way to control how long the memory of the neural network is, inorder to manage performance of the model.As identified by Verenich et al. in [49], LSTM networks have been in-creasingly leveraged to solve process mining prediction problems in recentyears. One of the early applications of LSTMs to this domain was pre-sented by Navarin et al. [31]. They proposed an LSTM architecture thataimed to predict the remaining time while also dealing with event logs withrepeated activities. When comparing this LSTM approach to transitionsystem methods, the LSTM resulted in increased accuracy when predict-ing remaining time. These results are supported by Tax et al. [42], whofound that LSTM architectures outperformed typical process mining base-lines when predicting next activities and future event sequences. Similarly,Tax et al. [42] found that LSTMs often outperformed the baseline methodswhen predicting remaining time in the process.112.3. COMPLICATIONS RELATED TO HIGHLY PARALLEL PROCESSES2.3 Complications related to highly parallelprocessesThe various sub-processes associated with the application process canbe completed simultaneously, which results in inconsistent completion orderfor the events in the event log. This makes the application process morecomplex than a typical process mining problem, which usually aims to derivea structured process model that represents the sequence of events in orderto predict factors related to one process rather than a group of processes[46]. This means that it is prohibitive to try to mine a typical process modelwith the data, as there are too many processes and possible event traces.Most process mining techniques assume that the event logs are ordered,and rely on time and date stamps to order the events in the log [3]. Awad etal. [3] propose a pipeline based model to handle event logs with unorderedevents, where these date stamps cannot be relied upon to represent the ac-tual order of events. However, this solution is based on a known or expectedorder of events, and an online event stream where incoming events can haveorder imposed upon them.As an example of the challenges faced, we can consider the flow analysisapproach proposed by Verenich et al. in [50]. The idea was to use flowanalysis, a calculation method based on averaging, to add interpretabilityto a traditionally black-box approach via notation that clearly outlined theprocess. However, the method hinged on being able to accurately mine theprocess, and the flow analysis calculations required assumptions to be madeabout parallel processing times. This approach could not accurately minehighly parallel processes, or those with many loops, due to the exponen-tial increase in possible process paths introduced by the parallel or loopedprocesses.When applying this method to our BC-ER data, the flow analysis ap-proach was unable to derive a legible model from the full process data. Otherattempts to derive Petri nets from the BC-ER data were also unsuccessfulusing ProM1, an industry standard tool for process mining. This experiencesupports how difficult it is for these mining techniques to handle parallelprocesses.These challenges extend to process mining techniques with neural net-works. Peeperkorn et al. [36] evaluated whether LSTM based process miningmodels could effectively learn the structure of process models with loops andparallel processes. They found that comprehensive parameter tuning was1https://promtools.org/122.4. STATISTICAL METHODSrequired in order for the networks to learn even trivial process models withparallel behaviour. Specifically, they were able to identify that models withparallel sections were challenging for LSTM models.These results illustrated that typical process mining techniques wouldnot be appropriate for the BC-ER data due to the highly parallel process.As a result, it was necessary to explore combining process mining conceptswith alternative methods of predicting the remaining time in the process.2.4 Statistical methodsA typical alternative approach for predicting a quantitative value such asduration based on known information would be statistical regression. Thereare many different approaches to regression, such as linear or polynomialregression, splines, and generalized additive models [23]. In this sectionwe will focus on tree-based methods based on performance expectations[12, 53] and the results of our preparatory work (see Section 3.1). Thesetree-based regression methods can be either black-box, where the relation-ship between input variables and the prediction is unknown, or white-box,where the relationship can be explained [20]. Generally, white-box methodsare more desirable, due to the interpretability of results generating moretrust. Black-box models are often more complex, which contributes to theirstatus as black-box models, and allows them to solve difficult problems moreaccurately.Input variables, also known as features or attributes represent observa-tional data that can be used to generate predictions. These features can beused to produce Classification and Regression Trees (CART), an implemen-tation of decision trees [20]. An individual CART is interpretable, as thedecision path that forms the tree can be reviewed. CARTs typically rely onbuilding trees to a maximum depth, then pruning the tree to the smallesttree that has the lowest cost, which is typically some measure of error. Thistechnique allows complex regression problems to be solved quickly, but ishighly susceptible to variance. A small change can result in a different de-cision path, or error propagated through the tree [20]. One solution for thisissue is constructing a forest from multiple trees.Trees are particularly well suited to averaging in order to reduce vari-ance [20]. One of these averaging methods, bootstrap aggregating (bagging),takes repeated samples from a data set to generate multiple training sets,which can each be used to train a model that produces an averaged predic-tion [23]. Trees generated through bagging are identically distributed, which132.4. STATISTICAL METHODSallows for the average of the results to represent the expected value of anyone of the trees. By randomly selecting the input variables, the variance canbe decreased by reducing the correlation between the trees. As a result, ran-dom forests typically perform well, although they lose the interpretabilityof individual trees, as the decision path can no longer be extracted [22].The results of random forests are affected by the same bias that affectsindividual trees. In terms of the bias-variance trade-off, this means thatas the number of variables increases, the bias decreases as the varianceincreases. Typical performance indicators used in forecasting and predictionproblems include RMSE, MAE, and R2 [48].1. R2The R2 statistic is a measure of the quality of fit of the model, byquantifying the variance explained by models. Although it is typi-cally used with linear regression, it can be extended to understand thevariance explained in data where linear fits are not realistic [23]. Itrepresents the proportion of variance explained as a value between 0and 1, where values close to 1 indicate a good fit for linear models,but lower values are an indication that the data might not support alinear model.2. Mean Absolute Error (MAE)MAE is a typical measure of forecast accuracy [48],MAE =1n∑n|et|, (2.1)where et is the error computed as the difference between the predictionand the truth for one data point, and n is the number of available errormeasurements. As it is not scaled, it is helpful to consider it in relationto the average of the response.3. Root Mean Squared Error (RMSE)RMSE is a measure of accuracy that assigns more importance to highererror values [48],RMSE =√1n∑ne2t , (2.2)where et has the same definition as the MAE equation. When data isstrongly affected by outliers, having a secondary accuracy metric thatrepresents the effect of these higher errors can help represent the biasin the prediction.142.4. STATISTICAL METHODSIn general, MAE can be considered an indicator of the median accuracy,and RMSE an indicator of the average accuracy [48].2.4.1 Feature engineering vs. feature encodingSometimes, the features captured in a data set do not appropriately cap-ture or represent the full behaviour of a system, requiring additional featuresto be engineered [1, 44]. Features are meant to bridge between problems andsolutions, and can be created by activities associated with the life-cycle ofthe data production [44]. Domain analysis can be used to help understandthe problem domain and identify interactions between features. Feature en-gineering is used to generate features that might help bridge the knowledgegap between domain knowledge and measured variables in the data. Thisprocess can result in better solutions when building prediction models [1].The concept of reverse engineering can be used as a feature engineeringtechnique. Investigating the available measured data, and supplementingthe information with domain knowledge can produce features that representstatic and dynamic dependencies in the data [44].Feature engineering should not be confused with feature encoding, whichis a method to translate categorical or textual features into numerically en-coded values [21, 43]. Feature selection and other machine learning methodsrequire numerical data, which drives the need to feature encode categoricaldata [21]. Common feature encoding methods include one-hot encoding,label encoding, and frequency encoding [21, 34, 43].With one-hot encoding, each categorical data feature is encoded into kdifferent binary indicator feature columns, where k is the number of categor-ical types in the feature [43]. The column corresponding to the categoricallabel is set to 1, while all other indicators are set to zero. This has thebenefit that all of the categorical types have the same order of magnitude[21]. However, this can also result in a significant dimensionality increasewhen there are a large number of categorical types [21, 43].Label encoding is a simpler method of feature encoding that translatescategorical labels into corresponding numerical values [21]. For example,applying label encoding to the months of the year would convert January to1, February to 2, etc. This method is popular because of its simplicity, andthe fact that it does not increase the dimensionality of the data. However,this means that magnitude is being assigned to a categorical value, whichmight influence the model [43].Categorical features with a high number of categorical types are consid-ered high-cardinality, and can be difficult to handle with feature encoding152.4. STATISTICAL METHODS[34]. One possible solution is to consider frequency encoding, where eachcategory is mapped to the frequency of observations in the training data.This is a method that can reduce the dimensionality required when encod-ing the features, however, it creates an assumption that there is correlationbetween the number of observations of each type, and the response [34].2.4.2 Improvements to statistical methodsOne common approach for improving random forest results is to sim-plify the data by performing Principal Components Analysis (PCA) as afeature selection method. The concept is to improve the interpretability ofa model by removing irrelevant features from the data [23]. The principalcomponents that account for the most variance in the data are identified,then the set of principal components that explain the desired amount ofvariance in the data can be used as prediction data for a model. Reducingthe dimensionality of the feature set to a smaller set that explains most ofthe variability ensures that the model result is more easily interpreted, whilemost of the variance is still explained [23]. Although Principle ComponentAnalysis (PCA) is sometimes used as a feature selection method to improverandom forests, it does not explain the relative importance of the features,nor does it consider the interactions between features.While tree-based prediction methods are less affected by the scale ofpredictors than clustering methods, it is still good practice to consider stan-dardizing data. As CARTs are often pruned based on a measure of error,features with particularly large scale can sometimes affect the error and theresulting trees [20]. In order to minimize this possibility, the data can bestandardized so that each feature has a mean value of zero, and a standarddeviation of 1.Typical tree-based ensemble models cannot be optimized using standardoptimization methods, but they can be improved using gradient boosting[12]. These methods leverage the first- and second-order gradients of theleaves on the tree to help greedily improve the scoring used in building thetrees.In 2016, Chen et al. [12] describe the XGBoost algorithm as a ‘scalablemachine learning system for tree boosting’. Gradient tree boosting is apopular method for improving tree based algorithms, and has been knownto frequently win Kaggle2 competitions for machine learning [12, 53]. Inaddition to providing accurate results, the XGBoost implementation can2https://www.kaggle.com/162.4. STATISTICAL METHODSFigure 2.1: An approximate comparison of accuracy and interpretability ofcommon models.handle large amounts of tabular data, allowing it to be competitive withresults from deep neural networks.When we compare the use of XGBoost to traditional random forests ontabular data sets, this means that the key difference is performance anddata set size. In a situation where the data set is not unreasonably large,the two methods are comparable. However, if the data set is too large, orcomputational resources are scarce, then the optimization found in XGBoostwould be more appropriate.An alternative method of boosting tree algorithms is compared to theuse of random forests by Mathur et al. in [28]. They implemented theCatBoost, or categorical boosting, gradient boosting method. This methodimproves traditional boosting methods by avoiding target leakage and han-dling complex categorical data without requiring numerical feature encoding[39]. They then leveraged an external tool called SHAP that provides inter-pretability for the black-box tree methods [28].Other types of boosting approaches include the Explainable BoostingMachine (EBM)3, which adds some interpretability to the results. EBMsare tree-based generalized additive models (GAM) that are highly inter-pretable due to the explanation of relative feature importance along withcontributions of the features [33]. EBMs improve traditional GAMs due tothe consideration of pairwise interaction between features while maintaining3https://interpret.ml/docs/ebm.html172.5. PREDICTION METHODS IN OIL AND GASinterpretability [27].Figure 2.1 shows an approximate comparison of relative accuracy and in-terpretability of the models discussed, to help clarify the comparative utilityof the various approaches. This is only a guideline, as the accuracy of eachmodel is highly dependent on the data it is applied to.2.5 Prediction methods in oil and gasCurrent work in the oil and gas industry seeks to solve alternative prob-lems to the one that we face. For example, Aslam et al. [2] compared theability of different machine learning models to detect anomalies in eventsassociated with oil wells. They found that random forests were the mostaccurate for their classification task, and required leveraging external ex-plainable frameworks to help explain the results of the model.When considering research related to permitting processes, Pika et al.[37] present a case study that uses process mining techniques to study theexecution of ‘permit to work’ processes in industrial applications. These aresafety procedures that must be followed in strict order, in order to approvea workplace to resume work. Process mining allowed these processes to beextracted, which were then checked for conformance to expectations.Interpretable models, including process mining techniques, have gainedpopularity in the financial and healthcare sectors recently due to the needfor explainable results [15, 19, 22, 50, 55]. The complexity of the dataused in healthcare means that cases might follow different process paths,even though they have similar circumstances [15, 19]. This presents aninteresting problem, with similar characteristics to the problem we are tryingto solve. Additionally, predicting the duration of the permit applicationprocess for governmental oversight of the oil and gas industry represents adeviation away from these common directions, and introduces the challengeof determining which approaches to consider.18Chapter 3MethodologyThe primary motivation for this research has been to identify a highlyinterpretable and accurate prediction method. This will not only providea reasonable estimate of the duration of the relevant processes, but alsoprovide information about which parts of the processes and sub-processescontribute the most time and complexity to the predicted duration. Bydeveloping a model that can explain these factors, we can verify that ourpredictions can be trusted. This chapter explains the preparatory work thatwas completed to compare possible prediction methods. We then explain theapproach we selected to balance interpretability and accuracy, and presentthe modifications made to enhance the model.3.1 Background workIn this thesis, we will use R2, MAE, and RMSE as indicators to help usjudge the fit of our models. Although R2 is not a great indicator for ourdata, as we know that it cannot be modelled with linear regression, it helpsus to identify how much of the variance in our data is explained with eachmodel. The MAE most closely aligns with our goal of targeting accuracywithin a set number of days of truth, as a median indicator of accuracy. Thedata that we are working with is strongly affected by outliers, so we alsoleverage RMSE as a secondary metric that indicates the average accuracyof our predictions.Preparatory work was done to compare a few common models to de-termine the best approach for predicting the application duration. As itwas unclear what data was relevant, a simple data set was gathered thatincorporated the count of different application tasks associated with an ap-plication, along with the number of revisions and the amount of idle timeduring the process. Four common prediction techniques were evaluated: anMLP neural network, EBM, random forest, and XGBoost. A 5-layer, fullyconnected MLP was chosen as the data available was too simple to justifythe computational cost and complexity associated with more complex neu-ral networks typical of process mining applications. The EBM was chosen193.1. BACKGROUND WORKModel R2 RMSE (days) MAE (days)5MLP 0.524 104.3 46.70EBM 0.535 103.1 41.2XGBoost 0.507 106.1 45.3Random Forest 0.561 100.1 43.7Table 3.1: Results of predicting the full application duration using commonbasic models, applied to basic task count data with idle time considered.as a basic regression technique with added explainability. XGBoost andrandom forest were chosen as tree-based algorithms that typically performwell.4 [12, 22, 53].The results of the preparatory work are provided in Table 3.1, andshowed that although around half of the variance was explained, the ac-curacy was poor. The best MAE was obtained from an EBM model thatprovided some interpretability, but the MAE of 41.2 days was more than75% of the mean duration of 53 days, and the RMSE of 103.1 days wasalmost double the mean duration. The random forest achieved the bestRMSE of 100.1, and explained the most variance with an R2 of 0.561. Theamount of information gained from the EBM did not justify the poor overallperformance.This preparatory work showed us that a simple approach would likelynot be suitable. The complexity of the overall application process requiredus to break the application prediction problem down into smaller predic-tions, in order to build a model that could be used to target predictions forthe most important parts of the process. An outline of a simple startingmodel is illustrated in Figure 3.1. Although the application process hasmultiple community consultation tasks, the EBM results indicated that theFN Consultation task was a key step of the application process and drovea good portion of the time required for the application. Our interpreta-tion of the information on the BC-ER website supported this, as they statethat they ‘encourage the participation of Indigenous Nations and peoplesin processes affecting them’ [8]. The FN Consultation also has additionaldata available related to the task stream associated with the consultationsub-process, which supported the development of an independent predictionmodel for this sub-process. Therefore our initial focus was building a model4The preparatory work was completed by Brandon Graham-Knight.5The mean duration in the data is 53 days.203.1. BACKGROUND WORKFigure 3.1: Outline of plan to feed FN Consultation prediction into appli-cation duration prediction.Model R2 RMSE (days) MAE (days)6MLP 0.666 54.0 24.0EBM 0.774 44.5 20.9XGBoost 0.847 36.6 13.5Random Forest 0.899 29.8 12.2Table 3.2: Results of predicting the duration of the First Nation Consulta-tion tasks using common basic models.that would accurately predict the FN Consultation duration.All of these factors combine to highlight the importance of predicting theexpected duration of the FN Consultation, so that the prediction can be usedto add more information to the expected overall duration of the application.The prediction of overall application duration will have the most benefitfor the BC-ER when all of the application information is not yet known,so by predicting FN Consultation duration based on factors known whenthe consultation is initiated, we can maximize the expected utility of theresulting prediction. This also meant that it was necessary to minimize theerror associated with the prediction, to limit the propagation of error oncethe prediction was fed into the data used to predict the overall applicationduration.6The mean duration in the data is 26 days.213.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTThe MLP, EBM, XGBoost, and Random Forest models used for prepara-tory work for the application duration prediction were applied to the FNConsultation data, and the results are summarized in Table 3.2. These re-sults were promising, and showed that the tree based methods were able toexplain a lot of the variance, although the MAE of 12.2 days for the randomforest still represented nearly half of the mean consultation duration of 26days. These initial results showed that the tree-based algorithms performedthe best, and drove the motivation to see if we could identify a random forestapproach that added interpretability. Once a reasonably accurate predictionwas available for the FN Consultation duration, the goal was to then usethat prediction to augment the available data for the overall applicationduration.3.2 eXplainable Reasonably Randomized ForestIn September 2022, Jain and Jana [22] proposed the eXplainable Rea-sonably Randomized Forest (XRRF) model as a state-of-the-art method toimprove random forests by adding interpretability and explainability to themodel. The XRRF model considers the importance of features both individ-ually and as interactions with other features. The premise is to use multiplefeature selection algorithms to determine the relative importance of featuresbased on correlation. The forest is then manually constructed from ran-domly selected subsets of features, and the decision paths are extracted togain even more information about the prediction.There are four steps to the process, which are illustrated in Figure 3.2.Step 1 improves learning performance via data cleaning; Step 2 improvesinterpretability via feature selection; Step 3 increases model accuracy viarandom forest predictions; and Step 4 introduces explainability via decisionpath extraction [22]. The interactions of the four steps are illustrated inFigure 3.3, which shows how the model achieves responsibility, trustability,transparency, and acceptability. The figure also shows how a transparentand responsible model can be built that still provides accurate results. Thesteps of the XRRF method as proposed by Jain and Jana are explained indetail in the following subsections.3.2.1 Step 1: Learning performanceThe first step of the proposed method improves learning performancethrough data preprocessing to refine the quality of the tested data sets.Jain and Jana focused on three key aspects of data cleaning: handling the223.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTFigure 3.2: The XRRF method consists of four steps: Learning performance,Intrinsic interpretability, Model accuracy, and Explainability.233.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTFigure 3.3: The benefits gained from the interactions of the four steps inthe XRRF algorithm.243.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTmissing data, removing noisy data, and normalization of the data. In orderto handle the missing data, Jain and Jana cite the method proposed byMuzellec et al. in [30] as a cutting-edge method for data imputation. As itis based on optimal transport, Jain and Jana [22] suggest that it can infer asignificant volume of missing data, while outperforming other state-of-the-art methods.Noisy data typically adds complexity to tree-based algorithms like ran-dom forests, affecting the learning performance. Jain and Jana [22] cite ahyper-clique based data cleaning method called HCleaner proposed by Xionget al. [52] to remove noise. The HCleaner method targets data that wascollected in error, or that is considered less relevant. This is achieved byleveraging the correlation between two data points.The third aspect of data cleaning put forth by Jain and Jana is nor-malization of the data. This is a typical approach when using tree-basedlearning methods, and ensures that scale of the individual features does notaffect the outcome of the result. The XRRF method uses Integer Scaling,which scales the data between 0 and 1, and is not affected by the scale ofthe original data, or the scale of the data set [22, 35].3.2.2 Step 2: InterpretabilityJain and Jana propose a ‘statistically graph-based feature learning (SGFL)algorithm based on feature relationship accountability’ [22]. The premise isto use three different feature selection algorithms to determine each fea-ture’s impact on the model’s correctness via correlation values. The co-dependencies between the features are considered, which improves accuracyas well as the interpretability of the model.The SGFL starts with a correlation matrix of the features that takes theform of a graph G = (Fn, E,Wij) where Fn = (F1, F2, ..., Fn) is the set offeatures, E = {(Fi, Fj)|Fi, Fj ∈ Fn} are the edges of the graph, and Wij isthe associated edge weight between feature Fi and Fj based on the absolutePearson’s correlation coefficient shown in Equation (3.1)Wij =∣∣∣∣∣∑m(vi − vi)(vj − vj)√∑m(vi − vi)2√∑m(vj − vj)2∣∣∣∣∣ (3.1)where vi denotes the vector of feature Fi, and vi denotes the mean value ofthe feature vector vi averaged over m samples. Likewise, vj and vj denotethe vector of feature Fj , and the mean value of the feature vector vj averagedover m samples, respectively. The weights of the feature-to-class correlation253.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTare computed in the same way, between a feature Fi and the response vari-able Fj . Due to this graph structure, the graph G can be generated as theabsolute Pearson’s correlation matrix for the feature set of the data.The selection of relevant features is based on the premise that featureswith low feature-to-feature and high feature-to-class correlation are highlyrelevant for model accuracy [22]. In order to identify these features, Jainand Jana propose three different feature selection algorithms that are thenreviewed with Venn-diagram logic to classify the features as pivotal, domi-nant, or basic. Features that are in the result sets for all three algorithmsare considered pivotal, features that are present in any two of the featureselection algorithms are considered dominant, and features present in onlyone of the three result sets are considered basic.To preserve feature-to-class relevancy while removing feature-to-featureredundancy, a feature correlation threshold of a feature set Fn, Tfc, is shownin Equation (3.2)Tfc =Af−c(Af−c +Af−f )(3.2)where Tfc is the threshold, Af−c is the average feature-to-class correlationbased on Wij as the edge weight between feature Fi ∈ Fn and the responsevariable. Af−f is the average feature-to-feature correlation based on Wij asthe associated edge weight between feature Fi and Fj .To prepare for the feature classification algorithms, a spanning tree Gs isconstructed as a spanning graph of the undirected weighted graph G, whereall Fn ∈ Gs. Algorithm 1 is then applied to G, where the cost function ofGs is the sum of the weights of all of the edges of the tree, or∑Wij , andwe define the minimum spanning tree (MST)M with the lowest cost amongthe possible spanning trees Gs. The MST M is found by employing Prim’salgorithm, and the edges in M are pruned if their weights are smaller thanthe threshold Tfc. The trees generated by pruning edges in M are thenevaluated to find the minimum vertex cover (MVC), and the union of thefeatures in the MVC of all of the trees is output as the set of relevant featuresRs.Once Algorithm 1 is complete, the graph G is converted into a featurecorrelation matrix (FCM) where the edge weights Wij are replaced with 0when Wij is less than or equal to Tfc, and 1 when Wij is greater than Tfc.This FCM is then used to determine the most relevant features using thedegree of each of the vertices (features) in the graph. The max-degree vertexedge remover (MVER) algorithm (see Algorithm 2) stores the maximumdegree vertices in a list, then iteratively removes the edges connected to the263.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTAlgorithm 1: MST-based graph algorithmData: G = (Fn, E,Wij), the correlation matrix of the featuresResult: Rs, the resulting features identified by the algorithm1 begin2 Gs ←− G3 M ←− MST of G derived using Prim’s algorithm using Wij asedge weights4 Tfc ←− threshold calculated from Equation (3.2)5 for E ∈M do6 if Wij < Tfc then7 prune edge8 tk′ ←− tree created by pruning edge9 end10 end11 for tn ∈ (t1, t2, ..., tk′) do12 Rtn ←− features in MVC of tn13 end14 Rs ←− {Rt1 ∪Rt2 ... ∪Rtk′}15 endAlgorithm 2: MVER graph algorithmData: FCM , the feature correlation matrixResult: Re, the resulting features identified by the algorithm1 begin2 Re ←− empty list3 G′ ←− FCM4 while G′ is not edge-less do5 max degree←− all max-degree vertices of G′6 Re ←− Re +max degree7 G′ ←− remove all edges connected with the max-degreevertices from G′8 end9 end273.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTmaximum degree vertices until the graph becomes edge-less.The max-degree vertex neighbour remover (MVNR) algorithm functionsvery similarly to the MVER algorithm, but iteratively removes the neigh-bouring vertices of the maximum degree vertices, rather than the neighbour-ing edges (see Algorithm 3).Algorithm 3: MVNR graph algorithmData: FCM , the feature correlation matrixResult: Rn, the resulting features identified by the algorithm1 begin2 Rn ←− empty list3 G′ ←− FCM4 while G′ is not vertex-less do5 max degree←− all max-degree vertices of G′6 Rn ←− Rn +max degree7 G′ ←− remove all neighbouring vertices connected with themax-degree vertices from G′8 end9 endFigure 3.4 illustrates an example Jain and Jana [22] used to explain theMVER algorithm. In this example, the starting graph has four vertices(features), Fn = {F1, F2, F3, F4}. The max degree vertex is F4, so F4 isstored in the result set, and the edges connected to F4 are removed. Once theedges are removed, the new graph G′ has two new max degree vertices: F1and F2. These vertices are also added to the result set, and the neighbouringedge is removed. At that point, the graph G′ is edge-less, and the algorithmis complete.To better understand the differences between the MVER and MVNRalgorithms, Jain and Jana also provide an example to illustrate the MVNRalgorithm (see Figure 3.5). The starting graph is the same as the previousexample, and F4 is the first stored max degree vertex. However, in this casethe neighbouring vertices are removed from the graph. As F4 is a neighbourto all of the other vertices, they are all removed, and the graph becomesvertex-less, triggering the end of the algorithm. Although Algorithm 2 andAlgorithm 3 follow similar logic, we can see that the differentiation betweenneighbouring edges and vertices produces differing result sets.Once all of the result sets of relevant features (Rs, Re, and Rn) areobtained using the algorithms, they are reviewed with the Venn-diagram283.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTFigure 3.4: ‘MVER-based feature selection on a graph G, where featureof max degree vertex = F4 is selected and its neighbourhood edges areremoved. The same procedure is repeated on the remaining graph in order toobtain the output feature set A[Re] = {F4, F1, F2}, until the graph becomesedge-less.’ [22]Figure 3.5: ‘MVNR-based feature selection on a graph G, where feature ofmax degree vertex = F4 is selected and its neighbourhood nodes (vertices)are removed. The same procedure is repeated on the remaining graph inorder to obtain the output feature set A[Rn] = {F4}, until the graph becomesvertex-less.’ [22]293.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTFigure 3.6: A Venn diagram showing the intersections of feature sets ob-tained from the MST, MVER, and MVNR algorithms. The intersectionsillustrate the classification of features as PFs, DFs, or BFs [22]logic shown in Figure 3.6. The features that are present in the result setsof all 3 algorithms are stored as the Pivotal Feature set (PFs), indicatingthat they add the most correctness to the result due to higher correlation.The features that are present in any two of the three result sets are storedas the Dominant Feature set (DFs), indicating significant correlation to theresponse. Finally, any features that are only present in one of the result setsare stored as the Basic Feature set (BFs), indicating a weaker correlationto the response variable.3.2.3 Step 3: Model accuracyThis step holds the Reasonably Randomized Forest (RRF) algorithmthat forms the backbone of the XRRF method, an ensemble tree algorithmsimilar to traditional random forests. The main difference is that with tra-ditional random forests the feature samples used to construct the trees areuninterpretable [22]. Jain and Jana [22] propose a modified stochastic uni-versal sampling (MSUS) method that modifies the traditional stochasticuniversal sampling method designed by Baker [4]. Jain and Jana’s modi-fication is to increase the number of selection pointers from 1 to N, whereN is the number of selections needed. In this case, N = 3, the number offeature sets to select from. By using N pointers, each feature set is sampledindependently. This ensures that the highest priority features, PFs, aresampled in all trees, while minimizing the selection of the same features in303.2. EXPLAINABLE REASONABLY RANDOMIZED FORESTmultiple trees (reducing correlation between trees). Their reasoning is thatthis sampling method ‘increases the model’s overall average strength whiledecreasing the occurrence of the associated decision trees in the forest’[22].A colour-coded example of the RRF algorithm is illustrated in Figure 3.7.It shows the features split into the feature sets, before being randomly sam-pled, and building trees from each sampled feature set. The RRF detailsare provided in Algorithm 4.Algorithm 4: Reasonably randomized forest algorithmData: [PFs, DFs, BFs], feature classes; data, prediction dataResult: result, prediction results1 begin2 N ←− 33 for i in k trees do4 for each class in [PFs, DFs, BFs] do5 Fx ←− number of features in the class6 range←− [Fx4 , Fx2 ]7 sample←− random sample of features in range selectedfrom class8 end9 Ti ←− CART constructed from randomly sampled features10 end11 result←− result via majority voting (classification) oraveraging (regression) of k CARTs12 end3.2.4 Step 4: ExplainabilityIn typical tree-based methods, the decision paths and features that leadto the prediction results are unknown. In the XRRF method, each tree istrained and tested separately, which allows for details to be extracted fromeach tree. Each tree has a decision path from the root to the predictionwhich can be evaluated to determine the set of features Fz ∈ Fn that makeup the decision path DP = {F1, F2, ..., Fz}[22]. The DP for each tree canbe extracted, which provides a set of feature sets (TF1, TF2, ..., TFz) forthe forest. Looking at the intersection of each TFk with the PFs used tobuild the tree provides a set of contributing features, CFk. The final set ofcontributing features CF = {CF1 ∪ CF2... ∪ CFz} provides explainability313.2.EXPLAINABLEREASONABLYRANDOMIZEDFORESTFigure 3.7: An example of the RRF algorithm. The set of prioritized features is split into PFs, DFs, and BFs.These feature sets are then randomly sampled to produce decision trees.323.3. DATA CLEANINGof which features contribute the most to model accuracy in the forest. Dueto the sampling method used to select features for the trees, it is possiblethat two trees might have the same DP and CF . However, it is unlikelythat these trees will have the same overall structure [22]. The ability toextract structural details along with decision path features adds significantexplainability to the XRRF model that is missing from other black-boxensemble tree methods. In order to provide further explainability, additionaldetails could be extracted, such as the frequency with which CF s contributeto DP s in the forest.3.3 Data cleaningDue to the BC-ER’s status as a Crown Corporation, much of the in-formation available about past projects is freely available on the BC-ERwebsite. However, data related to specific applications is largely kept pri-vate. As a result, throughout this thesis, some references to private datawill be kept deliberately vague, and details about prediction results maybe simplified to maintain privacy. The data set used for this research willnot be made publicly available. Any requests for information not publiclyavailable can be made under the Freedom of Information and Protectionof Privacy Act (FIPPA) by contacting the BC-ER Freedom of InformationOffice directly [9].The data cleaning approach proposed by Jain and Jana was a genericsolution proposed to provide the widest utility, while also ensuring highlearning performance for the RRF algorithm. Due to the complex nature ofthe BC-ER application process, the data available for our prediction prob-lems was unsuited to their data cleaning methodology. As a result, a highlycustomized data cleaning approach that best fit the data was chosen, ratherthan one that prioritized generalizability for the method. Although the datacleaning approach is not generalizable, the feature engineering and featureencoding logic can be extended to alternative applications.3.3.1 Data preparationThe first deviation from Jain and Jana’s proposed data cleaning was theavoidance of data imputation as a method to supplement missing data. Withthe BC-ER data, missing data was a result of the processes themselves, andwas likely to continue. Rather than impute missing values, an ‘Unknown’category was used for missing categorical values to account for this expecteduncertainty. Missing boolean flags were filled in with false indicators, and333.3. DATA CLEANINGFigure 3.8: The distribution of FN Consultation duration values in the BC-ER data.missing continuous data was filled with zeroes. This approach was verifiedwith domain knowledge of expected values in these situations.Further, the decision was made to leave in the vast majority of noisy data.The process itself has noise and variance in the overall duration, due to avariety of process-driven and contextual factors. Figure 3.8 uses a boxplotto illustrate the noise in the FN Consultation duration. We can see thatmost of the data is centered close to zero, and a long tail shows processesthat took longer than the majority of the data. Similarly, Figure 3.9 usesboxplots to illustrate the noise in the application duration, as well as thetime that the application spends idle. This data has a similar distribution,with the majority of the data centered close to zero, with long tails.The goal of the project is to create a prediction model that can accountfor as much of these noisy processes with long duration as possible whilemaintaining accuracy, rather than simply prioritizing accuracy for a smallerset of use cases. A simple cutoff for application inclusion based on 1 year343.3. DATA CLEANINGFigure 3.9: The distribution of application duration, and idle time presentduring the application process. Lines have been added at 730 days for theduration values, and at 365 days for the idle time values, to indicate thecutoff for exclusion of outlier values.of idle time in the application and 2 years of overall duration was selectedto remove only the most egregious outliers in the data. While the majorityof applications were completed within 2 years, with less than a year of idletime associated, there were some extreme outliers that were removed. Theremoved outliers accounted for 48 of the 11103 cleaned observations, lessthan 0.5% of the data.No cutoff was applied to the FN Consultation duration, due to the pre-dictions being fed into the application duration model afterwards. If onlyspecific consultations were excluded, it could misrepresent the predictionfed up to the overall application. By exclusion based on overall applicationduration, the FN Consultations associated with outlier applications wouldbe excluded in the roll-up process.353.3. DATA CLEANING3.3.2 Development of data tables and viewsThe data available for this project spans the time period from July 2016to January 2023. The data was provided as SQL reports on an online portal.These reports were extracted and added to a local mySQL7 Docker8 image.We leveraged views and materialized views to perform data cleaning tasks.The system used to process applications changed in 2016, so some ob-servations with dates in 2016 were completed prior to the stored date, andloaded in to the new system after tasks had been completed. Additionally,much of the data used in this prediction relied upon computer date stampsfor when specific tasks were completed. However, as this relies on humancompletion of tasks following a specified process, there are some erroneousdates in the data. An example of an inaccurate date would be marking atask as ‘completed’ on the day the next step was initiated, rather than whenthe task was actually completed. For the most part, the data has been leftas-is, with an understanding that we are trying to predict the actual timetaken to process an application through the system, including time takenfor human error.One significant concern with the data was the large number of tasks thatwere completed with zero time (ie. the task was started and completed onthe same day in the system). The system does not record timestamps alongwith the datestamps, resulting in lost accuracy when a task may requirea partial day to complete. Traditional process mining techniques typicallyrely on dates with timestamps, and processes where the majority of tasksare completed sequentially. When too many tasks are completed in parallel,complexity is increased due to inconsistent paths through the process. Ad-ditionally, when multiple tasks are completed on the same day, the modelcannot determine the order in which the tasks were completed. In both ofthese situations, it is prohibitive to find a model that can mine all of thepossible combinations of events [46].In order to solve this issue, and ensure that process mining techniquescould be leveraged, views were established that ensured that the tasks com-pleted in parallel had a completion time of at least one day overall, and thatthe two sequential tasks that followed had their order enforced along witha completion time of at least one day each. These views were then used tocalculate the idle time in the application process9.The date padding to prevent zero completion time for tasks had mini-7https://www.mysql.com/8https://www.docker.com/9The idle time calculation was developed by Brandon Graham-Knight363.3. DATA CLEANINGmal impact on the accuracy of the predictions. When we consider the FNConsultation, the padding meant that any tasks that had a duration of zerodays were bumped up to have a completion time of one day. A completiontime of zero days is not realistic, so padding the date to reflect one workingday provided value for the prediction.The impact of padding on the idle time is difficult to quantify, as theworst-case scenario could have added one day of padding for every taskassociated with the application. This is highly unlikely, but would havemeant that the idle time was underestimated.To prevent any chance of these impacts resulting in aggregated error forthe application duration prediction, the true duration of applications wasbased on the date the application was received and the date that the decisionwas reached. This meant that the padding affected two of the predictors forthe model, but not the truth, so the reported accuracy was based on truthrather than aggregated error.In addition to human error, and tasks that seem to take zero time,the FN Consultation process has changed over the years. These changeshave resulted in inconsistent data in some known time periods. With thehelp of our BC-ER business partner, we were able to investigate the trendsand outliers in the application data using Tableau10 and Python11, anddetermine inclusion and exclusion criteria for the data.One of the key recent changes in the FN Consultation process was a BCSupreme Court ruling that affected how effects on the FN communities weremeasured. On June 29, 2021, the BC Supreme Court decided in favour ofthe Blueberry River First Nations (BRFN) in response to a claim made bythe BRFN that the province ‘has authorized industrial development with-out regard for BRFN’s treaty rights’ [54], resulting in cumulative effectsthat had a negative impact on the Nation. The result of this ruling wasthat it was necessary to reconsider how FN Consultations would proceed inthe future, to ensure that the negative impacts would not continue. Thisreconsideration caused a significant delay in the processing of applicationswhile new strategies for processing were developed.The complexity of the relationships between the data tables added an-other level of necessary cleaning. The parent data tables used to pull indata for the prediction are summarized in Table 3.3, along with the result-ing data tables used for prediction. When working with FN Consultationdata, there were three levels of relevant data: The application level, the ap-10https://www.tableau.com/11https://www.python.org/373.3.DATACLEANINGDataset Hierarchy Data type No. obs. No. attrib. PredictionData tables available before cleaningApplication Activities Revision C 109223 26 AApplication Status Log Revision C 58042 6 AApplication Tasks Revision C 122144 12 A, FNApplications Application C 14252 16 A, FNFN Organization Information N/A I 211 4 FNFN Task History Task C 332358 26 A, FNPipe Detail Activity C 31456 63 AData tables available for prediction after cleaningApplication prediction data (1) Application C 7427 43 AApplication prediction data (2) Revision C 11052 43 AFN Consultation prediction data Task C 25500 17 FNData type: ‘I’ = Informational, and ‘C’ = Case Specific.Prediction: ‘A’ = Application duration, ‘FN’ = FN Consultation duration.Table 3.3: Summary of the data tables used for this research.383.3. DATA CLEANINGplication task level, and a task stream level associated with the consultationitself. The back and forth process of verifying data availability was a time-intensive and complex step of the data cleaning process, but established astrong foundation for prediction.In order to prepare for the application of the EXRRF model to the FNConsultation and application data, views were established in the MySQLdatabase to gather the relevant features, followed by feature engineeringin Python. These views were built upon previously generated views thatcleaned the data, based on known circumstances that would exclude thedata. For example, only FN Consultation task stream data that included‘create’, ‘assign’ and ‘in progress’ statuses were included. Any task streamdata that did not include all three of these status was considered invalid dueto data integrity, difference in process, or cancellation before the task hadbeen initiated. The view for the FN Consultation data rolled the subtaskstream information up to the task level, and coalesced any null categoricalvalues with ‘Unknown’. The data was also filtered to restrict it to dataassociated with applications that were, at some point, approved. In thisway, only data with completed workflows was considered.The view for the application data was much more complex, due to ex-tensive feature encoding and feature engineering. Notice in Table 3.3 thatsix different data tables were relevant to the application duration prediction,and represented four different levels of the hierarchy associated with the ap-plication. The join methods used to combine all of the views and tables intoone final view were chosen to intentionally drop any remaining dirty data,along with any observations where data was not available at every level.The observations that were excluded for not having data available at everylevel accounted for less than 0.5% of the observations, and represented ap-plications that follow an entirely different process related to forestry usage.The information gained by pulling the information from multiple data setsfar outweighed any gain from predicting these rare outlier application types.Constructing the view in MySQL and performing much of the featureencoding and engineering in the view creation helped to streamline the pre-diction process in Python. Complex actions that pulled data from previousor subsequent revisions into each revision were performed in Python for ef-ficiency and ease of code development. Contextual information about thenumber of applications open at the time the application was calculated inPython. The same was true for contextual information about the FN Con-sultations.393.3. DATA CLEANING3.3.3 Feature engineering and feature encodingPrior to initiating the feature engineering process, preliminary featureencoding was completed to preserve privacy in the database. None of thedata used in constructing the views or predicting the duration would al-low for individuals associated with the application process to be identified.However, the proponent that submitted the application was maintained, asan expected significant factor in the application process. These proponentswere considered as categorical values that were encoded as numerical val-ues during the data preparation stage. Similarly, when preparing the datato predict the FN Consultation duration, the names of the FN communi-ties being consulted were encoded as numerical categorical predictors. Soalthough the proponent or FN community might be identified, none of theBC-ER employee(s) processing the application, the employees of proponentssubmitting the applications, or the FN community members involved in con-sultation could be identified from the prediction data.In order to initiate the feature engineering process, a key first step wasto understand the available data through exploration of the relevant tables.This helped to identify possible features, and how best to encode the featuresto avoid any potential problems related to interpretability or data stability.Through available domain knowledge, we identified that there was a needto account for more information related to the process shifts that happenedover time, including the BRFN ruling that resulted in applications withlonger duration due to a pause in processing while process changes wereestablished. Further, there was an expectation that the specific FN commu-nity, or combination of FN communities, associated with an application orconsultation might affect the timeline. This resulted in development of morefeatures that tried to identify interactions between the FN communities inan application.Existing engineered features related to idle time and task counts werealso leveraged to improve results, as they were part of the motivation toidentify a more accurate model. Table 3.4 identifies the generated features,along with indicators of which method was applied to generate the feature.The data used for prediction of the FN Consultation duration consistedof 17 features. One feature was maintained in its original state from theavailable data, three features were pulled in from the application level, 7were feature encoded, and 6 were feature engineered.The data used for prediction of the application duration consisted of43 features. One of these features was maintained in their original statefrom the available data, and one was pulled from the FN Consultation level.403.3. DATA CLEANINGFeature Engineered EncodedFN Consultation dataFN Community name - xFN Treaty 812 membership x -Generalized FN postal code x -Review classified as not required x -FN Consultation type - xFN review status - xCirculation method - xFN reason code - xMonth consultation started - xYear consultation started - xInitiated after BRFN ruling x -Number of FN groups consulted x -Number of open consultations x -Application dataApplication type - xApplication priority level - xApplication major project - xApplication risk - xApplication proponent - xHigh volume indicator x -Month revision submitted - xYear revision submitted - xTime spent idle13 x -Number of times idle13 x -Counts of task types (14)13 x -Counts of activities (11) x -Number of activities x -Pipeline length x -Number of open applications x -Duration of previous FN Consultation x -Prediction of current FN Consultation x -Duration of previous revision x -12http://treaty8.bc.ca/13Features engineered by team member.Table 3.4: List of engineered and encoded features for FN Consultation andapplication data, with indication of method used.413.3. DATA CLEANINGSeven of the features were feature encoded, while the remaining 34 were allengineered features.Feature encodingA key feature of the EXRRF method is the interpretability and explain-ability that it provides over traditional random forests. Label encodingoffered us the opportunity to maintain simpler feature names that would beover-complicated by other feature encoding methods (see Section 2.4.1 forpros and cons of feature encoding approaches). The overarching concern wasthat if one-hot encoding was used, a certain level of interpretability would belost due to the vast number of features that would be added. For example,it would provide the most value to the BC-ER to know that the year a FNConsultation was initiated helped to explain the duration. It would provideless value to tell them that a specific year provided the most informationabout the duration.Although assigning magnitude to encoded features is a known concernthat is not present when one-hot encoding is applied, Tokuyama et al. [43]found that the magnitude did not meaningfully affect the result of pre-dictions, and was comparably accurate to one-hot encoding. Results fromJackson et al. [21] corroborated this finding, as they showed that label en-coding actually provided more accurate results when applied to their data.Jackson et al. [21] also found that in some cases one-hot encoding actuallyincreased dimensionality so much that it prevented models from running dueto memory issues.To support label encoding, we used the factorize function available inPython’s pandas library14. This function converted the categorical variablesinto numerical variables with numbering based on the order of observationof categorical values. In the case of the encoded months, the numbering wasdrawn directly from the date rather than from order of observation in orderto preserve interpretability (June was encoded as 6, July as 7, etc).Encoded features in FN Consultation dataOf the features that were encoded for the FN Consultation data, fourwere string categorical identifiers, one was a numerical categorical identifier,and two were date characteristics that were individually encoded. The cate-gorical identifier for FN Community name had 160 categories included, whilethe other three string categorical identifiers only had 5, 6, and 8 categories14https://pandas.pydata.org/docs/reference/api/pandas.factorize.html423.3. DATA CLEANINGeach. This volume of categorical identifiers supported the desire to use la-bel encoding rather than one-hot encoding, as it would have increased thenumber of data columns nearly ten-fold, in addition to the interpretabilityconcerns previously mentioned. Frequency encoding was ruled out, as thenumber of observations of categorical features in the data is not correlatedto the duration.The existing numerical categorical identifier was not comprised of se-quential numbers, so label encoding was applied there as well. This allowedus to force sequential numbering in order to minimize any magnitude differ-ences. The remaining two encoded features were date values. The date thateach consultation was initiated was used as a reference point, and the monthand year were extracted. Label encoding the month was straightforward, asthere were 12 consecutive values that could be extracted directly from thedate. There were only 8 years of data available, so the years were extracted,then encoded with label encoding to minimize the magnitude of the labels.Although it could be reasonable to consider whether encoding the datevalues and passing them in to the prediction might overfit the data, it is notthe case for these data sets. Specifically, the FN Consultation data wentthrough a process shift in 2016, 2018, and 2021. By passing in the encodedyear, we can ensure that we have indicators for those process shifts. Simi-larly, there is a certain amount of seasonality in the data that is representedby the months. If we were to avoid including these date characteristics, wewould be intentionally underfitting the data, and losing available informa-tion that could help with the prediction.Encoded features in application dataThe application data features that were encoded consisted of two stringcategorical identifiers, two date characteristics, and three boolean identi-fiers. Although the application type was a string categorical identifier thatonly had four possible values, the application proponent identifier had 159categories. Similarly to the FN Consultation data, label encoding was cho-sen as the best option to avoid the excessive increase in dimensionality thatone-hot encoding would have introduced.The date characteristics of month and date were encoded in the sameway as the FN Consultation month and year features, but in this case werebased on the date that the application itself was submitted. The reasoningbehind encoding the date values is the same as the reasoning for the FNConsultation data. The process shifts over the years are also represented atthe higher application level, along with the seasonality in processing times.433.3. DATA CLEANINGThe boolean identifiers were simply encoded as 0 for false, and 1 for true.Feature engineeringWith the BC-ER data, it became immediately apparent that althoughthere was a wide variety of observational data available through the system’slogging processes, we would need to engineer additional features to providecontextual information for whatever models we were proceeding with. Forexample, when working with such an intricate process, it is reasonable toconsider the volume of work being assigned, and how that might affect theprocessing times.When deciding which contextual information to engineer, we leverageddomain knowledge to help us determine which information could be engi-neered from existing data, and how that might correspond to the desiredprediction. However, there is no certainty that engineered features wouldadd information to the model. This meant that we avoided engineeringany features that were too similar to existing information available, or anyfeatures that seemed unrelated to the problem that we were solving.We used an iterative approach, which leveraged the information gainedfrom the feature importance classification and the correlation values pulledfrom the feature weight information in step 2 of the XRRF process. Thisallowed us to quantify the added value of engineered features while alsoensuring that the features we engineered were not overly correlated. Inthis section, we present the engineered features and explain their perceivedpotential value. Although the specific features are not necessarily general-izable, the iterative approach is.Engineered features in FN Consultation dataThe driving force behind engineering features for the FN Consultationdata was to find some way to quantify any complexity added by the locationof the consultation. It was not possible to encode specific region codes thatmight provide a measure of overlapping interests associated with the appli-cation. Instead, the postal codes associated with each First Nation weregeneralized in order to provide a location proxy for the FN Consultationprediction. The first three characters were extracted, and any FN commu-nities without an available postal code in the data were filled with ‘XXX’ togroup the unknown codes together. As these codes were alphanumeric, theywere then encoded using label encoding in the same way as described in theprevious section. This location proxy was only used in the FN Consultation443.3. DATA CLEANINGprediction, as it would have been inappropriate to roll it up to the appli-cation level. Each application could span many postal regions due to theirsize, making it impractical and inaccurate to try to choose one representativepostal region for each application.To measure overlapping interests in an application that could affect theconsultation, two further features were engineered to add further context.Our examination of the data indicated that Treaty 815 member nations inBritish Columbia might have a more complex consultation process. As a re-sult, any FN Consultations associated with a Treaty 8 member were flaggedusing a numerical boolean indicator, to indicate this differing process. Ap-plications could have a large or small number of FN Consultations associatedwith them, depending on the region of the application. In order to capturethis, in addition to the location proxy, a count of FN groups being consultedwas added.We previously discussed how the volume of work might affect the du-ration of the process, and this was captured by engineering a feature tocount the number of consultations in process at a specific FN communityat the time a new consultation was initiated. Smaller communities mightface staffing or workload issues if too many consultations are in process atthe same time, so this was a way for us to try to quantify the amount ofwork that a specific community might be facing when a new consultationwas requested.In some cases, a full consultation might not be required, such as whenthere is an administrative error that just requires a small change to theapplication. In these cases, a status code is associated with the FN Con-sultation indicating this different process. A numerical boolean feature wasengineered to flag the data associated with this differing process, so that wecould track it, and so that the model could account for the variance causedby this data.The biggest change in process, along with the greatest cause of variancein the data was caused by the BRFN ruling that caused the BC-ER toput all applications on hold pending development of a process change thatwould satisfy the outcome of the ruling. As a result, we engineered a date-based numerical boolean indicator that flagged all applications received afterJune 29, 2021 (the date of the ruling). This acted to differentiate the dataassociated with that process shift and the associated extended consultationdurations.15http://treaty8.bc.ca/453.3. DATA CLEANINGEngineered features in application dataWhen deciding how to engineer features for the application data, themost pressing concern was the complexity of the application process itself,along with the high number of parallel tasks that were completed. There are18 different task types associated with the applications. Two of these onlyapply to historical record changes (a different process that was excluded fromthis research), and two more are completion tasks that happen in sequenceat the end of the application once a decision has been reached. This left14 valid application tasks that each represented an entire process in itself.These tasks are represented by the blue process steps shown in Figure 1.1.Feature engineering related to these tasks was initiated during the dataexploration stage, as an attempt to understand the possible directions forprediction models. The preparatory work introduced in Section 3.1 wasbuilt upon engineered features that counted the number of different tasksassociated with each application. For example, an application might havea few dozen different FN Consultations associated with it, based on theregion(s) affected. A feature was created for each of the application tasksthat could happen in parallel, and the engineered value was a count of theobserved number of tasks at the application level. The reasoning was thatthe count of different activity types would help us understand the relativecomplexity of each application, by providing a proxy measure of the numberof processes associated with it.Further to this, data exploration during preparatory work showed thatthere were sometimes significant periods where the application ended up‘idle’, where no tasks related to the application were open or active. Thisidle time represents time that the application does not progress, when thereis nothing in the data explaining why no tasks are being completed. Theamount of time that each application remained idle was measured and addedas a feature, along with another feature that counted the number of timesthe application entered an idle period. While the data cannot provide ananswer as to why the application might go idle, measuring the time spentidle and the number of idle periods allows us to provide information thatis not measured in the data, that directly corresponds to the amount oftime an application might take. There was some concern about potentialoverfitting caused by this information being made available, as we wereproviding a measure of duration in order to predict duration. However,we reasoned that the idle time was a highly relevant measure that wouldbe available/dynamic during the application process, and was an avenuefor us to consider for future prediction (see Section 6.2 for more details).463.3. DATA CLEANINGStudying the effect of the inclusion of these features in preliminary modelsprovided assurance that they were not causing overfitting, and that theyadded valuable information to the model.This preliminary engineering added 16 features to our available data.The task type counts used for the preparatory work were rolled back tothe application revision level, rather than the overall application level, inorder to provide more granular details. We then leveraged the same logicto engineer features related to the count of activities associated with eachapplication revision. These activities included specific details related to theapplication, which helped place them in the domain of the type of appli-cation. For example, some applications had pipelines or roads associatedwith them, along with associated right of way concerns. Further applicationcomplexity could be tied to geophysical activities, wells, or changes nearwater sources.An overall count of activities associated with each application was added,along with counts of each specific activity type. Domain knowledge allowedus to identify that pipelines added additional complexity that could be quan-tified by the length of the associated pipeline, so an additional feature wasengineered to add up the overall length of pipeline associated with the ap-plication. This added 13 features specific to application activities to thedata available for prediction. This is not to be confused with the engineeredfeatures associated with application task and activity counts with featureencoding. Although each type is added as a feature with an associated in-dicator for its presence in the application, it is not one-hot encoding, asthe value being encoded is a count of observations that adds magnitudeand context. The specific number of activities and tasks associated witheach application revision is highly variable, and specific to the permit(s) theapplication is associated with.Another common factor affecting the success of the application was theproponent submitting the application, specifically the proponent’s familiar-ity with the application process. Although the proponent itself was encodedas a categorical variable, more general information could be added to themodel by classifying proponents by the volume of applications they hadsubmitted. Consider a proponent who has submitted a large volume of ap-plications, compared to a proponent who has submitted comparatively fewapplications. A higher volume proponent might have more staff available towork on applications, in addition to more experience with expectations tiedto the application process, such as supplementary paperwork, anticipatedconcerns and questions, etc. A lower volume proponent might not have thisexperiential knowledge, or the staffing to deal with concerns that come up in473.3. DATA CLEANINGa timely manner. This resulted in a reasonable expectation that there mightbe a correlation between higher volume proponents and shorter applicationduration. Due to a system change in 2016, engineering a feature based onthe number of applications a proponent had submitted before the applica-tion submission date would not have been representative of a proponent’strue volume. However, we were able to use the total number of applicationssubmitted since 2016 as a representative value. Trial and error identified250 cumulative applications as a reasonable cutoff point to classify a propo-nent as high volume, and a corresponding numerical boolean indicator wasengineered to flag the high volume proponents.Similar to the FN Consultation data, we wanted to capture the volume ofwork that was being processed during the time the application was received.We engineered a feature to count the number of applications that were open(not resolved) at the time that the application process was started. We alsowanted to consider specific contextual information about the application re-vision itself, so we added features that added the duration of the previousrevision(s), along with the duration of the FN Consultation in the previousrevision. The duration of the previous revision would help to provide infor-mation about the work completed so far in the application, and measuredthe cumulative duration of all previous revisions, including any dormancyperiod in between the time a revision was requested, and when the new re-vision was received. The duration of the previous FN Consultation workedin a similar way, measuring the actual amount of time spent for FN Con-sultation in the revision immediately before the current revision. This wasmeasured as the number of days between the first consultation being started,and the last consultation being completed, for the previous revision.The last engineered feature was derived from the prediction of FN Con-sultation duration for the current revision. The FN Consultation durationprediction was at a task level, meaning that every FN Consultation associ-ated with an application had an associated prediction. This was rolled upto the application revision level by taking the maximum predicted durationamong all of the FN Consultations associated with that revision. The max-imum was chosen as a representation of the longest expected time for theconsultation process to take, as the consultations were largely conducted inparallel. In the case of missing predictions caused by missing data, 1227synthetic values were created by applying noise consistent with the residualdistribution of the test and validation predictions to the true value. Thisrepresented approximately 11% of the values used.Overall, engineering features added 34 features to the available data asso-ciated with the application. This greatly increased the information available483.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)for prediction, and as all of the engineered features were already numericaltypes, no further encoding was required.3.4 Extended eXplainable ReasonablyRandomized Forest (EXRRF)While the XRRF method is comprehensive, it was designed to providea model that would be widely accurate, interpretable, and explainable [22].In addition to the comprehensive data cleaning detailed in Section 3.3, weadapted the method to facilitate alternative logic for feature selection, alongwith adding accuracy improvements tailored to our domain. As a result, wehave renamed the method Extended eXplainable Reasonably RandomizedForest (EXRRF), to differentiate from the original XRRF method proposedby Jain and Jana. The adaptation details are described in the followingsubsections.3.4.1 Threshold for feature selectionThe threshold chosen by Jain and Jana was effective, but was calculatedbased on logic that was not fully elaborated. Further, the threshold pro-posed typically resulted in all of the features being identified as importantby the MST algorithm, and fewer features being identified by the MVERand MVNR algorithms. Due to this, the MVER and MVNR algorithms wereunable to identify basic features, as they did not have any features that wereonly present in one of their result sets. We proposed testing four different fea-ture selection thresholds for pruning the MST and the FCM for the featureselection algorithms, and comparing their respective interpretable resultsto our domain knowledge gained through the research to determine whichthreshold provided the most interpretability for our EXRRF model. In addi-tion to the added interpretability, providing multiple thresholds would allowfor more balanced feature selection sets.The original threshold, FTpaperThe first feature selection threshold tested is the original Tfc calculationproposed by Jain and Jana,FTpaper = Tfc =Af−c(Af−c +Af−f ), (3.3)which we have renamed FTpaper.493.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)The harmonic mean threshold, FTharmonicThe FTpaper threshold was suggested as a way to preserve highly rel-evant features with ‘low feature-to-feature correlation and high feature-to-class correlation’ [22]. However, initial tests noticed that consistently thethreshold was too low to prune any features in the MST algorithm. The Tfccalculation was very similar to the harmonic mean for two numbers, x1 andx2,H =2x1x2x1 + x2. (3.4)The harmonic mean is traditionally used to provide averages where onevariable might provide more influence [17]. By replacing x1 with the averagefeature-to-class correlation Af−c and x2 with the average feature-to-featurecorrelation Af−f , we get an updated harmonic mean equationH =(2Af−f )Af−cAf−c +Af−f= 2Af−fTfc. (3.5)Notice that the difference between Equation (3.5) and Equation (3.3) is theaddition of the 2Af−f term in the numerator. As 0 ≥ Af−f ≥ 1, thisterm typically acts to reduce the Tfc. This harmonic mean with updatednotation,FTharmonic =2Af−cAf−fAf−c +Af−f, (3.6)was considered as a threshold to investigate the effect on the explainability.The mean-sd threshold, FTmean−sdFurther, as the average feature-to-feature correlation was being consid-ered, it was reasonable to consider a threshold that would be more likely toprune some edges in the MST by subtracting the standard deviation of thefeature-to-feature correlation from the average. This represents a scenariobased on the mean where pruning is guaranteed.This resulted in the third threshold tested, FTmean−sd, the average of thefeature-to-feature correlation minus one standard deviation of the feature-to-feature correlation (SDf−f )FTmean−sd = Af−f − SDf−f . (3.7)503.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)The min2 threshold, FTmin2We continue to use the graph definitions from Section 3.2.4, with graphG = (Fn, E,Wij) where Fn is the set of features, E are the edges of thegraph, and Wij are the associated edge weights, and M is the set of edgesin the MST of G derived using Prim’s algorithm, generated by Algorithm1. We then define the minimal weight in the MST to be W ′ = min{W (E) |E ∈M}, which corresponds to an edge E′ with weight W (E′) =W ′.This allows us to consider a threshold set to be equal to the minimumedge or second minimum edge of M, forcing the MST itself to be pruned.Initial tests during implementation showed that using the second minimumedge as a threshold consistently provided more accuracy, so further experi-ments only proceeded with FTmin2,FTmin2 =W′′ = argmin{W (E) | E ∈M \ {E′}}, (3.8)where the threshold is set to the second minimum weight of the edges in theMST M for simplicity.3.4.2 Range of features selectedIn Algorithm 4, a range of features to be randomly sampled is definedas [Fx4 ,Fx2 ], where Fx is the number of features in each of the PFs, DFs,and BFs classes. However, due to the limited number of features associatedwith the BC-ER data, this resulted in a significant loss of accuracy for thetrees. It was reasonable that the random sampling could still be achievedwith different ratios while continuing to minimize correlation, as long as thesampling method with separate pointers was maintained. Our implementa-tion allowed for different ratios to be used: [Fxx ,Fxy ], where x and y could bedefined as hyperparameters.3.4.3 Threshold for tree inclusionSimilar to the problem solved by adjusting the range of features selectedby the RRF algorithm, a problem was faced where some very poorly per-forming trees were included in the forest due to certain combinations offeatures, or particularly small sets of features. A solution was to set a user-defined threshold for inclusion of a tree in the forest, based on the amountof variance explained by the tree (R2) for the training set. We have de-noted this threshold as TIR2, and it should not be confused with the featureselection thresholds discussed in Section 3.4.1.513.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)The addition of TIR2 naturally caused concern related to overfitting thetrees, so a generalizability test was added to ensure that the threshold cho-sen did not cause overfitting that negatively impacted the results of testingon the trees. This meant that the train/test split of the data changed to atrain/test/validation split. Comparing performance of the testing and vali-dation set further ensured that overfitting was not affecting the results, andallowed for a truly blind test of the prediction on the validation set. Over-fitting was determined to be present when there was a drop in performancebetween the testing and validation sets.3.4.4 Implementation of EXRRF methodThe EXRRF method was implemented in Python, with the data cleaningand preprocessing (Step 1) happening outside of the packaged implementa-tion, as the specific cleaning tasks varied depending on the data set. Theprediction of interest was a regression problem, so classification was not im-plemented. However, adding a classification prediction would be a trivialchange to the implementation. The key difference is that regression averagesthe result, while classification uses majority voting to come to a decision.When it came to implementation of Algorithm 1, Jain and Jana proposedusing Prim’s algorithm to identify the MST. The expectation is that thegraph G will always be dense due to the weights being based on correlation.Due to the ability of Prim’s algorithm to handle dense graphs well in relationto both memory usage and speed [29], we proceeded with implementationof Algorithm 1 using Prim’s algorithm.The implementation of the method accepts tuning hyperparameters todefine the number of trees, the feature selection threshold, the train/test/val-idation split, a set seed, the maximum depth of the trees, the TIR2 thresholdfor tree inclusion, and the ratio of features to be selected during the RRFalgorithm in Step 3. This meant that although the model was modified toaccount for different feature selection thresholds, tree inclusion thresholds,and ranges of features selected, the parameters could easily be set to theones suggested by Jain and Jana, if necessary.An additional parameter was added to allow for performing the featureclassification steps in isolation, to help with data development. In testing,we noticed that seeing which features were identified as important helped toinspire ideas for engineering features that could provide further context. Al-ternatively, by seeing which features were important, it could help prioritizedata investigations and cleaning. The results would change when the datachanged, but by working with Step 1 and Step 2 of the process iteratively,523.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)Figure 3.10: An example of the generalizability test, from the FN Consul-tation prediction using the FTharmonic feature selection threshold.it was possible to greatly improve the quality of the data prior to buildingthe forest.The implementation stored all of the feature sets returned by the MST,MVER, and MVNR algorithms (Rs, Re, and Rn, respectively), along withthe classifications of the features as PFs, DFs, or BFs, the calculated fea-ture selection threshold, and the interim graphs and trees that determinedthe importance of the features. This allowed for transparency in the deci-sion making that determined the features’ relative importance. The resultsfrom all of the trees in the forest were stored to allow for interpretability,along with each tree’s decision path, contributing features, training data,testing data, and resulting prediction. This allows for investigation of spe-cific tree details, including trees that were pruned from the forest by TIR2.Specifically, if required, trees could be inspected to see what may have ledto particularly poor or accurate predictions.The implemented function returns the overall prediction summary statis-tics that include R2 for each of the train, test, and validation sets, the RMSE,MAE, overall decision path features, overall contributing features, trainingdata, testing data, and the averaged results of the prediction. This allowstransparency of where the prediction came from, as the results can be viewedas the aggregate summary, or individual trees in the forest can be inspected533.4. EXTENDED EXPLAINABLE REASONABLY RANDOMIZED FOREST (EXRRF)to see what led to a prediction.Lastly, a method was added to perform the generalizability test and printout a graphical representation of the change in R2 based on different TIR2thresholds for tree inclusion. Examples of these generalizability tests havebeen included in Appendix B.1.The generalizability test functions as a ‘what-if’ scenario, which testsdifferent minimum R2 thresholds for tree inclusion to see the effect thatthey would have on the training and testing R2 results. This allows for a‘peak’ in variance explained to be identified. I.e., it is possible to identifythe threshold that would overfit the data to the point where the predictionresults would start to suffer on new data. This is visible in Figure 3.10, wherewe can see that although the r2 train is stable regardless of the number oftrees pruned, the r2 test line destabilizes after approximately 280 trees areremoved (when trees with R2 < 0.98 are pruned). This generalizability testis blind to the validation data, so that the results on the validation set canconfirm that the information gained from the generalizability test does notcreate overfitting in the test set as well as the validation set. This alsoensures that the results are not biased based on knowledge gained from thegeneralizability test.54Chapter 4Predicting process durationAs discussed in Section 3.1, preparatory work indicated that the FN Con-sultation was expected to provide a lot of information about the timeline ofthe overall application. Simple random forest predictions had been reason-ably accurate in preliminary results, so the EXRRF method was applied topredict the FN Consultation duration, and provide additional informationabout feature importance and contributions. The promising results werethen extended to predict the overall application duration.4.1 Predicting FN Consultation durationOne of the key issues faced when applying the EXRRF method to the FNConsultation data was that the XRRF approach as proposed by Jain andJana [22] tested the model on regression data sets that had 14 to 128 featureseach. The initial data available prior to feature engineering only included6 features that were likely to be important. Feature engineering added 8additional features to add contextual information to the consultation data,and a further 3 features were pulled in from the higher application level toprovide high-level context for the consultation. This resulted in predictionbased on 17 features. Although this is at the lower end of the scale testedby Jain and Jana, it was reasonable to expect that this would be enoughinformation to support the prediction based on the promising results of astandard random forest model, with the understanding that most, if not all,of the features might be identified as important. The results of Jain andJana’s comparison [22] showed that on average the method kept 78% of thefeatures, but this was based on 10 benchmark regression data sets where thenumber of actually relevant features cannot be known, so there is a possi-bility that some of the features included in the data added no information.In order to combat this potential issue, due to the known importance of thevast majority of the features used for prediction, the selection ratio for thenumber of features selected during the tree construction was increased from[Fx4 ,Fx2 ] to [Fx4 ,3Fx4 ], where Fx is the number of features in each of the PFs,DFs, and BFs classes. This resulted in the feature selection counts summa-554.1. PREDICTING FN CONSULTATION DURATIONFeature Selection Threshold Fn PFs DFs BFsExpected 14 3 4 7FTpaper 17 4 3 10FTharmonic 17 2 11 4FTmean−sd 17 7 10 0FTmin2 17 6 10 1Table 4.1: Count of FN Consultation features in the data set (Fn), pivotalfeature set (PFs), dominant feature set (DFs), and basic feature set (BFs)for each of the feature selection thresholds, compared to the expected valuebased on average ratio kept for regression data sets in results obtained byJain and Jana [22].rized in Table 4.1. The expected count based on the average ratio of featuresselected for each set in Jain and Jana’s regression results has been includedin the table for reference [22]. We could have expected 3 of the featuresto be excluded, but instead we see that regardless of the feature selectionthreshold, all of the features were including, justifying their importance.4.1.1 FN Consultation prediction resultsThe EXRRF approach was applied to the cleaned FN Consultation data,and the results were averaged across three runs for each feature selectionthreshold. The averaged results for each feature selection threshold areshown in Table 4.2, and show consistent accuracy across all of the featureselection thresholds. The best accuracy was achieved by the FTpaper thresh-old, providing an MAE of 7.5 days, and an RMSE of 28.5 days. The MAErepresents an accuracy of around 29% of the mean consultation duration of26 days. The RMSE is more vulnerable to larger errors, and was expected tobe higher due to noisy data. The R2 values for the validation set are consis-tent, and show that regardless of the threshold, around 83% of the varianceis explained. Generalizability tests were run using the training and testingsets to ensure that the threshold for inclusion of training trees (TIR2) wasnot too high, and showed that the trees were not overfitting the data (seeAppendix B.1). This is supported by the fact that the R2 value is slightlyhigher for the validation set than for the testing set. In general, due to thepresence of outliers in the data, the R2 is expected to be lower in the testand validation set than in the training set. Changing the location of the564.1. PREDICTING FN CONSULTATION DURATIONFeature SelectionR2 train R2 test R2 valRMSE MAEThreshold (days) (days)16FTpaper 0.984 0.793 0.834 28.5 7.5FTharmonic 0.986 0.791 0.833 29.4 7.6FTmean−sd 0.987 0.783 0.835 29.5 7.5FTmin2 0.987 0.786 0.836 29.7 7.5Table 4.2: Results of the EXRRF model predictions on FN Consultationdata.outliers by setting different seeds showed that the lower R2 values persistedin the set with the most outliers.The results of the feature selection algorithms have been summarized inFigure 4.1. The number of features classified as PFs are shown in green,DFs are shown in blue, and BFs are shown in pink. The results show thatthe MST algorithm consistently identified the features classified as basic,and that the differing logic behind the different feature selection algorithmsresulted in different results for each of the MST, MVER, and MVNR algo-rithms. The classification of the features’ relative importance was consistentbetween runs of the same feature selection threshold. Table 4.3 shows thecontributing features for each feature selection threshold, and shows consis-tent features between the FTmean−sd and FTmin2 thresholds. Notice alsothat three of the feature selection algorithms identified the year the consul-tation started and whether the consultation was initiated after the BRFNruling as contributing features. This indicates that regardless of the featureselection algorithm, those features add a lot of information to the model.This is supported by domain knowledge. The interpretability of the resultsallowed this information about relative importance and contributing featuresto be identified, along with the features in the decision paths of trees in theforest. The EXRRF results verified the expectation that every feature in thedata set was in the decision path for the trees, supporting that there wereno superfluous features. The implications and value of this information arediscussed in more detail in Section 5.1.2.The distribution of the test and validation residuals for the EXRRF pre-diction with FTpaper threshold applied is shown in Figure 4.2. The trainingdata has been excluded, and the histogram window has been set to show the16The mean consultation duration in the data is 26 days.574.1. PREDICTING FN CONSULTATION DURATION(a) Breakdown of feature classifi-cation results for the FTpaper fea-ture selection threshold(b) Breakdown of feature classifi-cation results for the FTharmonicfeature selection threshold.(c) Breakdown of feature classifica-tion results for the FTmean−sd fea-ture selection threshold(d) Breakdown of feature classifi-cation results for the FTmin2 fea-ture selection threshold.Figure 4.1: Illustration of FN Consultation feature distribution across theRs, Re, and Rn sets chosen by each of the MST, MVER, and MVNR featureselection algorithms, respectively. Green numbers represent pivotal features,blue numbers represent dominant features, and pink numbers represent basicfeatures.584.1.PREDICTINGFNCONSULTATIONDURATIONFeature FTpaper FTharmonic FTmean−sd FTmin2Application priority level - - X -FN Consultation type - - X XFN review status - - X XCirculation method - - X XGeneralized FN postal code X - - -Number of days FN given to respond - X - -Number of FN groups consulted - X - -Month consultation started - - X XYear consultation started X - X XInitiated after BRFN ruling X - X XReview classified as not required X - - -Table 4.3: Features classified as contributing features (CF) in the XRRF FN Consultation prediction.594.2. PREDICTING APPLICATION DURATIONFigure 4.2: The residual distribution for the FN Consultation duration pre-diction with the FTpaper feature selection threshold applied.distribution within ±50 days of zero, with 50 days being approximately dou-ble the average FN Consultation duration. The strong presence of residualsaround zero supports the accuracy of this model. Residuals were similarlydistributed for the other three thresholds, see Appendix B.2, Figures B.11,B.12, and B.13.We establish a target accuracy within half of a week, or ±3 days. Thisrepresents approximately 12% of the mean FN Consultation duration of 26days. Investigating the residual distribution with FTpaper threshold appliedshows that of the 10,200 residuals in the combined test and validation set,6715 are within ±3 days of the true value, representing 66% of the predic-tions. Of those, 23 are equal to the true value (residual = 0), 809 are anunderestimation of the truth (positive residual), and 5883 are an overesti-mation of the truth (negative residual). Overestimations of the durationare desirable for this problem, as they avoid creating shortfalls by underes-timating the predicted duration.4.2 Predicting application durationBuilding a model to accurately predict the application duration is thefinal goal of this research, and the success of the EXRRF model in predict-ing the duration of the FN Consultation supported applying the model topredict the application duration. Having a model to accurately predict theFN Consultation provides an additional feature to consider when predicting604.2. PREDICTING APPLICATION DURATIONthe application prediction process. While it was unlikely that the EXRRFmethod would be able to explain all of the variance in the application data,the results from the preliminary work, the explainability of the method, andthe possibility of building a strong baseline model to improve upon were allstrong reasons to proceed with the EXRRF method.The features used for application duration prediction included the predic-tions for expected duration of the FN Consultation process in the currentrevision. The FN Consultation duration predicted the duration for everyFN community consulted on an application. This meant that each revisionof the application might have more than one FN Consultation predictionvalue. Three runs predicting the FN Consultation duration were completedfor each feature selection threshold, and the results were averaged acrosseach threshold to provide a result for each FN community on the applica-tion. To aggregate these predictions, the maximum prediction of any of theconsultations was extracted for each application revision, where available.All values where there were no FN Consultation tasks recorded for the appli-cation were filled with a zero value for the prediction. The actual durationof FN Consultation tasks in previous revisions was included as contextualinformation that would be available during the application process.There were multiple possible approaches to take when choosing how topredict the application duration. Preliminary work discussed in Section 3.1was based on predicting the duration of the overall application, based ontime taken for all revisions, and based on a cumulative count of activitiesand idle time that had occurred over the length of the entire application.This approach was valid, but did not take into account the informationthat might be available in early revisions of the application, or the differentstages where prediction might add the most utility to the BC-ER. We con-sidered two possible models for prediction: one that would make a simpleprediction based on all available information for the application; and onethat used past and current information available in each revision. This sec-ond method allowed for prediction during any revision of the application,based on whatever information was available at the time the revision wasprocessed.4.2.1 Model 1: Predicting overall application durationThe first model in consideration was selected to provide a predictionthat could be approximately compared to the results obtained by the pre-liminary work. The timeline for prediction is outlined in Figure 4.3, andproposed using all of the information related to an application to predict614.2.PREDICTINGAPPLICATIONDURATIONFigure 4.3: Explanation of the prediction window for predicting the application duration during the final revision,based on information known at the time of the final revision. Specific application details are a hypotheticalexample.624.2. PREDICTING APPLICATION DURATIONFeature SelectionR2 train R2 test R2 valRMSE MAEThreshold (days) (days)17FTpaper 0.970 0.514 0.545 39.9 20.6FTharmonic 0.968 0.533 0.535 41.5 21.1FTmean−sd 0.972 0.534 0.535 41.3 20.9FTmin2 0.965 0.473 0.543 43.4 23.1Table 4.4: Results of the EXRRF method predictions for the entire appli-cation duration, during the last revision (Model 1).the overall application duration. This figure outlines a hypothetical exam-ple of the application process, with 3 revisions. Each preliminary revisionhas an amount of time taken to process application activities associated withthe model, followed by a dormant period while the applicant revised the ap-plication. This dormant time represents the time between an applicationrevision being requested, and the time the new revision is received. Theamount of time taken for these combined steps is considered the revisionduration, which is measured as the time elapsed between revisions beingreceived. The last, approved, revision of an application measures durationas the time elapsed between the revision being received and approved. Thismodel aims to predict only the cumulative duration of the application. Thismodel was unlikely to have as much utility for the BC-ER, as it requiresknowledge that the application is in its final revision, and does not provideany interim predictions during previous revisions.The differentiation of this model was achieved by limiting the data toobservations representing the final revision of each application. The resultsof the model are summarized in Table 4.4. Although the results were againconsistent between different feature selection thresholds, the best result wasachieved by the FTpaper threshold, with MAE of 20.6 days. This repre-sented an accuracy of around 32% of the mean application duration of 65days, while the RMSE of 39.9 days represented approximately 61% of themean duration. These results cannot be directly compared to the prelim-inary model results in Table 3.1, due to differences in data caused by theextensive data cleaning and feature engineering implemented for the EXRRFmethod. However, these EXRRF results represent an obvious improvementin accuracy. This model functioned primarily to show improvement over pre-17The mean duration in the data is 65 days.634.2. PREDICTING APPLICATION DURATIONvious methods, and to confirm the value of using EXRRF for the applicationduration prediction.4.2.2 Model 2: Predicting application revision durationModel 2 provides the most practical and useful model of prediction. Itallows for prediction of the duration of the current revision, based on in-formation available at the time of the current revision. As a result, theprediction could be reliably used during any point in the application pro-cess, and offers the chance to eventually augment it with predictions forwhether a revision might be needed, and potentially a prediction for howmany more revisions might be needed. The timeline for prediction withthis model is outlined in Figure 4.4. The timeline of the application andrevisions themselves are the same as those described in Section 4.2.1. Notethat the key difference between the model described in Figure 4.4 and themodel described in Figure 4.3 is the presence of multiple possible predictionwindows based on revision, denoted as q, r, and s in the figure.Feature SelectionR2 train R2 test R2 valRMSE MAEThreshold (days) (days)18FTpaper 0.942 0.529 0.536 36.5 19.0FTharmonic 0.942 0.534 0.522 37.5 19.5FTmean−sd 0.946 0.593 0.567 37.1 19.1FTmin2 0.939 0.573 0.560 37.1 19.1Table 4.5: Results of the EXRRF method predictions for the duration atthe end of the current revision, based on information available in the currentrevision (Model 2).The EXRRF method was applied to the full set of application revisiondata, resulting in a shorter mean duration of 54 days compared to the meanoverall application duration of 65 days in Model 1. The results are shownin Table 4.5, and show that the most accurate results were obtained by theFTpaper threshold. The MAE of 19.0 days represents approximately 35%of the mean duration of 54 days, while the RMSE of 36.5 days representsapproximately 68% of the mean duration. Although the results for Model1 had slightly higher accuracy, it is reasonable to expect this drop in per-formance for Model 2 due to the change in the data caused by the inclusion18The mean duration in the data is 54 days.644.2.PREDICTINGAPPLICATIONDURATIONFigure 4.4: Explanation of the prediction windows for predicting the application duration during any revision,based on information known during revision being predicted. Specific application details are a hypotheticalexample.654.2. PREDICTING APPLICATION DURATIONof all possible revisions. Including all revisions in the data increased thenumber of available observations, but introduced more variance in the datacaused by the applications where revisions were required. This was likelydue to tasks being interrupted or halted when it was determined that a re-vision would be necessary. The higher utility of Model 2 drives the supportfor this model and justifies the slight decrease in accuracy.As with the FN Consultation, generalizability tests were run using thetraining and testing sets to ensure that the tree inclusion threshold (TIR2)was not too high. The generalizability tests are available in Appendix B.1.Parameters were tuned to verify that although the trees were intentionallyoverfitting the training set, the overfitting was minimized as much as possiblein the interest of balancing the bias-variance trade-off in the trees. Reducingthe overfitting in the training set resulted in too general of a model, andunderfitting on both the validation and testing sets. The generalizabilitytests show that the R2 values are similar between the validation and testsets, verifying that although the training set was overfit, the amount ofvariance explained in the train and test sets was reasonable, and the MAEwas still reasonable. Although extreme outliers were removed from the data,there was still significant variability in the application duration (see Figure3.9). As a result, the R2 values achieved by the model were lower thanthe FN Consultation model, but exceeded the goal of R2=0.5 that we hadtargeted based on our knowledge of the available data. The long tail in thedata distribution, combined with the low R2 value indicates that there issome unknown contextual information, that should be explored in futuredevelopment.The results of the different feature selection algorithms have been sum-marized in Figure 4.5. Similarly to the results from the FN Consultationprediction, the MST algorithm identified the most features in the BFs. Withthe application data, we can see that the different feature selection thresh-olds provide a bit more variance in the classification of features by the algo-rithms. Notice that the MVER algorithm with FTmin2 threshold was ableto identify some basic features. This was the only example in this researchwhere the MVER algorithm was able to identify basic features. The con-tributing features have been summarized in Table 4.6, which shows similarcrossover between the FTmean−sd and FTmin2 thresholds. Note that three ofthe feature selection thresholds identified the risk index and the number ofidle periods in the application as contributing features. Specific applicationtasks and activities were not listed, but each of the thresholds considered atleast one of these engineered features as important, supporting that the com-plexity added by the tasks and activities is being considered in the model.664.2. PREDICTING APPLICATION DURATION(a) Breakdown of feature classifi-cation results for the FTpaper fea-ture selection threshold(b) Breakdown of feature classifi-cation results for the FTharmonicfeature selection threshold.(c) Breakdown of feature classifica-tion results for the FTmean−sd fea-ture selection threshold(d) Breakdown of feature classifi-cation results for the FTmin2 fea-ture selection threshold.Figure 4.5: Illustration of application feature distribution across the Rs,Re, and Rn sets chosen by each of the MST, MVER, and MVNR featureselection algorithms, respectively. Green numbers represent pivotal features,blue numbers represent dominant features, and pink numbers represent basicfeatures.674.2.PREDICTINGAPPLICATIONDURATIONFeature FTpaper FTharmonic FTmean−sd FTmin2Application proponent - X - -Risk index - X X XNumber of idle periods - X X XIdle time - - X XNumber of other open applications - - X XDuration of FN Consult in previous rev. - - X XApplication received after ruling - - X XPipeline length - - X XYear application submitted - - X XCount of each application task type 4 3 4 3Count of activities associated with app. X - - -Count of each application activity type 2 1 4 4Table 4.6: Features classified as contributing features (CF) in the EXRRF application prediction with Model 2.In order to preserve data privacy, in addition to the large number of application and activity types, applicationtasks and activity tasks deemed important are aggregated.684.2. PREDICTING APPLICATION DURATIONFeature Selection Threshold Fn PFs DFs BFsExpected 34 7 10 17FTpaper 43 7 0 36FTharmonic 43 7 23 13FTmean−sd 43 16 27 0FTmin2 43 15 25 3Table 4.7: Count of application features in the data set (Fn), pivotal featureset (PFs), dominant feature set (DFs), and basic feature set (BFs) for eachof the feature selection thresholds, compared to the expected value based onaverage ratio kept for regression data sets in results obtained by Jain andJana [22].The decision path features were also extracted, and matched the full listof features considered for the model, showing that although some featureshad differing levels of importance in the prediction, all of the features werenecessary to come to the final prediction value.The expected and actual counts of features in each set, for each threshold,are shown in Table 4.7. Based on the average number of features kept in theresults provided by Jain an Jana, we might expect up to 9 of the featuresto have been pruned from the model. However, the extensive data cleaningand engineering performed on this data extracted features that were knownto add little to the model. This domain knowledge, combined with theperformance of the model, verifies that it was valid for the feature selectionmethod to not prune any features completely out of the model.To further investigate the accuracy of Model 2, we can consider theresidual distributions. The distribution of the test and validation residualsfor the EXRRF prediction with FTpaper threshold applied is shown in Figure4.6. The training data has been excluded, and the histogram window hasbeen set to show the distribution within ±100 days of zero, with 100 daysbeing approximately double the average application duration. We establish atarget accuracy range for the application duration prediction of ±7-10 days,or between a week and a week and a half. This represents approximately13-19% of the mean application duration of 54 days, when considering allrevisions. These targets have been added to the residual distribution inFigure 4.6, as red dashed lines for the outer target of ±10 days of truth, andas green dashed lines for the more strict ±7 days target. Notice that theresiduals are slightly skewed due to the outliers in the data. The mean of694.3. COMPUTATIONAL TIMEFigure 4.6: The residual distribution for the application duration predictionwith the FTpaper threshold applied. Red dashed lines indicate target of ±10days of truth, green dashed lines indicate final target of ±7 days of truth.the residuals is 0.05, and the median of the residuals is -4.42 days, showingthat this is due to slight skew in the residuals, rather than bias in themodel. Residuals were similarly distributed for the other three thresholds(see Appendix B.2, Figures B.14, B.15, and B.16).Looking more closely at the residual distribution with FTpaper thresholdapplied shows that of the 4422 residuals in the combined test and validationset, 2355 are within ±10 days of the true value. Of those, 1680 are an over-estimation of the truth (negative residual), and 675 are an underestimationof the truth (positive residual). This means that 53% of the residuals meetour broader accuracy target. Looking at the more strict accuracy target of±7 days, 1678 of the residuals fall within that range, representing 38% ofthe residuals. Of the residuals that meet the more strict target, 1155 are anoverestimation of the truth (negative residual), and 523 are an underesti-mation of the truth (positive residual). Overestimated predicted durationsare more desirable, for the same reasons mentioned for the FN Consultationprediction in Section 4.1.1.4.3 Computational timeAll of the EXRRF model predictions were run using a laptop with a 1.8GHz Intel Core i7 processor and 16GB of RAM. Despite the basic specifi-cations, the EXRRF method ran quickly, typically within a minute. Themost time consuming process in the prediction was the generalizability test,704.3. COMPUTATIONAL TIMEwhich required recomputing the results k times, where k is the number oftrees in the forest. Each generalizability test took between 1 and 3.5 min-utes, depending on the size of the forest. As such, running the full results forall 4 feature selection thresholds with generalizability tests could take up to18 minutes. Due to the large number of hyperparameters, tuning the modelwas a time-intensive process. As familiarity with the parameters’ impacton model performance increased, it was faster to tune the model based onexperience rather than a grid search of parameters.71Chapter 5DiscussionWhen discussing the results obtained by the EXRRF method, there aremultiple considerations that determine the validity of the results. The firstconsideration is whether the changes made to the proposed XRRF modelprovided added information or accuracy over the originally proposed XRRFmodel. To add further depth to this evaluation, it is necessary to comparethe results obtained to a traditional random forest approach, to determinewhether the EXRRF method had validity in comparison to a traditionallyaccepted method. Finally, limitations associated with the EXRRF methodwill be explored, as an additional consideration when evaluating the results.5.1 Validity of changes to XRRF methodThe changes to the original XRRF method proposed by Jain and Janaare detailed in Section 3.4, and primarily consist of a differing data cleaningapproach, addition of multiple feature selection thresholds, a broader rangeof features used to build the trees in the forest, and a minimum R2 thresholdfor tree inclusion in the forest.5.1.1 Impact of changes to data cleaningAny change to model accuracy or performance cannot be measured be-tween the original XRRF method, and the EXRRF method implemented inthis research, as the original data cleaning methods proposed were not imple-mented. However, due to domain knowledge gained through collaborationwith our BC-ER partner, it was possible to determine the most appropriatedata cleaning method for our specific data set. The number of data pointslost by excluding incomplete records was minimal, so the information gainedfrom including them with imputed data would be similarly minimal. Thepotential for the strongest difference between the proposed method and theactual implementation is the exclusion of noisy data. The BC-ER data hasa lot of noise, as evidenced by Figure 3.8 and Figure 3.9 that show the dis-tribution of FN Consultation duration, and the distribution of application725.1. VALIDITY OF CHANGES TO XRRF METHODFeature SelectionR2 train R2 test R2 valRMSE MAEThreshold (days) (days)19FTpaper 0.916 0.529 0.543 43.0 19.3FTharmonic 0.916 0.519 0.524 43.8 19.6FTmean−sd 0.916 0.512 0.524 43.8 19.5FTmin2 0.919 0.510 0.544 42.1 19.0Table 5.1: Results of the EXRRF method predictions for the duration atthe end of the current revision, based on information available in the currentrevision, limited to applications received prior to the BRFN ruling in 2021(Model 2).duration and idle time respectively. This noise is due to how complex theprocesses are, resulting in a wide variety of factors that could influence theduration and idle time values.As an example situation, let us consider the BRFN ruling in 2021 [54].When this ruling occurred, any permits with FN Consultations were puton hold, in order for BC-ER to find an appropriate way to accommodatethe ruling’s requirements. This caused an expected and measurable shiftin behaviour. Similarly, data studies of application processing counts andtimes in 2020 reflected the work stoppage that occurred at the beginningof the COVID-19 pandemic. If we were to use a data cleaning methodto remove noisy data, much of the applications that were being processedduring that time would have been excluded, potentially limiting the validityand usefulness of the final prediction model. In order to test the robustnessof the model to the noise introduced by the BRFN ruling, prediction wasalso completed with data limited to applications received prior to the ruling.Table 5.1 shows the results when we limit the data.Excluding the applications received after the ruling excluded 10% of thedata shortened the mean application duration from 54 to 45 days. Whilethe magnitude of the MAE values are similar to the MAE from Table 4.5at around 19 days, the accuracy actually decreased when considering theMAE compared to mean duration in the data. Due to the shorter meanapplication duration, the MAE as a proportion of mean duration increasedfrom 35% when including all data to 42% when excluding data received afterthe ruling. The lower R2 values indicate that less variance is explained over-19The mean duration in the data is 45 days.735.1. VALIDITY OF CHANGES TO XRRF METHODall, and the higher RMSE values are further indications of loss of accuracy.The decrease in accuracy and variance explained shows that although thenoise was included in the original results with all data, the model is able toaccount for it, and explain the changes. This is supported by the model’sclassification of the engineered features used as indications of the processshift as important, showing that the model recognized the shift and usedthe feature information to explain it. This validates that the data cleaningimplemented was appropriate for this data set specifically.The results were not re-computed with data that excluded applicationsaffected by COVID-19. The exact timeline for effect is less clear for thepandemic than for the ruling, but we know that the majority of the influenceon the data happened in 2020. However, there is some crossover betweenthe data affected by the pandemic and the data affected by the BRFNruling. We expect that the model utilizes the engineered feature indicatingthe year to account for that process shift, in a similar way to how the modelaccounted for the BRFN process shift. This has created an avenue of futurework, where we would like to consider whether we can identify contextualfactors that are not present directly in the data, and account for them in amodel.5.1.2 Impact of additional feature selection thresholdsWhen considering the multiple feature selection thresholds that were im-plemented, first we need to evaluate Jain and Jana’s claim that the FTpaperthreshold considers ‘features with low feature-to-feature correlation and highfeature-to-class correlation’ as ‘highly relevant’ [22].We will consider the FN Consultation prediction as an example, due toa lower number of features that allow for a summary comparison of results.For the purpose of our example, the feature names have been replaced withnumbers for simplicity of comparison. Table 5.2 shows the average feature-to-feature correlation, and each feature’s average correlation to the responsevariable (as this is a regression problem, not a classification problem). Wewill use the average as a guide to consider which values might be consideredlow or high.The CFs from Table 4.3 have also been included in Table 5.2 for easeof referencing which features each feature selection threshold identified asimportant. The counts of each CF’s presence in tree DP have been addedto further quantify relative importance. The feature selection thresholdproposed by Jain and Jana, FTpaper, considers features 3, 4, 5, and 14 to beimportant. If we look at the correlation values shown in Table 5.2, we see745.1. VALIDITY OF CHANGES TO XRRF METHODFeature Correlation Average corr.P H M-SD MTNumber to response to features1 0.078 0.178 - - - -2 0.003 0.031 - - 215 2163 0.007 0.119 67 - 217 2124 0.047 0.123 50 - 185 1845 0.195 0.246 51 - - -6 0.090 0.086 - - - -7 0.156 0.158 - - - -8 0.031 0.032 - - 185 -9 0.205 0.101 - - 189 19610 0.170 0.189 - - 205 20411 0.028 0.134 - - - -12 0.170 0.221 - - 189 18513 0.071 0.175 - - - -14 0.033 0.186 54 - - -15 0.406 0.147 - 173 - -16 0.085 0.194 - 166 - -17 0.102 0.179 - - - -Avg.0.111 0.147Corr.Num.70 220 250 230TreesFeature selection thresholds: ‘P’ = FTpaper, ‘H’ = FTharmonic,‘M-SD’ = FTmean−sd, and ‘MT’ = FTmin2.Table 5.2: Feature-to-class, average feature-to-feature correlation, andcounts of CF presence in tree DPs for each numbered FN Consultationfeature. Features with higher than average feature-to-class and lower thanaverage feature-to-feature correlation are bolded for ease of identification.755.1. VALIDITY OF CHANGES TO XRRF METHODthat out of the features deemed important, two of them have the expectedlower than average feature-to-feature correlation, with the other two havinghigher than average correlation. Similarly, only one of the four featureshas higher than average feature-to-class correlation, with the other threehaving much lower correlation to the response. One possible explanation forthis behaviour is that the feature interactions considered by the algorithmsJain and Jana use for feature selection are not visible by simply reviewingthe correlation values, meaning that we cannot verify the effect of featureinteractions on the selection.The FTharmonic feature selection threshold consistently selects differentfeature sets than the FTpaper threshold, which justifies the differentiation ofFTpaper from the true harmonic mean calculation. However, by reviewingthe features selected by the harmonic mean threshold with domain knowl-edge, the features selected are important to the data and would logicallybe important factors in how long an application might take. Additionally,by looking at feature 15, we can see that the FTharmonic threshold is theonly one that identified that feature as important, even though it meetsthe target lower than average feature-to-feature correlation and higher thanaverage feature-to-class correlation.The FTmean−sd and FTmin2 thresholds consistently returned similar re-sults, which indicates that although they are different calculations, they’reresulting in similar decision making via pruning. These thresholds tend toprioritize more features as important than the other two thresholds (see Fig-ures 4.1, 4.5, and Tables 4.1, 4.7, 4.3, and 4.6). Based on the results fromall of the feature selection thresholds, we can confidently say that all of thefeatures included in the prediction are important in some way. When weconsider the domain knowledge available, we can also confirm that the fea-tures selected by the FTmean−sd and FTmin2 thresholds identify the featuresthat explain most of the variance in the data caused by process changes.We then consider whether the differences in behaviour between the fea-ture selection thresholds address the concern relating to the inability of theMVER and MVNR algorithms to identify any basic features. When weconsider the results shown in Figure 4.1 and Figure 4.5, we can see thatthe differing logic of the FTmean−sd and FTmin2 does address the issue ofFTpaper always selecting every feature. However, in the majority of cases,the MVER and MVNR algorithms are still unable to identify any basic fea-tures. We conclude that this is intended behaviour based on the specificfeature selection algorithms used, and is not a result of the feature selectionthresholds.Overall, the information gained through all of these feature selection765.1. VALIDITY OF CHANGES TO XRRF METHODthresholds is important, and can provide similar functionality to identifyingthe contributing features of the trees. The contributing features consistentlyidentify all of the pivotal features, and Jain and Jana proposed that thenumber of times contributing features are in the decision paths of treescould help further contextualize relative importance. This same logic canbe applied by considering the contextual information gained from all of thefeature selection methods.For example, Table 5.2 shows that features 3 and 4 are identified asimportant by 3/4 of the feature selection thresholds. We can consider thesetwo features to be ‘more’ important than the other contributing features,which is supported by our domain knowledge. Feature 3 is an indicator ofthe year that the consultation was started, and feature 4 is an indicationof whether the application was received after the BRFN ruling. Together,these features identify a significant process shift, which also supports thatthe EXRRF method considers feature interactions in addition to isolatedfeature importance.With our predictions, we would consider the FTsd or FTmin2 thresh-olds to have the most value, for both the FN Consultation and applicationduration predictions. The results are comparably accurate to the FTpapermethod, but they identify features that our domain knowledge confirms arelikely important, and that appear in a higher proportion of tree decisionpaths (DPs). Although it would be possible and reasonable to leverage theinformation gained from all four feature selection thresholds, by averagingthe predictions and considering the relative importance of features identi-fied by all thresholds, we can reasonably move forward by simply identifyingwhich threshold provided the most information and accuracy.5.1.3 Impact of broadening range of features included inCARTsIn the specific application of the EXRRF method to the BC-ER data, itis highly relevant to broaden the range of features included in the CARTs.While it makes sense to narrow down the features for data sets where thenoisy data has been removed, more features are required to explain thenoise in the BC-ER data. We have already established that the featuresassociated with both the FN Consultation and application data sets do notcontain any superfluous information that can easily be pruned out. Addi-tionally, especially when considering the FN Consultation data, the numberof features available (17) is not very high. By broadening the random sam-pling range from [Fx4 ,Fx2 ] to [Fxx ,Fxy ], we have the ability to increase the775.1. VALIDITY OF CHANGES TO XRRF METHODFigure 5.1: The generalizability test result for the FN Consultation predic-tion using the FTpaper feature selection threshold.number of features sampled in the trees, which will help explain more of thevariance in our data. The counts of each CF’s presence in tree DPs shownin Table 5.2 show that regardless of the number of trees kept in the forest,none of the features appear in all of the DPs. This validates that althoughthe number of features included increased, it did not result in homogeneityin the forest. Attempting experiments to compare the differences in modelperformance when the range is changed for noisy versus noiseless data couldbe an interesting future research direction for this model.5.1.4 Validity of setting a minimum R2 threshold for treeinclusionThe generalizability tests are a good tool to illustrate the validity ofthe TIR2 threshold for tree inclusion. Figure 5.1 shows an example of thegeneralizability test for the FTpaper feature selection threshold applied to theFN Consultation duration prediction. This example was chosen as the TIR2threshold had the greatest impact on this prediction. We select the TIR2value by identifying the point where the testing R2 peaks or destabilizes, andsetting TIR2 to the smallest value that maintains stability. We can see that785.2. INTERPRETABILITY GAINED FROM EXRRF METHODif no TIR2 threshold was applied, the training R2 would sit around 0.90, andthe testing R2 would sit around 0.65. However, by setting TIR2 to 0.98, wehave gained nearly 15% of additional explained variance in the training set,while minimizing the amount of overfitting caused by throwing out trees thatpoorly predicted the training set. The generalizability test allows us to saywith certainty that this threshold is not too high, and balances informationgained with added accuracy. This is supported by the relative stability ofthe training R2, regardless of TIR2. This process could be optimized andautomated in the future, by leveraging calculations that will compare theresults of the trees with different TIR2 values.5.2 Interpretability gained from EXRRF methodThe interpretability gained from the EXRRF method is widespread andimmediately evident from the information provided in Section 5.1.2. All ofthe information provided was available because the EXRRF method allowsfor in-depth analysis of the relative importance of features.The classification of features and extraction of contributing features al-lowed us to narrow down and prioritize the number of features of interestfrom 17 to 11 for the FN Consultation data, and from 43 to 24 for the ap-plication data. This allowed us to use the information gained in meetingswith the BC-ER to explain specific examples of the model performing wellvs performing poorly, which allowed us to identify that there are additionalcontextual factors that affect the processes and that are not currently avail-able in the data we explored. Overall, the interpretability gained meant thatthe model was not black-box like traditional random forests. This justifiedthat the information gained from the EXRRF model provided more valuethan a traditional random forest, along with satisfying the explainabilityrequirements put forth by the BC-ER. A future avenue for research couldbe to identify potential external factors that might affect application timeby exploring the connections between cases, and by considering keywordspresent in notes associated with applications.5.3 Accuracy compared to traditional randomforestsThe key gain over traditional random forests is the additional informa-tion gained from the comprehensive data cleaning and feature engineeringdriven by the EXRRF process. Table 5.3 shows the results comparison795.3. ACCURACY COMPARED TO TRADITIONAL RANDOM FORESTSMethod R2 test (days) RMSE MAE20Initial random forest 0.899 29.8 12.2EXRRF with FTharmonic 0.793 28.5 7.5Random forest with EXRRF data 0.896 28.3 7.7Table 5.3: Comparison of EXRRF FN Consultation duration predictionresults to Random Forest prediction results before iterative EXRRF featureengineering, and after.Method R2 test (days) RMSE MAE21Initial random forest 0.561 100.1 43.7EXRRF with FTpaper 0.529 36.5 19.0Random forest with EXRRF data 0.772 37.2 18.8Table 5.4: Comparison of EXRRF application duration prediction resultsto Random Forest prediction results before iterative EXRRF feature engi-neering, and after.between the initial random forest prediction of FN Consultation durationwith cleaned data prior to the addition of engineered features, the bestEXRRF results, and the random forest prediction with the data set used forthe EXRRF method. Without the extra features engineered as part of theEXRRF process, the random forest results are significantly less accurate,with an MAE of 12.2 days. The accuracy of the random forest results withthe cleaned data from the EXRRF process are improved, and are similar tothe EXRRF results, but are tempered by the fact that the random forest isa black-box approach, and provides none of the additional interpretabilityor explainability that the EXRRF method offers.Similarly to the results for the FN Consultation duration prediction,the EXRRF and random forest results (with and without EXRRF data)are shown in Table 5.4. As expected based on the trend seen with the FNConsultation data, the initial random forest prediction is significantly lessaccurate than the EXRRF results. The random forest using EXRRF dataprovides comparable results to the EXRRF results, but is again lacking20The mean consultation duration in the data is 26 days.21The mean duration in the data is 54 days.805.4. LIMITATIONSinterpretability and explainability of the features affecting the predictiondue to the black box model.Overall, a lot of the improvement in both predictions was directly tiedto the work done for the EXRRF model. Although we can see similarperformance on the same data, we can consider the improvement betweenthe original random forest and the EXRRF as specifically caused by theEXRRF process.5.4 LimitationsThere are a few key limitations that affect this research. It should bemade very clear that the modifications to the XRRF method are only verifiedfor application to the BC-ER data. Additionally, the validity of results hasonly been confirmed for application to this specific problem.The data used for this research spanned the period from mid 2016 to early2023. During this time, the BC Supreme Court ruled in favour of the BRFNregarding concerns related to permit impacts on treaty rights, causing asignificant delay in application processing while the FN Consultation processwas re-evaluated. The change in process driven by this ruling will impactthe ability of the model to explain the variance in the data as time goeson. The applications have only recently started to be processed again, soresults should be re-run once more data is available, in order to verify thatthe model still accounts for this shift.The XRRF method itself, and by extension the EXRRF method, haslimited efficacy when considering data sets with a small number of features,and a small number of observations. The noise of the BC-ER data set islikely at the outer limits of what can be explained by the EXRRF method.In order to balance the bias-variance trade-off, we opted to set a lower goalof 0.5 for our R2 values on testing and validation sets. This allowed usto prioritize slightly higher accuracy over more variance explained. Whencomparing the results of the EXRRF method to traditional random forestson the BC-ER data sets, we can see that traditional random forests are ableto explain more variance while maintaining similar accuracy. It is not clearwhy this is possible, as the traditional random forest is a black-box model.However, it does indicate that there are likely contextual factors affecting theprocesses that are not being captured in the randomized trees built duringthe EXRRF method. This provides strong motivation for future work toimprove the model.81Chapter 6Conclusion and Future WorkDeveloping computing solutions that bridge the gap between traditionalbusiness process mining and machine learning applications is a challengingissue. As the popularity of neural networks increases the complexity andaccuracy of machine learning approaches, developing interpretable methodsthat achieve similar accuracy is gaining importance. In many industries, es-pecially those with human impact, the ability to explain the decision makingbehind a prediction is critical in order for the result of the prediction to beunderstood and trusted.6.1 Summary contributionThe objective of this work was to solve the problem of how long a permitapplication might take to process at the BC-ER. The application process ishighly complex, which allowed us to start by predicting the FN Consultationprocess. By making the prediction at a lower level, and rolling the result upto the overall application as an additional feature, we were able to providegreater context and information to the overall application prediction.The BC-ER seeks insight into the permit application process, and theability to accurately predict the expected duration of the application pro-cess, in order to keep applicants informed. The BC-ER’s status as a reg-ulatory body lends even more importance to the need for an interpretableand explainable model. We identified a viable state-of-the-art interpretablerandom forest method, called XRRF, which was modified to suit the BC-ERprediction problem. This method leverages the accuracy of a random forest,while adding interpretability, and explainability via feature selection thatprovides information about the relative contribution of each feature to themodel’s prediction [22]. This approach allows the decision paths of the treesin the forest to be extracted, providing interpretability of the predictionlogic that supports the explainability added by the feature selection.We then implemented a modified version of the XRRF method, calledEXRRF, by comprehensively updating the data cleaning methodology tosupport application to the data associated with our prediction problem. We826.2. FUTURE WORKimplemented alternative logic for feature selection, and leveraged domainknowledge to verify the efficacy of our logic. This, combined with our domainknowledge, allowed us to increase the information gained from the EXRRFmethod. Finally, we extended the hyperparameters for tuning the EXRRFmethod, in order to increase the accuracy of the method, while maintainingthe improved interpretability.This EXRRF approach allowed us to build a strong model that accu-rately predicts the FN Consultation duration based on data extracted fromBC-ER systems, supplemented with engineered features that add furthercontextual information. Using this model, we have been able to provideinformation to the BC-ER about the factors that affect the consultationduration, and extract specific examples of consultations that followed anexpected pattern, as well as examples of consultations that deviated fromthe expected process pattern.These results were then extended to predict the entire application du-ration. We were able to meet our accuracy goals, and achieve our targetedvariance explained. We were also able to extend the explainablility of theEXRRF method to extract and investigate examples of expected and un-expected behaviour associated with the application process. The ampleinformation gained from the EXRRF model to support the predictions al-lowed us to build a strong plan for future development of a more complexmodel that will further improve our results.6.2 Future workThe primary goal of the future work associated with this research is tofurther improve the results of the overall application duration prediction.The prediction of the FN Consultation duration is strong, and meets ouraccuracy goals. The prediction of the application duration is a solid start,and provides better than expected results. However, this is still a highlycomplex process that will likely benefit from further breaking the processdown into smaller prediction problems.We initially identified the FN Consultation as the most critical sub-process to predict, but between domain knowledge and the results of theEBM in the preparatory work, we identified further sub-processes that addenough information to the model to justify their own prediction models. Wepropose a piecewise model that incorporates and extends the strong EXRRFprediction of FN Consultation. The model plan in Figure 6.1 combines whatwe learned from the preparatory work with the promising results of the836.2. FUTURE WORKFigure 6.1: The proposed piecewise model to predict the application dura-tion.846.2. FUTURE WORKEXRRF model.As discussed, the FN Consultation duration can be accurately predictedusing the EXRRF model. The Forestry, Archaeology, and Composite Ap-plication review tasks were all identified as important by the preparatorywork, and should be combined into a simple model due to the minimal dataavailable for each of their sub-processes. We previously incorporated the idletime within the application process as an engineered feature. We proposebreaking this idle time down further by considering the idle time that occursduring the sequentially executed application tasks separately from the idletime that occurs during the parallel execution of application tasks.We propose incorporating a method that considers inter-case and ex-ternal factors that may affect processing time, such as the inter-case fea-tures model proposed by Grinvald et al. [18]. This inter-case approach wasconsidered a potential way to account for delays caused by the COVID-19pandemic, as well as any further changes to the FN Consultation processresulting from the BRFN ruling. There are also a few attributes of the datathat could be accounted for with this process, specifically the applicationsthat are associated with major energy projects in BC.Another avenue for improvement related to missing contextual informa-tion would be to build an NLP model to evaluate the case notes associatedwith applications. This model may be able to generate contextual flags orindicators to supplement the data, and act as a prediction tool to determinewhether additional revisions might be required in the application. Knowingwhether a new revision will be required will add valuable information to themodel, but is currently not possible with a random forest approach, as thereis not enough relevant data.Further feature engineering could be considered, such as engineering acount of the number of applications proponents have submitted up to thepoint of the application. As mentioned in Section 3.3.3, a system changein 2016 would mean that using the data in our data set for this countwould not be representative. However, further exploration of data from theprevious system might improve the feasibility of this option. A differentapproach could also consider the number of applications that proponentshave in progress at the time of submission, similar to how the number ofopen FN Consultations were considered for the FN Consultation durationprediction.The proposed piecewise model also considers the potential of incorporat-ing a method that aims to add order to uncertain process segments, such asthe one proposed by Pourbafrani et al. [38]. This would allow the model toevaluate whether the tasks executed in parallel can be forced into a reason-856.2. FUTURE WORKable or legible order, in order to facilitate application of traditional processmining techniques to add further information to the model. Additionally,this approach could potentially allow for the idle time that occurs duringthese simultaneous tasks to be predicted.When it comes to the data managed by the BC-ER, and used in sup-port of the predictions developed here, it would be worthwhile to considerwhether further development of the data collection systems would providelong-term benefit to the commission. For example, the date stamps usedfor prediction do not include time, and currently result in some lost infor-mation when processes may only take a partial day. Adding a timestampmight clarify why so many processes seem to take ‘zero’ time, when they arecompleted on the same day they are started. Additional information couldbe gained by identifying the contextual information that is currently notrecorded in the system, and adding attributes to the system that can recordthat information. The BC-ER data is also imbalanced, as seen in Figure 3.8and Figure 3.9. Future work with the data should address the imbalance inthe data, either through further experimentation once more data is availablethat allows the data set to be balanced, or through generation of syntheticdata [26].A neural network approach was not considered suitable for this work,due to the relatively poor performance of simple neural networks shownin Table 3.1 and Table 3.2, combined with the limited data availability,the known imbalance in the data set, and the inherent black-box nature ofneural networks. However, a future avenue of research could be to explorethe validity of applying few-shot learning. LSTMs typically perform wellin process mining applications [42, 49], so leveraging few-shot learning as amethod to address imbalanced learning [51] could help to address the issueswith our imbalanced and limited data available for training. To address thenecessary interpretability, we could leverage Fisher Kernels [24] to identifywhich training data had the most influence on the model’s predictions.Future work related to the EXRRF method would focus on finding meth-ods to automate the tuning process. It is currently labour intensive, andwould benefit from implementation of tuning methods that would identifythe point where the bias-variance trade-off is optimized based on the predic-tion goals. One way of achieving this could be by performing a grid searchof the hyperparameters, and augmenting the current generalizability test toidentify and return the point where the testing R2 destabilizes. Anotherapproach could be to use a discrete black-box method for tuning, such asthe HyperNOMAD method presented by Lakhmiri et al. [25].The current EXRRF method uses a greedy approach to build a single866.2. FUTURE WORKminimum vertex cover from the MST, as we did not see measurable rele-vance in adding computational time to identify possible alternative MVCs.Further study could be completed to empirically evaluate whether identify-ing multiple MVCs from the MST in Algorithm 1 might affect the amountof information gained, and what computational load might be associated.When considering improvements to the accuracy of the method, it wouldbe valuable to evaluate optimization of the decision trees, to determine howmuch accuracy can be gained, while also preventing overfitting. One pos-sible approach could be to leverage a mixed-integer optimization approachsimilar to the one presented by Bertsimas et al. [5].The EXRRF method would also benefit from testing the extensions onadditional data sets to ascertain whether the added value is specific to thisdata set, or of benefit to a broader range of applications.87Bibliography[1] Arroba, P., Risco-Mart´ın, J. L., Zapater, M., Moya, J. M.,and Ayala, J. L. Enhancing regression models for complex systemsusing evolutionary techniques for feature engineering. Journal of GridComputing 13 (2015), 409–423. → pages 15[2] Aslam, N., Khan, I. U., Alansari, A., Alrammah, M., Algh-wairy, A., Alqahtani, R., Alqahtani, R., Almushikes, M., andHashim, M. A. Anomaly Detection Using Explainable Random Forestfor the Prediction of Undesirable Events in Oil Wells. Applied Compu-tational Intelligence and Soft Computing 2022 (Aug. 2022), e1558381.Publisher: Hindawi. → pages 18[3] Awad, A., Weidlich, M., and Sakr, S. Process Mining over Un-ordered Event Streams. In 2020 2nd International Conference on Pro-cess Mining (ICPM) (Oct. 2020), pp. 81–88. → pages 12[4] Baker, J. E., et al. Reducing bias and inefficiency in the selectionalgorithm. In Proceedings of the second international conference ongenetic algorithms (1987), vol. 206, pp. 14–21. → pages 30[5] Bertsimas, D., and Dunn, J. Optimal classification trees. MachineLearning 106 (2017), 1039–1082. → pages 87[6] British Columbia Energy Regulator. About — BC Energy Reg-ulator. Retrieved 2023/03/06 from https://www.bc-er.ca/about/,2023. → pages 1, 2[7] British Columbia Energy Regulator. Consulta-tion & Engagement — BC Energy Regulator. Re-trieved 2023/03/06 from https://www.bc-er.ca/how-we-regulate/engage-with-indigenous-communities/consultation-engagement/, 2023. → pages 3[8] British Columbia Energy Regulator. Engagement & Consul-tation with Indigenous Communities — BC Energy Regulator. Re-88Bibliographytrieved 2023/03/06 from https://www.bc-er.ca/how-we-regulate/engage-with-indigenous-communities/, 2023. → pages 20[9] British Columbia Energy Regulator. Freedom of Information— BC Energy Regulator. Retrieved 2023/03/06 from https://www.bc-er.ca/about/freedom-of-information/, 2023. → pages 33[10] British Columbia Energy Regulator. Review Applications — BCEnergy Regulator. Retrieved 2023/03/06 from https://www.bc-er.ca/how-we-regulate/review-applications/, 2023. → pages 2[11] Carmichael, Z., Langroudi, H. F., Khazanov, C., Lillie, J.,Gustafson, J. L., and Kudithipudi, D. Performance-efficiencytrade-off of low-precision numerical formats in deep neural networks.In Proceedings of the Conference for Next Generation Arithmetic 2019(New York, NY, USA, 2019), CoNGA’19, Association for ComputingMachinery. → pages 10[12] Chen, T., and Guestrin, C. Xgboost: A scalable tree boosting sys-tem. In Proceedings of the 22nd ACM SIGKDD International Confer-ence on Knowledge Discovery and Data Mining (New York, NY, USA,2016), KDD ’16, Association for Computing Machinery, p. 785–794. →pages 9, 13, 16, 20[13] Chowdhary, K. R. Natural language processing. Fundamentals ofartificial intelligence (2020), 603–649. → pages 10[14] Corallo, A., Lazoi, M., and Striani, F. Process mining andindustrial applications: A systematic literature review. Knowledge andProcess Management 27, 3 (2020), 225–233. → pages 7, 11[15] Dallagassa, M. R., dos Santos Garcia, C., Scalabrin, E. E.,Ioshii, S. O., and Carvalho, D. R. Opportunities and challengesfor applying process mining in healthcare: a systematic mapping study.Journal of Ambient Intelligence and Humanized Computing 13, 1 (Jan.2022), 165–182. → pages 18[16] Evermann, J., Rehse, J.-R., and Fettke, P. Predicting processbehaviour using deep learning. Decision Support Systems 100 (2017),129–140. → pages 11[17] Ferger, W. F. The nature and use of the harmonic mean. Journalof the American Statistical Association 26, 173 (1931), 36–40. → pages5089Bibliography[18] Grinvald, A., Soffer, P., and Mokryn, O. Inter-case Propertiesand Process Variant Considerations in Time Prediction: A ConceptualFramework. In Enterprise, Business-Process and Information SystemsModeling (Cham, 2021), A. Augusto, A. Gill, S. Nurcan, I. Reinhartz-Berger, R. Schmidt, and J. Zdravkovic, Eds., Lecture Notes in BusinessInformation Processing, Springer International Publishing, pp. 96–111.→ pages 85[19] Guzzo, A., Rullo, A., and Vocaturo, E. Process mining appli-cations in the healthcare domain: A comprehensive review. WIREsData Mining and Knowledge Discovery 12, 2 (2022), e1442. eprint:https://onlinelibrary.wiley.com/doi/pdf/10.1002/widm.1442. → pages18[20] Hastie, T., Tibshirani, R., and Friedman, J. H. The elementsof statistical learning: data mining, inference, and prediction, vol. 2.Springer, 2009. → pages 10, 13, 16[21] Jackson, E., and Agrawal, R. Performance evaluation of differentfeature encoding schemes on cybersecurity logs. In 2019 SoutheastCon(2019), pp. 1–9. → pages 15, 42[22] Jain, N., and Jana, P. K. XRRF: An eXplainable Reasonably Ran-domised Forest algorithm for classification and regression problems. In-formation Sciences 613 (Oct. 2022), 139–160. → pages x, 3, 14, 18, 20,22, 25, 26, 28, 29, 30, 31, 33, 49, 50, 55, 56, 69, 74, 82[23] James, G., Witten, D., Hastie, T., and Tibshirani, R. An in-troduction to statistical learning, vol. 112. Springer, 2013. → pages 13,14, 16[24] Khanna, R., Kim, B., Ghosh, J., and Koyejo, S. Interpret-ing black box predictions using fisher kernels. In The 22nd Inter-national Conference on Artificial Intelligence and Statistics (2019),PMLR, pp. 3382–3390. → pages 86[25] Lakhmiri, D., Digabel, S. L., and Tribes, C. Hypernomad: Hy-perparameter optimization of deep neural networks using mesh adaptivedirect search, 2019. → pages 86[26] Lin, Z., Jain, A., Wang, C., Fanti, G., and Sekar, V. Generatinghigh-fidelity, synthetic time series datasets with doppelganger. arXivpreprint arXiv:1909.13403 (2019). → pages 8690Bibliography[27] Lou, Y., Caruana, R., Gehrke, J., and Hooker, G. Accurateintelligible models with pairwise interactions. In Proceedings of the19th ACM SIGKDD international conference on Knowledge discoveryand data mining (2013), pp. 623–631. → pages 18[28] Mathur, P., Sethi, T., Mathur, A., Maheshwari, K., Cywin-ski, J. B., Khanna, A. K., Dua, S., and Papay, F. Explain-able machine learning models to understand determinants of COVID-19mortality in the United States, June 2020. → pages 17[29] Moret, B. M. E., and Shapiro, H. D. An empirical analysis ofalgorithms for constructing a minimum spanning tree. In Algorithmsand Data Structures (Berlin, Heidelberg, 1991), F. Dehne, J.-R. Sack,and N. Santoro, Eds., Springer Berlin Heidelberg, pp. 400–411.→ pages52[30] Muzellec, B., Josse, J., Boyer, C., and Cuturi, M. Missingdata imputation using optimal transport. In International Conferenceon Machine Learning (2020), PMLR, pp. 7130–7140. → pages 25[31] Navarin, N., Vincenzi, B., Polato, M., and Sperduti, A. LSTMnetworks for data-aware remaining time prediction of business processinstances. In 2017 IEEE Symposium Series on Computational Intelli-gence (SSCI) (Nov. 2017), pp. 1–7. → pages 9, 11[32] Neu, D. A., Lahann, J., and Fettke, P. A systematic literaturereview on state-of-the-art deep learning methods for process prediction.Artificial Intelligence Review (2022), 1–27. → pages 9, 11[33] Nori, H., Jenkins, S., Koch, P., and Caruana, R. Interpretml: Aunified framework for machine learning interpretability. arXiv preprintarXiv:1909.09223 (2019). → pages 17[34] Pargent, F., Pfisterer, F., Thomas, J., and Bischl, B. Regu-larized target encoding outperforms traditional methods in supervisedmachine learning with high cardinality features. Computational Statis-tics 37, 5 (2022), 2671–2692. → pages 15, 16[35] Patro, S., and Sahu, K. K. Normalization: A preprocessing stage.arXiv preprint arXiv:1503.06462 (2015). → pages 25[36] Peeperkorn, J., vanden Broucke, S., and De Weerdt, J. CanDeep Neural Networks Learn Process Model Structure? An Assessment91BibliographyFramework and Analysis. In Process Mining Workshops (Cham, 2022),J. Munoz-Gama and X. Lu, Eds., Lecture Notes in Business InformationProcessing, Springer International Publishing, pp. 127–139. → pages12[37] Pika, A., ter Hofstede, A. H. M., Perrons, R. K., Grossmann,G., Stumptner, M., and Cooley, J. Analysing an Industrial SafetyProcess Through Process Mining: A Case Study. In Asset Intelligencethrough Integration and Interoperability and Contemporary VibrationEngineering Technologies (Cham, 2019), J. Mathew, C. Lim, L. Ma,D. Sands, M. E. Cholette, and P. Borghesani, Eds., Lecture Notes inMechanical Engineering, Springer International Publishing, pp. 491–500. → pages 18[38] Pourbafrani, M., Kar, S., Kaiser, S., and van der Aalst, W.M. P. Remaining Time Prediction for Processes with Inter-case Dy-namics. In Process Mining Workshops (Cham, 2022), J. Munoz-Gamaand X. Lu, Eds., Lecture Notes in Business Information Processing,Springer International Publishing, pp. 140–153. → pages 85[39] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush,A. V., and Gulin, A. Catboost: unbiased boosting with categor-ical features. Advances in neural information processing systems 31(2018). → pages 17[40] Rama-Maneiro, E., Vidal, J., and Lama, M. Deep learning forpredictive business process monitoring: Review and benchmark. IEEETransactions on Services Computing (2021). → pages 11[41] Rogge-Solti, A., and Weske, M. Prediction of business process du-rations using non-markovian stochastic petri nets. Information Systems54 (2015), 1–14. → pages 9[42] Tax, N., Verenich, I., La Rosa, M., and Dumas, M. Predic-tive Business Process Monitoring with LSTM Neural Networks. InAdvanced Information Systems Engineering (Cham, 2017), E. Duboisand K. Pohl, Eds., Lecture Notes in Computer Science, Springer Inter-national Publishing, pp. 477–492. → pages 11, 86[43] Tokuyama, Y., Miki, R., Fukushima, Y., Tarutani, Y., andYokohira, T. Performance evaluation of feature encoding methodsin network traffic prediction using recurrent neural networks. In Pro-ceedings of the 2020 8th International Conference on Information and92BibliographyEducation Technology (New York, NY, USA, 2020), ICIET 2020, As-sociation for Computing Machinery, p. 279–283. → pages 15, 42[44] Turner, C. R., Fuggetta, A., Lavazza, L., and Wolf, A. L.A conceptual basis for feature engineering. Journal of Systems andSoftware 49, 1 (1999), 3–15. → pages 15[45] Van Der Aalst, W., Adriansyah, A., De Medeiros, A.K. A., Arcieri, F., Baier, T., Blickle, T., Bose, J. C., VanDen Brand, P., Brandtjen, R., Buijs, J., et al. Process MiningManifesto. In Business Process Management Workshops: BPM 2011International Workshops, Clermont-Ferrand, France, August 29, 2011,Revised Selected Papers, Part I 9 (2012), Springer, pp. 169–194. →pages 6[46] van der Aalst, W., and Weijters, A. Process mining: a researchagenda. Computers in Industry 53, 3 (2004), 231–244. Process / Work-flow Mining. → pages 6, 12, 36[47] Van der Aalst, W. M., Schonenberg, M. H., and Song, M.Time prediction based on process mining. Information systems 36, 2(2011), 450–475. → pages 9[48] Vandeput, N. Data science for supply chain forecasting, vol. 2. DeGruyter, 2021. → pages 14, 15[49] Verenich, I., Dumas, M., Rosa, M. L., Maggi, F. M., andTeinemaa, I. Survey and cross-benchmark comparison of remainingtime prediction methods in business process monitoring. ACM Trans.Intell. Syst. Technol. 10, 4 (jul 2019). → pages 6, 7, 9, 11, 86[50] Verenich, I., Dumas, M., Rosa, M. L., and Nguyen, H. Pre-dicting process performance: A white-box approach based on processmodels. Journal of Software: Evolution and Process 31, 6 (2019). →pages 9, 12, 18[51] Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. Generalizingfrom a few examples: A survey on few-shot learning. ACM computingsurveys (csur) 53, 3 (2020), 1–34. → pages 86[52] Xiong, H., Pandey, G., Steinbach, M., and Kumar, V. Enhanc-ing data analysis with noise removal. IEEE Transactions on Knowledgeand Data Engineering 18, 3 (2006), 304–319. → pages 2593Bibliography[53] Xu, H., Kinfu, K. A., LeVine, W., Panda, S., Dey, J.,Ainsworth, M., Peng, Y.-C., Kusmanov, M., Engert, F.,White, C. M., et al. When are deep networks really better thandecision forests at small sample sizes, and how? arXiv preprintarXiv:2108.13637 (2021). → pages 10, 13, 16, 20[54] Yahey v. British Columbia. 2021 BCSC 1287. https://www.bccourts.ca/jdb-txt/sc/21/12/2021BCSC1287.htm. → pages 37, 73[55] Zerbino, P., Stefanini, A., and Aloini, D. Process Science in Ac-tion: A Literature Review on Process Mining in Business Management.Technological Forecasting and Social Change 172 (Nov. 2021), 121021.→ pages 1894Appendices95Appendix ASupporting tablesParameter ValueTrain/test/validation split 60/20/20Seed 1Number of runs averaged 3TIR2 for training trees 0.98Max tree depth 15Number of trees 500Range of features selected Fx4 to3Fx4Table A.1: Hyperparameters used for the successful EXRRF prediction ofFN Consultation duration. Fx is the number of features in each of the PFs,DFs, and BFs classes.Parameter ValueTrain/test/validation split 60/20/20Seed 1TIR2 for training trees 0.962Max tree depth 14Number of trees 500Range of features selected Fx4 to3Fx4Table A.2: Hyperparameters used for the EXRRF prediction of the entireapplication duration, during the last revision (Model 1). Fx is the numberof features in each of the PFs, DFs, and BFs classes.96Appendix A. Supporting tablesParameter ValueTrain/test/validation split 60/20/20Seed 1TIR2 for training trees 0.925Max tree depth 13Number of trees 500Range of features selected Fx4 to3Fx4Table A.3: Hyperparameters used for the successful EXRRF prediction ofapplication duration (Model 2). Fx is the number of features in each of thePFs, DFs, and BFs classes.Feature SelectionR2 train R2 test R2 valRMSE MAEThreshold (days) (days)22FTpaper 0.944 0.548 0.539 33.4 17.6FTharmonic 0.947 0.608 0.575 35.4 18.0FTmean−sd 0.945 0.597 0.575 34.3 17.9FTmin2 0.941 0.590 0.573 34.2 17.9Table A.4: Results of the EXRRF method predictions for the duration at theend of the current revision, with addition of flag for whether another revisionwas needed (Model 2). These results support that accurately predictingwhether a revision will be required will add accuracy to the model.22The mean duration in the data is 54 days.97Appendix BSupporting figuresFigure B.1: The tuning results of differing tree depths for the EXRRF modelprediction of application duration.98AppendixB.SupportingfiguresFigure B.2: Results of the EBM model completed during preparatory work. The importance of the Idle Time,Forestry Review, and FN Consultation process are indicated.99B.1. Generalizability testsB.1 Generalizability testsFigure B.3: The generalizability test result for the FN Consultation predic-tion using the FTpaper feature selection threshold.Figure B.4: The generalizability test result for the FN Consultation predic-tion using the FTharmonic feature selection threshold.100B.1. Generalizability testsFigure B.5: The generalizability test result for the FN Consultation predic-tion using the FTmean−sd feature selection threshold.Figure B.6: The generalizability test result for the FN Consultation predic-tion using the FTmin2 feature selection threshold.101B.1. Generalizability testsFigure B.7: The generalizability test result for the application durationprediction using the FTpaper feature selection threshold.Figure B.8: The generalizability test result for the application durationprediction using the FTharmonic feature selection threshold.102B.1. Generalizability testsFigure B.9: The generalizability test result for the application durationprediction using the FTmean−sd feature selection threshold.Figure B.10: The generalizability test result for the application durationprediction using the FTmin2 feature selection threshold.103B.2. Residual distributionsB.2 Residual distributionsFigure B.11: The residual distribution for the FN Consultation durationprediction with the FTharmonic feature selection threshold applied.Figure B.12: The residual distribution for the FN Consultation durationprediction with the FTmean−sd feature selection threshold applied.104B.2. Residual distributionsFigure B.13: The residual distribution for the FN Consultation durationprediction with the FTmin2 feature selection threshold applied.Figure B.14: The residual distribution for the application duration predic-tion with the FTharmonic feature selection threshold applied.105B.2. Residual distributionsFigure B.15: The residual distribution for the application duration predic-tion with the FTmean−sd feature selection threshold applied.Figure B.16: The residual distribution for the application duration predic-tion with the FTmin2 feature selection threshold applied.106
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Lasserre, P.
Lucet, Yves
Predicting remaining time for an energy regulator’s highly parallel process
Text