Applied Science, Faculty of
Science, Faculty of
Earth, Ocean and Atmospheric Sciences, Department of
Mining Engineering, Keevil Institute of
DSpace
Geosciences 14 (3): 67 (2024)
Yang, Beverly
Heagy, Lindsey
Morgenroth, Josephine
Elmo, Davide
Technological advancements have made rock engineering more data-driven, leading to increased use of machine learning (ML). While the use of ML in rock engineering has the potential to transform the industry, several methodological issues should first be addressed: (i) rock engineering’s use of biased (poor quality) data, resulting in biased ML models and (ii) limited rock mass classification and characterization data. If these issues are not addressed, rock engineering risks using unreliable ML models that can have potential real-life adverse impacts. This paper aims to provide an overview of these methodological issues and demonstrate their impact on the reliability of ML models using surrogate models. To take full advantage of the benefits of ML, rock engineers should make sure that their ML models are reliable by ensuring that there are sufficient unbiased data to develop reliable ML models. In the context of this paper, the term sufficient retains a relative meaning since the amount of data that is sufficient to develop reliable a ML models depends on the problem under consideration and the application of the ML model (e.g., pre-feasibility, feasibility, design stage).
Citation: Yang, B.; Heagy, L.J.;Morgenroth, J.; Elmo, D. AlgorithmicGeology: Tackling MethodologicalChallenges in Applying MachineLearning to Rock Engineering.Geosciences 2024, 14, 67.https://doi.org/10.3390/geosciences14030067Academic Editors: Anika Braun,Stratis Karantanellis andJesus Martinez-FriasReceived: 14 January 2024Revised: 25 February 2024Accepted: 28 February 2024Published: 4 March 2024Copyright: © 2024 by the authors.Licensee MDPI, Basel, Switzerland.This article is an open access articledistributed under the terms andconditions of the Creative CommonsAttribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).geosciencesArticleAlgorithmic Geology: Tackling Methodological Challenges inApplying Machine Learning to Rock EngineeringBeverly Yang 1, Lindsey J. Heagy 2, Josephine Morgenroth 1 and Davide Elmo 1,*1 NBK Institute of Mining Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada;jsmorgen@mail.ubc.ca (J.M.)2 Department of Earth, Ocean and Atmospheric Sciences, University of British Columbia,Vancouver, BC V6T 1Z4, Canada; lheagy@eoas.ubc.ca* Correspondence: davide.elmo@ubc.caAbstract: Technological advancements have made rock engineering more data-driven, leading toincreased use of machine learning (ML). While the use of ML in rock engineering has the potential totransform the industry, several methodological issues should first be addressed: (i) rock engineer-ing’s use of biased (poor quality) data, resulting in biased ML models and (ii) limited rock massclassification and characterization data. If these issues are not addressed, rock engineering risksusing unreliable ML models that can have potential real-life adverse impacts. This paper aims toprovide an overview of these methodological issues and demonstrate their impact on the reliabilityof ML models using surrogate models. To take full advantage of the benefits of ML, rock engineersshould make sure that their ML models are reliable by ensuring that there are sufficient unbiaseddata to develop reliable ML models. In the context of this paper, the term sufficient retains a relativemeaning since the amount of data that is sufficient to develop reliable a ML models depends on theproblem under consideration and the application of the ML model (e.g., pre-feasibility, feasibility,design stage).Keywords: rock engineering; machine learning; rock mass characterization; rock mass classification1. IntroductionMachine learning (ML) has become unavoidable in rock engineering, with academicsand industry professionals increasingly incorporating ML into their projects. As withany new tool, its increasing popularity has increased ML methodological issues, suchas failing to split the data into a train and test split, using subjective data, and workingwith extremely limited datasets. A summary of recent ML applications to rock masscharacterization and classification data, including concerns with their methodology, isprovided in Appendix A and demonstrates the need for a critical discussion on the topic ofML in rock engineering. While other industries have similar problems, they are workingtowards bringing awareness and finding solutions. Rock engineering, on the other hand,continues to overlook these problems, which has the potential to result in unreliable MLmodels. In this context, reliable is defined as providing consistently good quality results;therefore, a reliable ML model is an ML model that provides consistently good qualityresults. A reliable ML model requires high quality (unbiased) data to develop a model thatoffers accurate predictions, as compared to the “ground truth” or targets, and a sufficientquantity of this high-quality data in order to provide consistent results as measured byan appropriate performance metric [1]. To date, there has been limited discussion on theimpacts of poor quality, biased data in insufficient quantities for the development of MLmodels in rock engineering. The resulting unexplainable or even poor model performanceshave the potential to hinder the adoption of ML in rock engineering. This paper aims toprovide an overview of these methodological issues and, using synthetic data and surrogatemodeling, demonstrate their impact on the reliability of ML models. The code and dataGeosciences 2024, 14, 67. https://doi.org/10.3390/geosciences14030067 https://www.mdpi.com/journal/geosciencesGeosciences 2024, 14, 67 2 of 31used in this paper can be found at the following link: https://github.com/beverlyyang/geosciences-machine-learning-in-engineering-geology.Key DefinitionsBefore proceeding with the discussion, terminology used in ML and statistics is clari-fied, as common usage in rock engineering sometimes deviates from the precise definitions:• Imbalanced data: the distribution of classes or bins is not equal [2]. Note that if acontinuous variable is binned, the resulting distribution can be imbalanced, althoughthe severity of the imbalance would depend on the bin width chosen.• Skewed data or distribution: a distribution of data points in a dataset that is notsymmetric around the dataset’s mean [3].• Outliers: broadly defined as data points that differ significantly from other observa-tions (e.g., data points found in the tail of a distribution). However, there is no setmathematical or statistical definition of what constitutes an outlier [4].• Training and test data: as part of the machine learning workflow, the dataset is splitinto its training and test data (referred to as the train/test split in this paper). TheML model is first trained on the training data, then the trained model is evaluated onthe test data to better understand how well the model will perform on data it has notseen before. The dataset is often randomly shuffled before its train/test split to ensurethat the training and test data have the same distribution. The train/test split servesto evaluate the ability of a machine learning model to perform effectively on newand unseen data. Additionally, this practice helps us to identify and reduce the riskof overfitting, a scenario in which a model excels in accurately predicting outcomesbased on the training data but struggles to generalize its predictions to new instances(see below).• Validation data: the training dataset is split into training and validation data. Themodel is trained on the training data and then refined or tuned on the validation databefore being evaluated on the test data. Including a validation dataset provides a morerobust method of evaluating the model performance. When working with smallerdatasets, a single train/validation split may misrepresent the test data, so it is oftenrecommended that cross-validation be performed. In cross-validation, the trainingset is divided into k number of folds (five and ten folds are commonly used) andeach fold takes a turn at being the validation set. A validation score is determinedfor each fold, and the scores for all folds are then averaged to determine an averagevalidation score [1].• Overfitting: the ML model cannot generalize well as it is too complex and fits tothe noise in the data. Overfitting can be identified when the model performs sig-nificantly better on the training data than the test data and commonly occurs insmaller datasets [5].While the terms imbalanced and skewed data are often used interchangeably, itis important to differentiate between them as different techniques are used to handlethem. Figure 1 depicts a uniform, normal, and lognormal distribution and describes theirdistribution characteristics.Geosciences 2024, 14, x FOR PEER REVIEW 3 of 35 Figure 1. Characteristics of uniform, normal, and lognormal distributions. 2. Data Quality Issues In machine learning and numerical modeling, incorrect or poor-quality input will result in a misleading output. Likewise, if the data used to develop the model are biased, then the model’s results will be biased [6]. Examples of this include Amazon’s ML recruit-ment algorithm, which was trained on resumes submitted mostly by men, leading to a gender bias in its predictions, as well as the Correctional Offender Management Profiling for Alternative Sanctions, which was an algorithm used in criminal sentencing to predict which people are most likely to re-offend, that was trained on incomplete data with race as an input, resulting in a racial bias in its predictions [6,7]. Contrary to popular belief, rock engineering also suffers from biased and subjective data; many of the rock mass char-acterization and classification parameters collected during a site investigation (e.g., rock quality designation, rock mass rating, Q-system, and geological strength index) are sub-jective and imprecise [8–11]. These subjective rock mass characterization and classification parameters are com-mon input features and target variables in rock engineering ML applications (e.g., [12–15], etc.) and their continued inclusion in ML models, as well as the propensity of (often more senior) rock engineering practitioners to blindly and incorrectly believe in their objectiv-ity, pose an obstacle in the development of reliable rock engineering ML models. Addi-tional obstacles to using these parameters in ML include a lack of standardization in col-lecting them (as each mining and consulting company has its own confidential guide-lines), practitioners using them incorrectly, and a general tendency among practitioners to ignore the parameter limitations. The development of reliable ML models in rock engi-neering requires that rock engineers first acknowledge the subjective and imprecise nature of rock mass characterization and classification parameters, followed by developing more objective parameters that can reliably be used with ML. 3. Data Quantity Issues The data collection process in rock engineering is often time and cost-intensive and prone to human error due to the nature of the parameters we collect. This can make ac-quiring good quality data for ML challenging, resulting in smaller datasets relative to other industries. Recent examples of rock engineering ML models have been trained on only a few hundred data points (e.g., [12,13,16–19], etc.), with some on as few as 80 data points [20]. This starkly contrasts ML models in other industries that have been trained on hundreds of thousands of data and are constantly updating (e.g., NASA’s Prediction of Worldwide Energy Resources). One study, [1], investigated the impacts of data quan-tity on the performance and reliability of ML models using a synthetic dataset for a slope stability problem. This study demonstrated that ML models trained on smaller datasets can lead to unreliable ML models. More specifically, ML models trained on smaller da-tasets (i.e., a few hundred data points) are sensitive to how the data was randomly shuf-fled before its train/test split (controlled by the random_state parameter in scikit-learn’s train_test_split function), resulting in significant variation in both the test R2 and root Figure 1. Characteristics of uniform, normal, and lognormal distributions.Geosciences 2024, 14, 67 3 of 312. Data Quality IssuesIn machine learning and numerical modeling, incorrect or poor-quality input willresult in a misleading output. Likewise, if the data used to develop the model are biased,then the model’s results will be biased [6]. Examples of this include Amazon’s ML recruit-ment algorithm, which was trained on resumes submitted mostly by men, leading to agender bias in its predictions, as well as the Correctional Offender Management Profilingfor Alternative Sanctions, which was an algorithm used in criminal sentencing to predictwhich people are most likely to re-offend, that was trained on incomplete data with raceas an input, resulting in a racial bias in its predictions [6,7]. Contrary to popular belief,rock engineering also suffers from biased and subjective data; many of the rock masscharacterization and classification parameters collected during a site investigation (e.g.,rock quality designation, rock mass rating, Q-system, and geological strength index) aresubjective and imprecise [8–11].These subjective rock mass characterization and classification parameters are commoninput features and target variables in rock engineering ML applications (e.g., [12–15], etc.)and their continued inclusion in ML models, as well as the propensity of (often more senior)rock engineering practitioners to blindly and incorrectly believe in their objectivity, pose anobstacle in the development of reliable rock engineering ML models. Additional obstaclesto using these parameters in ML include a lack of standardization in collecting them (aseach mining and consulting company has its own confidential guidelines), practitionersusing them incorrectly, and a general tendency among practitioners to ignore the parameterlimitations. The development of reliable ML models in rock engineering requires that rockengineers first acknowledge the subjective and imprecise nature of rock mass characteriza-tion and classification parameters, followed by developing more objective parameters thatcan reliably be used with ML.3. Data Quantity IssuesThe data collection process in rock engineering is often time and cost-intensive andprone to human error due to the nature of the parameters we collect. This can makeacquiring good quality data for ML challenging, resulting in smaller datasets relative toother industries. Recent examples of rock engineering ML models have been trained ononly a few hundred data points (e.g., [12,13,16–19], etc.), with some on as few as 80 datapoints [20]. This starkly contrasts ML models in other industries that have been trainedon hundreds of thousands of data and are constantly updating (e.g., NASA’s Prediction ofWorldwide Energy Resources). One study, [1], investigated the impacts of data quantity onthe performance and reliability of ML models using a synthetic dataset for a slope stabilityproblem. This study demonstrated that ML models trained on smaller datasets can leadto unreliable ML models. More specifically, ML models trained on smaller datasets (i.e., afew hundred data points) are sensitive to how the data was randomly shuffled before itstrain/test split (controlled by the random_state parameter in scikit-learn’s train_test_splitfunction), resulting in significant variation in both the test R2 and root mean square error(RMSE) for each random shuffling (or random_state value). In other words, how the datawere randomly shuffled prior to train/test splitting will have a variable and somewhatrandom impact on the ML performance.An example of this variation is shown in Table 1 for a dataset of 250 data pointsfrom [1]. A total of 80% of the data was used for training (200 data points), while theremaining 20% was used as the test dataset (50 data points).Table 1. Test R2 and RMSE for different random shufflings of a dataset of 250 data points from [1].Performance Metric Random_State 1 = 0 Random_State = 1 Random_State = 42 Random_State = 123Test R2 0.33 0.49 0.54 −4.26Test RMSE 1.35 0.33 0.23 1.001 Note that random_state is a parameter in scikit-learn’s train_test_function that controls how the data is randomlyshuffled prior to splitting the data into training and test data.Geosciences 2024, 14, 67 4 of 31This unreliability can also be found using real-world data in rock engineering MLapplications. Using the dataset from [12], we developed random forest models withfour different random_state values without hyperparameter tuning, with hyperparametertuning and 10-fold cross-validation via scikit-learn’s RandomizedSearchCV, and withhyperparameter tuning and 10-fold cross-validation via scikit-optimize’s BayesSearchCV.The dataset consists of 138 data points, with 80% used to train the model and the remaining20% used to test the model. The results in Table 2 demonstrate the variability of the modelresults depending on how the data were randomly shuffled before the train/test split (i.e.,the random_state value).Table 2. Variation in test performance metrics for four different random shufflings of the dataset in [12].HyperparameterTuningPerformanceMetric Random_State = 0 Random_State = 1 Random_State = 42 Random_State = 123NoneTest R2 0.19 0.39 0.55 0.50Test RMSE 1.97 2.02 2.34 2.35RandomizedSearchCVTest R2 0.35 0.39 0.44 0.5Test RMSE 1.77 2.02 2.61 2.34BayesSearchCV Test R2 0.35 0.40 0.49 0.59Test RMSE 1.77 2.01 2.49 2.13In previous work by some of the current authors, [1] also noted that ML modelsdeveloped on smaller datasets can result in overfitting and instances where the modelperforms better on the test data than the training data, depending on how the data wereshuffled before the train/test split (i.e., the random_state value). This study demonstratedthat a dataset of 250 data points resulted in overfitting for a random_state of 0, but the modelperformed better on the test data for a random_state of 42. The ML model performingbetter on the test data is also found in real-world data such as [12]. The consequence of themodel performing better on the test data in datasets with only a few hundred data points isthat it can lead to some rock engineers thinking that their model is able to generalize well,when in reality it is most likely an artefact of how the data were shuffled before they weresplit, and the model is both unable to generalize well and is unreliable.Of note is that the example in [1] demonstrates more extreme unreliability comparedto the results in Table 2. This can be attributed to more extreme outliers in the datasetin [1]; the effect of extreme outliers and skewed data will be discussed in greater detail inSections 3.3 and 3.4.3.1. MethodologyThe analysis performed by [1] was expanded to investigate how data preparationand different machine learning algorithms, in conjunction with data quantity, impact theperformance and reliability of ML models. The subsequent ML modeling used the samesynthetic dataset generated from a SWedge probabilistic analysis [21]. Five thousand datapoints were generated in SWedge from a Latin Hypercube sampling method with a pseudo-random number generator (using a seed of 1). Any data points where the safety factorwas not applicable were removed (8 data points). As noted in [1], surrogate modelingwas employed, where the inputs of the SWedge model (the dip and dip direction of thetwo joints, the friction angle of the two joints, and the dip and dip direction of the slopeface) became the input features of the ML model and the target (safety factor) became thetarget variable of the ML model. Table 3 and Figure 2, taken from [1], show the ranges anddistributions of the input parameters of the SWedge and ML model and the stereonet of thetwo joints and slope faces.Geosciences 2024, 14, 67 5 of 31Table 3. Distribution of input parameters into SWedge to generate synthetic data [1].Input Parameter Mean Min Max Standard Deviation DistributionSlope Dip (◦) 60 50 70 2 NormalDip direction (◦) 180 170 190 2 NormalJoint 1Dip (◦) 40 - - - Fisher K = 40Dip direction (◦) 130 - - -Joint 2Dip (◦) 55 - - - Fisher K = 40Dip direction (◦) 230 - - -Joint 1Cohesion (MPa) 0 - - - -Friction angle (◦) 30 20 40 2 NormalJoint 2 Cohesion (MPa) 0 - - - -Geosciences 2024, 14, x FOR PEER REVIEW 5 of 35 3.1. Methodology The analysis performed by [1] was expanded to investigate how data preparation and different machine learning algorithms, in conjunction with data quantity, impact the per-formance and reliability of ML models. The subsequent ML modeling used the same syn-thetic dataset generated from a SWedge probabilistic analysis [21]. Five thousand data points were generated in SWedge from a Latin Hypercube sampling method with a pseudo-random number generator (using a seed of 1). Any data points where the safety factor was not applicable were removed (8 data points). As noted in [1], surrogate model-ing was employed, where the inputs of the SWedge model (the dip and dip direction of the two joints, the friction angle of the two joints, and the dip and dip direction of the slope face) became the input features of the ML model and the target (safety factor) be-came the target variable of the ML model. Table 3 and Figure 2, taken from [1], show the ranges and distributions of the input parameters of the SWedge and ML model and the stereonet of the two joints and slope faces. Table 3. Distribution of input parameters into SWedge to generate synthetic data [1]. Input Parameter Mean Min Max Standard Deviation Distribution Slope Dip (°) 60 50 70 2 Normal Dip direction (°) 180 170 190 2 Normal Joint 1 Dip (°) 40 - - - Fisher K = 40 Dip direction (°) 130 - - - Joint 2 Dip (°) 55 - - - Fisher K = 40 Dip direction (°) 230 - - - Joint 1 Cohesion (MPa) 0 - - - - Friction angle (°) 30 20 40 2 Normal Joint 2 Cohesion (MPa) 0 - - - - Figure 2. Stereonet of the wedge problem in SWedge used to generate the synthetic data [1,21]. Using the synthetic data, various supervised ML models with different pre-pro-cessing steps were developed for varying dataset sizes, ranging from 100 to 4950 data Figure 2. Stereonet of the edge proble i S t t t , .si t synthetic data, various supervised ML models with different pre-processingsteps were developed for varying dataset sizes, ranging from 100 to 4950 data points witha step of 50. Like the methodology outlined in [1], 80% of each dataset was used fortraining, the remaining 20% was used for testing, and the subsequent training and testscores were determined for each dataset. A 5-fold cross-validation was performed for someof the models with hyperparameter tuning (using scikit-learn’s RandomizedSearchCV andGridSearchCV functions). Four different random_state values (0, 1, 42, and 123) in thetrain_test_split function in scikit-learn were examined for each model. A learning curve(i.e., dataset size vs. test score) for each random_state value was plotted to examine theimpact of the random_state value on the model results and reliability.The amount of data needed to develop reliable ML models is also dependent on thecomplexity of the algorithm chosen; more complex algorithms (such as neural networks)will require more data than simpler algorithms (such as linear regression or k-nearestneighbors). As a result, two ML algorithms with varying complexities were examined in theML modeling: k-nearest neighbors (kNN—a simple, computationally efficient, and easilyinterpretable algorithm) and multilayer perceptron (MLP—a basic neural network thatis better able to identify complex relationships among data compared to kNN). Principalcomponent analysis (PCA) and hyperparameter tuning with 5-fold cross-validation wereGeosciences 2024, 14, 67 6 of 31also performed; hyperparameter tuning for the kNN models was performed with scikit-learn’s GridSearchCV function, while hyperparameter tuning for the MLP models wasperformed with scikit-learn’s RandomizedSearchCV function. Note that hyperparametertuning for the MLP models was only performed on dataset sizes of 100, 150, 200, 250, 750,and 2000 for computational efficiency.The ML modeling consisted of generating regression, classification, and pseudo-regression models. While the target data in this example are numeric and thus appropriatefor regression models, the data can be binned into classes for classification models. Thedata were binned according to the target variable (safety factor) into bins of 0.3 up to asafety factor of 3, and any safety factors greater than 3 were grouped into one bin. Pseudo-regression models can also be developed from the classification models, where the mean ofthe predicted class is compared with the actual numeric value, and R2 and RMSE can bedetermined and compared with the R2 and RMSE values from the traditional regressionmodels. Performing classification may reduce the impact of extreme outliers in the targetvariable, especially in skewed data.Different methods to handle the imbalanced nature of the target variable were alsoexamined: stratified sampling was performed in the regression and classification models,and data balancing techniques (oversampling and synthetic minority oversampling tech-nique (SMOTE) [22]) were performed in the classification models. Note that oversamplingand SMOTE are primarily used in classification problems (such as in the classificationmodels in this paper), however, they have recently been applied to regression problems aswell (although they are not as well developed as their classification counterparts). Due tothe right skewed lognormal distribution of the target variable, an ln transformation wasperformed on the target variable for the regression models. This ln transformation wasonly performed with the kNN algorithm for computational efficiency.An important aspect to note about ML modeling is the performance metrics used toevaluate the performance and reliability of the models. There are several performancemetrics to evaluate regression (as well as pseudo-regression) and classification models,and they may lead to different interpretations regarding the performance and reliability ofthe models. Common performance metrics for regression and pseudo-regression modelsare the coefficient of determination R2 and RMSE; while RMSE is more robust than R2,the latter is more commonly used and better understood among rock engineers. Similarly,while the F1-score and AUC (the area under the receiver operating characteristic curve) aremore robust performance metrics for classification models (especially for imbalanced data),accuracy is better understood among rock engineers. As a result of this dichotomy betweenperformance metrics, the ML modeling in this paper uses one rock engineering-friendlymetric and one robust metric when evaluating the models: R2 and RMSE were used forthe regression and pseudo-regression models, while accuracy and F1-score were used forthe classification models. To better assess overfitting and the model performing betteron the test data, the difference between the train and test accuracy and F1-scores weredetermined for each dataset size in the classification models. In contrast, only the differencebetween the test and train RMSE was determined for each dataset size in the regression andpseudo-regression models. The R2 difference was not examined due to the large negativeR2 values. The thresholds used to define overfitting in this paper are train–test accuracy≥ 0.25, train–test F1-score ≥ 0.25, and test–train RMSE ≥ 0.75. Note that these thresholdswere arbitrarily chosen for this specific problem.Figure 3 shows the general methodology of the ML modeling, and Tables 4 and 5 showa list of the ML models that were developed.Geosciences 2024, 14, 67 7 of 31Geosciences 2024, 14, x FOR PEER REVIEW 7 of 35 used for the classification models. To better assess overfitting and the model performing better on the test data, the difference between the train and test accuracy and F1-scores were determined for each dataset size in the classification models. In contrast, only the difference between the test and train RMSE was determined for each dataset size in the regression and pseudo-regression models. The R2 difference was not examined due to the large negative R2 values. The thresholds used to define overfitting in this paper are train–test accuracy ≥ 0.25, train–test F1-score ≥ 0.25, and test–train RMSE ≥ 0.75. Note that these thresholds were arbitrarily chosen for this specific problem. Figure 3 shows the general methodology of the ML modeling, and Tables 4 and 5 show a list of the ML models that were developed. Figure 3. Workflow of the ML modeling process used in this paper. Table 4. Regression ML models developed and examined. kNNRegressor MLPRegressor Model 1a Standardization Model 1b Standardization Model 2a Standardization, PCA Model 2b Standardization, PCA Model 3a Standardization, hyperparameter tun-ing with 5-fold cross-validation Model 3b Standardization, hyperparameter tun-ing with 5-fold cross-validation Model 4a Standardization, stratified sampling Model 4b Standardization, stratified sampling Model 5 Standardization, ln transformation Table 5. Classification (and subsequently pseudo-regression) ML models developed and exam-ined. kNNClassifier MLPClassifier Model 6a Standardization Model 6b Standardization Model 7a Standardization, PCA Model 7b Standardization, PCA Model 8a Standardization, hyperparameter tun-ing with 5-fold cross-validation Model 8b Standardization, hyperparameter tun-ing with 5-fold cross-validation Model 9a Standardization, stratified sampling Model 9b Standardization, stratified sampling Model 10a Standardization, oversampling Model 10b Standardization, oversampling Model 11a Standardization, SMOTE Model 11b Standardization, SMOTE Figure 3. orkflo of the L odeling process used in this paper.Table 4. Regression ML models developed and examined.kNNRegressor MLPRegressorModel 1a Standardization Model 1b StandardizationModel 2a Standardization, PCA Model 2b Standardization, PCAModel 3a Standardization, hyperparametertuning with 5-fold cross-validation Model 3bStandardization, hyperparametertuning with 5-fold cross-validationModel 4a Standardization, stratified sampling Model 4b Standardization, stratified samplingModel 5 Standardization, ln transformationl . l ssific ti ( s s tl s -r r ssi ) ls l ined.kNNClassifier MLPClassifierModel 6a Standardization Model 6b StandardizationModel 7a Standardization, PCA Model 7b Standardization, PCAModel 8a Standardization, hyperparametertuning with 5-fold cross-validatio Model 8bStandardization, hyperparametertuning with 5-fold cross-validationModel 9a Standardization, stratified sampling Model 9b Standardization, stratified samplingModel 10a Standardization, oversampling Model 10b Standardization, oversamplingModel 11a Standardization, SMOTE Model 11b Standardization, SMOTE3.2. Data VisualizationBefore implementing machine learning, exploratory data analysis and data visualiza-tion should be performed to better understand the data and help guide some pre-processingdecisions. The histogram and probability density function of the target variable (safetyfactor) are shown in Figure 4.The target variable (safety factor) is both imbalanced and positively skewed. It roughlyfollows a right-skewed log-normal distribution, which is to be expected for a safety factordistribution for a slope stability problem as the lowest safety factor value is always zero,and its highest value has no limit.Geosciences 2024, 14, 67 8 of 31Geosciences 2024, 14, x FOR PEER REVIEW 8 of 35 3.2. Data Visualization Before implementing machine learning, exploratory data analysis and data visuali-zation should be performed to better understand the data and help guide some pre-pro-cessing decisions. The histogram and probability density function of the target variable (safety factor) are shown in Figure 4. Figure 4. (a) Histogram of the target variable (safety factor) and (b) probability density function of the target variable (safety factor). Figure 4. (a) Histogram of the target variable (safety factor) and (b) probability density function ofthe target variable (safety factor).The mean, median, minimum, maximum, and standard deviation of the input featuresand target variable in the dataset are summarized in Table 6, while the correlation matrix isshown in Figure 5.Geosciences 2024, 14, 67 9 of 31Table 6. Basic statistics of the input features and target variable.Parameter Mean Median Min Max Standard DeviationInput featuresDip of joint 1 (◦) 41 41 10 76 9Dip direction of joint 1 (◦) 130 130 77 186 14Dip of joint 2 (◦) 55 55 18 86 9Dip direction of joint 2 (◦) 230 230 184 272 11Dip of slope (◦) 60 60 52 67 2Dip direction of slope (◦) 180 180 173 187 2Friction angle of joint 1 (◦) 30 30 22 37 2Friction angle of joint 2 (◦) 30 30 23 38 2Target Safety factor 1.16 1.06 0.38 12.56 0.52Geosciences 2024, 14, x FOR PEER REVIEW 9 of 35 The target variable (safety factor) is both imbalanced and positively skewed. It roughly follows a right-skewed log-normal distribution, which is to be expected for a safety factor distribution for a slope stability problem as the lowest safety factor value is always zero, and its highest value has no limit. The mean, median, minimum, maximum, and standard deviation of the input fea-tures and target variable in the dataset are summarized in Table 6, while the correlation matrix is shown in Figure 5. Table 6. Basic statistics of the input features and target variable. Parameter Mean Median Min Max Standard Deviation Input features Dip of joint 1 (°) 41 41 10 76 9 Dip direction of joint 1 (°) 130 130 77 186 14 Dip of joint 2 (°) 55 55 18 86 9 Dip direction of joint 2 (°) 230 230 184 272 11 Dip of slope (°) 60 60 52 67 2 Dip direction of slope (°) 180 180 173 187 2 Friction angle of joint 1 (°) 30 30 22 37 2 Friction angle of joint 2 (°) 30 30 23 38 2 Target Safety factor 1.16 1.06 0.38 12.56 0.52 Figure 5. Correlation matrix for the dataset examined. Cells highlighted in blue indicate that there is a positive correlation between those parameters, while cells highlighted in red indicate that there is a negative correlation between those parameters. The correlation matrix shows that none of the input features are strongly correlated with the safety factor, however, the dip and dip direction of joint 1 show a weak negative correlation to the safety factor, while the dip direction of joint 2 shows a weak positive correlation. 3.3. Results The learning curves for the base models (Models 1a, 1b, 6a, and 6b) are found in Ap-pendix B (Figures A1–A6), while the learning curves for the other ML models can be found in the Github link (Section 1). The R2, RMSE, and accuracy learning curves for the base models demonstrate that the models are unreliable at smaller dataset sizes (a few hundred data points) due to their variation depending on the random_state value (i.e., how the data were randomly shuffled before the train/test split). There is less variation between these learning curves for different random_state values after approximately 1500 data points, indicating that there are enough data such that how the data are randomly shuffled prior to data splitting does not significantly impact ML model results. Similar to the other Figure 5. Correlation matrix fo e dataset xamined. Cells highlighted in blue indicate that there isa positive correlati between those parameters, while cells highlighted in red indicate that there is anegative correlation between those parameters.The correlation matrix shows that none of the input features are strongly correlated withthe safety factor, however, the dip and dip direction of joint 1 show a weak negative correlationo the safet ctor, w ile the dip dir ction of joint 2 shows a weak positive correlation.3.3. ResultsThe learning curves for the base models (Models 1a, 1b, 6a, and 6b) are found inAppendix B (Figures A1–A6), while the learning curves for the other ML models can befound in the Github link (Section 1). The R2, RMSE, and accuracy learning curves forthe base models demonstrate that the models are unreliable at smaller dataset sizes (afew hundred data points) due to their variation depending on the random_state value(i.e., how the data were randomly shuffled before the train/test split). There is less varia-tion between these learning curves for different random_state values after approximately1500 data points, indicating that th re are enough data such that how the data are randomlyshuffled prior to data splitting does not significantly impact ML model results. Similar tothe other learning curves, the F1-score learning curves also demonstrate that the modelsare unreliable at smaller dataset sizes, however, the variation between the learning curvesfor different random_state values is generally smaller than the variation in the accuracylear ing curves and remains constant t roughout t e entire dataset range. The differencebetween the accuracy and F1-score learning curves demonstrates the importance of theperformance metric used in evaluating the model; at smaller dataset sizes (in the order ofa few hundred data points), the accuracy learning curves show a more unreliable modelGeosciences 2024, 14, 67 10 of 31compared to the F1-score learning curves, but at larger dataset sizes, the accuracy learningcurves show a more reliable model that plateaus at a higher value than the F1-score learningcurves. This trend can be seen in the learning curves for the other classification modelswith different pre-processing techniques and reinforces the importance of choosing anappropriate performance metric.The base model learning curves in Figures A1, A2, A5 and A6 also demonstrate that theunreliability (i.e., variation between learning curves for different random_state values) ismore pronounced in regression and pseudo-regression models and can also occur at largerdataset sizes (e.g., at a dataset size of 2000 in Figures A1a, A2a, A5a, and A6a), resulting inthe “spikiness” in the learning curves. This spikiness can be attributed to extreme outliersin the dataset and is present in the learning curves for the other regression and pseudo-regression models with different data preprocessing steps. Similarly, overfitting is alsogenerally found in smaller datasets (a few hundred data points) and is more pronouncedin the regression and pseudo-regression models. The regression and pseudo-regressionmodels also have more instances of the model performing better on the test data than thetraining data, compared to the classification models.Incorporating PCA into the modeling workflow did not improve the reliability of themodels compared to their base model counterparts. It had a minimal impact on overfittingand the model performing better on the test data. Hyperparameter tuning with cross-validation also did not improve the reliability of the models (as there is still variationbetween the learning curves for different random_state values), however, it did reducethe difference between the train and test scores and subsequently overfitting in the MLPclassification. Performing hyperparameter tuning in the kNN classification and pseudo-regression models did not impact overfitting. While cross-validation generally providesa more robust evaluation of the ML model than a traditional single train/test split, it isnot immune to any extreme values found in the test dataset that are not captured in thetraining and validation data that lead to unreliable models.The literature on ML applications in rock engineering tends to focus on neural net-works, even when the dataset is small (e.g., [18,23,24]). While neural networks are powerfulalgorithms that can better identify complex relationships between data than other MLalgorithms, they are computationally intensive and require significantly more data thanother, simpler algorithms [25]. Due to their data quantity requirements, MLPs and otherneural networks may not be the most appropriate algorithm when working with thesmaller datasets encountered in rock engineering. This is demonstrated in the learningcurves in Figures A2, A4, and A6, as well as the learning curves for the other models;the learning curves for the MLP models are just as unreliable as their kNN counterpartsfor smaller datasets (less than 750 data points for this example) despite being more com-putationally intensive and difficult to interpret. Furthermore, the MLP classification andpseudo-regression models result in a more significant difference between the training andtest scores in dataset sizes smaller than 250 data points compared to the kNN classifica-tion and pseudo-regression models, even after hyperparameter tuning. As a result, theMLP classification and pseudo-regression models developed on dataset sizes smaller than250 data points exhibit more overfitting than the kNN counterparts. Interestingly, the fre-quency and magnitude of overfitting are similar for the MLP and kNN regression models,with overfitting occurring at smaller dataset sizes (less than 500–750 data points).Similar to overfitting, the frequency of the model performing better on the test data issimilar between the MLP and kNN regression and pseudo-regression models and occursacross all dataset sizes examined, however, the magnitude of the model performing betteron the test data is smaller in the MLP models. Conversely, the MLP classification modelsdid not result in the model performing better on the test data, unlike the kNN classificationmodels that had a few instances occurring in datasets smaller than 1000 data points.Of note is that the MLP models result in a better performance metric than the kNNmodels above a certain dataset size (around 1500 data points in this example), thus demon-strating that the MLP models perform better than the kNN models if enough data areGeosciences 2024, 14, 67 11 of 31available. This improvement is more prominent in the classification models than the re-gression models (as an example, the kNN and MLP regression learning curves all showspikes at larger datasets), which is most likely due to the extreme outliers in the regressionmodels impacting the MLP and kNN regression models similarly. While the MLP modelsresult in a better performing model than the kNN models at larger dataset sizes, they donot improve the model performance or reliability for smaller datasets, and using MLPs canresult in more overfitting in certain scenarios (such as in the classification models in thisexample) compared to simpler models like kNN.Comparing the regression, classification, and pseudo-regression models demonstratesthat they are all unreliable at smaller dataset sizes (generally a few hundred data pointsin this example), however, the regression and pseudo-regression models tend to be moreunreliable at these smaller dataset sizes (i.e., they have more variation in their R2 and RMSEvalues for different random_state values for the same dataset) and can also be unreliableat larger dataset sizes (i.e., the spikes in the regression and pseudo-regression learningcurves), both of which can be attributed to the increased sensitivity of regression andpseudo-regression models to extreme outliers in the dataset. This sensitivity to outliersin the regression and pseudo-regression models also manifests in increased frequency ofoverfitting and the model performing better on the test data for the same dataset comparedto classification. While the regression and pseudo-regression models are susceptible tooutliers in the dataset, the classification models are more susceptible to the choice in thealgorithm. The MLP models resulted in better performance metric values than their kNNcounterparts when enough data was present, and there were fewer instances of the modelperforming better on the test dataset than on the training dataset. An important note is thatthe results of the classification and pseudo-regression models are sensitive to how the datawas binned; larger bins (i.e., fewer classes) will result in better classification performancemetrics and generally worse pseudo-regression performance metrics, while smaller bins(i.e., more classes) will result in worse classification performance metrics and generallybetter pseudo-regression performance metrics.The results of the ML models examined demonstrate the importance of data quantityin developing reliable models. Note that all of the models investigated were supervisedmodels, and the observations in this paper are not applicable to unsupervised models.While the reliability and performance of ML models may be improved by performingclassification instead of regression, as well as performing hyperparameter tuning, all ofthe ML models examined tended to remain unreliable when developed on smaller datasetsizes (a few hundred data points in this example) as they are susceptible to how the datawere shuffled before the train/test split (i.e., the random_state value). Overfitting and themodel performing better on the test data are also more commonly found in ML modelstrained on smaller dataset sizes, thus reinforcing the importance of data quantity in ML. Itis important to note that the random_state parameter examined in these analyses representshow the data were randomly shuffled before the train/test split and is not a parameter thatshould be optimized to try to achieve a better model. If the results of an ML model varysignificantly depending on the random_state parameter, then that indicates that there isnot enough data to develop a reliable model.3.4. Skewed and Imbalanced DataMost real-world data are imbalanced, which can negatively impact the results of MLmodels. There are different approaches to handling imbalanced data, such as stratifiedsampling—where the class frequencies in the dataset are maintained in the training andtest data—and using balancing techniques. The balancing techniques examined in theML modeling are oversampling (where new samples in the underrepresented classes aregenerated by randomly sampling from the available data points) and SMOTE (wheresynthetic data are generated for the underrepresented classes). An example of the learningcurves for the models with stratified sampling, oversampling, and SMOTE are shown inFigures A7 and A8.Geosciences 2024, 14, 67 12 of 31Due to the nature of stratified sampling and SMOTE, they require a minimum amountof data in each class, resulting in a larger minimum dataset size. As a result, these tech-niques may not apply to rock engineering datasets that are too small and where the classfrequencies are too small. The data balancing techniques (oversampling and SMOTE) didnot improve the reliability of the models compared to the base models, as including themin the modeling process did not reduce the variation between the learning curves, espe-cially in smaller datasets. The data balancing techniques slightly increased the differencebetween the train and test accuracy, F1-score, and RMSE scores, resulting in marginallymore instances of overfitting compared to the base models. Similarly, stratified samplingdid not significantly impact the models’ reliability, however, unlike the data balancingtechniques, stratified sampling reduced the difference between the train and test accuracy,F1-score, and RMSE scores and subsequently reduced overfitting. Both stratified samplingand the data balancing techniques removed the (few) instances of the model performingbetter on the test data in the classification models.Another important feature of the target variable distribution is its skewness. Skeweddistributions are commonly found in rock engineering. They can significantly impactstatistical analyses and machine learning due to the potential for more outliers and moreextreme outliers. The safety factor distribution in Figure 4 is a positively skewed (right-tailed) lognormal distribution. Its longer tail contains more extreme outlier values, resultingin spikes in the regression learning curves (Figures A1 and A2). A common method ofhandling skewed data in regression models is to transform it into a normal distribution.Some examples of commonly used transformations include ln and box cox transformations.However, when evaluating the model, the predicted transformed target variables shouldbe back transformed into the original scale. For this example, an ln transformation wasperformed on the target variable (safety factor) for the kNN regression model, whichhad a minimal impact on the learning curves for both the regression and classificationmodels, as shown in Figures A9–A12. The ln transformation also did not impact themagnitude and frequency of overfitting or the model performing better on the test datawhen compared to not performing an ln transformation. Another method of handling theextreme outlier values that are commonly found in skewed data is to bin the data andperform a classification (Models 6–11), which reduced the spikes in the learning curves andgenerally resulted in a more reliable model than performing an ln transformation in theregression model.3.5. LimitationsSimilar to what was noted in [1], the results of the ML modeling performed in thissection demonstrate the issues associated with data quantity are specific to the syntheticdata generated in SWedge and the ML modeling workflow used. The dataset sizes thatresult in unreliable ML models in this paper should not be taken as a universal guideline,as they may change depending on the problem, data, and ML methodology. Furthermore,it is worth repeating that the random_state value in scikit-learn’s train_test_split function(i.e., how the data are randomly shuffled before the train/test split) is not a hyperparameterand should not be optimized to obtain a better model. Suppose the results of the ML modelvary significantly due to how the data were shuffled before the train/test split. In that case,that indicates that there is not enough data to develop a reliable ML model.4. Knowledge IssuesA fundamental aspect of developing reliable ML models is ensuring that rock engi-neers involved in an ML project, from those developing the model to the project managersand senior engineers overseeing the development, have sufficient knowledge of crucialML concepts and methodological issues. Like other industries outside of computer sci-ence/data science that have begun applying ML to their domain-specific problems, rockengineering has a limited number of practitioners who understand ML. This is especiallyapparent in rock engineering academic publishing; since ML is so new in rock engineering,Geosciences 2024, 14, 67 13 of 31there are a limited number of experts, making it difficult to find reviewers who are knowl-edgeable about the topic and leading to flawed methodologies and/or overly complexmodels developed on simple or data-poor problems being published. For example, ref. [12]developed a model that performs better on the test data and trained the model on incorrectrock mass classification data. Another study, ref. [26], developed models based on theirentire dataset instead of splitting the data. Other examples include misunderstandingsof ML/statistics-specific terms [27], which led to incorrectly identifying non-normal dis-tributions. As discussed by [28], language is not neutral, and even more important is theperception that specific words leave in the mind of an audience that are neither engineersnor geoscientists. Indeed, in the author’s experience, rock engineering practitioners oftenmisunderstand the terms “validation” and “skewed data”. Potential solutions to theseknowledge issues include staffing rock engineering ML projects with data and computerscientists, updating mining and geological engineering curricula to include courses on datascience, and mandating that ML publications publish their data and codes so that lessonslearned and best practices become easier to access.5. Conclusions and RecommendationsThe development of ML models in rock engineering is not immune to the importanceof data quality and quantity. The use of subjective rock engineering parameters in MLmodels will result in an equally subjective model and model predictions, and the ML modelsexamined in this paper demonstrate that ML models run the risk of being unreliable ifthey are trained on small datasets, regardless of the data pre-processing steps includedin the modeling workflow. This unreliability is exacerbated when extreme outliers arepresent in the dataset (which tends to occur more frequently in skewed data). It can resultin unreliable ML models even when trained on larger dataset sizes. While binning thedata to perform classification can help to mitigate the effects of extreme outliers on modelreliability at larger dataset sizes, classification models (and pseudo-regression models)are dependent on how the data are binned, and they still result in unreliable ML modelswhen trained on smaller dataset sizes. It is important to remember that the dataset sizesoutlined in this paper are specific to this example and there is no universal definition ofa small dataset. We recommend that any rock engineers considering applying ML to aproblem follow the methodology outlined in this paper and in [1] to determine if there areenough data to develop a reliable ML model. Additional recommendations to ensure thedevelopment of reliable ML models in rock engineering include:• Understanding the methodological issues at play.• Visualizing the data and performing exploratory data analysis to gain a better under-standing of the data.• Starting simple and increasing the complexity of your model as needed. Increas-ing the complexity of the model does not necessarily improve the model and iscomputationally intensive.If the findings discussed herein are overlooked, ML has the potential to create unnec-essary computational expense and model complexity, reinforce and bury data biases, andslow down or even derail engineering decision-making. However, if developed prudentlyand with an understanding of blind spots in the data, ML is a powerful tool that may beleveraged by rock engineers to relieve the burden of data analysis, uncover hidden databiases, and expedite decision-making.Author Contributions: Conceptualization, B.Y. and D.E.; methodology, B.Y. and L.J.H.; modeling,B.Y.; writing—original draft preparation, B.Y.; writing—review and editing, B.Y., L.J.H., J.M. and D.E.;supervision, L.J.H. and D.E. All authors have read and agreed to the published version of the manuscript.Funding: The authors would like to acknowledge the financial support of: Natural Sciences andEngineering Research Council of Canada (NSERC PGS-D), NSERC Grant LJGQ GR000440, and MitacsGrant POEH/GR022346.Geosciences 2024, 14, 67 14 of 31Data Availability Statement: The data used in this study were generated from SWedge and areavailable from the following link https://github.com/beverlyyang/geosciences-machine-learning-in-engineering-geology.Conflicts of Interest: The authors declare no conflicts of interest.Appendix ATable A1 below provides a summary of recent ML applications to rock mass character-ization and classification data, including the concerns with their methodology.Table A1. Summary of recent ML applications to rock mass characterization and classification data..Reference Application Dataset Size Results Data andCode Availability Concerns[29]Used support vector machineand back-propagation neuralnetwork to predict basic rockquality (BQ) class frominputs, including JRC,volumetric joint count (Jv),groundwater condition, etc.25 data points (20 fortraining, 5 for test). Alsoincluded a separatedataset of 20 data pointsto “verify” the accuracyof their model.Accuracy on the 20 datapoints to “verify” theaccuracy of their modelis 95%.The dataset is notavailable. The code isnot available.Small and imbalanced dataset (BQclass ranges from 3–5, with themajority being 4). The data islimited to Anhui province inChina. The ML model isredundant, as many of the inputfeatures are used in the equationsto calculate BQ class.The training and test accuracy arenot reported.[30]Used naïve Bayes, randomforest, artificial neuralnetwork, and support vectormachine to predict the rockmass rating (RMR) rock massclass from the RMR ratingsfor intact rock strength,discontinuity spacing,weathering, persistence,aperture, and presence ofwater. The inputs werelabel encoded.3216 data points (2144training, 1072 test). Theauthors performed theirown version ofcross-validation, wherethe model was trained30 times on a differenttrain/test split and themodel performance foreach split was averaged.Average test accuracyranges from 0.81–0.89 forall models.The dataset is available.The code isnot available.The authors did not make it clearwhich RMR version was used(RMR89 was assumed). Theauthors’ version ofcross-validation could result indata leakage (where the test datainfluence the training of themodel). The training results werenot reported. The models areredundant, as the model inputsmake up 68% of RMR89, however,it is possible to estimate the RQDrating from the discontinuityspacing rating since the twoparameters are linked, meaningthat there is enough informationto estimate the RMR class withoutthe use of ML.[12]Used support vector machine(SVM), decision trees (DT),random forest (RF), Gaussianprocess regression (GPR),and ensemble learning (EL)using the previous 4 modelsto predict the end-bearingcapacity of rock-socketedshafts from the unconfinedcompressive strength ofintact rock, geologicalstrength index (GSI), lengthof shaft within the soil layer,length of shaft within therock layer, andshaft diameter.151 data points (121training, 30 test points).SVM: training R2 of 0.56,test R2 of 0.55, trainingRMSE of 2.13, testRMSE of 2.33.DT: training R2 of 0.80,test R2 of 0.85, trainingRMSE of 1.38, testRMSE of 1.21.RF: training R2 of 0.82,test R2 of 0.87, trainingRMSE of 1.38, testRMSE of 1.42.GPR: training R2 of 0.79,test R2 of 0.83, trainingRMSE of 1.50, testRMSE or 1.54.EL: training R2 of 0.84,test R2 of 0.89, trainingRMSE of 1.39, testRMSE of 1.17.The dataset is availablebut contains 138 datapoints instead of 151.The code is availableupon request.The dataset is small and some ofthe models perform better on thetest dataset than the trainingdataset. Refer to Section 3 foradditional details.[13]Used relevance vectorregression (RVR) andsupport vector regression(SVR) to predict RMR valuesfrom seismic velocity (Vpand Vs), seismic wave type,orientation, polarity, wavemagnitude, andreflection depth.132 data points (92training, 40 test). Novalidation orcross-validationwas performed.RVR: training R of 0.99,test R of 0.94, trainingRMSE of 1.4, testRMSE of 4.3.SVR: training R of 0.995,test R of 0.96, trainingRMSE of 1.1, testRMSE of 3.6.The dataset and code arenot available.The dataset is small andlimited—the test RMR values areconcentrated between 0 and 20and 60–85. The data is also basedon two case studies in Iran. TheRMR version was not specified bythe authors.Geosciences 2024, 14, 67 15 of 31Table A1. Cont.Reference Application Dataset Size Results Data andCode Availability Concerns[26]Used classification andregression tree (CART) aswell as genetic programming(GP) to predict tunnel boringmachine (TBM) performance(FPI—field penetration index)from uniaxial compressivestrength (UCS), rock qualitydesignation (RQD, jointspacing, partial jointcondition rating in RMR89,and rock type code.580 data points from7 tunneling projects(4 from Iran, 1 fromNew Zealand, 1 fromIndia, 1 fromSwitzerland; 35%metamorphic, 37%igneous, 28%sedimentary). Atrain/test split was notperformed (to the best ofmy knowledge).CART: entire dataset R2of 0.91, entire dataset ofRMSE 6.67.GP: entire dataset R2 of0.85, entire datasetRMSE of 8.46.The dataset and code arenot available.A train/test split was notperformed (i.e., the model resultsare for the entire dataset), makingit difficult to determine how wellthe models will generalize (whichis the end goal of ML). Theauthors did not explain what apartial joint condition rating fromRMR89 meant.[31]Used k-nearest neighbors(kNN), naïve Bayes (NB),random forest (RF), artificialneural network (ANN), andsupport vector machine(SVM) to predict rockcompressive strength fromacoustic characteristicsstemming from hitting thecore with a geologicalhammer (amplitudeattenuation coefficient (AAC)and high and low frequencyratio (HLFR)).2104 data points(1614 training, 400 test).kNN: training R2 of 0.98,test R2 of 0.98, trainingRMSE of 0.6, testRMSE of 0.6.NB: training R2 of 0.88,test R2 of 0.88, trainingRMSE of 3.84, testRMSE of 3.94.RF: training R2 of 0.98,test R2 of 9.07, trainingRMSE of 1.1, testRMSE of 1.8.ANN: training R2 of 0.99,test R2 of 0.99, trainingRMSE of 0.3, testRMSE of 0.3.SVM: training R2 of 0.99,test R2 of 0.99, trainingRMSE of 0.08, testRMSE of 0.1.The dataset and code arenot available.AAC and HLFR showed goodcorrelation with rock compressivestrength without the use of ML(R2 of 0.91 for AAC and R2 of 0.93for HLFR), putting into questionthe practicality of using ML forthis application.[32]Used random forest topredict Is50 classes fromfracture frequency, RQD,mineralized veins per meter,rock density, intact rockstrength hammer test class,alteration strength index,mineralization strengthindex, mineralization percentsum, fracture spacing,fracture frequency index,discontinuity frequencyindex, discontinuityfrequency index weighted byorientation, joint set index,rock colour, lower contact,rock texture, rock type,selvage mineralization,mineral metric, rockstructure, rock fabric, PLTmachine ID.7687 data points. Theauthors performed theirown version ofcross-validation, wheretheir model was trainedon all boreholes exceptfor one, and then testedon the remainingborehole. This wasrepeated until allboreholes had been usedas the ”test dataset”, andthe score for eachborehole was averaged.The average accuracy ofthe boreholes is 39%.The dataset and code arenot available.The authors’ version ofcross-validation could result indata leakage (where the test datainfluences the training of themodel). The training results werenot reported. Some of the inputfeatures are correlated with oneanother (ex; fracture frequencyand fracture frequency index).The reported accuracy is low anddoes not represent the true “test”accuracy (which should be lowerthan the reported 39%).[33]Used stacked autoencoders(SAEs—a type of deeplearning) to predict RMRrock mass class from its inputparameters (UCS, RQD,spacing, persistence,aperture, roughness, infilling,weathering, groundwater,and orientation). The ratingclasses of each inputparameter was one-hotbinary encoded (ex; the UCSrating class for UCS > 250MPa was transformed into1000000, while the UCSrating class for UCS between100 and 250 MPa wastransformed into0100000, etc.)309 data points(232 training, 77 test).The train and testaccuracy are both 100%.The dataset and code arenot available.The dataset is small, especially fora neural network. The model isredundant as it uses the inputs ofRMR to determine RMR, whichcan be done (and is done) easilywithout ML in Excel. The authorsalso did not specify which RMRversion was used (RMR89 wasassumed). The authors alsomisused the word calibration, asthey referred to one-hot encodingthe input features as “calibration”,despite it being a form ofdata preparation.Geosciences 2024, 14, 67 16 of 31Table A1. Cont.Reference Application Dataset Size Results Data andCode Availability Concerns[34]Used ANN to predict RMRvalues from its inputparameters (strength,spacing, RQD, jointconditions,and groundwater).The dataset sizeis unclear.The MAPE was less than1%. It was unclear if thiswas for the training ortest set.The dataset and code arenot available.The dataset size is unclear, and itis unclear if the modelperformance is for the training ortest set. The model is redundantas it uses the inputs of RMR todetermine RMR, which can bedone (and is done) easily withoutML in Excel. The authors also didnot specify which RMR versionwas used (RMR89 was assumed).[18]Used Gaussian processregression (GPR), supportvector regression (SVR),decision tree (DT), and longshort-term memory(LSTM—a type of neuralnetwork) to predict cohesionand friction angle ofsandstone from uniaxialcompressive strength,uniaxial tensile strength, andconfining stress.233 data points fromRockData (195 training,49 test). The authorsperformed a 5-foldcross-validation.For cohesion, the R2 ofall of the models rangedfrom 0.95 to 0.98, whilethe RMSE ranged from1.3 to 2.38.For friction angle, the R2of all the models rangedfrom 0.6 to 0.85, whilethe RMSE ranged from1.86 to 9.88. It wasunclear if this was for thetraining set, test set, orentire dataset.The dataset is availablein RockData. The code isnot available.The dataset size is small,especially for LSTM, and limitedto only sandstone. It is unclear ifthe model performance values arefor the training set, test set, orentire dataset, making it difficult,if not impossible, to determinehow well the model generalizes.[35]Used convolutional neuralnetworks (CNN) to predictRQD from core photos.124 images (99 trainingimages, 25 test images).Average test error of3.24%. RQD for trainingwas determined by twoexperienced engineers.The dataset is availableupon request. The codeis available.The dataset is small, especially fora CNN. The model has issuesdifferentiating between naturaland mechanical fractures. Thehard and soundness requirementwas not mentioned in the paper.[15]Used convolutional neuralnetworks (CNN) to predictRQD from core photos.7030 images (6400training images, 630 testimages). The trainingimages were ofsandstone, while the testimages contained 540sandstone images and90 limestone images.Test error for sandstoneis 2.58%, while the testerror for limestoneis 3.17%.The dataset is notavailable. The codeis available.The model does not differentiatebetween natural vs. mechanicalfractures and the hard andsoundness requirement is notmentioned. The model is limitedto sandstone and limestone.[23]Used artificial neuralnetworks to predict intactrock elastic modulus fromuniaxial compressivestrength and the unit weightof the rock.609 data points(487 training, 122 test).Training and test resultsare unclear.The dataset and code arenot available.The training and test resultsare unclear.[20]Used support vectormachines with heuristicoptimization algorithms (forhyperparameter tuning) topredict the rock mass grade(based on a scale of 1 to 5using qualitativedescriptions) from saturatedrock compressive strength,RQD, rock mass integrityfactor, and water inflow.80 data points fromChina (64 training, 16test). The datais available.Training accuracy of 91%and test accuracy of 94%.The dataset is available.The code isnot available.The dataset is very small, and themodel performs better on the testdataset than the training dataset(despite its limited dataset size).[36]Used support vectorregression to predict themodulus of deformationfrom the dynamic modulusof elasticity, uniaxialcompressive strength, RQD,joint conditions, andjoint spacing.88 data points. Theauthors only performedcross-validation on theentire dataset. There isno test dataset.Cross-validation resultsfor the SVR include R2 of0.87 and RMSE of 1.01.The dataset and code arenot available.The authors did not explain howthe joint condition parameter wasdetermined. The dataset is smalland limited to one site. The resultsof cross-validation cannot be usedto determine how well the modelwill perform on new, unseen data.A test dataset is missing.Geosciences 2024, 14, 67 17 of 31Table A1. Cont.Reference Application Dataset Size Results Data andCode Availability Concerns[17]Used k-nearest neighbor(kNN), random forest (RF),multi-layer perceptron(MLP), random tree (RT), andstacked RT-RF-kNN-MLP topredict Young’s modulus(modulus of elasticity) fromporosity, Schmidt hammerrebound number, pulsevelocity, and Is50.92 data points.Performed a sensitivityanalysis for each modelusing differentpercentages of trainingdata (80%, 85%, and 90%of the entire dataset).For an 80/20train/test split:RT: training R2 of 0.84,test R2 of 0.67, trainingRMSE of 14.92, testRMSE of 20.47.kNN: training R2 of 0.82,test R2 of 0.78, trainingRMSE of 15.81, testRMSE of 17.02.RF: training R2 of 0.84,test R2 of 0.71, trainingRMSE of 14.80, testRMSE of 19.04.MLP: training R2 of 0.90,test R2 of 0.77, trainingRMSE of 11.78, testRMSE of 16.97.Stacked: training R2 of0.83, test R2 of 0.82,training RMSE of 15, testRMSE of 15.01.The dataset and code arenot available.The dataset is small. Each inputalready shows a strong linearcorrelation with the output.[37]Used support vector machine(SVM), k-nearest neighbor(kNN), random forest (RF),gradient boosting decisiontree (GBDT), decision tree(DT), logistic regression (LR),multi-layer perceptron(MLP), and a stackingensemble classifier to predictrock mass class (on a scale of1 to 5) from 10 TBMoperational parameters(cutterhead rotational speed,pitch angle of gripper shoes,gear sealing pressure,pressure of gripper shoes,output frequency of maindrive motor, internal pumppressure, penetration rate,control pump pressure,torque penetration index,and roll position ofgripper shoes).7538 data points (6784training, 754 test). A10-fold cross-validationwas performed.SVM: test accuracy of89%, test F1 of 0.89.kNN: test accuracy of88%, test F1 of 0.87.RF: test accuracy of 91%,test F1 of 0.90.GBDT: test accuracy of92%, test F1 of 0.92.DT: test accuracy of 87%,test F1 of 0.87.LR: test accuracy of 79%,test F1 of 0.75.MLP: test accuracy of81%, test F1 of 0.80.Stacked ensembleclassifier: test accuracyof 93%, test F1 of 0.93.The dataset and code arenot available.Training results werenot provided.[38]Used a back-propagationneural network to predict therock mass deformationmodulus from the uniaxialcompressive strength ofintact rock, RQD, dry density,porosity, number of joints permeter, and GSI. A geneticalgorithm was used tooptimize the neural network.120 data points. Theamount of data in thetraining and testdatasets was not clear.Data are limited to foursites in Iran and consistpredominantly ofsedimentary rocks.Training R of 0.981, testR of 0.4. Training MSE of3.16, test MSE of 5.21.The dataset and code arenot available.The dataset size is small,especially for a neural network,and limited to sedimentary rocks.The size of the training and testdatasets were not mentioned. Theauthors did not mention how therock mass deformation modulusvalues were determined and howreliable these values are, despitementioning that the credibility ofthe results of in situ tests arequestionable.Geosciences 2024, 14, 67 18 of 31Appendix BThe learning curves for Models 1a, 1b, 5, 6a, 6b, 9a, 10a, 11a, and 12 are shown here.Geosciences 2024, 14, x FOR PEER REVIEW 23 of 35 Appendix B The learning curves for Models 1a, 1b, 5, 6a, 6b, 9a, 10a, 11a, and 12 are shown here. Figure A1. Regression learning curves for Model 1a (kNNRegressor base case): (a) R2 learning curves, (b) RMSE learning curves, (c) RMSE difference learning curves. (a) (b) (c) Figure A1. Regression learning curves for odel 1a (kNNRegressor base case): (a) R2 learning curves,(b) RMSE learning curves, (c) RMSE difference learning curves.Geosciences 2024, 14, 67 19 of 31Geosciences 2024, 14, x FOR PEER REVIEW 24 of 35 Figure A2. Regression learning curves for Model 1b (MLPRegressor base case): (a) R2 learning curves, (b) RMSE learning curves, (c) RMSE difference learning curves. (a) (b) (c) Figure A2. Regression learning curves f el 1b (MLPRegressor base case): (a) R2 learning curves,(b) RMSE learning curves, (c) RMSE difference learning curves.Geosciences 2024, 14, 67 20 of 31Geosciences 2024, 14, x FOR PEER REVIEW 25 of 35 Figure A3. Classification learning curves for Model 6a (kNNClassifier base case): (a) accuracy learn-ing curves, (b) F1-score learning curves, (c) accuracy difference learning curves, (d) F1-score differ-ence learning curves. (a) (b) (c) (d) 3. Classification lear ing curves for Model 6a (kNNClassifier b case): (a) accuracylearning curves, (b) F1-sco e learning curves, (c) accuracy difference learning curves, (d) F1-sco edifference learning curves.Geosciences 2024, 14, 67 21 of 31Geosciences 2024, 14, x FOR PEER REVIEW 26 of 35 Figure A4. Classification learning curves for Model 6b (MLPClassifier base case): (a) accuracy learn-ing curves, (b) F1-score learning curves, (c) accuracy difference learning curves, (d) F1-score differ-ence learning curves. (a) (b) (c) (d) 4. Classification learning curves for Model 6b (MLPClassifi r base case): ( ) accuracylearning curves, (b) F1-score learning curves, (c) accuracy diff re ce learning curves, (d) F1-scoredifference learning curves.Geosciences 2024, 14, 67 22 of 31Geosciences 2024, 14, x FOR PEER REVIEW 27 of 35 Figure A5. Pseudo-regression learning curves for Model 6a (kNNClassifier base case): (a) R2 learn-ing curves, (b) RMSE learning curves, (c) RMSE difference learning curves. (a) (b) (c) . s o-regres ion learning curves for Model 6a (kN Classifier base case): (a) R2 learningcurves, (b) RMSE learning curves, (c) RMSE difference learning curves.Geosciences 2024, 14, 67 23 of 31Geosciences 2024, 14, x FOR PEER REVIEW 28 of 35 Figure A6. Pseudo-regression learning curves for Model 6b (MLPClassifier base case): (a) R2 learn-ing curves, (b) RMSE learning curves, (c) RMSE difference learning curves. (a) (b) (c) r 6. se o-regres ion learning curves for Model 6b (MLPClassifier base case): (a) R2 learningcurves, (b) RMSE learning curves, (c) RMSE difference learning curves.Geosciences 2024, 14, 67 24 of 31Geosciences 2024, 14, x FOR PEER REVIEW 29 of 35 Figure A7. Accuracy learning curves for (a) Model 9a (kNNClassifier with stratified sampling), (b) Model 10a (kNNClassifier with oversampling), and (c) Model 11a (kNNClassifier with SMOTE). (a) (b) (c) 7. A curacy learning curves for (a) Model 9a (kNNClassifier with stratified sampling ,(b) Model 10a (kNNClassifier with oversampling), and (c) Model 11a (kNNClassifier with SMOTE).Geosciences 2024, 14, 67 25 of 31Geosciences 2024, 14, x FOR PEER REVIEW 30 of 35 Figure A8. RMSE (pseudo-regression) learning curves for (a) Model 9a (kNNClassifier with strati-fied sampling), (b) Model 10a (kNNClassifier with oversampling), and (c) Model 11a (kNNClassifier with SMOTE). (a) (b) (c) . ( seudo-regres ion) learning curves for (a) Model 9a (kN Classifier with stra ifiedsampling), (b) Model 10a (kNNClassifier with oversampling), and (c) Model 11a ( l ifiith S OTE).Geosciences 2024, 14, 67 26 of 31Geosciences 2024, 14, x FOR PEER REVIEW 31 of 35 Figure A9. R2 regression learning curves for (a) kNNRegressor base case (Model 1a) and (b) kNNRegressor with ln transformation and the predictions back transformed (Model 5). (a) (b) Figure A9. R2 re ression learni g curves for (a) kNNRegressor base case (Model 1a) and (b) kNNRe-gressor with ln transformation and the pre ictions back transformed (Model 5).Geosciences 2024, 14, 67 27 of 31Geosciences 2024, 14, x FOR PEER REVIEW 32 of 35 Figure A10. RMSE regression learning curves for (a) kNNRegressor base case (Model 1a) and (b) kNNRegressor with ln transformation and the predictions back transformed (Model 5). (a) (b) Figure A10. RMSE regression learning curves for (a) k Regressor base case (Model 1a) and(b) kNNRegressor with ln transformation and the predictions back transformed (Model 5).Geosciences 2024, 14, 67 28 of 31Geosciences 2024, 14, x FOR PEER REVIEW 33 of 35 Figure A11. Accuracy classification learning curves for (a) kNNClassifier base case (Model 6a) and (b) kNNClassifier with ln transformation and the predictions back transformed (Model 12). (a) (b) Figure A11. Accuracy classification learning curves for (a) kNNClassifier base case (Model 6a) and(b) k NClassifier with ln transfor ation and the predictions back transformed (Model 12).Geosciences 2024, 14, 67 29 of 31Geosciences 2024, 14, x FOR PEER REVIEW 34 of 35 Figure A12. F1-score classification learning curves for (a) kNNClassifier base case (Model 6a) and (b) kNNClassifier with ln transformation and the predictions back transformed (Model 12). References 1. Yang, B.; Tsai, A.; Mitelman, A.; Tsai, R.; Elmo, D. The importance of data quantity in machine learning—How small is too small? In Proceedings of the GeoSaskatoon 2023, Saskatoon, SK, Canada, 1–4 October 2023. 2. Krawczyk, B. Learning from imbalanced data: Open challenges and future directions. Prog. Artif. Intell. 2016, 5, 221–232. 3. National Institute of Standards and Technology. Measures of skewness and kurtosis. In Engineering Statistics Handbook; National Institute of Standards and Technology: Gaithersburg, MD, USA, 2012. 4. Grubbs, F.E. Procedures for Detecting Outlying Observations in Samples; National Institute of Standards and Technology: Gaithersburg, MD, USA, 1974. 5. What Is Overfitting? Available online: https://www.ibm.com/topics/overfitting (accessed on 1 December 2023). 6. Debrusk, C. The risk of machine learning bias (and how to prevent it). MIT Sloan Manag. Rev. 2018, 15, 1. 7. Manyika, J.; Silberg, J.; Presten, B. What do we do about the biases in AI? Harv. Bus. Rev. 2019. 8. Pells, P.J.; Bieniawski, Z.T.; Hencher, S.R.; Pells, S.E. Rock quality designation (RQD): Time to rest in peace. Can. Geotech. J. 2017, 54, 825–834. 9. Elmo, D.; Stead, D. The role of behavioural factors and cognitive biases in rock engineering. Rock Mech. Rock Eng. 2021, 54, 2109–2128. 10. Yang, B.; Mitelman, A.; Elmo, D.; Stead, D. Why the future of rock mass classification systems requires revisiting their empirical past. Q. J. Eng. Geol. Hydrogeol. 2022, 55, qjegh2021-039. https://doi.org/10.1144/qjegh2021-039. 11. Yang, B.; Elmo, D. Why engineers should not attempt to quantify GSI. Geosciences 2022, 12, 417. (a) (b) ( ) l ssifi r it l tr sf r ti t e re ictio s back transfor ed ( odel 12).References1. Yang, B.; Tsai, A.; Mitelman, A.; Tsai, R.; Elmo, D. The importance of data quantity in machine learning—How small is too small?In Proceedings of the GeoSaskatoon 2023, Saskatoon, SK, Canada, 1–4 October 2023.2. Krawczyk, B. Learning from imbalanced data: Open challenges and future directions. Prog. Artif. Intell. 2016, 5, 221–232.[CrossRef]3. National Institute of Standards and Technology. Measures of skewness and kurtosis. In Engineering Statistics Handbook; NationalInstitute of Standards and Technology: Gaithersburg, MD, USA, 2012.4. Grubbs, F.E. Procedures for Detecting Outlying Observations in Samples; National Institute of Standards and Technology: Gaithersburg,MD, USA, 1974.5. What Is Overfitting? Available online: https://www.ibm.com/topics/overfitting (accessed on 1 December 2023).6. Debrusk, C. The risk of machine learning bias (and how to prevent it). MIT Sloan Manag. Rev. 2018, 15, 1.7. Manyika, J.; Silberg, J.; Presten, B. What do we do about the biases in AI? Harv. Bus. Rev. 2019.8. Pells, P.J.; Bieniawski, Z.T.; Hencher, S.R.; Pells, S.E. Rock quality designation (RQD): Time to rest in peace. Can. Geotech. J. 2017,54, 825–834. [CrossRef]9. Elmo, D.; Stead, D. The role of behavioural factors and cognitive biases in rock engineering. Rock Mech. Rock Eng. 2021,54, 2109–2128. [CrossRef]Geosciences 2024, 14, 67 30 of 3110. Yang, B.; Mitelman, A.; Elmo, D.; Stead, D. Why the future of rock mass classification systems requires revisiting their empiricalpast. Q. J. Eng. Geol. Hydrogeol. 2022, 55, qjegh2021-039. [CrossRef]11. Yang, B.; Elmo, D. Why engineers should not attempt to quantify GSI. Geosciences 2022, 12, 417. [CrossRef]12. Chen, H.; Zhang, L. A machine learning-based method for predicting end-bearing capacity of rock-socketed shafts. Rock Mech.Rock Eng. 2022, 55, 1743–1757. [CrossRef]13. Gholami, R.; Rasouli, V.; Alimoradi, A. Improved RMR rock mass classification using artificial intelligence algorithms. Rock Mech.Rock Eng. 2013, 46, 1199–1209. [CrossRef]14. Rechlin, A.J.; Luth, S.; Giese, R. Rock mass classification based on seismic measurements using support vector machines. InProceedings of the 12th ISRM Congress, Beijing, China, 18–21 October 2011.15. Alzubaidi, F.; Mostaghimi, P.; Si, G.; Swietojanski, P.; Armstrong, R.T. Automated rock quality designation using convolutionalneural networks. Rock Mech. Rock Eng. 2022, 55, 3719–3734. [CrossRef]16. Shen, J.; Jimenez, R. Predicting the shear strength parameters of sandstone using genetic programming. Bull. Eng. Geol. Environ.2018, 77, 1647–1662. [CrossRef]17. Koopialipoor, M.; Asteris, P.G.; Mohammed, A.S.; Alexakis, D.E.; Mamou, A.; Armaghani, D.J. Introducing stacking machinelearning approaches for the prediction of rock deformation. Transp. Geotech. 2022, 34, 100756. [CrossRef]18. Mahmoodzadeh, A.; Mohammadi, M.; Salim, S.G.; Ali, H.F.H.; Ibrahim, H.H.; Abdulhamid, S.N.; Nejati, H.R.; Rashidi, S. Machinelearning techniques to predict rock strength parameters. Rock Mech. Rock Eng. 2022, 55, 1721–1741. [CrossRef]19. Fathipour-Azar, H. Shear strength criterion for rock discontinuities: A comparative study of regression approaches. Rock Mech.Rock Eng. 2023, 56, 4715–4725. [CrossRef]20. Hu, J.; Zhou, T.; Ma, S.; Yang, D.; Guo, M.; Huang, P. Rock mass classification prediction model using heuristic algorithms andsupport vector machines: A case study of Chambishi copper mine. Sci. Rep. 2022, 12, 928. [CrossRef]21. Rocscience. SWedge—Surface Wedge Analysis of Slopes. 2023. Available online: https://www.rocscience.com/software/swedge(accessed on 1 December 2023).22. Chawla, N.V.; Bowyer, K.W.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic minority over-sampling technique. J. Artifical Intell.Res. 2002, 16, 321–357. [CrossRef]23. Sonmez, H.; Gokceoglu, C.; Nefeslioglu, H.A.; Kayabasi, A. Estimation of rock modulus: For intact rocks with an artificial neuralnetwork and for rock masses with a new empirical equation. Int. J. Rock Mech. Min. Sci. 2006, 43, 224–235. [CrossRef]24. Xu, H.; Zhou, J.; Asteris, P.G.; Armaghani, D.J.; Tahir, M.M. Supervised machine learning techniques to the prediction of tunnelboring machine penetration rate. Appl. Sci. 2019, 19, 3715. [CrossRef]25. Rossbach, P. Neural Networks vs Random Forests—Does It Always Have to Be Deep Learning? 2018. Available online:https://blog.frankfurt-school.de/neural-networks-vs-random-forests-does-it-always-have-to-be-deep-learning/ (accessed on1 December 2023).26. Salimi, A.; Rostami, J.; Moormann, C. Application of rock mass classification systems for performance estimation of rock TBMsusing regression tree and artificial intelligence algorithms. Tunneling Undergr. Space Technol. 2019, 92, 103046. [CrossRef]27. Lin, Y.; Zhou, K.; Li, J. Application of cloud model in rock burst prediction and performance comparison with three machinelearning algorithms. IEEE Access 2018, 6, 30958–30968. [CrossRef]28. Elmo, D.; Mitelman, A.; Yang, B. Examining rock engineering knowledge through a philosophical lens. Geosciences 2022, 12, 174.[CrossRef]29. Liu, K.; Liu, B.; Fang, Y. An intelligent model based on statistical learning theory for engineering rock mass classification. Bull.Eng. Geol. Environ. 2018, 78, 4533–4548. [CrossRef]30. Santos, A.E.M.; Lana, M.S.; Pereira, T.M. Evaluation of machine learning methods for rock mass classification. Neural Comput.Appl. 2021, 34, 4633–4642. [CrossRef]31. Ren, Q.; Wang, G.; Li, M.; Han, S. Prediction of rock compressive strength using machine learning algorithms based on spectrumanalysis of geological hammer. Geotech. Geol. Eng. 2018, 37, 475–489. [CrossRef]32. Thielsen, C.; Furtney, J.K.; Valencia, M.E.; Pierce, M.; Orrego, C.; Stonestreet, P.; Tennant, D. Application of machine learning tothe estimation of intact rock strength from core logging data: A case study at the Newcrest Cadia East Mine. In Proceedings ofthe 56th US Rock Mechanics/Geomechanics Symposium, Santa Fe, NM, USA, 26–29 June 2022.33. Sheng, D.; Yum, J.; Tan, F.; Tong, D.; Yan, T.; Lv, J. Rock mass quality classification based on deep learning: A feasibility study forstacked autoencoders. J. Rock Mech. Geotech. Eng. 2023, 15, 1749–1758. [CrossRef]34. Brousset, J.; Pehovaz, H.; Quispe, G.; Raymundo, C.; Moguerza, J.M. Rock mass classification method applying neural networksto minimize geotechnical characterization in underground Peruvian mines. Energy Rep. 2023, 9, 376–386. [CrossRef]35. Su, R.; Zhao, Q.; Zheng, T.; Han, G.; Jiang, J.; Hu, J. A framework for RQD calculation based on deep learning. Min. Metall. Explor.2023, 40, 1567–1583. [CrossRef]36. Meybodi, E.E.; DastBaravarde, A.; Hussain, S.K.; Karimdost, S. Machine-learning method applied to provide the best predictivemodel for rock mass deformability modulus. Environ. Earth Sci. 2023, 82, 149. [CrossRef]Geosciences 2024, 14, 67 31 of 3137. Hou, S.; Liu, Y.; Yang, Q. Real-time prediction of rock mass classification based on TBM operation big data and stacking techniqueof ensemble learning. J. Rock Mech. Geotech. Eng. 2022, 14, 123–143. [CrossRef]38. Majdi, A.; Beiki, M. Evolving neural network using a genetic algorithm for predicting the deformation modulus of rock masses.Int. J. Rock Mech. Min. Sci. 2010, 47, 246–253. [CrossRef]Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individualauthor(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury topeople or property resulting from any ideas, methods, instructions or products referred to in the content.
Article
eng
Reviewed
Vancouver : University of British Columbia Library
Multidisciplinary Digital Publishing Institute
CC BY 4.0
Faculty
Researcher
rock engineering
machine learning
rock mass characterization
rock mass classification
Algorithmic Geology: Tackling Methodological Challenges in Applying Machine Learning to Rock Engineering
Text