Applied Science, Faculty of
Mechanical Engineering, Department of
DSpace
UBCO
Barriault, Matthew
Master of Applied Science - MASc
University of British Columbia
Electronic noses are used in diverse applications where sensory information about chemical compounds in the air can provide valuable insight, just as humans use their sense of smell this way. In this thesis, the role of machine learning for data analysis in three main applications of electronic noses is examined. These applications are the regression of arbitrary binary mixtures, detection of natural gas leaks, and detection of THC in human breath. To emphasize the importance of machine learning in the overall workflow of an electronic nose, the data collection apparatus selected was a single metal oxide semiconducting sensor embedded in a coated microchannel to provide selectivity. This apparatus produces a time-series response as the target gas sample diffuses through the microchannel toward the sensor. A comparison of machine learning models and time-series data representation methods is examined for each application. Using this method, a 98.75% accuracy in classifying between methane, ethane, and binary mixtures is obtained, as well as a 12.0% mean relative error in regression estimates for arbitrary mixtures. For the natural gas detection application, simulated natural gas mixtures containing 1% and 3% ethane were discriminated from samples of pure methane with 86.7% and 93.3% accuracy, respectively, and concentrations of both methane and ethane were regressed with a maximum of a 19.3% mean relative error. Finally, samples of methanol were discriminated from samples of denatured THC dissolved in methanol with concentration 1.0 mg/mL. This classification was performed with 100% accuracy using a support vector machine.
Applications of Artificial Olfaction and Machine Learning for Detection of Volatile Gas by Matthew Barriault B.A.Sc., University of British Columbia, 2016 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE COLLEGE OF GRADUATE STUDIES (Mechanical Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Okanagan) March 2020 © Matthew Barriault, 2020 ii The following individuals certify that they have read, and recommend to the College of Graduate Studies for acceptance, a thesis/dissertation entitled: Applications of Artificial Olfaction and Machine Learning for Detection of Volatile Gas submitted by Matthew Barriault in partial fulfillment of the requirements of the degree of Master of Applied Science. Dr. Homayoun Najjaran, School of Engineering Supervisor Dr. Mina Hoorfar, School of Engineering Supervisor Dr. Abbas Milani, School of Engineering Supervisory Committee Member Dr. Zheng Liu, School of Engineering Supervisory Committee Member Dr. Richard Klukas, School of Engineering University Examiner iii Abstract Electronic noses are used in diverse applications where sensory information about chemical compounds in the air can provide valuable insight, just as humans use their sense of smell this way. In this thesis, the role of machine learning for data analysis in three main applications of electronic noses is examined. These applications are the regression of arbitrary binary mixtures, detection of natural gas leaks, and detection of THC in human breath. To emphasize the importance of machine learning in the overall workflow of an electronic nose, the data collection apparatus selected was a single metal oxide semiconducting sensor embedded in a coated microchannel to provide selectivity. This apparatus produces a time-series response as the target gas sample diffuses through the microchannel toward the sensor. A comparison of machine learning models and time-series data representation methods is examined for each application. Using this method, a 98.75% accuracy in classifying between methane, ethane, and binary mixtures is obtained, as well as a 12.0% mean relative error in regression estimates for arbitrary mixtures. For the natural gas detection application, simulated natural gas mixtures containing 1% and 3% ethane were discriminated from samples of pure methane with 86.7% and 93.3% accuracy, respectively, and concentrations of both methane and ethane were regressed with a maximum of a 19.3% mean relative error. Finally, samples of methanol were discriminated from samples of denatured THC dissolved in methanol with concentration 1.0 mg/mL. This classification was performed with 100% accuracy using a support vector machine. iv Lay Summary An electronic nose is a device that mimics the human sense of smell by using sensors to perform odour detection. This is usually accomplished by using several different types of sensors and combining their outputs. This is similar to how humans have different types of odour receptors for different scents. This thesis examines the impact of different methods of data analysis for an electronic nose that uses only a single sensor, and demonstrates success on tasks that would usually require two of more sensors. These tasks include real-world applications like pipeline leak detection and marijuana detection in breath. v Preface Much of the work presented in this thesis is an extension of that published in the Conference Proceedings of the Canadian Society of Mechanical Engineers (CSME) in 2018. This paper was titled “Quantitative Natural Gas Discrimination For Pipeline Leak Detection Through Time-Series Analysis of an MOS Sensor Response”. The results presented in Chapter 6 are taken partially from this publication. The manuscript was drafted by myself and Mahyar Mohaghegh Montazeri, with editorial support provided by Dr. Homayoun Najjaran, Dr. Mina Hoorfar, and Allen O’Brien. Design and scope of this research were developed in collaboration with and under the supervision of Dr. Homayoun Najjaran and Dr. Mina Hoorfar. Field testing for natural gas leaks as presented in Chapter 5 was performed in collaboration with Hetek Solutions Inc. and tests were conducted under the supervision of Gerald Mauthe. vi Table of Contents Abstract .................................................................................................................................... iii Lay Summary ........................................................................................................................... iv Preface ....................................................................................................................................... v Table of Contents ..................................................................................................................... vi List of Tables ............................................................................................................................ ix List of Figures .......................................................................................................................... xi Acknowledgements ............................................................................................................... xiii Dedication .............................................................................................................................. xiv Chapter 1: Introduction ............................................................................................................ 1 1.1 Research overview ................................................................................................... 1 1.2 Thesis outline ........................................................................................................... 2 Chapter 2: Background and Literature Review ....................................................................... 3 2.1 MOS Sensors ........................................................................................................... 3 2.2 Electronic Noses ....................................................................................................... 4 2.3 Natural Gas Detection .............................................................................................. 6 2.4 Human Breath Analysis ............................................................................................ 8 2.5 Machine Learning with Electronic Noses .................................................................10 2.5.1 Learning from Time-Series Data ........................................................................12 2.5.2 E-nose Data Pre-processing ..............................................................................13 2.5.3 Dimensionality Reduction ...................................................................................14 2.5.3.1 Feature Extraction ......................................................................................14 2.5.3.2 Feature Selection .......................................................................................17 2.5.3.3 Dimensionality-Reducing Transforms .........................................................23 2.5.4 Machine Learning Models ..................................................................................24 vii 2.5.4.1 k-Nearest Neighbours ................................................................................24 2.5.4.2 Random Forest ..........................................................................................26 2.5.4.3 Support Vector Machine .............................................................................29 2.5.4.4 Multi-Layer Perceptron ...............................................................................31 2.5.4.5 Convolutional Neural Network ....................................................................33 2.5.5 Hyper-Parameter Selection ................................................................................34 Chapter 3: Experimental and Sensing Apparatus................................................................. 36 3.1 Sensing Apparatus ..................................................................................................36 3.2 Automatic Sample Generation .................................................................................38 3.3 Natural Gas Detector ...............................................................................................44 3.4 Breath THC Detector ...............................................................................................48 Chapter 4: Analysis Methodology .......................................................................................... 51 4.1 Data Pre-processing ................................................................................................51 4.2 Data Representation and Feature Extraction ...........................................................51 4.2.1 Feature Extraction ..............................................................................................51 4.2.2 Feature Selection ...............................................................................................53 4.2.3 Dimensionality-reducing Transformations ..........................................................53 4.3 Hyper-parameter Selection ......................................................................................54 4.3.1 k-nearest Neighbours .........................................................................................54 4.3.2 Random Forest ..................................................................................................54 4.3.3 Support Vector Machine .....................................................................................56 4.3.4 Multi-layer Perceptron ........................................................................................56 4.3.5 Convolutional Neural Network ............................................................................57 Chapter 5: Experimental Methodology .................................................................................. 59 viii 5.1 Experiment I: Classification and Regression of Arbitrary Mixtures with Feature-based Representation ..................................................................................................................61 5.2 Experiment II: Classification and Regression of Arbitrary Mixtures with Raw Data Representation ..................................................................................................................62 5.3 Natural Gas Discrimination ......................................................................................62 5.3.1 Experiment III: Simulated Natural Gas Mixtures .................................................62 5.3.2 Experiment IV: Outdoor Leak Detection .............................................................63 5.4 Experiment V: THC Discrimination ...........................................................................64 Chapter 6: Results .................................................................................................................. 65 6.1 Experiment I: Classification and Regression of Arbitrary Mixtures with Feature-based Representation ..................................................................................................................65 6.2 Experiment II: Classification and Regression of Arbitrary Mixtures with Raw Data Representation ..................................................................................................................71 6.3 Natural Gas Discrimination ......................................................................................73 6.3.1 Experiment III: Simulated Natural Gas Mixtures .................................................73 6.3.2 Experiment IV: Outdoor Leak Detection .............................................................76 6.4 Experiment V: THC Discrimination ...........................................................................79 Chapter 7: Conclusions and Future Work ............................................................................. 81 Bibliography ............................................................................................................................ 84 Appendices ............................................................................................................................. 89 Appendix A: Complete list of hand-designed features .......................................................89 ix List of Tables Table 1: Hyper-parameter grid search values for k-NN, RF, SVM, MLP and CNN models ........58 Table 2: Hyper-parameter optimization and test results for classification of arbitrary mixtures using PCA feature representation .............................................................................................65 Table 3: Hyper-parameter optimization and test results for classification of arbitrary mixtures using LDA feature representation ..............................................................................................66 Table 4: Hyper-parameter optimization and test results for classification of arbitrary mixtures using SA feature representation ................................................................................................66 Table 5: Hyper-parameter optimization and test results for classification of arbitrary mixtures using the full feature set ............................................................................................................67 Table 6: Summary of the best classification results using each combination of model and representation methods ............................................................................................................67 Table 7: Hyper-parameter optimization and test results for regression of arbitrary mixtures using PCA feature representation .......................................................................................................67 Table 8: Hyper-parameter optimization and test results for regression of arbitrary mixtures using SA feature representation .........................................................................................................68 Table 9: Hyper-parameter optimization and test results for regression of arbitrary mixtures using the full feature set .....................................................................................................................68 Table 10: Hyper-parameter optimization and test results for classification of arbitrary mixtures using raw data representation ...................................................................................................71 Table 11: Hyper-parameter optimization and test results for regression of arbitrary mixtures using raw data representation ...................................................................................................72 Table 12: Test results for classification of simulated natural gas mixtures using raw data representation ...........................................................................................................................74 x Table 13: Test results for regression of simulated natural gas mixtures with 3% ethane using raw data representation ............................................................................................................75 Table 14: Test results for regression of simulated natural gas mixtures with 1% ethane using raw data representation ............................................................................................................76 Table 15: Results of outdoor natural gas leakage detection in London, Ontario ........................78 Table 16: Cross-validated test results for classification of THC in methanol using raw data representation ...........................................................................................................................80 xi List of Figures Figure 1: Example MOS sensor from Figaro [2] ......................................................................... 4 Figure 2: E-Nose comparison to human sense of smell [9] ........................................................ 5 Figure 3: Remote methane detection diagram [14] ..................................................................... 7 Figure 4: SensAbues breath sampling mouthpiece [32] ............. Error! Bookmark not defined. Figure 5: k-NN working principle [61] ........................................................................................25 Figure 6: How a random forest combines predictions from individual trees to make a final decision [66] ..............................................................................................................................28 Figure 7: Maximal-margin separation between classes [70] ......................................................30 Figure 8: MLP structure [73] ......................................................................................................32 Figure 9: Cross section view of the sensing apparatus ..............................................................37 Figure 10: Example time-series sensor response, showing exposure and recovery phases .....38 Figure 11: Automated sample generation apparatus .................................................................39 Figure 12: Voltage divider circuit for the MOS sensors [2] .........................................................40 Figure 13: CAD model of the sensing apparatus in the actuator housing .................................41 Figure 14: Actuator housing mounted onto sensing chamber ....................................................42 Figure 15: Magnetic plunger, shown from underside .................................................................43 Figure 16: Repeatability of automatic sample generation ..........................................................44 Figure 17: Simplified diagram of how the exposure and recovery chambers interact through the magnetic plunger ......................................................................................................................46 Figure 18: Natural gas detection device exterior .......................................................................47 Figure 19: External view of the exposure and recovery chambers.............................................48 Figure 20: CAD rendering of quad-sensor sensing apparatus for THC breathalyzer .................50 Figure 21: Sensor’s time response showing the differences between methane and ethane ......52 Figure 22: Average accuracy and variances of 30 random forests trained on the same data ....55 xii Figure 23: A) PCA-transformed train data. B) PCA-transformed test data. C) LDA-transformed train data. D) LDA-transformed test data ...................................................................................70 xiii Acknowledgements I would like to express my gratitude to my degree supervisors Dr. Homayoun Najjaran and Dr. Mina Hoorfar for providing guidance and support for my research and writing efforts. Without them I could not have even begun this work. I am grateful to the lab management staff members that have assisted in all aspects of this work, from grant writing to experiment design to writing, and even listening to my frustrated ramblings. Finally, I would like to thank all of the graduate and undergraduate students who have worked with me over the past years for their hard work, insights, stimulating conversations, and excellent company. xiv Dedication To my family and friends, who balanced encouraging words with just enough prodding to inspire me to finally finish this thesis. 1 Chapter 1: Introduction The human sense of smell consists of an interaction between chemical receptors in the olfactory system, and neural memory patterns. Chemical receptors react to different compounds that make up the odours that humans are able to sense. A particular combination of receptor reactions defines a “smell-print” for one given odour. This smell-print, generated by the olfactory system, is stored in the brain as a memory, allowing recall of familiar odours. The more times a certain odour is encountered, the stronger the memory will become. In the same way, it is possible to develop an artificial memory for odours detected using a variety of sensing technologies. Once this memory database is established, it can be used to train a predictor to classify new samples into one of the odour categories in its memory. There are many different applications for such a system, since odours can be indicative of condition in a multitude of industries including food and beverage, oil and gas, and air quality monitoring. 1.1 Research overview In this work, three different applications of odour learning and memory are examined, in the context of techniques for both effective odour sensing, as well as effective learning from the odour memory database. All three of these applications were based on the use of a sensing apparatus that consists of one or more commercial metal oxide semiconducting sensor(s) (MOS) embedded in a specially coated microchannel. This microchannel provides selectivity to the apparatus based on diffusion and physical adsorption rates of different analytes. The first application involves the discrimination and quantification of binary hydrocarbon mixtures. The quantification of a binary mixture with unknown concentrations would normally be impossible with the use of only a single non-selective sensor such as an MOS, but can be accomplished through the use of the microchannel as well as machine learning algorithms for 2 pattern detection. The hydrocarbons examined were methane and ethane in order to lead into the second application, which was detection of natural gas pipeline leaks. A difficulty with this type of work is false positives on pipeline leaks due to background sources of methane emissions. Part of the difficulty comes from the fact that the proportion of ethane in natural gas is small compared to methane, so the overall mixture is very similar to methane. Another element of difficulty comes from a requirement for portable sensing solutions usable for walking or fly-over surveys of pipelines. Finally, the feasibility of using this sensing apparatus for detection of THC in human breath is examined. With the legalization of marijuana in Canada, the issue of identifying impaired drivers has become more important than ever. Of the ways to test for THC use in humans, breath analysis is one of the least invasive, and is thus very attractive as a method for roadside or home testing. In this thesis, the feasibility of using the diffusion-based sensing apparatus to detect THC is tested and discussed. 1.2 Thesis outline The remainder of this thesis is structured as follows. Chapter 2 presents the relevant background information and a literature review for the topics discussed. In Chapter 3, the experimental apparatus including the sensing device, data generation setup, and prototypes for each of the applications are presented. Chapter 4 details the methodology used for processing and analyzing the data obtained from the sensing apparatus. Specific details on experimental methods and results are given in Chapters 5 and 6, respectively. Finally, conclusions are presented in Chapter 7. 3 Chapter 2: Background and Literature Review 2.1 MOS Sensors Metal oxide semiconductor (MOS) sensors rely on gas adsorption to perform sensing. Oxygen adsorbed on the surface of the sensor attracts electrons from the semiconducting material. A high electron density creates a large potential barrier at the grain boundaries, which inhibits electron flow. However, the reaction of this adsorbed oxygen in the presence of reducing gases lowers the potential barrier. The reduction manifests as an increase in the conductance of the MOS, meaning that the signal can be read using a simple voltage divider circuit. Some common metal oxides used in the fabrication of such sensors are tin oxide (SnO2), zinc oxide (ZnO), titanium oxide (TiO2), and tungsten oxide (WO3). Other factors that influence the response characteristics of the sensor are the sensing film thickness and the operating temperature [1]. Sensors of this type have the advantages of being low cost and having high sensitivity. However, selectivity is a shortcoming with this type of sensor, which usually makes it necessary to incorporate multiple sensors and pattern recognition algorithms to perform the actual predictions. An example of a gas sensor of this type is shown in Figure 1. 4 Figure 1: Example MOS sensor from Figaro [2] 2.2 Electronic Noses Machine olfaction is the term used to describe an attempt to mimic the human olfactory (smell) sense using sensors and computers. Decades of research has been conducted into new types of sensors and algorithms to perform this task. Devices that perform this odour recognition are commonly referred to as electronic noses (E-Noses) [3]. Since the smell sense is quite complex in its sensing and interaction with our memory, most E-Noses are trained only to recognize certain odours and perform classifications on a small variety of gases. Examples of typical tasks for E-Noses are classifying olive oils [4], spice odours [5], spoiled meat [6], and teas [7]. In fact, food-related monitoring tasks are among the most common for E-Noses [8]. A comparison of an E-Nose to the human sense of smell is given in Figure 2. 5 Figure 2: E-Nose comparison to human sense of smell [9] Odour discrimination is accomplished through the use of several different individual sensors, each being sensitive to certain compounds. An ideal E-Nose being used to classify n gases would be composed of n sensors, each sensitive to only one of the target gases. Of course, it is rare that this level of selectively can be achieved. In practice, all sensors will have some cross-sensitivity to other gases, so a great volume of sensor research is dedicated to improving selectivity. In addition, most odours involve complex mixtures, so attempting to detect and quantify every compound in the odour would be infeasible without tools like gas chromatography. To deal with cross-sensitivity, pattern recognition algorithms are common to use along with E-Noses. Discovering such a pattern from raw sensor data is an example of something that would be very difficult for a human to do, but possible to train a computer to do. Algorithms to 6 accomplish tasks of this kind are commonly referred to as “machine learning”, because the model which discriminates between the samples “learns” how to do so by looking at the available data. This is a huge benefit, because it eliminates the need for a human expert to attempt to codify their domain knowledge. 2.3 Natural Gas Detection Natural gas pipeline leak detection can be broadly categorized into “internal” and “external” methods, depending on whether the detector resides inside or outside the pipeline. Internal methods include acoustic measurement, pressure/flow monitoring, and statistical analysis. These methods often use mathematical modeling options to predict when a leak has occurred. External methods are typically hardware-based, relying on, for example, acoustic, optical, soil monitoring, or vapor sampling sensors. Hardware solutions can either be permanently installed in a fixed location or used in conjunction with a mobile monitoring apparatus, such as a handheld detector or even an unmanned aerial vehicle (UAV) [10]. Of particular interest to this study are methods that support inexpensive and portable sensing devices, which restricts the scope of candidate technologies to those classified as “external” methods. Acoustic instruments have been integrated into portable devices, pipeline pigs [10], and permanent installations affixed outside pipelines [11]. However, acoustic type sensors are vulnerable to background acoustic noise, which can overpower a leak signal. Similar to acoustic, optical methods offer portability and are flexible due to the number of available technologies including LIDAR, diode laser absorption, backscatter imaging, and thermal imaging. One of the greatest benefits of an optical system is the ability to sense leaks remotely [12], but many of the available technologies are quite expensive [10]. Recently, a device capable of being carried by a human operator and sensing leaks at a distance of 100ft 7 has been developed using Tunable Diode Laser Absorption Spectroscopy (TDLAS) [13]. This device has been mounted onto a quad-rotor UAV for landfill emission monitoring, as well as onto a Cessna 206 fixed-wing aircraft for detecting pipeline emissions [14]. The authors report excellent results in regressing the concentration of methane plumes from metered leak simulations and landfill emissions, but no mention is made of discriminating pipeline natural gas from background sources of methane. A diagram describing the principle of this type of detection is given in Figure 3. Figure 3: Remote methane detection diagram [14] Vapour sampling is another available method for portable detection which can be a smaller and less expensive alternative to acoustic or optical due to the advantages of the various types of sensing elements (metal oxide semiconductor, catalytic bead, electrochemical, etc.). This makes it especially attractive for applications using portable systems like UAVs [15]. A portable vapour-sampling device for discrimination between petroleum distillates has been developed using a fluorescence-based micro-sensor array. A Complementary-Metal-Oxide-Semiconductor 8 (CMOS) camera was used to capture the light output of the micro-sensor array, and the data was classified using a Support Vector Machine (SVM) [16]. An example of a stationary vapour sampling apparatus is the LEOS leak detection system [17], in which a tube permeable to hydrocarbons is installed alongside a pipeline. In the event of a leak, some of the fugitive gas will diffuse into the permeable tube. At periodic intervals, a pump circulates the air filling this tube, forcing it to flow past an array of semiconducting gas sensors. By measuring the time delay between the pump start and the array sensing, the leak can be localized within the tube. Regardless of the sensing method, discriminating between natural gas leaks and background methane (e.g., emissions from agriculture and landfills) is challenging. The composition of natural gas is highly variable at the point of extraction, with the methane proportion varying from ~30% to over 95%. Even after processing, typical standards only require a minimum methane mole fraction of 75%, with up to 10% ethane allowed [18]. The difference in the effects of such similar hydrocarbon gases on a sensor’s response may not be immediately clear, especially with non-selective sensors such as many of the inexpensive ones found in vapour sampling devices. In much of the described leak detection literature, this issue is not discussed, but several patents have addressed it using infrared absorption [19][20]. This approach has the potential for a compact solution to the selectivity problem, but it is difficult to extend the functionality. This is because infrared absorption is a narrow band technique, so the detection of new gases would require the addition of extra sensors that are selective to them. 2.4 Human Breath Analysis Analysis of human breath through laboratory experiments using gas chromatography and mass spectroscopy has allowed for the characterization of 3481 volatile organic compounds found in breath [21]. This has led to the detection of biomarkers indicative of conditions such as diabetes [22], lung cancer [23], and liver disease [24]. Another important area of study involves the 9 detection of impairment and illicit substance use via breath testing. For example, the use of handheld breathalysers for alcohol detection is commonplace, but these devices do not have the benefit of large, expensive equipment such as in spectroscopy experiments. Most alcohol breathalysers use a small fuel cell to detect ethanol in breath, which correlates to blood alcohol content (BAC), with actual blood testing being more accurate. With a trend toward legalization, another substance of particular interest for detection is THC from marijuana use. Quantification of THC to indicate usage in humans has been carried out through analysis of blood [25], urine [26], hair [27], saliva [28], and sweat [29]. The task of designing a portable breathalyser is currently an active research topic. A reliable breath analysis device for the purpose of roadside or personal detection of THC in breath would be incredibly useful as a less invasive (as compared with analysis of blood, urine, etc) indication of impairment. Several challenges exist when attempting to design a portable device for detection of THC in breath. First, the concentration of THC in breath, even directly following smoking, is quite low. Beck et al (2011) sampled exhaled breath from 10 regular marijuana users 1-12 hours after smoking, and 8 control subjects. Breath samples were passed through a 47mm Empore C18 disk for 10 minutes using a pump. They were able to detect THC in the breath samples of all 10 smokers using ultra performance liquid chromatography - tandem mass spectroscopy (UPLC-MS/MS) [30]. Himes et al. (2013) tested 13 chronic and 11 occasional smokers using a very similar analysis method to [30], and collected using the SensAbues breath sampling bag. A picture of this sampling system is shown in Figure 4. Among chronic users tested, 100% of samples provided 0.89 hours after smoking were positive for THC, but only 76.9% at 1.38 hours, and 53.8% at 2.38 hours (times after smoking are averages for all subjects). Among the occasional smokers, 90.9% of samples were positive for THC at 0.95 hours, 63.6% after 1.49 hours, and none for any times after that. In addition, one occasional smoker showed no THC in 10 breath at any time after smoking. This suggests that the detection window for THC in breath after smoking is short, and may in fact be even shorter for occasional smokers [31]. Beck et al (2013) tested for traces of 12 different drugs in breath, plasma, and urine in patients who were recovering from acute intoxication in a drug addiction emergency clinic. THC was detected in breath in 8/9 samples where THC was also detected in plasma. This finding is very interesting, because patients were sampled between 12 and 24 hours after intake into the clinic, so the THC-positive breath samples seem to conflict with studies that indicated a 1-2 hour detection window [32]. Coucke et al (2016) also used the SensAbues breath collection apparatus and observed the correlation of measured THC in breath with physiological indicators (pupil size and heart rate). Although they used the same collection and analysis methods as in [30], they found much higher amounts of THC per sample, in some cases multiple orders of magnitude. In addition, samples from 12/13 subjects were found to be positive for THC before smoking [33]. Figure 4: SensAbues breath sampling mouthpiece [32] 2.5 Machine Learning with Electronic Noses Humans unconsciously learn complex relationships about their surroundings every day. From birth, without any dedicated effort on our part, we learn how to walk, talk, run, identify and manipulate objects, identify smells, and perform innumerable other tasks simply by observing 11 others, or by trial and error. Humans excel at learning these types of tasks, and easily outperform even the most sophisticated computers on tasks such as object recognition and grasping. Of course, the advantage of teaching computers how to perform these same tasks lies in the ability to deploy them to do so automatically, for as long as needed, well past the physical limits of the human body. A good example is car manufacturing. It would be much simpler to teach humans to assemble the components of a car instead of using robots, but humans cannot work 24 hours a day, and require food, water, and sleep. On the other hand, there are also tasks that computers can do much more effectively than humans. One of these is the detection and identification of odours. A human might have moderate success in correctly identifying smells like spices, flowers, or perfumes, but in many cases the best that can be done is a faint familiarity. Also, in many interesting scenarios, the gas under consideration may not even be detectable by the human nose. Such is the case when working with many hydrocarbons, where the odour is undetectable by humans. All applications of machine learning involve defining a model, training the model to perform some task, and then evaluating its performance with some quality metric. The two most common tasks for a model to perform are classification and regression. In both cases, the model learns on data samples (the training dataset) for which the correct labels are known. This type of learning, where the labels of the training set are known, is called supervised learning. Let N be the number of samples in the training dataset, and let d be the number of features describing each sample, also called the dimensionality of the data. The model is then trained on an N x d matrix 𝑋𝑡𝑟𝑎𝑖𝑛, where each row contains a sample, and each column contains all observations of a single feature. Each sample is associated with a label 𝑦𝑛. In the case of classification, the label is 𝑦𝑛 ∈ {1,2,… , 𝐶} where C is the number of classes. In the regression case, the label is a real vector with length b, where b is the number of regression targets. An 12 example of a classification task might be to determine what object is depicted in an image, while a regression task might be to predict the price of a house based on features like location and size. 2.5.1 Learning from Time-Series Data Time series classification is a specialized subset of the classification task in which the feature vector for each sample is an ordered time series. A classification model is learned on time series with known labels, and new time series are labeled based on their similarity to those in the training set. This topic can be further broken down into three general categories: whole sequence classification, segment classification, and temporal classification. Whole sequence classification refers to the process of assigning a single label to a time-series. This may be the case when some data is generated from a sensor in response to a certain stimulus, and the stimulus needs to be identified. In cases where the overall sequence must be broken down into small sub-sequences, each requiring a label, but with the proper segmentation known in advance, segment classification is more appropriate. This situation arises frequently in written text or speech classification. The most general case is one where the segmentation of the input sequence is not known in advance, and the lengths of the sub-sequences cannot be assumed. This case is referred to as temporal classification [34]. By nature, e-nose classification tasks frequently fall into the category of whole sequence classification, since one data series is usually generated from a single vapour sample. Due to the added complexities of working with time-series as feature vectors, there is a vast volume of research dedicated to classification of time-series data. Topics in this area include time series pre-processing, representation, segmentation, similarity measures, clustering, and indexing to improve classification [35]. 13 2.5.2 E-nose Data Pre-processing Data pre-processing is an important step for dealing with noise, outliers, and signal correlations, as well as for ensuring that the learning algorithm converges quickly and accurately. For a time-series of MOS sensor data, most of the issues encountered involve sensor noise or baseline drift. Noise can be mitigated by proper attention to sources of electromagnetic interference and integrity of electrical connections, but baseline drift is an inherent drawback of using MOS sensors [36]. Since a MOS sensor is fundamentally a variable resistor, the physical quantity measured is the voltage drop across the MOS sensor, which is then used to calculate the resistance. It is this resistance value in response to clean air that drifts over time, resulting in possibly unreliable readings. This drift is usually compensated for with the use of differential, relative, or fractional compensation. It has been found that in the case of MOS sensors, the most effective compensation for baseline drift is to convert the sensor’s resistance value to a fractional change in conductance [37]. The fractional change in conductance is described by Eqn (1): 𝐺𝑓𝑟𝑎𝑐 =𝐺𝑔𝑎𝑠 − 𝐺𝑎𝑖𝑟𝐺𝑎𝑖𝑟 (1) where 𝐺𝑎𝑖𝑟 is the sensor’s conductance in clean air, 𝐺𝑔𝑎𝑠 is the sensor’s conductance in response to the target sample, and conductance is simply the inverse of resistance. With respect to pre-processing of the time-series as a whole, there are different methods depending on the end goal. In the case of classification, shape and structure of the time-series is typically more important than scale. For this reason, transforms that remove scale differences while preserving overall structure are used. One such transform is Z-normalization, in which each time-series is mean-subtracted and divided by the series’ standard deviation. In fact, it has 14 been shown in a study of 45 publicly available time-series datasets, application of the Z-normalization improved classification performance on every single one [38]. In cases where scale information is important, such as in concentration estimation for e-noses, another possible approach is to normalize the signals so that the entire dataset takes values between 0 and 1, with differences in individual magnitudes preserved. 2.5.3 Dimensionality Reduction Many machine learning models perform poorly with time-series data due to the high dimensionality of the input. This is a well-documented problem in machine learning, commonly referred to as the “curse of dimensionality” [39]. There are several different methods of reducing the dimensionality of data. Feature extraction and selection are highly domain-dependent, since different tasks will require different features. For example, image data and time-series data will have very different feature extraction methods. Transformation-based dimensionality reduction is another common method, which typically requires less domain-specific knowledge than feature extraction. 2.5.3.1 Feature Extraction Feature extraction is a common method of reducing the dimensionality of e-nose data. An ideal set of 𝑑 features would convey the same amount of information as the full time-series curve containing 𝑛 points, but with 𝑑 ≪ 𝑛. For example, consider the simple case of a perfectly first-order time-series response. No matter how many time points the series contains, the entire series could be compressed to just three numbers: the initial value, the steady-state value, and the time constant, without any loss of information. For image data, designing these features by hand would be very difficult, since correctly classifying an image depends on a huge number of subtleties, both local and global. It would be 15 almost impossible to match the success of modern image recognition software using only features that were hand-designed. The most common method of obtaining useful features is to train a machine learning model to extract them automatically. This is done using a model called a convolutional neural network. In this architecture, the model learns to represent the image through matching small filters to different sections of the input image. As the network depth increases (more hidden layers), the network is able to learn more complex features hierarchically, by building them out of simple features. Similar to image data, it is often difficult to hand-design features from time-series data because of the inherent complexity of the signals. The signals may have different lengths, be corrupted by noise, and be either periodic or non-periodic. This means that design of time-series features is highly domain-dependent, and features that work in one task may be useless for another. Attempts to design a general set of features that will work with most time-series data usually involve splitting the whole sequence into sub-sequences, and extracting a set of summary features from each of these. Some of the main challenges of this approach are the selection of sub-sequence length and summary feature design. Once extracted, the feature sets can be used to train a classifier, on either the features themselves or a representation such as a histogram. Deng et al. (2013) address the issue of interval length with their Time Series Forest (TSF) method by using randomly sampled lengths and starting points for √𝑀 intervals, where 𝑀 is the length of the whole sequence. From each of these intervals, the features extracted were the mean, standard deviation, and the slope of the least-squares-fitted line for that interval. A random forest was then trained on this feature dataset [40]. Baydogan et al. (2013) first divide each full time-series into sub-sequences, and then each sub-sequence into intervals. The motivation for this method comes from the insight that global characteristics may not always capture relevant local information at various smaller 16 time temporal scales than the full series length. Once each series was divided into sub-sequences, and each sub-sequence into intervals, the summary features were extracted from each interval. A new dataset was then constructed, where the instances were sub-sequences, and the attributes were the interval features. A random forest (RF) was trained on this new dataset, and the output class probability estimates (CPEs) for each sub-sequence were discretized using binning. For each full series, the discretized CPEs for sub-sequences in that series were bagged together to form a histogram of predictions for each class. These histograms were then concatenated to create a new feature vector for the full series. This forms yet another new dataset, on which a second RF classifier was trained. This was the classifier used to make the final predictions [41]. A noise-tolerant feature extraction and classification algorithm called Bag-of-SFA-Symbols (BOSS) was proposed by Schäfer [42]. This approach applied a sliding window to each series to generate sub-sequences. Each of these sub-sequences was converted into a string of symbols (a “word”) using Symbolic Fourier Approximation (SFA) [43], which provides robustness to noise in two ways: firstly by performing a Discrete Fourier Transform (DFT) which acts as a low-pass filter, and secondly by quantising the coefficients of the DFT into bins, which are then assigned symbols. This approximation allows real-valued sub-sequences to be compared by using string-matching algorithms on their SFA representations. Once the sub-sequences were transformed into SFA words, the time series could be described using an unordered histogram of these words. This has the effect of removing phase dependencies from the representation. Another approach for a method of extracting features from a time-series is to directly use the sub-sequences, and classify the time-series based on whether or not that sub-sequence appears in them. This is the idea behind shapelet features [44]. For a given training dataset consisting of time-series data, a brute force approach is used to determine which sub-sequence from within the entire dataset provides the greatest information gain when segregating the training dataset based on a distance metric 17 between each series and the shapelet. This process is repeated to create a decision tree structure where each node contains a shapelet that further partitions the data. This decision tree can then be used to predict the label of new data. In the field of e-nose detection, there are a few advantages involving the regularity of the signal. For example, many e-nose experiments involve distinct “exposure” and “recovery” phases, where the sensor is exposed to the target gas, and then allowed to recover to its baseline value by being placed in clean air. In these cases, there has also been a great deal of research into choosing good hand-designed features. Commonly used e-nose features can be broadly placed into one of three categories: geometric features, model parameters, and transform-based features. Geometric features express geometric properties of the time-series curve, such as peak values, slopes, and areas under the curve. To use model parameters as features, a dynamical model of the sensor’s response is constructed based on the observed data. Then, the model parameters that provided the best fit are used as features. For example, if the sensor’s responses were to be modelled as a simple first-order dynamical system, the features from each curve would be the gain and time constant (K and τ, respectively) of a first-order system model that best matched the actual data. Finally, transform-based features express properties of the signal that cannot be found in the time domain. Some of the most popular of these are the wavelet and the Fourier transform, which can extract either local or global frequency information, respectively [45]. 2.5.3.2 Feature Selection Because there are so many different possible features that can be extracted from most types of data, it is often necessary to reduce the dimensionality even further by discarding features which are not helpful in making good predictions. The process of reducing a feature set to a smaller, optimal subset is called “feature selection”. There are two broad categories of 18 techniques that can be used to select features: filters and wrappers. Both of these rank features based on their ability to predict the target variables, but wrappers do so by directly evaluating the performance of the prediction algorithm on that feature subset. In contrast, filters do not take the prediction algorithm into account, instead computing a subset quality metric based on statistical correlations between the features and the output variables. The advantage of wrappers is that they are usually able to give better overall prediction results, since the subset selection is implicitly tailored to the strengths of the chosen prediction model. The compromise is that wrapper methods are much more computationally expensive than filters, since a new model must be trained on every feature subset under consideration [46]. In both cases, because the target variables must be known for the quality to be computed, they are only applicable in supervised learning situations. Given a set of features that describe a dataset, a subset of features may be selected by computing either the quality of each individual feature, or the subset as a whole. Individual feature rankings fail to take feature interactions into account, which can sometimes be important depending on the domain [47], but they can be computed in linear time, since d qualities are calculated for d input features. Ranking all possible subsets of a given feature set, however, requires evaluating 2𝑑 − 1 possible subsets. The RELIEF algorithm was one of the successful early works in feature selection. It is fairly unique in the sense that it is an individual feature ranking algorithm, but is still able to indirectly account for feature interactions. It does this by iteratively updating the feature rankings based on how well they correctly discriminate between selected instances that are near each other in the feature space [48]. An individual ranking method that does not take feature interactions into account is the Fast Correlation-Based Filter (FCBF), which assigns a score to each feature based on symmetric uncertainty (SU), which is an information-theoretic measure of correlation that can handle non-linearity. SU is defined by the following equations: 19 𝑆𝑈(𝑋, 𝑌) = 2𝐼𝐺(𝑋/𝑌)𝐻(𝑋) + 𝐻(𝑌) (2) 𝐼𝐺(𝑋/𝑌) = 𝐻(𝑌) + 𝐻(𝑋) − 𝐻(𝑋, 𝑌) (3) 𝐻(𝑋) = −∑𝑃(𝑥𝑖)𝑙𝑜𝑔𝑃(𝑥𝑖)𝑛𝑖=1 (4) 𝐻(𝑋, 𝑌) = −∑∑𝑃(𝑥𝑖, 𝑦𝑗)𝑙𝑜𝑔𝑃(𝑥𝑖 , 𝑦𝑗)𝑛𝑗=1𝑚𝑖=1 (5) In the above relations, 𝐼𝐺(𝑋/𝑌) is the information gain of one random variable 𝑋 obtained from observing a second random variable 𝑌. It is calculated in terms of the entropies 𝐻 of each of the random variables, as well as their joint entropy 𝐻(𝑋, 𝑌). The goal in calculating the SU is to find variables that provide the maximum reduction in uncertainty of predictions made using those variables. This method is effective for removing redundant features even without computing subset qualities [49]. Similarly, the INTERACT algorithm also uses SU as a quality measure for features, but then performs an additional step that allows it to handle feature interactions. Once all of the features have been ranked by their SU scores, a consistency contribution (c-contribution) is calculated for each feature, beginning at the bottom of the ranked list. The c-contribution is a measure of how much the consistency of the dataset would be affected by removing that feature. Features with a c-contribution lower than a user-designed threshold are removed from the dataset, and the remaining features form the subset [50]. An example of a method that assigns a score to an entire feature subset is Correlation-based Feature Selection (CFS). In this method, a merit score 𝑀𝑠 for the subset is calculated in the following manner: 𝑀𝑠 =𝑘𝑟𝑐𝑓̅̅ ̅̅√𝑘 + 𝑘(𝑘 − 1)𝑟𝑓𝑓̅̅ ̅̅ (6) 20 in which 𝑘 is the number of features in the subset, 𝑟𝑐𝑓̅̅ ̅̅ is the average correlation between the features and the output variable(s), and 𝑟𝑓𝑓̅̅ ̅̅ is the average correlation between each of the features. This ranking method is biased towards subsets where the features are highly correlated with the outputs and uncorrelated with each other. The low feature-feature correlation bias helps to remove redundant features to provide the greatest reduction in dimensionality. [51]. This redundancy removal is one of the benefits of scoring a subset rather than each feature individually. The greatest trade-off, however, is the time required to search through all possible subsets. The exponential time cost makes a ranking of all possible subsets intractable for even a modest feature set size. This necessitates the use of a search method to find an acceptable solution. Different search methods use heuristics to sample the feature subset space in different ways. The simplest of these would be a sequential forward search (SFS) or sequential backward search (SBS). These are performed in a greedy manner, meaning that the algorithm only searches in the direction that maximally improves the objective function. Forward search begins with an empty subset, and checks the quality of every individual feature, and adds the one that gives the largest improvement. This process is then repeated, at each step adding whichever feature provides the most improvement. The algorithm terminates once no further improvement will be made by the addition of any more features. Backward search does the opposite, beginning with a feature subset containing every possible feature, and sequentially removing one at a time in a greedy manner. The advantage of the forward search method is that it is faster than backward search because it is faster to evaluate smaller subsets. However, backward search handles feature dependencies better. To clarify, some features work well as predictors only when paired with certain other features. In the case where an empty feature set is the starting point, it is impossible to predict which features will work well together. In addition 21 to the simple forward and backward search methods, there are also multiple similar search algorithms designed to address some of the shortcomings of SFS and BFS. One of these shortcomings is that SFS and SBS do not allow features to be replaced after removal or removed after adding. “Floating” methods such as sequential forward floating search (SFFS) and sequential backward floating search (SBFS) attempt to address this issue by incorporating extra steps for conditional feature removal (in SFFS) or addition (in SBFS) [52]. The brand and bound algorithm has been used to efficiently search the feature subset space by assuming monotonicity of the subset quality improvement. This means it is assumed that adding additional features to any subset will not decrease its quality. This method also requires the subset size to be specified in advance [53]. Sequential forward and backward search are greedy, deterministic algorithms, meaning that they will return the same solution if run multiple times on the same feature set, and can only search in directions where improvements are made. However, when a very large subset space is to be searched, stochastic algorithms can be very successful. Stochastic algorithms incorporate randomness into the search methodology, usually by sparsely sampling the subset space, and refining the search in areas that seem promising. Many methods of this type draw inspiration from biology, including genetic algorithms (GA) and particle swarm optimization (PSO). Genetic algorithms are based on the concept of incorporating random changes (called mutations) to “good” solutions, in the hopes of evolving it in a direction of improvement. These algorithms typically maintain a pool of candidate solutions, which are analogous to a genetically diverse biological population. Then, mimicking biological reproduction, elements of randomly selected members of the population can be exchanged with those of other members in a process called crossover. Crossover and mutations form the basis of most genetic algorithms, both of which allow the feature space to be searched randomly. An advantage of these methods is that they lend themselves naturally to feature selection, since a subset of a feature set 22 containing n features can be easily coded as a binary string of length n, with a 1 indicating that the feature is included, and a 0 meaning that the feature is excluded. The elements of these binary strings then undergo crossover and mutations [54]. PSO randomly generates a population of solutions that move across the feature subset space searching for the optimal solution. Each of these solutions has a “velocity” with which it traverses the solution space. In the general formulation, the velocity vector of each population member is influenced by its random initialization, by the current best solution found by that member, and by the current best solution found by all members. Each of these factors adds a pull to the flight direction, which are weighted and summed to give the overall direction. However, in the feature selection application, there are only two possible values in each of the n dimensions of the feature subset space – either the feature represented by each bit in the string is included, or it isn’t. This means that a velocity in each dimension cannot be properly defined, so the velocity of each member is represented only with an integer, representing the rate (in number of bits changed) at which that member approaches the current best known solution. The distance between two solutions is the number of bits that are different [55]. In contrast to the biologically-inspired algorithms, another very popular stochastic search method is simulated annealing (SA), which takes its inspiration from metallurgy. Similar to the gradual cooling of the annealing process, SA allows for highly random search of the subset space while the “temperature” is initially high. This means that the algorithm can search in directions where the solution quality is decreasing, enabling it to escape local maxima. As the running time progresses and the simulated temperature lowers, the probability of searching in directions of lower quality solutions decreases, and the algorithm becomes closer to greedy hill-climbing. The motivation is that the high-temperature portion of the search will bring the solution into the vicinity of the global optimum, and the greedier low temperature portion will arrive at the optimal solution. The subset space search is performed by generating neighbours of the current 23 state, and probabilistically deciding whether to transition to this neighbour state based on the qualities of the current and neighbouring states, and also the current simulated temperature. If the quality of the neighbouring state is better than the current state, the transition is always accepted. If it is lower, it will be accepted with probability: 𝑃 = 1 − 𝑒∆𝐸𝑘𝑇 (7) where ∆𝐸 is the change in energy from the current to the neighbouring state, 𝑇 is the current simulated temperature, and 𝑘 is a parameter that relates the temperature to the energy of a state. Analogously to metallurgical systems, lower energy states are preferred, so as ∆𝐸 increases the acceptance probability decreases exponentially. It can also be seen that a larger 𝑇 offsets the effect of a large ∆𝐸, and so increases probability of accepting a worse state [56]. This method requires a way of quickly assessing the quality of a feature subset and formulating an energy value in order to calculate the acceptance probability. This drawback is shared by many stochastic search methods since they typically perform a broader search of the subset space than greedy methods, and thus require a greater computational effort. 2.5.3.3 Dimensionality-Reducing Transforms Another method of reducing the dimensionality of data is to apply a transformation that does not involve extracting specified features. These methods have the advantage of requiring less domain knowledge about the data in question, since many of these transformations can be naively applied to datasets of almost any type. However, the ability to express the reduced-dimensionality data in terms of quantities that are meaningful to humans is lost. Many different transformations for reducing data dimensionality have been proposed in the literature, including Principal Components Analysis (PCA), Sammon mapping, autoencoders, 24 and Local Linear Embedding (LLE) [57]. One of the most popular of these is PCA, which has been applied often in the field of electronic noses [8]. PCA is a linear, unsupervised transformation that projects the data into a lower-dimensional space by finding a basis in which the variance in the data is maximized. The dimensionality of the reduced space can be chosen based on how much of the original information needs to be preserved. The fact that PCA is unsupervised means that the transformation it applies does not depend on the class labels of the data. In contrast, Linear Discriminant Analysis (LDA) uses the class labels to apply a transformation that maximizes inter-class variance, while minimizing intra-class variance. Once this transformation has been applied, the actual LDA method is to then apply a linear classifier to the reduced-dimensionality data, but in practice any classification algorithm could be used instead [58]. Like PCA, LDA has also found extensive use in the field of electronic nose research [8]. 2.5.4 Machine Learning Models There are many choices for which model to use for a given machine learning task, each with their own advantages and disadvantages related to training time, complexity, and sensitivity to outliers, noisy data, and size of dataset. There is no model which universally performs best for every type of task, so several types of models are usually validated before a solution is deployed. There are also some models which are not suitable for both classification and regression, so the type of learning task must be taken into account when selecting one. This review is limited to the most common types of classification algorithms used for E-nose data. 2.5.4.1 k-Nearest Neighbours A simple algorithm that has seen remarkable success in time-series classification is k-nearest neighbours (k-NN) [59]. For each sample in the test set to be classified, this classifier looks at 25 the 𝑘 nearest (as defined by some distance metric) samples in the training set, and assigns a label based on the labels of these 𝑘 neighbours through either weighted or un-weighted majority voting. There are a few options for the distance function used to find the nearest neighbours. Examples include the Euclidean distance, Mahalanobis distance, and Minkowski distance. In the case of time-series classification, it is also very common to allow the input sequences to be warped in time, making predictions more robust to temporal scaling. This Dynamic Time Warping (DTW) has been extensively studied in terms of ways to determine the optimal warping windows and speed up computations [60]. In the absence of DTW, the only hyper-parameter of the k-NN classifier is 𝑘. Figure 5 shows the working principle, where a new sample 𝑥𝑢 is compared to the nearest 𝑘 neighbours in order to make a class prediction. Figure 5: k-NN working principle [61] k-NN has been widely applied to E-nose data. Güney and Atasoy compared the performance of the k-NN and support vector machine (SVM) classifiers with an E-nose consisting of 12 MOS 26 sensors for the purpose of classifying different concentrations of n-butanol ranging from 5 to 20,000 ppm. Classification rates achieved were 87% for the k-NN and 86% for the SVM. These rates were improved to 93% for the k-NN and 96% for the SVM through the use of a decision tree to remove some of the less relevant sensor data [62]. Schiffman et al used a 15 MOS sensor E-nose to classify different types of bacteria, fungi, and volatile organic compounds related to them. The studied species were related to common airborne contaminants in buildings. Before classifying, the raw sensor data was first transformed using LDA to maximize class separability. The authors were able to obtain excellent classification results for both cultured bacteria and fungi samples, including up to a 96% classification rate for the five fungal species [63]. In the field of medical diagnosis, Blatt et al used a 6 sensor MOS E-nose to attempt to diagnose lung cancer. In their study, volunteers provided breath samples into a nalophan bag, and this sample was then introduced to their E-nose. The actual lung health of all of these volunteers was known in advance, so this was a feasibility study on the use of this E-nose for diagnosis. The classification rates were obtained using the leave-one-out cross validation strategy, and the results of the k-NN, two modifications of k-NN, linear discriminant analysis, quadratic discriminant analysis, and an artificial neural network were compared. The best result for discriminating between healthy and sick volunteers was 92.6%, but a student t-test indicated that there were no significant differences between the results of any of the six algorithms [64]. 2.5.4.2 Random Forest Random forest (RF) is a classification algorithm that uses ensembles of decision trees. Decision trees use recursive partitioning of the training data to build a classification model. At each non-leaf node of the tree, starting from the top, the training data is split based on the value of a single feature that provides the maximal information gain. The leaf nodes are labeled with 27 classes, such that if the test sample follows that particular branch of the decision tree all the way to the leaf, it will be assigned the class label at that leaf. Decision trees can also be applied to regression, by replacing information gain with standard deviation reduction as the splitting criteria. A more desirable split will have lower standard deviations of the target variable within each partition. Once a stopping criterion has been reached (standard deviation of all partitions is below a threshold, or further partitioning would not reduce standard deviation further), the leaf node is labeled with the mean of the target variable values for the subset of training samples at that node. A random forest is simply a collection of many decision trees ensembled through a method called “bagging”, which builds trees based on a randomly sampled subset of the training data. The purpose of this ensembling is that decision trees tend to overfit the training data, and ensembling allows for the averaging of many potentially overfit models, which reduces the overall variance of the classifier [65]. The voting process is shown in Figure 6. The hyper-parameters for the RF classifier pertain to both the ensemble as a whole, and to the individual trees included in it. 𝑛_𝑡𝑟𝑒𝑒𝑠 is the number of decision trees ensembled together, and 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ controls the maximum depth of each individual tree. The performance of a RF typically improves with a greater 𝑛_𝑡𝑟𝑒𝑒𝑠, with computational time as a trade-off. However, this improvement can be difficult to see with a simple hyper-parameter grid search, since the improvement is in terms of the classifier’s variance, which means that multiple random forests built from the same training dataset will give more similar predictions if more trees are included in each. Low variance translates to better generalization to test data, because it reduces the chance that one RF, via statistical anomaly, performs amazingly well on the validation data while still generalizing poorly. 28 Figure 6: How a random forest combines predictions from individual trees to make a final decision [66] Li, Gu, & Wang evaluated the performance of the RF classifier against a linear discriminant (LDA) classifier, multi-layer perceptron (MLP) and support vector machine (SVM) for the purpose of identifying Chinese liquor flavours. Their E-nose consisted of eight quartz crystal microbalance (QCM) sensors. Similar to the common procedure with MOS sensor, a flow controller was used to pass the liquor headspace over the sensor array. Compounds released from the liquors adsorb to the QCB coating layer and produce a change in resonant frequency of the sensors. Following this, the array is cleaned by passing a carbon absorbent through the headspace, which removes the liquor compounds. By using a RF classifier, the authors were able to achieve 100% classification accuracy for twelve types of liquors, but were unable to do so with either the LDA, MLP, or SVM models [67]. Pardo & Sberveglieri used pre-existing datasets to compare the performances of RF, SVM, and nearest shrunken centroids (NSC) classifiers. The two datasets were for fungus classification and olive oil defect detection. In all 29 cases, optimal hyper-parameters were selected using stratified cross validation. Their findings indicated that the performances of the RF and SVM classifiers were comparable on both datasets, with the NSC performing significantly less well [68]. 2.5.4.3 Support Vector Machine A support vector machine (SVM) is a binary classification model that is based on the goal of achieving the maximum margin of separation between classes. This principle is shown in Figure 7. They can construct a very complex, non-linear classification surface through the use of a kernel function, which maps inputs 𝑥 into a higher-dimensional space 𝜑(𝑥). A linear decision boundary is constructed in the high-dimensional space. Then, once the inputs are mapped back from 𝜑(𝑥) to 𝑥, the decision boundary becomes non-linear. A common choice for the kernel function is the radial basis function (RBF) [69]. The solution to construct the decision surface is deterministic, but depends on a few hyper-parameters that control the shape. C is the cost trade-off parameter that controls how strongly incorrect classifications are penalized in the training stage. Higher values of C can result in excellent fitting on the training data, but can also cause poor generalization (test set performance). A parameter that is unique to the RBF kernel is 𝛾, which controls the extent to which each training sample controls the shape of the decision surface. SVM’s are not inherently multi-class compatible, so when discriminating between three or more classes, a one-against-one or one-against-all training procedure is used. Similarly, if multi-target regression is desired, multiple single-target regression SVM’s are trained. 30 Figure 7: Maximal-margin separation between classes [70] SVM’s have found several uses in conjunction with E-nose systems, especially in the food quality testing field. El Barbri et al. examined samples of beef and sheep meat and attempted to use an E-nose to detect spoilage. Their E-nose consisted of six MOS sensors and a flow control system that used nitrogen to pass the meat headspace over the sensor array. Measurements from the sensors were taken every second for 50 minutes, and temporal features were then extracted from the time-series responses. These features were used as inputs to the SVM classifier, with true labellings (spoiled or unspoiled) obtained using bacteriological analysis. The authors obtained excellent results in identifying spoilage, with the SVM giving accuracies of 98.81% and 96.43% for the beef and sheep meats, respectively [71]. Also in the area of meat spoilage detection, Papadopoulou et al. conducted a similar study on beef fillet samples, 31 examining spoilage rate compared to storage temperature. However, in their experiments, the quality of the meat was determined by a panel of taste testers who categorized the samples as “fresh”, “semi-fresh”, or “spoiled”, and the authors tested the SVM classifier’s ability to correctly guess these labels. Their E-nose was an array of eight quartz crystal microbalance (QCB) sensors known as the LibraNose, and the SVM classifier was able to determine the correct labels with accuracies ranging from 89.3% to 92.8%, depending on which replicates of the data were used for training and testing. They also examined the performance of the SVM as used for regression, to try and predict the population numbers of select microbial groups. The SVM was also well suited to this task, with average prediction errors ranging from 14.4% to 16.3% depending on the type of bacteria examined [72]. 2.5.4.4 Multi-Layer Perceptron A multilayer perceptron (MLP) is a type of feed-forward neural network consisting of at least three layers, all but the first of which apply a nonlinear transform (activation) to a weighted sum of the previous layer’s activations. This nonlinear transform function is usually either the logistic sigmoid or the hyperbolic tangent function. The first layer is called the input layer or visible layer, and it is where the features are input. The last layer is called the output layer, and it is where the network’s predictions appear. Any layers in between these two are called hidden layers. This structure is depicted in Figure 8. Since a nonlinear transform is applied at every layer, the addition of more hidden layers means that the network can learn more complex functions, with greater non-linearity. More complex functions, however, are not always better. With more complexity there is more possibility to overfit the training data. To offset this tendency, regularization of the network weights is a common strategy. This adds a penalty to the network’s cost function based on large values for individual weights. These penalties are usually 32 proportional to the L1 or L2 norm of each layer’s weight vector. The strength of this penalty is controlled with the regularization coefficient 𝜆. Figure 8: MLP structure [73] An MLP can be used either as a classification or a regression model, depending on the type of output neuron used. Typically, for classification, the output layer contains one neuron for each possible class, and the activation function is the softmax function, which outputs a probability distribution over the classes (the sum of the activations of the output neurons is one). The predicted class is the one for which the activation of its corresponding output neuron is greatest. Sundgren et al. used two different e-noses, each with 6 different sensors, to predict concentrations in either two- or four-component mixtures. They used 3 Pd and 3 Pt gate type sensors for mixtures of hydrogen, ammonia, ethylene, and ethanol. An array of 2 Pd, 2 Pt, and 2 Ir gate types was used for mixtures of hydrogen and acetone. In their study, the concentrations 33 of hydrogen and ammonia in the four-component mixture were predicted well, but those of ethylene and ethanol were not. In the two-component mixture, success was shown in predicting concentrations of both hydrogen and acetone [74]. Llobet et al. used four tin oxide sensors to discriminate between ethanol, toluene, and 0-xylene. They trained one multilayer perceptron (MLP) to classify the target into one of the three classes, and a separate MLP to then predict the concentration as one of three pre-determined amounts. This method gave a 100% classification rate, and a 95% rate of prediction of the correct concentration [75]. Zhang et al. were able to obtain excellent results in predicting the concentrations of both ammonia and formaldehyde in mixtures at room temperature. This was done using an array of three graphene-based sensors decorated with different metal oxides. The concentrations were estimated using a multi-layer perception with inputs being the steady-state resistances of each sensor [76]. Ali et al. used the responses of six Piezoelectric Quartz sensors as inputs to a Radial-Basis Function Neural Network (RBFNN) to perform predictions about quality of various vegetable oils. The data was first visualized using Principal Component Analysis (PCA) and it was apparent that natural clusters formed between classes. This explains why the RBFNN was able to obtain a 99.13% correct classification rate [4]. 2.5.4.5 Convolutional Neural Network A Convolutional Neural Network (CNN) shares the general structure of an MLP, in that it contains an input layer, output layer, and some number of intermediary hidden layers that perform non-linear operations on the activations of the previous layer. The main difference is that instead of simply using a weighted sum of the activations of every neuron in the previous layer, the CNN learns a set of filters that are passed over (convolved with) the data, and produce an activation map of how strongly those filters are activated in each local region of the input. This type of network is very popular in image processing such as face detection [77], 34 using 2-dimensional convolutions. However, since the data in this work is time-series, the filters will be 1-dimensional. As mentioned, these filters are learnable, so the model will automatically adjust them to improve the network’s predictions. This is accomplished through backpropagation, just as in the MLP. CNN’s are the building blocks of many modern deep learning models due to their ability to learn features automatically. 2.5.5 Hyper-Parameter Selection An important step in machine learning is to select the optimal hyper-parameters for the model being used. Hyper-parameters are non-trainable parameters of the model. The process of finding good values for these parameters is highly application-dependent, and there are no values that are universally better than others. A simple method of hyper-parameter selection is a grid search, in which every possible combination from a specified list of values is tried for each hyper-parameter. For example, if three values each for three different hyper-parameters were tried, the grid search would involve training 3x3x3=27 models and evaluating which one performed the best. The quality of each set of hyper-parameters is quantified by the performance of the model trained using those parameters, evaluated a set of data not used for training, called the validation dataset. After training, the performance of the model is usually evaluated by looking at prediction accuracy on a new set of data that was not used for training or hyper-parameter testing. This is to avoid having the model simply memorize the training data while generalizing poorly, and data used for this purpose is called the test set. Commonly the test set is simply a random sample of the overall dataset. In cases where the amount of data is small, cross-validation is an alternate method of evaluating performance. For this method, the entire dataset is split into C partitions. Each partition is used as the test set once, with the other 𝐶 − 1 partitions used as the training 35 set. In total, C models are trained, and the overall performance is estimated as the average performance of these models. 36 Chapter 3: Experimental and Sensing Apparatus In this chapter, details of several prototype gas detection devices are given, as well as details of the sensing apparatus that forms the basis for all of them. 3.1 Sensing Apparatus The sensing apparatus used in this work is based on previous work in our lab to design a compact, flexible VOC sensor [78]. It consists of a Figaro MOS sensor embedded at the end of a 3D-printed micro-channel. Any target gas that is introduced to the inlet must diffuse through the micro-channel before reaching the MOS sensor. This gives the apparatus selectivity based on the diffusion rates of different gases, because lighter molecules will appear on shorter time scales than heavier ones. To further increase the selectivity, the micro-channel is coated with parylene-C to encourage physical adsorption. Its small size enables it to be used for both stationary and mobile applications, which is extremely beneficial in the context of both natural gas pipeline leak detection and human breath analysis, where both types of devices may be needed, depending on the individual situation. A CAD rendering of the sensing apparatus is shown in Figure 9, with a cross-section to show the micro-channel. The rod-mount holes are to allow the entire apparatus to slide when moved by a linear actuator. This mechanism is described further in Section 3.2. This apparatus relies heavily on reliable gas diffusion through the micro-channel. This has the disadvantage that any induced flow will make the results non-comparable. However, relying on diffusion from stationary gas samples means that greater advantage can be taken of the transient response of the sensor, which is much richer in information than only using the steady-state response. In addition, it makes the apparatus simple to use in field applications, where a sample can simply be collected via a pump, and held stationary for analysis. It is also quite 37 flexible, and target gases can be altered simply by changing which MOS sensors are used. This is made easier by the fact that Figaro manufactures sensors with similar dimensions, but for a variety of target gases. Of course, the apparatus can also be adapted to fit custom in-house designed sensors as well. Figure 9: Cross section view of the sensing apparatus Every test conducted with this sensing apparatus consists of baseline, exposure, and recovery phases. The baseline phase is the first ten seconds of each test, during which the sensor is left in clean air, and the baseline value recorded. During the exposure phase, the target gas is introduced to the inlet of the sensing apparatus, and begins to diffuse toward the sensing element. Finally, during the recovery phases, the sensor apparatus is placed in clean air, and the target gas diffuses back out of the micro-channel. A typical time-series response is given in Figure 10, showing the exposure and recovery phases. Microchannel Sample inlet Figaro MOS sensor Rod-mount holes 38 Figure 10: Example time-series sensor response, showing exposure and recovery phases 3.2 Automatic Sample Generation Machine learning models typically require a fairly large dataset to effectively learn from. For this reason, an automated sample generation and data collection apparatus was designed. This apparatus uses mass flow controllers to control the dosing of individual target gases into a mixing chamber. A diagram of this entire apparatus is given in Figure 11. Before each test, valves 1 and 2 are opened for five minutes, so that the mixing chamber is thoroughly flushed with clean air. After flushing is complete, valves 1 and 2 close, and valves 3, 4, 5, and 6 open, allowing methane and ethane to flow from the cylinders through the mass flow controllers and into the sensing chamber at a pre-determined rate. The time that the valves are left open for depends on the known flow rate and desired concentration of the final mixture. Valves 7 and 8 are used to vent unwanted pressure from the lines immediately before the dosing step begins. 39 Figure 11: Automated sample generation apparatus Each of these tests consists of a 10 second baseline phase, 40 second exposure phase, and a 150 second recovery phase. Each test therefore lasts 200 seconds, with the sensor’s response sampled once every 0.1 seconds. The data collected from the sensor is actually the voltage taken across a load resistor in a voltage divider circuit. Exposure with a target gas changes the resistance of the MOS, which can be calculated using the following formula: 𝑅𝑆 = (𝑉𝐶𝑉𝑅𝐿− 1) ∗ 𝑅𝐿 (8) where 𝑅𝑆 is the resistance of the MOS, 𝑅𝐿 is the resistance of the load resistor, 𝑉𝐶 is the applied voltage, and 𝑉𝑅𝐿 is the measured voltage across the load resistor. This circuit is shown visually in Figure 12. 1 7 2 3 4 5 6 8 40 Figure 12: Voltage divider circuit for the MOS sensors [2] The exposure of the sensor to the target mixture is accomplished using a linear actuator. Figure 13 shows a CAD rendering of the actuator housing connected to the sensing apparatus. A picture of the sensing apparatus and actuator housing mounted onto the sensing chamber is also shown in Figure 14, and a picture from the underside showing the magnetic plunger is given in Figure 15. At 𝑡 = 10⁡ seconds the actuator extends, and the sensor apparatus inlet depresses the magnetic plunger, thus allowing gas to diffuse into the micro-channel, and toward the sensor. At 𝑡 = 50 seconds, the actuator retracts, the plunger seals the exposure chamber, and the gas inside the microchannel diffuses back out into clean air. The process is then repeated in a completely automated way. The flange mounts onto the sensing chamber, which is where the gas samples are mixed, and the rods allow the sensing apparatus to slide smoothly between exposure and recovery. In order to prevent ambient airflow from influencing the experimental results, the entire actuator housing (Figure 13) is enclosed in a small, sealed chamber, referred to as the “recovery chamber” because it contains only clean air. 41 A test of the repeatability of the samples generated by this apparatus is presented in Figure 16. For each concentration increment in {50, 100, … 250} ppm of methane, three repeats were performed and plotted. For each repeat, the sensing chamber was completely flushed and re-filled to the desired target. This figure shows that the desired concentrations can be replicated with a high degree of repeatability. Figure 13: CAD model of the sensing apparatus in the actuator housing Magnetic plunger Figaro 2610 MOS sensor Linear actuator Rods Micro-channel inlet Flange Actuator housing 42 Figure 14: Actuator housing mounted onto sensing chamber 43 Figure 15: Magnetic plunger, shown from underside 44 Figure 16: Repeatability of automatic sample generation 3.3 Natural Gas Detector A compact, portable sensing apparatus was designed for field use in detecting natural gas leaks. The functionality of the device includes automated sample intake, sensor exposure, and data collection/storage. First, a small pump draws a sample of the surrounding air into the exposure chamber. Check valves at the inlet and outlet to the exposure chamber ensure that there is no air flow unless the pump is running. The gas in the exposure chamber is prevented from entering the recovery chamber by the gasket on the magnetic plunger. Once the exposure chamber is completely filled with the sample, the linear actuator pushes the inlet of the sensing apparatus against the plunger. This motion displaces the plunger, and gas from the exposure chamber is allowed to flow around the plunger, and into the sensing apparatus inlet. After the exposure time has passed, the linear actuator retracts, moving the sensor back into the clean air of the recovery chamber. Figure 17 shows a simplified diagram of the design of the exposure and recovery 50ppm 250ppm 45 chambers, as well as the magnetic plunger and sample intake system. The linear actuator and the rest of the sensing apparatus are not pictured. The sensing apparatus and plunger are designed so that no gas should flow around the sides of the sensor inlet into the recovery chamber while the sensor is exposed to the target gas. However, in case of imperfect sealing between the two chambers, two ports were added to the recovery chamber that allow it to be purged with clean air in between each test. Check valves were also added to these ports, to ensure that the air in the chamber was stationary unless it was being purged with clean air. A solenoid valve switches the outlet of the pump between the exposure and recovery chambers. When the pump is connected to the recovery chamber, the purpose is cleaning, so the pump inlet should be placed in fresh air. In this way the intake pump was dual-purposed for sample collection and chamber cleaning. The entire detection device, including sensing apparatus, actuator housing, recovery and exposure chamber, electronics, valves, and micro-controller are enclosed in a portable casing, and controlled through a touch screen. An exterior view of the entire device is given in Figure 18. 46 Figure 17: Simplified diagram of how the exposure and recovery chambers interact through the magnetic plunger 47 Figure 18: Natural gas detection device exterior The exposure and recovery chambers, which sit inside the external case, are made from a combination of PVC piping and 3D-printed components. The recovery chamber is quite a bit larger in size than the exposure chamber since it needs to surround the entire actuator housing, so for cost reasons it was made from inexpensive PVC pieces. The exposure chamber was 3D-printed, and designed to bolt onto the bottom of the PVC pieces. The wiring to the sensing apparatus and linear actuator enter the recoery chamber via a strain relief connector, which is air-tight. A picture of the recovery/exposure chamber assembly outside the external case is given in Figure 19. Touch screen Outer case 48 Figure 19: External view of the exposure and recovery chambers 3.4 Breath THC Detector The requirements for a portable marijuana breathalyser prototype were much the same as for the natural gas detector, so most of the hardware design was re-used. For example, the breathalyser prototype incorporated the same exterior device case, the same exposure and recovery chambers, and the same actuator housing. A small pump was still used for the purpose of flushing the exposure and recovery chambers with clean air, but the pump is no longer needed for sample intake, since the subject actively blows into the exposure chamber. The largest difference lies in the sensing apparatus. Since human breath is a much more Recovery chamber Recovery chamber flushing inlet/outlet Exposure chamber Exposure chamber inlet/outlet Sensor and actuator wiring 49 complex mixture than natural gas, a single MOS sensor was not likely to be able to adequately detect the pattern associated with individual compounds such as THC. For this reason, the sensing apparatus described in Section 3.1 was adapted for use with multiple different MOS sensors. Each of these sensors was selected to have sensitivity to a different possible compound in breath, with the goal that the combination of all of them might reveal a pattern unique to THC. It is also important to include sensors that can detect compounds that can detect other compounds common in breath so that the effect of THC can be better isolated. Four Figaro MOS sensors were chosen for this purpose: • TGS2600 – sensitivity to hydrogen and carbon monoxide • TGS2602 – sensitive to volatile organic compounds in breath with low sensitivity to alcohol • TGS2610 – sensitive to alcohol • TGS861 – sensitive to acetone A CAD rendering of the sensing apparatus adapted to use these four sensors is shown in Figure 20. 50 Figure 20: CAD rendering of quad-sensor sensing apparatus for THC breathalyzer 51 Chapter 4: Analysis Methodology In this section, the process of organizing, pre-processing, and representing the time-series data from the sensing apparatus is discussed, as well as the methodology for evaluating different classification and regression models. 4.1 Data Pre-processing A different pre-processing method is used based on the objective of the experiment. For those involving classification, the shape of the response curve is more important than amplitude, so Z-normalization is used. For experiments where the goal is concentration regression, scale-dependent information must be preserved, so the datasets are scaled into the [0,1] interval. 4.2 Data Representation and Feature Extraction Methods of representation of time-series data can have an important impact on the performance of machine learning models. In this section, the different representation methods compared in this work are outlined. 4.2.1 Feature Extraction Since it is difficult to know a priori which features will be the most salient, the approach of extracting a wide variety of features and using feature selection to filter them afterward is taken. For this reason, both geometric and transform-based features are extracted, in an attempt to preserve a diverse amount of information. In order to determine which geometric features discriminate well between methane and natural gas, the comparison of the sensor’s response to methane and ethane is visually examined (Figure 21). Each sample is at a concentration of 1000ppm. The significant difference in the slope of the response, especially during the recovery 52 phase, suggests that temporal features such as the time at which the signal reaches 50% of its peak value (during both exposure and recovery phases) might be useful. Along with good discrimination between gases, it is also important for the magnitudes of predicted concentrations to be accurate. For this reason, features such as the peak value and the area under the curve will be useful because these features relate directly to the magnitude of the target gas concentration. In total, visual inspection combined with pre-existing literature on E-nose feature extraction yielded geometric features such as the area under the curve, peak value, slopes, first and second derivatives, normalized signal values, and times for the signal to reach a certain percentage of the peak value. A full description of all 120 features is provided in Appendix A. Figure 21: Sensor’s time response showing the differences between methane and ethane The wavelet features were generated by using a Daubechies mother wavelet (db6), decomposing each time series signal to the maximum level, and taking the 22 approximation coefficients as features. The approximation coefficients were used because the sensor 53 response signal is at low frequencies, so the approximation coefficients were the greatest. Wavelet decomposition was accomplished using the PyWavelets Python package [79][80]. In order to obtain the Fourier domain features, a Fast Fourier Transform (FFT) was applied to the data, and the values were squared to obtain an estimate of the Power Spectral Density (PSD). As expected, the vast majority of the power in the time-series signals existed in the low-frequency components. For this reason, the lowest eight PSD components were chosen to be used as features. 4.2.2 Feature Selection In this work, simulated annealing (SA) is used as the feature subset search method, and symmetric uncertainty as the subset quality metric. Symmetric uncertainty is an information-theoretic measure of correlation that can handle non-linearity. This approach allows for the ranking of feature subsets based on the correlations between the features contained in the subset, and the target output(s). This metric is used, for example, in the Fast Correlation-Based Filter [81]. Symmetric uncertainty requires discrete features and target variables, so before applying it, the features are first discretized using the method of Fayyad and Irani [82], which decides how to partition features based on information gain. For the experiments involving multi-target regression, the symmetric uncertainty of each feature was calculated as the average symmetric uncertainty between that feature and each of the regression targets. 4.2.3 Dimensionality-reducing Transformations In addition to using simulated annealing to reduce the size of the feature set, two different dimension reduction transformations were tested as alternatives: PCA and LDA. For the PCA 54 transform, the number of components selected was such that the explained variance was >99%, whereas the maximum number of components was chosen for the LDA transformation, since this is only two components for a three class discrimination problem. 4.3 Hyper-parameter Selection The hyper-parameters for each model are described in the following subsections; the grid search values for each parameter are provided in Table 1. 4.3.1 k-nearest Neighbours The only hyper-parameter of the k-NN classifier is 𝑘, the number of nearest neighbours used for classifying new samples. In these experiments, 𝑘 values of 1, 3, and 5 are used for both classification and regression tests. 4.3.2 Random Forest The hyper-parameters for the RF classifier pertain to both the ensemble as a whole, and to the individual trees included in it. 𝑛_𝑡𝑟𝑒𝑒𝑠 is the number of decision trees ensembled together, and 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ controls the maximum depth of individual trees. The performance of RF typically improves with a greater 𝑛_𝑡𝑟𝑒𝑒𝑠, with computational time as a trade-off. However, this improvement can be difficult to quantify with a simple hyper-parameter grid search, since the improvement comes in the form of reducing the classifier’s variance. This means that multiple random forests built from the same training dataset will provide more similar predictions if more trees are included in each. This ultimately reduces overfitting and improves generalization to unknown data. Because the generalization stability is so important, the variances of RFs constructed with 𝑛_𝑡𝑟𝑒𝑒𝑠 from 25 to 800 were examined, and the point at which the variance reaches a stable minimum was identified. To do this, 30 random forests were constructed for 55 each 𝑛_𝑡𝑟𝑒𝑒𝑠 in the set of {25, 50, 100, 200, 400, 800} using the training dataset without extracting features, and with 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ = 3. The 30 RFs for each 𝑛_𝑡𝑟𝑒𝑒𝑠 were then evaluated on the validation dataset, and the average and standard deviation of their performances were recorded (Figure 22). As shown in this figure, increasing the number of decision trees in each forest clearly causes a reduction in the variance, which eventually reaches 0 at 𝑛𝑡𝑟𝑒𝑒𝑠 = 800. As explained, the reduction in variance increases computation time; the average time to construct each random forest increased from 44ms with 25 trees, to 1184ms with 800 trees. Despite the significant increase, the time to build each classifier is modest due to the small dataset size (120 training dataset samples, discussed in Section 3.4.1), so 𝑛𝑡𝑟𝑒𝑒𝑠 is set to 800 in all following experiments. Figure 22: Average accuracy and variances of 30 random forests trained on the same data 56 4.3.3 Support Vector Machine As discussed in Section 2.5.4.3, C and 𝛾 are the hyper-parameters for the SVM. Compared to the classifiers that incorporate random elements (MLP, RF, CNN), SVMs are easier for finding the optimal hyper-parameters, as they provide a deterministic solution. As with the RF, the validation set was used to determine the best hyper-parameters before evaluating performance on the test set, and the optimization was repeated for each experiment. When discriminating between three or more classes, the one-against-one strategy was used, with the final classification performed through majority voting. Similarly, if multi-target regression is required, multiple single-target regression SVM’s are trained. 4.3.4 Multi-layer Perceptron In the case of MLP, the number and width of the hidden layers (𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 and 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ, respectively), the learning rate (𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒), the L2 regularization coefficient (𝜆), and the number of training epochs (𝑛_𝑒𝑝𝑜𝑐ℎ𝑠) are the hyper-parameters. The optimal number of training epochs is determined by evaluating the performance of the network on the validation set after each epoch, and examining when the validation performance degrades. An increasing validation error while the training error continues to drop is an indication of over-fitting. The ideal hyper-parameters are chosen based upon which achieve the lowest validation loss at any point during training, and the best 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 was the number of epochs after which this minimum loss was achieved. The maximum number of training epochs was selected to be 2000 to constrain training to a reasonable timeframe. For classification experiments, the number of nodes in the output layer is equal to the number of classes, and the activation used is a softmax with cross-entropy loss function. For regression 57 experiments, the output layer has two nodes (one for each regression target), and uses a mean-squared loss. 4.3.5 Convolutional Neural Network The structure of a CNN requires more hyper-parameters than a MLP. The CNN-specific hyper-parameters are the number of convolutional layers (𝑛_𝑐𝑜𝑛𝑣_𝑙𝑎𝑦𝑒𝑟𝑠), filter length (𝑓𝑖𝑙𝑡𝑒𝑟_𝑙𝑒𝑛𝑔𝑡ℎ), and number of filters (𝑛_𝑓𝑖𝑙𝑡𝑒𝑟𝑠). In addition to the convolutional layers, networks of this type commonly have fully-connected layers after them. This adds new hyper-parameters for the number and width of these layers. The number of parameters was reduced by making a few assumptions about an effective structure for the overall network, taken partly from AlexNet (i.e., a successful image classification CNN in 2012 [83]). A max pooling operator was used after each convolutional layer, and two fully-connected layers were stacked after the final convolutional layer. The sizes of the filters were also reduced for subsequent convolutional layers. For convolutional layers after the first, the filter size was halved, but the number of filters was kept constant. In addition, 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒, 𝜆, and 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 were used as in the MLP model. A coarser search was performed over values of 𝜆 (to compensate for the additional computation time) to search over the extra parameter from MLP. The procedure for finding the optimal 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠, as well as the training loss functions used, were both the same as for the MLP 58 Table 1: Hyper-parameter grid search values for k-NN, RF, SVM, MLP and CNN models Model Hyper-parameters Grid search values k-NN 𝐾 1, 3, 5 RF 𝑛_𝑡𝑟𝑒𝑒𝑠 25, 50, 100, 200, 400, 800 max⁡ _𝑑𝑒𝑝𝑡ℎ 3, 5, 7, 9, 11 SVM 𝐶 10-1, 100,..,105 𝛾 10-6, 10-5,..,101 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 1, 2 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ 8, 16, 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 0.001, 0.01 𝜆 10-4, 10-2, 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 Max 2000 CNN 𝑛_𝑐𝑜𝑛𝑣_𝑙𝑎𝑦𝑒𝑟𝑠 1, 2 𝑓𝑖𝑙𝑡𝑒𝑟_𝑙𝑒𝑛𝑔𝑡ℎ 32, 64, 96 𝑛_𝑓𝑖𝑙𝑡𝑒𝑟𝑠 8, 16, 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 0.001, 0.01 𝜆 10-4, 10-3, 10-2, 10-1, 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 Max 2000 59 Chapter 5: Experimental Methodology In order to prove the versatility of the sensing apparatus and procedure described above, a range of experiments are performed, including discrimination between individual gases, quantification of single gases, discrimination between different mixtures, quantification of binary mixtures, detection and quantification of natural gas, and detection of THC denatured in methanol. In these experiments, the performances of the k-Nearest Neighbours (k-NN), Random Forest (RF), Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), and are quantified using the Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), feature-based, and raw data representations. For the regression task, the same models are used. For each representation/model combination, the hyper-parameter optimization is repeated on the validation dataset. The method of hyper-parameter optimization used here is a grid search, where each combination of hyper-parameters is used to build a model, and the values giving the best results on the validation dataset are used to train one final new model, whose performance is evaluated on the test dataset. In order to evaluate the performance of each model on the defined tasks, a set of quality metrics must be chosen. The quality metric used to quantify classifier performance (Classification Accuracy (CA)) is the fraction of correct predictions (𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡 ) by the number of total predictions (𝑛𝑡𝑜𝑡𝑎𝑙): 𝐶𝐴 = ⁡𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑛𝑡𝑜𝑡𝑎𝑙⁡⁡ (9) Determining a suitable test quality metric for the regression experiments is slightly more nuanced than for classification. The most typical choice is the use of the Mean-Squared Error 60 (MSE), as in the training phase. However, Mean Absolute Error (MAE) is often preferable since the units are the same as the units of gas concentration (e.g., ppm). The regression errors can also be expressed as a percentage of the target concentration. For this reason, the Mean Absolute Percentage Error (MAPE) was used alongside the MAE. However, there is a complication with the use of MAPE as a quality metric, since some of the prediction targets are zero. To compensate for this and to determine explicitly how well the model is able to detect the absence of one of the target gases (zero concentration targets), the zero-valued concentrations are excluded from the MAPE calculation, and one other quality metric is included. This metric is referred to as the Mean Zero Target Prediction Error (MZTPE), which simply measures the average model prediction for zero-valued targets: 𝑀𝐴𝐸 =⁡1𝑛𝑝∑∑|𝑦𝑖𝑗 − 𝑦𝑝𝑟𝑒𝑑,𝑖𝑗|𝑝𝑗=0𝑛𝑖=0⁡⁡ (10) 𝑀𝐴𝑃𝐸 =⁡1𝑛𝑝∑∑|𝑦𝑖𝑗 − 𝑦𝑝𝑟𝑒𝑑,𝑖𝑗|𝑦𝑖𝑗𝑝𝑗=0𝑛𝑖=0, 𝑓𝑜𝑟⁡𝑦𝑖𝑗 ≠ 0 (11) 𝑀𝑍𝑇𝑃𝐸 =⁡1𝑛𝑝∑∑|𝑦𝑖𝑗 − 𝑦𝑝𝑟𝑒𝑑,𝑖𝑗|𝑝𝑗=0𝑛𝑖=0⁡ , 𝑓𝑜𝑟⁡𝑦𝑖𝑗 = 0 (12) In the above relations, 𝑦𝑖𝑗 refers to the 𝑗𝑡ℎ concentration target for the 𝑖𝑡ℎ data sample, 𝑦𝑝𝑟𝑒𝑑,𝑖𝑗 is the model’s prediction of 𝑦𝑖𝑗, 𝑛 is the number of samples, and 𝑝 is the number of concentration targets per sample. Since two gas concentrations are being predicted (methane and ethane), 𝑝 is set to two. The data representation methods examined here are the raw data, PCA-transformed data, LDA-transformed data, and the feature-based representation. For the raw data representation, a pre-61 processing transform was still applied, as described in Section 4.1. In the classification cases Z-normalization was used, and for regression the datasets were normalized between 0 and 1. For the PCA transform, the number of components kept was determined by the smallest number that exceeded 99% explained variance. For the LDA transform (which is only applicable to the classification cases), the maximum number of components was chosen, which is equal to the number of classes minus one. Finally, for the feature-based representation, two sub-cases were tested. One case is using the entire hand-designed feature set as described in Appendix A, and the second is the subset of features selected using SA 5.1 Experiment I: Classification and Regression of Arbitrary Mixtures with Feature-based Representation Classification tests quantify the model accuracy when distinguishing between pure methane, pure ethane, and arbitrary mixtures of the two, whereas regression tests determine the accuracy of concentration estimates. The training data used included methane and ethane concentrations (100, 200 … 1000 ppm), and binary mixtures with all concentration combinations. This amounted to 10 methane samples, 10 ethane samples, and 100 mixture samples. Validation and test datasets both included 20 random methane samples, 20 random ethane samples, and 40 random binary mixture samples. All concentrations for the random samples were in the range of [100,1000]. For the classification experiments, the prediction targets were the mixture class (methane, ethane, or mixture), and for the regression experiments, they were the known concentration values. 62 5.2 Experiment II: Classification and Regression of Arbitrary Mixtures with Raw Data Representation This experiment examines the more desirable case where raw sensor data is used, and no feature design is necessary. The only transformations performed on the data are the Z-normalization for the classification case, and the dataset compression between 0 and 1 for the regression case. The same hyper-parameter search values as in Experiment I were used here. 5.3 Natural Gas Discrimination 5.3.1 Experiment III: Simulated Natural Gas Mixtures As the feature extraction and processing methods examined in Experiment I and II might not be able to produce better results that simply using the raw data representation, in Experiment III only the raw data representation was used in the simulated natural gas experiments. The performance of the model was evaluated using two different proportions of the binary methane-ethane mixture to simulate commercial natural gas, which varies in composition. In this context we must distinguish between the concentration and the composition of the simulated natural gas samples. The composition refers to the relative amounts of methane and ethane that simulate industrial natural gas, and the concentration refers to the amount of this mixture in clean air, calculated from the sum of the methane and ethane concentrations. For example, for a concentration of 1000 ppm with a 1% ethane proportion, the binary mixture composition was 990 ppm methane and 10 ppm ethane in air. The training datasets consisted of pure methane with concentrations 100, 200 … 3000 ppm, and mixtures of either 1% or 3% ethane proportion in the same concentrations. An identical procedure for hyper-parameter optimization and testing as in Experiments I and II was followed, in which separate validation and test datasets were created. For the validation 63 datasets, 15 random concentrations between 100 and 3000ppm were generated. Each of these concentrations was used to create a sample with 0% (pure methane), 1% and 3% ethane. The pure methane samples were shared by the 1% and 3% ethane validation datasets. In this way, each of the two validation datasets contained 15 negative samples (pure methane) and 15 positive samples (either 1% or 3% ethane) with the same concentrations, allowing model performance comparison to be made solely based on ethane proportion. The same procedure was followed to generate the test datasets, with 15 new randomly generated concentrations. For instance, if one of the randomly generated test concentrations were 1345 ppm, the mixtures for the test sets would be generated as pure methane (1345 ppm methane, 0 ppm ethane); 1% ethane (1332 ppm methane, 13 ppm ethane); and 3% ethane (1305 ppm methane, 40 ppm ethane). 5.3.2 Experiment IV: Outdoor Leak Detection After training the sensing apparatus to discriminate between pure methane and natural gas, the apparatus was transferred into the detection device described in Section 3.3, and was brought to London, Ontario for field testing. The field tests consisted of sampling the air near where small ongoing natural gas leaks were known to exist. The locations of these leaks are tracked by Fortis BC, and they are monitored to ensure that they pose no threat. At each of the chosen locations, the existence of the leak was first confirmed using a device manufactured by Hetek Solutions Inc for use in conducting walking leak detection surveys. After confirmation, the detection device described here was used to take a sample at that location, which was then analyzed by the trained model. 64 5.4 Experiment V: THC Discrimination As a volatile organic compound, THC should be detectable by MOS type sensors. In the experiment presented here, the feasibility of using the detection apparatus described in Section 3.4 will be tested for detecting the presence of denatured THC dissolved in methanol. To collect the data for this experiment, liquid samples were dispensed into a closed 1L chamber using a micro-pipette and were allowed to evaporate and disperse into the chamber. Samples were liquid methanol or THC denatured in methanol with concentration 1μg/ μL. The goal of this experiment was to determine whether the samples containing THC could be discriminated from those of the pure methanol. Since the diffusion time of THC has not been studied in our lab, experiments were conducted using three different exposure times to determine if this made a significant impact. The exposure times examined were 40 seconds as used in Experiments I-IV, as well as 60 seconds and 80 seconds. The recovery times were fixed at 150 seconds, so the experiment lengths vary from 200-240 seconds depending on the exposure time used. Data from ten concentrations corresponding to 1-10 μL of solution dispensed into the sensing chamber was collected, with two repeats conducted each. In all, 20 THC samples and 20 methanol samples were collected for each exposure time. Due to the small dataset size, classification rates were obtained using a leave-one-out cross-validation method. 65 Chapter 6: Results 6.1 Experiment I: Classification and Regression of Arbitrary Mixtures with Feature-based Representation The results for the best hyper-parameters, validation set and test set CAs are given in Table 2 through Table 5. Each table presents results for a different data representation method (PCA, LDA, SA-selected features, or the full set of features). A summary of the best classification results for each model and representation method is also given in Table 6. The results for the best hyper-parameters, and test set Mean Absolute Error (MAE), Mean Absolute Percentage Error, (MAPE), and Mean Zero-Target Prediction Error (MZTPE) for the regression task are given in Table 7 through Table 9. Each table presents results for a different data representation method (PCA, SA-selected features, or the full set of features). The LDA method was not used for the regression experiments. Table 2: Hyper-parameter optimization and test results for classification of arbitrary mixtures using PCA feature representation Model Best hyper-parameters Best Validation Set CA (%) Test Set CA (%) k-NN 𝐾⁡ = 1 100 97.5 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 9 91.25 87.5 SVM 𝐶 = 10 𝛾⁡ = 0.1 100 96.25 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 = 2 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 16 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 1974 100 97.5 66 Table 3: Hyper-parameter optimization and test results for classification of arbitrary mixtures using LDA feature representation Model Best hyper-parameters Best Validation Set CA (%) Test Set CA (%) k-NN 𝐾⁡ = 1 55.00 55.00 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 3 71.25 68.75 SVM 𝐶 = 0.1 𝛾⁡ = 1 60.00 56.25 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 16 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.001 𝜆 = 0.0001 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 40 72.50 65.00 Table 4: Hyper-parameter optimization and test results for classification of arbitrary mixtures using SA feature representation Model Best hyper-parameters Best Validation Set CA (%) Test Set CA (%) k-NN 𝐾⁡ = 1 75.00 76.25 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 5 86.25 83.75 SVM 𝐶 = 1000 𝛾⁡ = 0.001 88.75 82.5 MLP 𝑛𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0.1 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 241 82.50 83.75 67 Table 5: Hyper-parameter optimization and test results for classification of arbitrary mixtures using the full feature set Model Best hyper-parameters Best Validation Set CA (%) Test Set CA (%) k-NN 𝐾⁡ = 5 87.50 86.25 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 9 97.50 98.75 SVM 𝐶 = 10 𝛾⁡ = 0.001 91.25 95.00 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0.01 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 407 97.50 91.25 Table 6: Summary of the best classification results using each combination of model and representation methods PCA LDA SA Full set k-NN 97.5 55.00 76.25 86.25 RF 87.5 68.75 83.75 98.75 SVM 96.25 56.25 82.5 95.00 MLP 97.5 65.00 83.75 91.25 Table 7: Hyper-parameter optimization and test results for regression of arbitrary mixtures using PCA feature representation Model Best hyper-parameters MAE (ppm) MAPE (%) MZTPE (ppm) k-NN 𝐾⁡ = 3 54.0 14.4 24.2 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 11 57.4 14.2 36.3 SVM 𝐶 = 10000 𝛾⁡ = 0.001 56.7 13.3 45.7 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠⁡ = 2 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠⁡ = 553 53.1 13.8 25.4 68 Table 8: Hyper-parameter optimization and test results for regression of arbitrary mixtures using SA feature representation Model Best hyper-parameters MAE (ppm) MAPE (%) MZTPE (ppm) k-NN 𝐾⁡ = 3 141.7 29.2 137.5 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 9 100.0 18.3 106.4 SVM 𝐶 = 100000 𝛾⁡ = 0.1 89.4 20.0 81.4 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠 = 32 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 1 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0.01 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 1026 83.5 18.1 64.7 Table 9: Hyper-parameter optimization and test results for regression of arbitrary mixtures using the full feature set Model Best hyper-parameters MAE (ppm) MAPE (%) MZTPE (ppm) k-NN 𝐾⁡ = 5 66.4 15.7 40.5 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 9 57.4 12.1 52.4 SVM 𝐶 = 10 𝛾⁡ = 0.001 56.5 10.8 86.1 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠⁡ = 32 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 2 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.001 𝜆 = 0.001 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠⁡ = 1948 43.2 11.5 12.7 The choice of data representation method had a significant impact on classification performance. The LDA representation, which should theoretically maximize inter-class variance, had poor classification accuracy, with the best test set performance being only 68.75%. In order to visualize the inferior performance of the LDA transform, the PCA- and LDA- transformed train and test data was plotted (Figure 23). 69 Both the PCA and LDA transforms are able to separate the training data well (Figure 6A and 6C respectively). However, while the PCA transform is also able to separate the test data (Figure 6B), the LDA transform is not (Figure 6D). The SA-selected features also did not perform well compared to the PCA and full feature set tests. Models trained with the PCA-transformed data performed very well during hyper-parameter selection, with the k-NN, SVM, and MLP models all achieving 100% CA on the validation set. However, none of them were able to do so on the test set. In fact, the best test set performance came from the RF model, trained using the full feature set. The fact that comparable results could be obtained between the original 120 features and the dataset compressed to two PCA components suggests that there is a high degree of correlation between the selected features. Examination of Appendix A provides an explanation for this: since the important information could not be known in advance, many of the hand-designed features are very similar to one another. For example, several of the features are the times at which the curve reaches 10%, 20% … of the peak value. This also demonstrates that the PCA transform is effective in dealing with this inter-feature correlation. A similar trend can be observed in the regression test results, where the performances of each of the models are relatively consistent for a given representation method. In this case, the best MAE, MAPE, and MZTPE came from models trained on the full feature set. Interestingly, the SVM achieved the lowest MAPE at 10.8%, but the MZTPE was relatively high at 86.1ppm. This suggests that while the SVM was excellent at predicting the regression targets, it was not able to reliably detect the absence of one of the vapours. In contrast, the MLP was excellent at recognizing zero-valued targets, averaging a prediction of just 12.7ppm. 70 Figure 23: A) PCA-transformed train data. B) PCA-transformed test data. C) LDA-transformed train data. D) LDA-transformed test data 71 6.2 Experiment II: Classification and Regression of Arbitrary Mixtures with Raw Data Representation The results for the best hyper-parameters, validation set and test set Classification Accuracies (CA) for the classification task are given in Table 10. The results for the best hyper-parameters, and test set Mean Absolute Error (MAE), Mean Absolute Percentage Error, (MAPE), and Mean Zero-Target Prediction Error (MZTPE) for the regression task are given in Table 11. Table 10: Hyper-parameter optimization and test results for classification of arbitrary mixtures using raw data representation Model Best hyper-parameters Best Validation Set CA (%) Test Set CA (%) k-NN 𝐾 = 1 98.75 98.75 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 3 100 91.25 SVM 𝐶 = 10000 𝛾⁡ = 0.1 100 98.75 MLP 𝑛𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 8 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.001 𝜆 = 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠⁡ = 1944 100 97.5 CNN 𝑛_𝑐𝑜𝑛𝑣_𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑓𝑖𝑙𝑡𝑒𝑟_𝑙𝑒𝑛𝑔𝑡ℎ = 32 𝑛_𝑓𝑖𝑙𝑡𝑒𝑟𝑠 = 8 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.001 𝜆 = 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 1766 100 97.5 72 Table 11: Hyper-parameter optimization and test results for regression of arbitrary mixtures using raw data representation Model Best hyper-parameters MAE (ppm) MAPE (%) MZTPE (ppm) KNN 𝐾 = 1 65.2 17.2 15.0 RF 𝑚𝑎𝑥⁡ _𝑑𝑒𝑝𝑡ℎ⁡ = 7 86.3 18.8 87.5 SVM 𝐶 = 10 𝛾⁡ = 1 58.9 13.9 48.0 MLP 𝑛_𝑙𝑎𝑦𝑒𝑟𝑠⁡ = 2 𝑙𝑎𝑦𝑒𝑟_𝑤𝑖𝑑𝑡ℎ = 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠⁡ = 1971 51.4 13.9 23.1 CNN 𝑛_𝑐𝑜𝑛𝑣_𝑙𝑎𝑦𝑒𝑟𝑠 = 1 𝑓𝑖𝑙𝑡𝑒𝑟_𝑙𝑒𝑛𝑔𝑡ℎ = 32 𝑛_𝑓𝑖𝑙𝑡𝑒𝑟𝑠 = 32 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 = 0.01 𝜆 = 0.0001 𝑛_𝑒𝑝𝑜𝑐ℎ𝑠 = 1604 46.2 12.0 12.1 For the classification tests, the raw data representation produced excellent results, with the RF, SVM, MLP, and CNN all able to achieve 100% accuracy, and the k-NN achieving 98.75%. The performances on the test set demonstrate that the models were also able to generalize well, with only the RF showing less than a 97.5% test set accuracy. In the regression tests, the three best-performing models (SVM, MLP, and CNN) gave similar results for both the MAE and MAPE metrics. The biggest difference was that MLP and CNN were much better in predicting the zero-valued targets, with each performing much better than SVM. All but the k-NN model achieved 100% accuracy on the validation set, but none were able to do so on the test set. This is an expected outcome, since many different sets of hyper-parameters are tested on the validation set, and thus there is a higher probability of getting lucky and 73 predicting every label correctly. Given that the test set accuracies are only slightly lower than those for the validation set, it can be concluded that the models generalize well to unseen data. This is important for deploying models into real applications, and having a good estimate of the accuracy that can be expected is essential. Overall, the results for the raw data representation are comparable with the best results from the feature-based representations, suggesting that tasks of classifying and regressing binary mixtures can be accomplished without any feature design, even using only a single MOS sensor. 6.3 Natural Gas Discrimination 6.3.1 Experiment III: Simulated Natural Gas Mixtures The results of performing classification between the pure methane samples and the 1% and 3% ethane samples with each of the studied models are given in Table 12. The optimal hyper-parameter values were left off of the tables in this section in the interest of compactness. The classification task in Experiment III was significantly more difficult than those in Experiments I and II, and hence the classification performances in the previous experiments were superior. All of the studied models followed a pattern of achieving better classification performance on the 3% ethane dataset than the 1%, with the exception of the k-NN, which performed equally well on both. This is to be expected, since a lower ethane proportion naturally leads to more difficulty in discrimination. 74 Table 12: Test results for classification of simulated natural gas mixtures using raw data representation Model Best Validation Set Accuracy (%) Test Set Accuracy (%) 1% ethane 3% ethane 1% ethane 3% ethane k-NN 86.7 100 86.7 86.7 RF 80 100 80 93.3 SVM 86.7 100 83.3 93.3 MLP 96.7 100 83.3 86.7 CNN 96.7 100 80 90 The results of regressing both the methane and ethane concentrations in the simulated natural gas samples with 3% and 1% ethane are given in As in the classification experiment, regression predictions for the 3% ethane dataset are significantly better than those for the 1% ethane. MLP provided the best predictions for the 3% ethane sample, achieving only 10.9% error for the methane and 14.9% error for the ethane concentrations. In almost every case, the methane predictions (as measured by MAPE) are better than those for ethane. This is due to the significantly smaller range of possible values that the ethane concentrations can take. For example, for the 3% ethane dataset, the possible values for the methane concentration are 97-2910 ppm, while the ethane values can range from only 3-90 ppm. One interesting observation is that the performances of both of the neural network approaches (MLP and CNN) degraded more significantly than the other models when the ethane proportion was decreased. In fact, for the 1% ethane dataset, all other studied regression models out-performed the MLP and CNN. Table 13 and Table 14, respectively. Due to the significantly different range widths for the concentrations of methane and ethane, the MAE metric would not be appropriate for determining overall quality (i.e., an error of 50ppm for ethane is much more significant than for 75 methane). For this reason, the MAE and MAPE metrics are calculated with respect to each individual component. Also, MZTPE is only applicable to ethane in this case, since there are no samples where the concentration of methane is zero. As in the classification experiment, regression predictions for the 3% ethane dataset are significantly better than those for the 1% ethane. MLP provided the best predictions for the 3% ethane sample, achieving only 10.9% error for the methane and 14.9% error for the ethane concentrations. In almost every case, the methane predictions (as measured by MAPE) are better than those for ethane. This is due to the significantly smaller range of possible values that the ethane concentrations can take. For example, for the 3% ethane dataset, the possible values for the methane concentration are 97-2910 ppm, while the ethane values can range from only 3-90 ppm. One interesting observation is that the performances of both of the neural network approaches (MLP and CNN) degraded more significantly than the other models when the ethane proportion was decreased. In fact, for the 1% ethane dataset, all other studied regression models out-performed the MLP and CNN. Table 13: Test results for regression of simulated natural gas mixtures with 3% ethane using raw data representation Model MAE (ppm) MAPE (%) MZTPE (ppm) k-NN Methane: 229.2 Ethane: 5.2 Methane: 13.7 Ethane: 18.2 0.0 RF Methane: 240.1 Ethane: 27.1 Methane: 13.3 Ethane: 29.4 8.4 SVM Methane: 169.0 Ethane: 21.6 Methane: 10.5 Ethane: 27.4 7.4 MLP Methane: 192.2 Ethane: 7.5 Methane: 10.9 Ethane: 14.9 2.0 CNN Methane: 228.0 Ethane: 10.4 Methane: 12.7 Ethane: 18.7 1.5 76 Table 14: Test results for regression of simulated natural gas mixtures with 1% ethane using raw data representation Model MAE (ppm) MAPE (%) MZTPE (ppm) k-NN Methane: 211.9 Ethane: 13.4 Methane: 14.5 Ethane: 34.7 7.2 RF Methane: 290.0 Ethane: 4.6 Methane: 19.3 Ethane: 12.6 2.3 SVM Methane: 215.7 Ethane: 10.4 Methane: 15.4 Ethane: 41.5 3.2 MLP Methane: 277.2 Ethane: 7.7 Methane: 21.5 Ethane: 44.3 2.7 CNN Methane: 392.2 Ethane: 11.4 Methane: 29.0 Ethane: 70.4 3.8 6.3.2 Experiment IV: Outdoor Leak Detection In total, data was sampled at three viable locations around London, Ontario for the purpose of testing the device’s ability to discriminate real natural gas leaks from background methane emissions. Several other locations were investigated, but the natural gas detector manufactured by Hetek Solutions Inc. was unable to detect any leakage there. Table 15 presents the results of using a 1-NN classifier to predict the compositions of the collected samples, of which there were two at each leak location, performed back to back. The training data used to train the classifier consisted of 13 natural gas samples and 10 pure methane samples, with concentrations ranging from 850 to 5500 ppm. Even with the limited training data available due to time constraints, 5 out of 6 collected samples were correctly identified as being natural gas. No test samples of pure methane were collected due to lack of a viable source. In the case of the misclassified sample, visual inspection of the 77 sensor response curve indicated a much slower recovery phase than seen in the training data. This was attributed to imperfect sealing between the exposure and recovery chambers, which would result in natural gas leaking into where there should be fresh air. After this test, the seal was tightened and all subsequent tests exhibit response curves closer to what is expected. In addition to predicting the binary classification of these samples, the concentrations were predicted using a separately trained MLP. As shown in Table 15, difficulty was encountered in predicting the concentration. The cause of this is multi-faceted. First, there was an outlier case where the first concentration estimate performed (Test 1) was much higher than the actual concentration. This was attributed to residual gas from the training phase trapped inside the exposure chamber. After Test 1, this chamber was thoroughly flushed, and all subsequent tests exhibited much lower response amplitudes. Another issue was that the actual target concentration could not be determined. This is because the device from Hetek is a continuous sampling device, and thus gives an instantaneous concentration value at its probe location. This concentration estimate varied significantly in response to how the probe was positioned, and thus the target concentration could only be roughly estimated. In all cases, the natural gas leak concentrations varied from approximately 200 to 600 ppm. This gives rise to the final issue, which was that, as mentioned above, the training data used to develop the regression model was in the concentration range of 850 to 5500 ppm. Thus, the model predicted a concentration value around 750 ppm for every test (excluding Test 1) because it could not properly extrapolate down to the actual measured concentrations. The challenges associated with performing predictions in changing outdoor conditions will factor into the training method for a real detector. The most important of these are changing temperature and humidity, both of which affect the output of MOS sensors. These effects can be mitigated in two different ways: by using a model trained to account for them, by or adjusting the 78 output data based on measured values. In the first method, the training data used to develop the models must include samples at a wide range of temperatures and humidities so that their effects can be learned. This would significantly increase training time in addition to requiring an environmental control chamber, likely making it infeasible. The second method simply involves applying a correction factor to the output of the MOS sensor based on the measured temperature and humidity. This would likely be the most practical solution, since Figaro already provides correction factors for several temperature and humidity values. Table 15: Results of outdoor natural gas leakage detection in London, Ontario FIELD TEST TEST Natural Gas ID Model Conc. estimation Comments Location 1 1 POSITIVE > 3000ppm Residual methane trapped in exposure chamber 2 NEGATIVE ~750ppm Slow recovery – imperfect sealing between exposure and recovery chamber Location 2 3 POSITIVE ~750ppm 4 POSITIVE ~750ppm Location 3 5 POSITIVE ~750ppm 6 POSITIVE ~750ppm 79 6.4 Experiment V: THC Discrimination The results of performing the leave-one-out validation tests for each of the three exposure times are given in Table 16. In every case, the classification rate improved from 40s to 80s exposure time, which makes sense given that extra information is contained in the response. However, for the SVM, MLP, and CNN classifiers the performance dropped with a 60s exposure time. Since the performance improved at 80s in all cases, this is likely due to experimental errors when collecting the 60s data. This strongly suggests that the THC molecule diffuses on a time-scale slow enough that an even longer exposure time than 80 seconds may be useful to improve the classification accuracy. With an 80 second exposure time, four out of the five studied models achieved greater than 90% classification accuracy when discriminating the pure methanol from THC denatured in methanol. The k-NN model did not perform as well, likely due to the small dataset size. When there are not as many samples available, checking the nearest neighbour is not as reliable of a test. From the perspective of developing a real-world detection device, the desired accuracy, false-positive and false-negative rates, and acceptable time required to complete each test are all up to the discretion of the industry partner. These must also be specified with law enforcement in mind, since the results of such a device will ultimately be used in court to determine impaired driving rulings. 80 Table 16: Cross-validated test results for classification of THC in methanol using raw data representation Model Cross-Validated Test Set Accuracy (%) 40 sec 60 sec 80 sec k-NN 71.4 75.0 78.6 RF 89.3 89.3 92.3 SVM 92.9 85.7 100 MLP 92.9 82.1 96.4 CNN 89.3 85.7 92.9 81 Chapter 7: Conclusions and Future Work The results presented in this thesis have demonstrated that the time-response from a single MOS sensor embedded in a microchannel can be used to discriminate between pure methane, pure ethane, and binary mixtures of methane and ethane even when the ethane proportion is as low as 1%, and quantify the concentrations of both gases in these mixtures. In Experiment I it was observed that once a large set of hand-designed features have been extracted, the processing method applied to these features has a significant impact on the quality of the resulting predictions. The most successful methods were applying a PCA transform to the feature set without any additional processing. In Experiment I and II, comparable results were observed from both the classification and regression tasks between the feature-based and raw data approaches. Eliminating the feature extraction step and working directly with the raw data showed improved recognition with the k-NN classifier, and equal performance with SVM and MLP. Only the RF model suffered in performance. In Experiments III and IV, it was demonstrated that classification of both simulated natural gas samples and real pipeline leaks can be accomplished using the same sensing apparatus. This finding is significant for two reasons. First, it shows that even with a large disparity in concentrations of two binary mixture components, the sensing apparatus and learning models are able to successfully detect the presence or absence of the trace component. Second, it provides confirmation that this approach is viable for the real-world application of natural gas detection and quantification. Such a task benefits greatly from the portability and cost-effectiveness of the detection apparatus, especially when drone-based approaches are considered. Finally, results from Experiment V confirm that a MOS-based breath analyzer is a viable possibility for detecting recent marijuana use. 82 The relative merit of using extracted features as opposed to raw time-series data is still inconclusive. In Experiment I and II, classification performance was not improved by any of the feature-based methods studied, and only a slight improvement was made to the regression performance. The results for feature-based methods also varied significantly across different feature transformation methods. The LDA transformation for example performed poorly in all studied cases. Due to the lack of a clear improvement, the preferred method for these experiments would be to use the raw data in the future. This approach greatly simplifies the prediction process by eliminating several processing steps and saving time on hyper-parameter optimization. Future application, however, should be evaluated on a case-by-case basis to determine which method works the best for them. The ability to discriminate between mixtures and pure gases, as well as mixture quantification, could be useful in many other applications, where different binary mixtures might need to be distinguished from each other before concentration estimation can be done. The work presented here demonstrates that such a task can be accomplished using a very simple sensing apparatus and machine learning. This methodology can be extended beyond studying mixtures of two simple hydrocarbons by incorporating the results of extra sensors with different sensitivities. In all studied cases, it could also be possible to improve prediction accuracy by examining sensor responses on shorter/longer timescales than the one proposed here. One possibility for accomplishing this study could be to vary the length of the microchannel and examine whether slower diffusion improves discriminative ability. The disadvantage of a longer microchannel is that test times would need to be longer to make up for a slower response. Another approach that could be promising would be to modify the sensing apparatus to include two sensors placed at the ends of microchannels with different lengths. This would add some bulk to the 3D-printed 83 housing but would likely provide improvement when analyzing mixtures in which the diffusion times of each of the components were significantly different. In order to reduce some of the trial and error associated with studies needed to optimize the sensor arrangements, the results of diffusion simulations should be incorporated when designing the experiments. Another design bottleneck is the speed of the machine learning training process. Prediction is usually quite fast once the model has been trained, but the process of hyper-parameter optimization involves training many different models and comparing their performance. This process could be accelerated in future by using GPUs to speed up training, or by reducing the size of each training sample by removing information that is not as useful. This would first require a study into which areas of the time-series curve contain the most useful information. Future work for the natural gas leak detector will primarily include steps involved in preparing a viable, marketable device. These steps include form factor minimization, robustness testing, patenting, and more thorough field testing. It should also be investigated whether odourization of the natural gas with mercaptan (rotten egg smell) will affect prediction accuracy, or if it could in fact be used to assist in the detection process. The THC detector is still in the earlier stages of development. The work presented here is a proof of concept that MOS sensors can reliably detect low concentrations of THC denatured in methanol, but the viability of a real device will require extensive field testing using real human breath samples. 84 Bibliography [1] A. Berna, “Metal oxide sensors for electronic noses and their application to food analysis,” Sensors. 2010, doi: 10.3390/s100403882. [2] Figaro, “TGS 2610-C00 Product Information.” . [3] J. W. Gardner and P. N. Bartlett, “A brief history of electronic noses,” Sensors Actuators B. Chem., 1994, doi: 10.1016/0925-4005(94)87085-3. [4] Z. Ali, D. James, W. T. O’Hare, F. J. Rowell, and S. M. Scott, “Radial basis neural network for the classification of fresh edible oils using an electronic nose,” J. Therm. Anal. Calorim., 2003, doi: 10.1023/A:1022222402328. [5] H. Zhang, M. Ö. Balaban, and J. C. Principe, “Improving pattern recognition of electronic nose data with time-delay neural networks,” Sensors Actuators, B Chem., 2003, doi: 10.1016/S0925-4005(03)00574-4. [6] M. Längkvist, S. Coradeschi, A. Loutfi, and J. B. Balaguru Rayappan, “Fast classification of meat spoilage markers using nanostructured ZnO thin films and unsupervised feature learning,” Sensors (Switzerland), 2013, doi: 10.3390/s130201578. [7] N. Bhattacharya, B. Tudu, A. Jana, D. Ghosh, R. Bandhopadhyaya, and M. Bhuyan, “Preemptive identification of optimum fermentation time for black tea using electronic nose,” Sensors Actuators, B Chem., 2008, doi: 10.1016/j.snb.2007.12.032. [8] A. Loutfi, S. Coradeschi, G. K. Mani, P. Shankar, and J. B. B. Rayappan, “Electronic noses for food quality: A review,” Journal of Food Engineering. 2015, doi: 10.1016/j.jfoodeng.2014.07.019. [9] T. eNose Company, “eNose Technology.” https://www.enose.nl/rd/technology/ (accessed Jun. 02, 2020). [10] P. S. Murvay and I. Silea, “A survey on gas leak detection and localization techniques,” J. Loss Prev. Process Ind., vol. 25, no. 6, pp. 966–973, 2012, doi: 10.1016/j.jlp.2012.05.010. [11] I. Brodetsky and M. Savic, “Leak monitoring system for gas pipelines,” 2002, doi: 10.1109/icassp.1993.319424. [12] ITT Corporation, “Enhanced Gas Leak Visualization Tools Accelerate Better Decision Making,” Pipeline Gas J., 2009. [13] M. B. Frish et al., “Standoff sensing of natural gas leaks: evolution of the remote methane leak detector (RMLD),” 2005, doi: 10.1109/qels.2005.1549335. [14] M. B. Frish et al., “Low-cost lightweight airborne laser-based sensors for pipeline leak detection and reporting,” in Next-Generation Spectroscopic Technologies VI, 2013, doi: 10.1117/12.2015813. [15] M. Rossi, D. Brunelli, A. Adami, L. Lorenzelli, F. Menna, and F. Remondino, “Gas-drone: Portable gas sensing system on UAVs for gas leakage localization,” in Proceedings of IEEE Sensors, 2014, doi: 10.1109/ICSENS.2014.6985282. [16] M. J. Aernecke, J. Guo, S. Sonkusale, and D. R. Walt, “Design, implementation, and field testing of a portable fluorescence-based vapor sensor,” Anal. Chem., 2009, doi: 10.1021/ac900505p. [17] P. W. Bryce, P. Jax, and J. Fang, “The Design and Installation of the LEOS Leak Detection and Location System for the Northstar Project,” 2009, doi: 10.1115/ipc2002-27042. [18] Arthur J. Kidnay and W. R. Parrish, Fundamentals of natural gas processing. mechanical engineering. 2006. [19] U. Bonne, “SELECTIVE LEAK-DETECTOR FOR NATURAL GAS,” 4,507,558, 1985. [20] U. Bonne and R. J. Matthys, “SELECTIVE NATURAL GAS DETECTING APPARATUS,” 85 4,958,076, 1990. [21] M. Phillips, J. Herrera, S. Krishnan, M. Zain, J. Greenberg, and R. N. Cataneo, “Variation in volatile organic compounds in the breath of normal humans,” J. Chromatogr. B Biomed. Sci. Appl., 1999, doi: 10.1016/S0378-4347(99)00127-9. [22] Y. Qiao et al., “Breath ketone testing: A new biomarker for diagnosis and therapeutic monitoring of diabetic ketosis,” Biomed Res. Int., 2014, doi: 10.1155/2014/869186. [23] M. Phillips et al., “Detection of lung cancer using weighted digital analysis of breath biomarkers,” Clin. Chim. Acta, 2008, doi: 10.1016/j.cca.2008.02.021. [24] S. S. Sehnert, L. Jiang, J. F. Burdick, and T. H. Risby, “Breath biomarkers for detection of human liver diseases: Preliminary study,” Biomarkers, 2002, doi: 10.1080/13547500110118184. [25] H. Z. Khiabani, J. G. Bramness, A. Bjørneboe, and J. Mørland, “Relationship between THC concentration in blood and impairment in apprehended drivers,” Traffic Inj. Prev., 2006, doi: 10.1080/15389580600550172. [26] R. A. Gustafson et al., “Urinary cannabinoid detection times after controlled oral administration of Δ9-tetrahydrocannabinol to humans,” Clin. Chem., 2003, doi: 10.1373/49.7.1114. [27] M. A. Huestis et al., “Cannabinoid concentrations in hair from documented cannabis users,” Forensic Sci. Int., 2007, doi: 10.1016/j.forsciint.2006.08.005. [28] S. J. Gross, T. E. Worthy, L. Nerder, E. G. Zimmermann, J. R. Soares, and P. Lomax, “Detection of recent cannabis use by saliva Δ9-THC radioimmunoassay,” J. Anal. Toxicol., 1985, doi: 10.1093/jat/9.1.1. [29] R. De La Torre and S. Pichini, “Usefulness of sweat testing for the detection of cannabis smoke,” Clinical Chemistry, 2004. [30] O. Beck, S. Sandqvist, I. Dubbelboer, and J. Franck, “Detection of Δ9-tetrahydrocannabinol in exhaled breath collected from cannabis users.,” J. Anal. Toxicol., 2011. [31] S. K. Himes, K. B. Scheidweiler, O. Beck, D. A. Gorelick, N. A. Desrosiers, and M. A. Huestis, “Cannabinoids in exhaled breath following controlled administration of smoked cannabis,” Clin. Chem., 2013, doi: 10.1373/clinchem.2013.207407. [32] O. Beck, N. Stephanson, S. Sandqvist, and J. Franck, “Detection of drugs of abuse in exhaled breath using a device for rapid collection: Comparison with plasma, urine and self-reporting in 47 drug users,” J. Breath Res., 2013, doi: 10.1088/1752-7155/7/2/026006. [33] L. Coucke, E. Massarini, Z. Ostijn, O. Beck, and A. G. Verstraete, “Δ9-Tetrahydrocannabinol concentrations in exhaled breath and physiological effects following cannabis intake – A pilot study using illicit cannabis,” Clin. Biochem., 2016, doi: 10.1016/j.clinbiochem.2016.06.003. [34] A. Graves, Supervised Sequence Labelling with Recurrent Neural Networks. 2012. [35] P. Esling and C. Agon, “Time-series data mining” ACM Comput. Surv., 2012, doi: 10.1145/2379776.2379788. [36] K. Arshak, E. Moore, G. M. Lyons, J. Harris, and S. Clifford, “A review of gas sensors employed in electronic nose applications,” Sensor Review. 2004, doi: 10.1108/02602280410525977. [37] E. L. L. Hines, E. Llobet, and J. W. W. Gardner, “Electronic noses: a review of signal processing techniques,” IEE Proc. - Circuits, Devices Syst., 1999, doi: 10.1049/ip-cds:19990670. [38] T. Rakthanmanon et al., “Searching and mining trillions of time series subsequences under dynamic time warping,” in Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’12, 2012, doi: 86 10.1145/2339530.2339576. [39] M. Verleysen and D. François, “The Curse of Dimensionality in Data Mining,” Analysis, 2005, doi: 10.1007/11494669_93. [40] H. Deng, G. Runger, E. Tuv, and M. Vladimir, “A time series forest for classification and feature extraction,” Inf. Sci. (Ny)., 2013, doi: 10.1016/j.ins.2013.02.030. [41] M. G. Baydogan, G. Runger, and E. Tuv, “A bag-of-features framework to classify time series,” IEEE Trans. Pattern Anal. Mach. Intell., 2013, doi: 10.1109/TPAMI.2013.72. [42] P. Schäfer, “The BOSS is concerned with time series classification in the presence of noise,” Data Min. Knowl. Discov., 2015, doi: 10.1007/s10618-014-0377-7. [43] P. Schäfer and M. Högqvist, “SFA: a symbolic fourier approximation and index for similarity search in high dimensional datasets,” in EDBT, 2012. [44] L. Ye and E. Keogh, “Time series shapelets: A new primitive for data mining,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2009, doi: 10.1145/1557019.1557122. [45] J. Yan et al., “Electronic nose feature extraction methods: A review,” Sensors (Switzerland). 2015, doi: 10.3390/s151127804. [46] J. Suto, S. Oniga, and P. P. Sitar, “Comparison of wrapper and filter feature selection algorithms on human activity recognition,” in 2016 6th International Conference on Computers Communications and Control, ICCCC 2016, 2016, doi: 10.1109/ICCCC.2016.7496749. [47] M. Hall and L. Smith, “Feature subset selection: a correlation based filter approach,” in Proceedings of International Conference on Neural Information Processing and Intelligent Information Systems, 1998. [48] K. Kira and L. A. Rendell, “A Practical Approach to Feature Selection,” in Machine Learning Proceedings 1992, 1992. [49] L. Yu and H. Liu, “Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution,” in Proceedings, Twentieth International Conference on Machine Learning, 2003. [50] Z. Zhao and H. Liu, “Searching for interacting features,” in IJCAI International Joint Conference on Artificial Intelligence, 2007. [51] M. Hall and L. a Smith, “Feature Selection for Machine Learning : Comparing a Correlation-based Filter Approach to the Wrapper CFS : Correlation-based Feature,” Int. FLAIRS Conf., 1999, doi: 10.1.1.50.2192. [52] D. Zongker and A. Jain, “Algorithms for feature selection: An evaluation,” in Proceedings - International Conference on Pattern Recognition, 1996, doi: 10.1109/ICPR.1996.546716. [53] P. M. Narendra and K. Fukunaga, “A Branch and Bound Algorithm for Feature Subset Selection,” IEEE Trans. Comput., 1977, doi: 10.1109/TC.1977.1674939. [54] D. Beasley, D. R. Bull, and R. R. Martin, “An overview of genetic algorithms : Part 1, fundamentals,” Univ. Comput., 1993. [55] X. Wang, J. Yang, X. Teng, W. Xia, and R. Jensen, “Feature selection based on rough sets and particle swarm optimization,” Pattern Recognit. Lett., 2007, doi: 10.1016/j.patrec.2006.09.003. [56] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science (80-. )., 1983, doi: 10.1126/science.220.4598.671. [57] L. van der Maaten, E. Postma, and J. van den Herik, “Dimensionality Reduction: A Comparative Review,” 2009. [58] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K. R. Mullers, “Fisher discriminant analysis with kernels,” 2003, doi: 10.1109/nnsp.1999.788121. [59] Y. S. Jeong, M. K. Jeong, and O. A. Omitaomu, “Weighted dynamic time warping for time series classification,” in Pattern Recognition, 2011, doi: 10.1016/j.patcog.2010.09.022. 87 [60] E. Keogh and C. A. Ratanamahatana, “Exact indexing of dynamic time warping,” Knowl. Inf. Syst., 2005, doi: 10.1007/s10115-004-0154-9. [61] W. Zhang, “Machine Learning Approaches to Predicting Company Bankruptcy,” J. Financ. Risk Manag., 2017, doi: 10.4236/jfrm.2017.64026. [62] S. Güney and A. Atasoy, “Multiclass classification of n-butanol concentrations with k-nearest neighbor algorithm and support vector machine in an electronic nose,” Sensors Actuators, B Chem., 2012, doi: 10.1016/j.snb.2012.03.047. [63] S. S. Schiffman, D. W. Wyrick, R. Gutierrez-Osuna, and H. T. Nagle, “Effectiveness of an Electronic Nose for Monitoring Bacterial and Fungal Growth,” in Proceedings of the 7th International Symp. On Olfaction and Electronic Nose, 2000. [64] R. Blatt, A. Bonarini, E. Calabró, M. Della Torre, M. Matteucci, and U. Pastorino, “Lung cancer identification by an electronic nose based on an array of MOS sensors,” in IEEE International Conference on Neural Networks - Conference Proceedings, 2007, doi: 10.1109/IJCNN.2007.4371167. [65] T. Hastie, R. Tibshirani, and J. Friedman, Springer Series in Statistics. 2009. [66] i-king-of-ml, “Random Forest and how it works [blog post],” 2019. https://medium.com/@rdhawan201455/random-forest-and-how-it-works-67f408e43a43 (accessed Jun. 02, 2020). [67] Q. Li, Y. Gu, and N. F. Wang, “Application of Random Forest Classifier by Means of a QCM-Based E-Nose in the Identification of Chinese Liquor Flavors,” IEEE Sens. J., 2017, doi: 10.1109/JSEN.2017.2657653. [68] M. Pardo and G. Sberveglieri, “Random forests and nearest shrunken centroids for the classification of sensor array data,” Sensors Actuators, B Chem., 2008, doi: 10.1016/j.snb.2007.12.015. [69] M. Pardo and G. Sberveglieri, “Classification of electronic nose data with support vector machines,” Sensors Actuators, B Chem., 2005, doi: 10.1016/j.snb.2004.12.005. [70] O. C. Carrasco, “Support Vector Machines for Classification [blog post],” 2019. https://towardsdatascience.com/support-vector-machines-for-classification-fc7c1565e3 (accessed Jun. 02, 2020). [71] N. El Barbri, E. Llobet, N. El Bari, X. Correig, and B. Bouchikhi, “Electronic nose based on metal oxide semiconductor sensors as an alternative technique for the spoilage classification of red meat,” Sensors, 2008, doi: 10.3390/s8010142. [72] O. S. Papadopoulou, E. Z. Panagou, F. R. Mohareb, and G. J. E. Nychas, “Sensory and microbiological quality assessment of beef fillets using a portable electronic nose in tandem with support vector machine analysis,” Food Res. Int., 2013, doi: 10.1016/j.foodres.2012.10.020. [73] F. Bre, J. M. Gimenez, and V. D. Fachinotti, “Prediction of wind pressure coefficients on building surfaces using artificial neural networks,” Energy Build., 2018, doi: 10.1016/j.enbuild.2017.11.045. [74] H. Sundgren, I. Lukkar, F. Winquist, and I. Lundstrom, “Artificial neural networks and gas sensor arrays: quantification of individual components in a gas mixture,” Measurement Science and Technology, vol. 2, no. 5. pp. 464–69, 1991, [Online]. Available: http://www.iop.org/EJ/article/0957-0233/2/5/008/mt910508.pdf. [75] E. Llobet, J. Brezmes, X. Vilanova, J. E. Sueiras, and X. Correig, “Qualitative and quantitative analysis of volatile organic compounds using transient and steady-state responses of a thick-film tin oxide gas sensor array,” Sensors Actuators, B Chem., vol. 41, no. 1–3, pp. 13–21, 1997, doi: 10.1016/S0925-4005(97)80272-9. [76] D. Zhang, J. Liu, C. Jiang, A. Liu, and B. Xia, “Quantitative detection of formaldehyde and ammonia gas via metal oxide-modified graphene-based sensor array combining with neural network model,” Sensors Actuators, B Chem., vol. 240, pp. 55–65, 2017, doi: 88 10.1016/j.snb.2016.08.085. [77] H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, “A convolutional neural network cascade for face detection,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, doi: 10.1109/CVPR.2015.7299170. [78] M. Paknahad, J. S. Bachhal, A. Ahmadi, and M. Hoorfar, “Characterization of channel coating and dimensions of microfluidic-based gas detectors,” Sensors Actuators, B Chem., vol. 241, pp. 55–64, 2017, doi: 10.1016/j.snb.2016.10.048. [79] N. H. Lee G, Gommers R, Wasilewski F, Wohlfahrt K, O’Leary A, “PyWavelets - Wavelet Transforms in Python.” [Online]. Available: https://github.com/PyWavelets/pywt [Online; accessed 2018-11-20]. [80] I. Batal and M. Hauskrecht, “A supervised time series feature extraction technique using DCT and DWT,” 8th Int. Conf. Mach. Learn. Appl. ICMLA 2009, pp. 735–739, 2009, doi: 10.1109/ICMLA.2009.13. [81] L. Yu and H. Liu, “Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution,” in Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003), 2003, doi: citeulike-article-id:3398512. [82] U. M. (Califonia I. of T. Fayyad and K. B. (University of M. Irani, “Multi-interval discretization of continuous-valued attributes for classification learning,” in Proceedings of the International Joint Conference on Uncertainty in AI (Q334 .I571 1993), 1993, doi: 10.1109/TKDE.2011.181. [83] A. Krizhevsky and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Neural Inf. Process. Syst., 2012, doi: http://dx.doi.org/10.1016/j.protcy.2014.09.007. 89 Appendices Appendix A: Complete list of hand-designed features Feature Number Feature Description 1 Peak value 2 Area under curve from t0 to thalf-peak,exposure 3 Area under curve from t0 to tpeak 4 Area under curve from tpeak to thalf-peak,recovery 5 Area under curve from tpeak to tend 6 Total area under curve 7 Slope of line from 5% of peak to 95% of peak 8 Slope of line from 95% of peak to end 9 - 18 Time to reach 10%, 20% … 100% of peak (exposure phase) 19 - 24 Time to reach 90%, 80% … 40% of peak (recovery phase) 25 - 41 Signal value at t=10s, 20s … 170s 42 - 57 Derivative of the signal at t=20s, 30s … 170s 58 - 73 Second derivative of the signal at t=20s, 30s … 170s 74 - 90 Value at t=10s, 20s … 170s after normalizing signal in [0,1] 91 - 112 DWT coefficient 1, 2 … 22 113 - 120 FFT coefficient 1, 2 … 8
Thesis/Dissertation
eng
Mechanical Engineering
Vancouver : University of British Columbia Library
Graduate
Applications of artificial olfaction and machine learning for detection of volatile gas
Text