Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Feasibility of discriminating between buried metallic spheroids by classification of their electromagnetic… Chesney, Robert Harvey 1982

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1982_A7 C48.pdf [ 5.67MB ]
Metadata
JSON: 831-1.0095648.json
JSON-LD: 831-1.0095648-ld.json
RDF/XML (Pretty): 831-1.0095648-rdf.xml
RDF/JSON: 831-1.0095648-rdf.json
Turtle: 831-1.0095648-turtle.txt
N-Triples: 831-1.0095648-rdf-ntriples.txt
Original Record: 831-1.0095648-source.json
Full Text
831-1.0095648-fulltext.txt
Citation
831-1.0095648.ris

Full Text

F e a s i b i l i t y of D i s c r i m i n a t i n g Between Buried M e t a l l i c Spheroids by C l a s s i f i c a t i o n of T h e i r E l e c t r o m a g n e t i c Response. by Robert Harvey Chesney B.A.Sc.,The U n i v e r s i t y of B r i t i s h Columbia, 1977 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENT FOR THE DEGREE OF MASTER OF APPLIED SCIENCE In the F a c u l t y of Graduate Stud i e s Department of E l e c t r i c a l E n gineering We accept t h i s t h e s i s as conforming t o the r e q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA September 1982 © Robert H. Chesney, 1982 I n p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t o f t h e r e q u i r e m e n t s f o r an a d v a n c e d d e g r e e a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a , I a g r e e t h a t t h e L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and s t u d y . I f u r t h e r a g r e e t h a t p e r m i s s i o n f o r e x t e n s i v e c o p y i n g o f t h i s t h e s i s f o r s c h o l a r l y p u r p o s e s may be g r a n t e d by t h e h e a d o f my d e p a r t m e n t o r by h i s o r h e r r e p r e s e n t a t i v e s . I t i s u n d e r s t o o d t h a t c o p y i n g o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l n o t be a l l o w e d w i t h o u t my w r i t t e n p e r m i s s i o n . D e p a r t m e n t Of E l e c t r i c a l Engineering The U n i v e r s i t y o f B r i t i s h C o l u m b i a 1956 Main Mall V a n c o u v e r , Canada V6T 1Y3 D a t e 13 October 1982 D E - 6 (3/81) A b s t r a c t A n i n v e s t i g a t i o n i n t o t h e f e a s i b i l i t y o f a p p l y i n g p a t t e r n r e c o g n i t i o n c o n c e p t s t o t h e c l a s s i f i c a t i o n o f m e t a l l i c o b j e c t s by t h e i r e l e c t r o m a g n e t i c r e s p o n s e w a s ; p e r f o r m e d . T h e e f f e c t on t h e r e s p o n s e o f v a r i o u s f a c t o r s s u c h a s o b j e c t s h a p e a n d o r i e n t a t i o n w a s e x a m i n e d a n d a p a t t e r n r e c o g n i t i o n s c h e m e w a s p r o p o s e d b a s e d on t h e s e r e s u l t s . I m p l e m e n t a t i o n o f t h e p r o p o s a l i n v o l v e d t h e d e v e l o p m e n t o f a n o v e l e x t e n s i o n t o t h e n e a r e s t m e a n v e c t o r t y p e o f c l a s s i f i e r i n w h i c h t h e c l a s s " c e n t r o i d " w a s g e n e r a l i z e d t o be a c u r v e i n t h e f e a t u r e s p a c e r a t h e r t h a n a p o i n t . T h e r e s u l t a n t p a t t e r n r e c o g n i t i o n s c h e m e w a s t e s t e d o n a r e p r e s e n t a t i v e t e s t s e t w h i c h i n c l u d e d 8 1 5 s i g n a t u r e s o f o b j e c t s , c o r r e s p o n d i n g t o 104 v a r i a t i o n s i n o b j e c t a n d o r i e n t a t i o n . A s u c c e s s r a t e o f g r e a t e r t h a n 98 p e r c e n t w a s a c h i e v e d . I t i s n o t e d t h a t t h e c l a s s i f i e r e x t e n s i o n d e v e l o p e d p r o v i d e s a v i a b l e a p p r o a c h t o c l a s s i f i c a t i o n o f r e s p o n s e s i g n a t u r e s t h a t v a r y c o n t i n u o u s l y w i t h r e s p e c t t o a n y s i n g l e p a r a m e t e r . i 1 T a b l e o f C o n t e n t s P a g e A b s t r a c t . . i i T a b l e o f C o n t e n t s * i i i L i s t o f T a b l e s v L i s t o f F i g u r e s v i A c k n o w l e d g e m e n t v i i C h a p t e r 1: I n t r o d u c t i o n 1.1 P u r p o s e 1 1 .2 S c o p e 2 C h a p t e r 2 : S e n s o r M e t h o d and A s s u m p t i o n s 2 . 1 S e n s o r S y s t e m . . . 4 2 . 2 I n i t i a l I n v e s t i g a t i o n s 8 2 . 3 A s s u m p t i o n s and R e s t r i c t i o n s . 12 C h a p t e r 3 : F e a t u r e E x t r a c t i o n 3 . 1 F e a t u r e T y p e s 16 3 . 2 F e a t u r e E v a l u a t i o n 19 C h a p t e r 4 : C l a s s i f i e r D e s i g n 4 . 1 C l a s s i f i e r T y p e 27 4 . 2 C l a s s i f i e r D i s t a n c e M e a s u r e 34 C h a p t e r 5 : E x p e r i m e n t s 5 . 1 I n t r o d u c t i o n 36 5 . 2 O r i e n t a t i o n S e n s i t i v i t y T e s t s 40 5 . 3 D e p t h S e n s i t i v i t y T e s t s 41 C h a p t e r 6 : R e s u l t s and E v a l u a t i o n 6 . 1 R e s u l t s o f O r i e n t a t i o n S e n s i t i v i t y T e s t s . 43 6 . 2 R e s u l t s o f D e p t h S e n s i t i v i t y T e s t s 49 i i i P a g e 6 . 3 E f f e c t s o f a M o d i f i e d D e s i g n S e t 51 6 . 4 E v a l u a t i o n o f R e s u l t s 53 C h a p t e r 7 : C o n c l u s i o n s and Summary 7 . 1 C o n c l u s i o n s — 55 7 . 2 R e c o m m e n d a t i o n s f o r F u r t h e r S t u d y 56 7 . 3 F i n a l R e m a r k s 57 A p p e n d i x A : E x p e r i m e n t a l A p p a r a t u s 58 A p p e n d i x B: W i l c o x o n Rank Sum T e s t f o r P a i r e d E x p e r i m e n t s 63 R e f e r e n c e s 66 i v L i s t o f T a b l e s P a g e T a b l e 5 .1 P r i m a r y D a t a S e t 39 5 . 2 A d d i t i o n a l D a t a S e t 39 T a b l e 6 . 1 C o n f u s i o n T a b l e s f o r T e s t 1 47 6 . 2 D i s t r i b u t i o n o f F a i l u r e s w i t h A n g l e 47 6 . 3 D i s t a n c e t o C l a s s Means f o r O b j e c t 2 48 6 . 4 C o n f u s i o n T a b l e s f o r T e s t 2 48 6 . 5 E c v e r s u s O b j e c t T y p e and D e p t h f o r T e s t 2 50 6 . 6 C o n f u s i o n T a b l e s f o r T e s t 3 50 6 . 7 C o n f u s i o n T a b l e s f o r T e s t 4 52 6 . 8 E c v e r s u s O b j e c t T y p e and D e p t h f o r T e s t 4 52 v L i s t o f F i g u r e s P a g e F i g u r e 2 . 1 S e n s o r g e o m e t r y 6 2 . 2 R e s p o n s e v a r i a t i o n w i t h o b j e c t t y p e . . . . 10 2 . 3 R e s p o n s e v a r i a t i o n w i t h o r i e n t a t i o n . . . . 11 2 . 4 R e s p o n s e v a r i a t i o n w i t h d i s t a n c e 13 2 . 5 O b j e c t s e t 15 F i g u r e 3 . 1 F e a t u r e e x t r a c t i o n • e x a m p l e s 1 8 3 . 2 E s t i m a t e d P m c ( E q u a l a r e a s e g m e n t s ) . . . . 25 3 . 3 E s t i m a t e d P m c ( E q u a l l e n g t h s e g m e n t s ) . . 26 F i g u r e 4 . 1 F e a t u r e v a r i a t i o n w i t h o b j e c t t y p e a n d o r i e n t a t i o n . . . . 2 9 4 . 2 D e s i g n s e t d e f i n i t i o n a n o m a l i e s 3 2 4 . 3 D e s i g n s e t f o r o b j e c t 1 c o m p a r e d t o d a t a f o r o t h e r a n g l e s 3 3 F i g u r e 6 . 1 C l a s s i f i e r p e r f o r m a n c e ( E q u a l a r e a s e g m e n t s ) 4 4 6 . 2 C l a s s i f i e r p e r f o r m a n c e ( E q u a l l e n g t h s e g m e n t s ) 4 5 F i g u r e A . l D a t a c o l l e c t i o n s y s t e m 5 9 A . 2 O v e r v i e w o f t e s t s t a n d a n d c o i l s 6 0 A . 3 D e t a i l o f c o i l s a n d o b j e c t j i g 61 v i A c k n o w l e d g e m e n t I w o u l d l i k e t o e x p r e s s my g r a t i t u d e t o my c o -s u p e r v i s o r s , P r o f e s s o r M a b o R . I t o a n d D r . Y o g a d h i s h D a s , f o r t h e i r i n v a l u a b l e s u g g e s t i o n s a n d s u p p o r t . I am a l s o g r a t e f u l t o my c o l l e a g u e D r . J o h n E . M c F e e f o r h i s c o m m e n t s a n d h e l p f u l d i s c u s s i o n . I w o u l d a l s o l i k e t o t h a n k , f o r t h e i r c o n t r i b u t i o n s , t h e t e c h n i c a l s t a f f o f t h e M i n e s a n d R a n g e C l e a r a n c e G r o u p o f t h e D e f e n c e R e s e a r c h E s t a b l i s h m e n t S u f f i e l d . M o s t e s p e c i a l l y , I w o u l d l i k e t o t h a n k J a c k T o e w s a n d M a r k T u r n b u l l f o r t h e i r a s s i s t a n c e i n t h e d e v e l o p m e n t o f t h e s e n s o r / d a t a a c q u i s i t i o n s y s t e m a n d i n t h e s u b s e q u e n t c o l l e c t i o n o f m e a s u r e m e n t s . T h i s w o r k w a s s u p p o r t e d by t h e D e p a r t m e n t o f N a t i o n a l D e f e n c e , C h i e f R e s e a r c h a n d D e v e l o p m e n t , u n d e r T e c h n i c a l S u b -p r o g r a m 2 7 B . T h e s u p p o r t r e c e i v e d f o r t h i s s t u d y f r o m t h e m a n a g e m e n t o f t h e D e f e n c e R e s e a r c h E s t a b l i s h m e n t S u f f i e l d a n d t h e D e p a r t m e n t o f N a t i o n a l D e f e n c e i s g r a t e f u l l y a c k n o w l e d g e d . v i i 1 C h a p t e r 1: I n t r o d u c t i o n  1.1 P u r p o s e A n e e d e x i s t s i n many f i e l d s t o a c c u r a t e l y l o c a t e a n d i d e n t i f y b u r i e d m e t a l l i c o b j e c t s . T h e s e o b j e c t s m i g h t r a n g e f r o m a r c h e o l o g i c a l a r t i f a c t s t o p i p e l i n e s . The s p e c i f i c n e e d a d d r e s s e d by t h i s w o r k , h o w e v e r , i s t h a t o f l o c a t i n g and i d e n t i f y i n g b u r i e d u n e x p l o d e d a r t i l l e r y s h e l l s . W h e r e v e r an a r e a has been u s e d as an a r t i l l e r y i m p a c t z o n e , a c e r t a i n p e r c e n t a g e (.1-10%) o f s h e l l s i m p a c t i n g do n o t e x p l o d e due t o f u z e f a i l u r e o r o t h e r c a u s e , y e t t h e y r e m a i n a l o n g t e r m e x p l o s i v e h a z a r d p r o h i b i t i n g t h e use o f t h e a r e a f o r o t h e r p u r p o s e s . The p r e s e n t s o l u t i o n t o t h i s p r o b l e m r e l i e s on m a n u a l l y s w e e p i n g t h e a r e a w i t h h a n d - h e l d m e t a l d e t e c t o r s . In i t s e l f t h i s i s a c o s t l y and v e r y t i m e c o n s u m i n g o p e r a t i o n w h i c h i s made w o r s e by t h e i n a d e q u a c i e s o f c o n v e n t i o n a l d e t e c t o r s . C u r r e n t l y a v a i l a b l e d e t e c t o r s do n o t a l l o w a u s e r t o d e t e r m i n e d e p t h and l o c a t i o n a c c u r a t e l y and t h e y p r o v i d e no i n f o r m a t i o n a b o u t t h e s i z e and t y p e o f t h e o b j e c t d e t e c t e d . H e n c e t h e e x p l o s i v e h a z a r d o f any o b j e c t d e t e c t e d i s unknown and r e m o v a l m e t h o d s mus t t h e r e f o r e be b a s e d on t h e g r e a t e s t known h a z a r d . S e v e r a l m e t h o d s t o p r o v i d e a c c u r a t e l o c a t i o n and d e p t h i n f o r m a t i o n o f b u r i e d m e t a l l i c o b j e c t s have been p r o p o s e d [ 8 ,10 ] .The mos t p r o m i s i n g o f t h e s e i n c l u d e : i ) M e a s u r e m e n t and a n a l y s i s o f t h e a n o m a l y i n t h e e a r t h ' s m a g n e t i c f i e l d i n d u c e d by t h e p r e s e n c e o^ a f e r r o u s o b j e c t . 2 i i ) Analys is of the response of a conducting object to a time varying electromagnetic f i e l d . E i t h e r of these methods can be used to l o c a l i z e an object and both approaches have been shown to be capable of providing depth information in the range (<2m) under cons iderat ion for this a p p l i c a t i o n . This thesis addresses the f e a s i b i l i t y of object i d e n t i f i c a t i o n by sui table analysis of i t s electromagnetic response. D irec t analysis of the response to a n a l y t i c a l l y determine the object parameters such as material composition, shape and volume has not proven promising [ 3 ] . Hence this research invest igates the potential of applying pattern recognit ion techniques to the analysis of the response, such that the object might be c l a s s i f i e d from within a known set of poss ible objects . For the purposes of this research the object set was r e s t r i c t e d to a set of steel prolate spheroids of varying shapes and volumes. This object set was chosen to provide .a r e a l i s t i c object shape with respect to a r t i l l e r y shel ls while re ta in ing a readi ly character!"zable geometry. A d d i t i o n a l l y , the responses studied were r e s t r i c t e d to those of the object in a i r , rather than in s o i l . 1.2 Scope The thesis covers the development of an ent ire recogni t ion s tructure , from determination of the potential of the object signatures for meaningful c l a s s i f i c a t i o n through to the development and tes t ing of a prototype c l a s s i f i e r . 3 C h a p t e r two d e s c r i b e s t h e s e n s o r m e t h o d o l o g y and d o c u m e n t s t h e r e s t r i c t i o n s and a s s u m p t i o n s made i n t h e w o r k . I n c l u d e d as w e l l i s a summary o f t h e r e s u l t s o f t h e i n i t i a l i n v e s t i g a t i o n u n d e r t a k e n t o d e t e r m i n e w h e t h e r t h e s i g n a t u r e s c o n t a i n e d s u f f i c i e n t i n f o r m a t i o n t o a l l o w d i s c r i m i n a t i o n b e t w e e n o b j e c t s . The e f f e c t o f c h a n g e s i n o b j e c t o r i e n t a t i o n on ' t h e r e s p o n s e i s a l s o e x a m i n e d . T h e t h i r d c h a p t e r d e s c r i b e s t h e f e a t u r e e x t r a c t i o n t e c h n i q u e s c o n s i d e r e d f o r i m p l e m e n t a t i o n and d e t a i l s a p r e l i m i n a r y i n v e s t i g a t i o n i n t o t h e i r r e l a t i v e e f f e c t i v e n e s s . C h a p t e r f o u r d e s c r i b e s t h e c l a s s i f i e r i m p l e m e n t a t i o n . The e x t e n s i o n s t o c o n v e n t i o n a l n e a r e s t mean v e c t o r t e c h n i q u e s n e c e s s a r y t o a c c o m m o d a t e t h e o r i e n t a t i o n d e p e n d e n c e o f t h e o b j e c t r e s p o n s e s a r e d i s c u s s e d . C h a p t e r f i v e d e s c r i b e s t h e t e s t s u s e d t o e v a l u a t e t h e c l a s s i f i e r and d e t a i l s t h e p e r f o r m a n c e s t a n d a r d u s e d i n e v a l u a t i n g t h e r e s u l t s . The p e r f o r m a n c e o f t h e s y s t e m on a r e p r e s e n t a t i v e t e s t d a t a s e t i s e x a m i n e d i n c h a p t e r s i x , i n c l u d i n g an a n a l y s i s o f t h e f a i l u r e t r e n d s and an e v a l u a t i o n o f t h e s i g n i f i c a n c e o f t h e s e r e s u l t s . C h a p t e r s e v e n e m b o d i e s an o v e r a l l summary and p r e s e n t s s u g g e s t i o n s f o r f u t u r e r e s e a r c h . 4 C h a p t e r 2 : S e n s i n g Method and System Assumptions 2.1 Sensor System While s e v e r a l methods of measuring the e l e c t r o m a g n e t i c r esponse of an o b j e c t e x i s t , the one t h a t has shown the most promise f o r o b j e c t l o c a l i z a t i o n and depth d e t e r m i n a t i o n i n a harsh e n v i r o n m e h t [ 9 ] i s pursued here. T h i s method i s c o n v e n t i o n a l l y known as p u l s e i n d u c t i o n . The p u l s e i n d u c t i o n t e c h n i q u e i s based on the p r i n c i p l e t h a t c o n d u c t i n g o b j e c t s s u b j e c t e d to a stepped magnetic f i e l d d evelop eddy c u r r e n t s w i t h i n them which s u b s e q u e n t l y decay i n a manner determined by the o b j e c t parameters. The decay of these eddy c u r r e n t s e s t a b l i s h e s a secondary magnetic f i e l d which can be sensed by a s i m p l e c o i l near the o b j e c t . The sensor geometry used f o r the measurements i s i l l u s t r a t e d i n F i g u r e 2 . 1 . I t c o n s i s t s of c o - p l a n a r t r a n s m i t and r e c e i v e c o i l s p l a c e d h o r i z o n t a l l y above the t a r g e t o b j e c t at a d i s t a n c e , d, from the c o i l c e n t r e to the c e n t r e of mass of the t a r g e t . The t r a n s m i t c o i l i s d r i v e n by a c u r r e n t p u l s e t r a i n and the r e c e i v e c o i l i s m onitored d u r i n g the t r a n s m i t t e r q u i e s c e n t p e r i o d a f t e r the f a l l of each c u r r e n t p u l s e . R e l e v a n t d e t a i l s of the e x p e r i m e n t a l a p p a r a t u s used to c o l l e c t data f o r t h i s t h e s i s are i n c l u d e d as Appendix A. A t h e o r e t i c a l s o l u t i o n a p p r o x i m a t i n g the step response of such a system e x i s t s f o r s p h e r i c a l o b j e c t s [ 7 ] . The v o l t a g e , v ( t ) , induced i n the r e c e i v e c o i l due to a t r a n s m i t t e r c u r r e n t step i s g i v e n by: 5 2 oo 2 v ( t ) = - - I A k ' e x p ( " t ) ( 2 , 1 ) 2 2 3 / 2 2 2 3 / 2 n _ 1 n 2 (R T+d ) (R +d ) o H 0 u r a a where a - i s the ra d i u s of the o b j e c t a - i s the c o n d u c t i v i t y of the o b j e c t \IQ- i s the p e r m e a b i l i t y of f r e e space u r- i s the r e l a t i v e p e r m e a b i l i t y of the o b j e c t d - c o i l - o b j e c t d i s t a n c e d e f i n e d above t - time from i n s t a n t of f a l l of c u r r e n t pulse C = » N T N R R T 1 2 RT - r a d i u s of the transmit c o i l R - r a d i u s of the r e c e i v e c o i l I - the change i n c u r r e n t i n the transmit c o i l N T - the number of turns i n the transmit c o i l - the number of turns i n the r e c e i v e c o i l 2 6 k \ = D n 2 ( ( u r - l ) ( u r + 2 ) + k n ) and the set of k n are the s o l u t i o n s to the tr a n s c e n d e n t a l equation tan k = k n ( V 1 * n V ( u r-1) As one can see, the response, v ( t ) , i s a sum of weighted damped e x p o n e n t i a l s . The weighting and the c o e f f i c i e n t s are determined by the primary f i e l d change, the Figure 2.1 Sensor geometry. 7 s p h e r e d i m e n s i o n s , a n d t h e s p h e r e m a t e r i a l p r o p e r t i e s . T h e e f f e c t o f v a r i a t i o n s i n a n y o f t h e s e p a r a m e t e r s c a n be q u i t e c o m p l e x ; h o w e v e r , s o m e g e n e r a l o b s e r v a t i o n s a r e i n o r d e r . T h e e f f e c t o f a c h a n g e i n t h e p r i m a r y f i e l d a m p l i t u d e m o d i f i e s o n l y t h e o v e r a l l c o n s t a n t t e r m C . H e n c e , v a r i a t i o n s f r o m t h i s s o u r c e c a n be e l i m i n a t e d by m o n i t o r i n g t h e t r a n s m i t t e r c u r r e n t a n d a p p l y i n g a m u l t i p l i c a t i v e c o r r e c t i o n t o t h e o b s e r v e d r e s p o n s e . T h e e f f e c t s a s s o c i a t e d w i t h o b j e c t d i m e n s i o n a r e m o r e c o m p l e x . A n i n c r e a s e i n r a d i u s b o t h i n c r e a s e s t h e i n i t i a l a m p l i t u d e o f t h e r e s p o n s e t e r m s a n d l e n g t h e n s t h e i r e x t e n t b y i n c r e a s i n g t h e t i m e c o n s t a n t o f t h e t e r m s . A n i n c r e a s e i n c o n d u c t i v i t y w o u l d be e x p e c t e d t o r e d u c e t h e i n i t i a l a m p l i t u d e s o f t h e e x p o n e n t i a l t e r m s , w h i l e a g a i n i n c r e a s i n g t h e t i m e c o n s t a n t s . T h e r e s p o n s e s i g n a t u r e i s a v e r y c o m p l e x f u n c t i o n o f t h e p e r m e a b i l i t y o f t h e o b j e c t a n d t h e e f f e c t o f a c h a n g e c a n n o t be s i m p l y d e s c r i b e d . H o w e v e r i n g e n e r a l , t h e r e s p o n s e f o r a n o n - p e r m e a b l e o b j e c t i s c h a r a c t e r i z e d by e a c h t e r m i n t h e r e s p o n s e h a v i n g e q u a l i n i t i a l a m p l i t u d e , b u t t h e h i g h e r o r d e r t e r m s h a v i n g p r o g r e s s i v e l y s h o r t e r t i m e c o n s t a n t s . F o r p e r m e a b l e o b j e c t s t h e t i m e c o n s t a n t s a l s o r e d u c e a s t h e o r d e r i n c r e a s e s ; h o w e v e r i n t h i s c a s e , t h e i n i t i a l a m p l i t u d e s o f t h e t e r m s i n c r e a s e . S u p e r f i c i a l l y , i t w o u l d a p p e a r t h a t t h i s m o d e l m i g h t a l l o w a n a l y s i s o f t h e s e n s o r s i g n a t u r e s t o d i r e c t l y d e r i v e a l l r e l e v a n t p a r a m e t e r s d e t a i l i n g t h e o b j e c t s i z e a n d m a t e r i a l p r o p e r t i e s . U n f o r t u n a t e l y o t h e r r e s e a r c h [ 3 ] h a s s h o w n t h a t t h i s a p p r o a c h b e a r s l i t t l e p r o m i s e , p r i m a r i l y d u e 8 t o t h e n o i s e s e n s i t i v i t y o f t h e a l g o r i t h m s u s e d t o e x t r a c t t h e s e p a r a m e t e r s f r o m t h e r e s p o n s e s i g n a t u r e . 2 . 2 I n i t i a l I n v e s t i g a t i o n s The a b o v e mode l a l l o w s one t o p r e d i c t t h e p u l s e i n d u c t i o n r e s p o n s e o f a s p h e r i c a l o b j e c t . H o w e v e r , no e q u i v a l e n t mode l e x i s t s f o r o b j e c t s o f a more c o m p l e x g e o m e t r y . T h e r e f o r e a p r e l i m i n a r y e x p e r i m e n t a l s t u d y was u n d e r t a k e n t o d e l i n e a t e t h e g e n e r a l t r e n d s i n b e h a v i o r o f p u l s e i n d u c t i o n s i g n a t u r e s f o r o b j e c t s w i t h p r o l a t e s p h e r o i d a l s h a p e . T h i s s t u d y e x a m i n e d c h a n g e s i n o b j e c t r e s p o n s e w i t h r e s p e c t t o : i ) C h a n g e o f o b j e c t s h a p e o r m a t e r i a l p r o p e r t i e s , ( o b j e c t s c o n s i d e r e d w e r e a l l p r o l a t e s p e r o i d s whose s h a p e was d e f i n e d by t h e i r m i n o r r a d i u s A and t h e a s p e c t r a t i o E) i i ) C h a n g e s i n t h e o r i e n t a t i o n o f an o b j e c t i n t h e v e r t i c a l p l a n e (6,F i g u r e 2 . 1 ) . i i i ) C h a n g e s i n t h e d e p t h o f an o b j e c t ( d , f i g u r e 2 . 1 ) . F rom t h e s y m m e t r y o f t h e s e n s o r s y s t e m , one can s e e t h a t no o r i e n t a t i o n d e p e n d e n c e e x i s t s f o r r o t a t i o n s o f t h e o b j e c t i n t h e h o r i z o n t a l p l a n e . A d d i t i o n a l l y t h e s y m m e t r y o f t h e o b j e c t s s t u d i e d a l l o w s one t o c o n s t r a i n t h e v e r t i c a l a n g l e t o a r a n g e o f 0 t o 9 0 ° . T h e s e i n v e s t i g a t i o n s w e r e l a r g e l y q u a l i t a t i v e a n d p a r a m e t e r i z e d t h e r e s p o n s e s o l e l y by i t s m a g n i t u d e and f o r m . C h a n g e s i n f o r m w e r e a s s e s s e d a f t e r t h e s i g n a t u r e s w e r e n o r m a l i z e d t o an c o n s t a n t RMS v a l u e . 9 The r e s u l t s o f t h e s e p r e l i m i n a r y i n v e s t i g a t i o n s w e r e as f o l l o w s : i ) The c h a n g e o f an o b j e c t t y p e ( i . e . s h a p e o r m a t e r i a l ) , g i v e n t h e same d e p t h and o r i e n t a t i o n , c h a n g e d t h e r e s p o n s e s i g n a t u r e b o t h w i t h r e s p e c t t o i t s o v e r a l l a m p l i t u d e and i t s f o r m . As one w o u l d e x p e c t f r o m t h e s p h e r e m o d e l , t h e a m p l i t u d e o f t h e s i g n a t u r e i n c r e a s e d w i t h t h e s i z e o f t h e t a r g e t ( m a t e r i a l p a r a m e t e r s h e l d c o n s t a n t ) and was s i g n i f i c a n t l y h i g h e r f o r s t e e l o b j e c t s t h a n f o r a l u m i n u m o b j e c t s o f s i m i l a r s h a p e . F i g u r e 2 . 2 shows t h e s i g n a t u r e s o f t h r e e o b j e c t s (2 s t e e l , 1 a l u m i n u m ) a t t h e same d e p t h and o r i e n t a t i o n t o d e m o n s t r a t e t h e e x t e n t o f t h e s e c h a n g e s . i i ) C h a n g e s i n t h e o r i e n t a t i o n o f an o b j e c t p r o d u c e d v a r i a t i o n s i n t h e r e s p o n s e s i g n a t u r e s as w e l l . F o r a l u m i n u m o b j e c t s t h e c h a n g e s w e r e l a r g e l y c o n f i n e d t o v a r i a t i o n s i n t h e a m p l i t u d e o f t h e w a v e f o r m w i t h r e l a t i v e l y i n s i g n i f i c a n t c h a n g e s i n t h e w a v e f o r m s h a p e . S t e e l o b j e c t s h o w e v e r , e x h i b i t e d s i g n i f i c a n t r e s p o n s e v a r i a t i o n s b o t h i n r e s p e c t t o a m p l i t u d e and f o r m . F i g u r e 2 . 3 shows t h i s e f f e c t by c o m p a r i n g t h e r e s p o n s e s o f a s i n g l e o b j e c t (made o f s t e e l ) , a t c o n s t a n t d e p t h , f o r t h r e e d i f f e r e n t o r i e n t a t i o n s i n t h e v e r t i c a l p l a n e ( 6 = 0 , 4 5 , 9 0 ° ) . i i i ) C h a n g e s i n t h e d e p t h o f t h e o b j e c t s p r o d u c e d an i n t e r e s t i n g and p o t e n t i a l l y b e n e f i c i a l o b s e r v a t i o n . N a m e l y , t h a t w h i l e a c h a n g e i n d e p t h o o oo co o Q_ CO UJ cc: o o CM 0 0 STEEL A - 3 3 M M E - 3 . 6 9 0 0 EG 40 CU STEEL A - 5 9 M M E - 2 . 0 90 DEG 40 CM ALUMINUM A - 3 3 M M E - 3 . 6 90 DEG 40 CM o J 200 400 600 TIME MICRO SECS 800 1000 Figure 2.2: Response variation with object. Depth and orientation held constant. o _j , 1 1 1 1 1 1 1 200 400 600 800 1000 TIME MICRO SECS Figure 2.3: Response variation with orientation. Normalized response with distance held constant. 12 a f f e c t e d t h e o v e r a l l a m p l i t u d e o f a r e s p o n s e , i t had m i n i m a l e f f e c t on t h e f o r m . As F i g u r e 2 . 4 s h o w s , t h e m a g n i t u d e o f v a r i a t i o n i n t h e n o r m a l i z e d o b j e c t r e s p o n s e s , d i f f e r i n g o n l y i n s e n s o r - o b j e c t d i s t a n c e , i s v e r y s l i g h t and i s c l o s e t o b e i n g masked by t h e n o i s e i n h e r e n t i n t h e m e a s u r e m e n t t a k e n a t t h e l a r g e r d i s t a n c e . The f i g u r e shown i s c h a r a c t e r i s t i c o f t h e r e s u l t s o b s e r v e d f o r a l l o b j e c t s s t u d i e d . T h e s e i n v e s t i g a t i o n s i n d i c a t e t h a t t h e r e a r e s i g n i f i c a n t d i f f e r e n c e s b e t w e e n t h e r e s p o n s e s o f d i s p a r a t e o b j e c t s a t t h e same o r i e n t a t i o n . W h i l e t h i s was n o t c o n c l u s i v e p r o o f t h a t s e p a r a t i o n o f t h e r e p o n s e o f two o b j e c t s was p o s s i b l e a t any two o r i e n t a t i o n s , some p r o m i s e c o u l d be h e l d f o r a me thod t o be d e v e l o p e d w h i c h w o u l d p r o v i d e o b j e c t t y p e i d e n t i f i c a t i o n . T h a t i s , by s u i t a b l e i n t e r p r e t a t i o n o f t h e p u l s e i n d u c t i o n r e s p o n s e f r o m an unknown o b j e c t , i t s m e m b e r s h i p f r o m a known s e t c o u l d be d e t e r m i n e d . 2 . 3 A s s u m p t i o n s and R e s t r i c t i o n s T h r o u g h o u t t h i s work c e r t a i n a s s u m p t i o n s have b e e n made t o l i m i t t h e c o m p l e x i t y o f t h e p r o b l e m t o be c o n s i d e r e d . T h e s e a s s u m p t i o n s a r e as f o l l o w s : i ) T h e r e i s no e r r o r i n s e n s o r p o s i t i o n i n g ( i . e . t h e c o i l a x i s i n t e r s e c t s t h e o b j e c t ' s c e n t r e o f m a s s ) . T h i s i s n o t u n r e a s o n a b l e as one c a n s h o w : t h a t , i n t h e s p h e r e c a s e , t h e maximum a m p l i t u d e r e s p o n s e w i l l o c c u r i n t h i s p o s i t i o n . o , 1 1 1 1 1 1 1 200 400 600 800 1000 TIME MICRO SECS Figure 2.4: Response variation with distance. Normalized response with orientation held constant. 14 i i ) No e x t r a n e o u s m e t a l l i c c l u t t e r i s p r e s e n t ; i . e . no s h e l l f r a g m e n t s o r o t h e r m e t a l o b j e c t s a r e n e a r b y , and h e n c e t h e r e s p o n s e m e a s u r e d i s t h a t o f t h e o b j e c t i n i s o l a t i o n . I t i s a s s u m e d t h a t i n an o p e r a t i o n a l e n v i r o n m e n t , s u r f a c e c l u t t e r , i n c l u d i n g mos t s h e l l f r a g m e n t s , w o u l d be r e m o v e d b e f o r e i d e n t i f i c a t i o n i s a t t e m p t e d . i i i ) O n l y a l i m i t e d number o f t a r g e t s a r e p o s s i b l e , a l t h o u g h t h e i r o r i e n t a t i o n s a r e i n d e t e r m i n a t e . T h i s i s c o n s i s t e n t w i t h t h e p r o b l e m s t a t e m e n t i i i t h a t a r t i l l e r y s h e l l s a r e l i m i t e d i n t h e i r v a r i e t y . S p e c i f i c a l l y , t h e s e t o f t a r g e t s u s e d c o m p r i s e s f o u r o b j e c t s . A l l t h e o b j e c t s a r e p r o l a t e s p h e r o i d s made o f m i l d s t e e l . The o b j e c t s d i f f e r i n t h e i r m i n o r r a d i i d i m e n s i o n (A) and t h e i r a s p e c t r a t i o ( E = B / A , B - m a j o r r a d i i ) . The o b j e c t s used are shown i n f i g u r e 2 . 5 . i v ) C l a s s i f i c a t i o n v a l i d i t y i s j u d g e d o n l y on r e c o g n i t i o n o f t a r g e t t y p e and n o t on d e t e r m i n a t i o n o f i t s o r i e n t a t i o n . T h i s m e r e l y r e f l e c t s t h e f a c t t h a t t a r g e t o r i e n t a t i o n i s n o t a m a j o r c o n c e r n . v) A d d i t i o n a l l y , and p e r h a p s mos t i m p o r t a n t l y , i t i s a s s u m e d t h a t t h e unknown s i g n a t u r e s u s e d t o t e s t t h e s y s t e m c a n d e r i v e o n l y f r o m t h e known s e t o f o b j e c t s . T h a t i s , r e j e c t i o n o f unknown t a r g e t s i s n o t a r e q u i s i t e r e s p o n s e f o r t h e s y s t e m . T h i s a s s u m p t i o n d e r i v e s f r o m t h e p r o h i b i t i v e r i s k a s s o c i a t e d w i t h m i s c i a s s i f y i n g an e x p l o s i v e h a z a r d as a n o n - h a z a r d o u s o b j e c t . A \J STEEL A - 6 0 M M E - 4 . 3 e STEEL A - 3 3 M M E - 3 . 6 STEEL A - 3 7 M M E - 7 . 1 0 STEEL A - 5 9 M U E - 2.0 Figure 2 .5 : Object set . 16 Chapter 3: Feature Extraction 3.1 Feature Types As in a l l pattern recognition problems, the prerequisites for feature selection comprise compression of the data to the fewest possible features while retaining important discrimatory information. While i t would be possible to consider the sampled response signature as a feature vector in i t s e l f , the computational and memory size requirements of that approach are p r o h i b i t i v e . Hence, some algorithm to compress the response is required. The methodology chosen for feature extraction in this thesis is a straightforward one, extending the approach used by Das et al [ 4 ] . The technique consists of normalizing the response signature to a constant RMS value, segmenting the response into k time segments and developing a single feature element for each segment. The time sequence of these elements then comprise the feature vector. Normalization of the overall response is required in this application as the amplitude of the signal is much more a function of the depth of the object than i t s type. Small offsets in depth, which one cannot expect accurate a p r i o r i knowledge of, would otherwise swamp the r e l a t i v e l y small response variations associated with differences in the object type. The original work with this concept used equal length segments and established as the feature element the time constant of a single exponential term f i t t e d to the data i 17 w i t h i n t h e s e g m e n t . T h i s p r o d u c e d e n c o u r a g i n g r e s u l t s on s i m u l a t e d d a t a ( u s i n g t h e s p h e r e m o d e l ) f o r c a s e s o f d i f f e r i n g r a d i i and m a t e r i a l p r o p e r t i e s . E x t e n s i o n s o f t h i s s c h e m e w e r e h e n c e i m p l e m e n t e d f o r t h i s t h e s i s . The v a r i a t i o n s t e s t e d i n c l u d e d c h a n g e s i n b o t h t h e s e g m e n t a t i o n scheme arid t h e f e a t u r e d e v e l o p e d f o r t h e i n d i v i d u a l s e g m e n t . F o u r f e a t u r e t y p e s w e r e c o n s i d e r e d , t h e s e w e r e : F l - The t i m e c o n s t a n t o f a s i n g l e d e c a y i n g e x p o n e n t i a l t e r m f i t t e d t o t h e d a t a w i t h i n t h e s e g m e n t . F2 - The s l o p e o f a l i n e s e g m e n t f i t t e d t o t h e s e g m e n t d a t a . F3 - The mean v a l u e o f t h e s e g m e n t . F4 - The d i f f e r e n c e b e t w e e n t h e mean v a l u e s o f s u c c e s s i v e s e g m e n t s , t h e l a s t f e a t u r e e l e m e n t b e i n g t h e mean o f t h a t s e g m e n t i t s e l f . The f i r s t two f e a t u r e t y p e s ( F 1 . F 2 ) a r e c h o s e n s i n c e one w o u l d e x p e c t t h e s e f e a t u r e s t o a c c u r a t e l y c h a r a c t e r i z e t h e f o r m o f t h e r e s p o n s e f o r s i g n a t u r e s o f t h i s t y p e . T h e i r a b i l i t y t o c h a r a c t e r i z e t h e s i g n a l i s shown i n F i g u r e s 3 .1 a & b, w h e r e f i t s a s s o c i a t e d w i t h t h e s e two f e a t u r e s a r e o v e r l a i d on t h e d a t a u s e d t o d e v e l o p t h e m . H o w e v e r , t h e s e two f e a t u r e e x t r a c t i o n t e c h n i q u e s s u f f e r f r o m t h e p r o s p e c t o f p o o r p e r f o r m a n c e i n h i g h n o i s e c a s e s due t o t h e l i m i t a t i o n s i n t h e f i t t i n g t e c h n i q u e s . T h i s i s e s p e c i a l l y t r u e when t h e o o o rO W g § s Q . CO CD I - Feature • - Data points o o o KJ too c ° aT co 0) a: o o o _ to a) too rO OL CO CD 250 500 750 Time (microseconds) (a) exponential tit (F1) 1000 O o R -• CD Q . co CD 1 1 1 250 500 750 Time (microseconds) (c) segment mean (F3) 250 Time 500 750 (microseconds) (b) linear fit (F2) 1000 , 1 1 1000 0 250 500 750 Time (microseconds) . (d) mean difference (F4) Figure 3-1: Feature extraction examples. Data is from object 1 at d-60 cm.,0=45. Three equal length segments are used. 1000 00 19 number o f p o i n t s u s e d t o e s t i m a t e t h e f e a t u r e i s s m a l l , as w o u l d be t h e c a s e i f many s e g m e n t s a r e u s e d t o d e v e l o p t h e f e a t u r e v e c t o r . The l a s t two f e a t u r e t y p e s ( F 3 , F 4 ) c a n be c o n s i d e r e d a s ' e s s e n t i a l l y r e d u c e d s a m p l e v e r s i o n s o f t h e r e s p o n s e ( F i g u r e s 3 . 1 c & d ) . The l a s t t y p e (mean d i f f e r e n c e , F 4 ) e m p h a s i z e s t h e c h a n g e s i n r e s p o n s e f r o m s e g m e n t t o s e g m e n t r a t h e r t h a n t h e r e s p o n s e i t s e l f . T h e s e two f e a t u r e t y p e s a r e i n h e r e n t l y n o i s e i n s e n s i t i v e and v e r y e a s y t o i m p l e m e n t , b u t c a n n o t be e x p e c t e d t o c h a r a c t e r i z e t h e r e s p o n s e w e l l f o r t h e c a s e o f few s e g m e n t s . E a c h o f t h e s e f e a t u r e t y p e s was c o n s i d e r e d f o r n u m b e r s o f s e g m e n t s i n t h e r a n g e o f two t o f i f t e e n u n d e r two s e g m e n t a t i o n p l a n s . The f i r s t s e g m e n t a t i o n p l a n u s e d e q u a l l e n g t h s e g m e n t s ; t h e s e c o n d b a s e d t h e s e g m e n t l e n g t h on a c h i e v i n g e q u a l s e g m e n t a r e a s . T h i s s e c o n d s e g m e n t a t i o n p l a n i s d e s i g n e d s u c h t h a t t h e s i g n a l t o n o i s e o f t h e m e a n - b a s e d f e a t u r e s ( F 3 , F 4 ) w o u l d be r o u g h l y c o n s t a n t i n a l l c o m p o n e n t s o f t h e f e a t u r e s p a c e . The s e g m e n t l i m i t s r e q u i r e d t o a c h i e v e e q u a l a r e a s e g m e n t s a r e d e p e n d e n t on t h e o b s e r v e d r e s p o n s e a n d h e n c e t h e s e c o n d s e g m e n t a t i o n p l a n a s s u m e d a r e s p o n s e a v e r a g e d o v e r t h e e n t i r e d a t a s e t i n c a l c u l a t i n g t h e s e g m e n t a t i o n r e q u i r e d . T h i s s e g m e n t a t i o n p l a n i s t h e n f i x e d f o r t h e c h o s e n number o f s e g m e n t s and u s e d t o p r o c e s s b o t h t h e d e s i g n and t e s t s e t . 3 . 2 F e a t u r e E v a l u a t i o n W h i l e i t w o u l d be c o n c e i v a b l e t o t a k e e v e r y c o m b i n a t i o n o f t h e s e f e a t u r e e x t r a c t i o n t e c h n i q u e s and s u b m i t 2 0 t h e m t o t h e c l a s s i f i e r t o d e t e r m i n e t h e i r r e l a t i v e e f f e c t i v e n e s s , t h a t w o u l d be a v e r y t i m e c o n s u m i n g e x e r c i s e . T o a v o i d t h i s , a p r e l i m i n a r y e v a l u a t i o n o f t h e f e a t u r e s w a s u n d e r t a k e n . F o r t h i s e v a l u a t i o n e a c h c o m b i n a t i o n o f f e a t u r e e x t r a c t i o n t e c h n i q u e a n d s e g m e n t a t i o n w a s a p p l i e d t o a t y p i c a l d e s i g n d a t a s e t . A d e s i g n f e a t u r e s e t ( c o m p r i s i n g t h e c l a s s m e a n s ) a n d an e s t i m a t e o f t h e v a r i a n c e a t e a c h p o i n t i n t h e d e s i g n s e t w a s d e v e l o p e d f o r e a c h c o m b i n a t i o n . T h e s e v a r i o u s d e s i g n f e a t u r e s e t s w e r e t h e n e v a l u a t e d u s i n g a p e r f o r m a n c e m e a s u r e ( d i s c u s s e d l a t e r ) t o e s t a b l i s h w h a t t r e n d s i n c l a s s i f i e r p e r f o r m a n c e c o u l d be e x p e c t e d w i t h r e s p e c t t o c h a n g e s i n : i ) t h e n u m b e r o f s e g m e n t s u s e d . / i i ) t h e t y p e o f f e a t u r e e x t r a c t e d . i i i ) t h e t y p e o f s e g m e n t a t i o n u s e d . T h e d e s i g n d a t a s e t u s e d w a s d e v e l o p e d by s e p a r a t e l y a v e r a g i n g a l l d a t a r e c o r d s a v a i l a b l e f o r e a c h c o m b i n a t i o n o f o b j e c t a n d o r i e n t a t i o n . T h e r e s u l t i n g s e t o f r e c o r d s w e r e t h e n c o n s i d e r e d n o i s e - f r e e p r o t o t y p e s o f o b j e c t r e s p o n s e s f o r e a c h o b j e c t a n d e a c h o r i e n t a t i o n . E a c h r e c o r d i n t h i s s e t i s t h e n r e p l i c a t e d 20 t i m e s w i t h t h e a d d i t i o n o f w h i t e G a u s s i a n n o i s e ( A W G N ) s u c h t h a t t h e s i g n a l t o n o i s e r a t i o i s 7 5 . T h i s p r o c e s s y i e l d s a s i m u l a t e d n o i s y d a t a s e t w i t h c o n s t a n t l e v e l n o i s e w i t h t h e n o i s e l e v e l b e i n g c o m p a r a b l e t o t h a t o f t h e w o r s t c a s e f o r w h i c h c l a s s i f i c a t i o n w o u l d be a t t e m p t e d . T h e f e a t u r e e x t r a c t o r u n d e r c o n s i d e r a t i o n i s t h e n a p p l i e d t o t h i s d a t a s e t a n d t h e c l a s s m e a n s m i 5 j ( c l a s s i , o r i e n t a t i o n j ) , a n d v a r i a n c e v e c t o r s j h * , j i n t h e f e a t u r e s p a c e a r e 21 e s t i m a t e d . The c l a s s means t h u s c o m p u t e d f o r m t h e p o i n t s o f t h e d e s i g n s e t f o r t h e e v a l u a t i o n . The use o f t h i s a r t i f i c i a l d a t a s e t a l l o w s one t o make two a s s u m p t i o n s f o r t h e f o l l o w i n g a n a l y s i s . T h e s e a r e : i ) The f e a t u r e s w i l l e x h i b i t no c o v a r i a n c e ( f o l l o w s f r o m AWGN). i i ) The v a r i a n c e v e c t o r s ^ i , j w i l l b e r o u g h l y e q u a l ( f o l l o w s f r o m c o n s i s t e n t S / N ) . The p e r f o r m a n c e m e a s u r e ( ' P m c ) u s e d i s an e s t i m a t e o f t h e p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n (Pmc) a s s o c i a t e d w i t h a p a r t i c u l a r d e s i g n s e t i n t h e c o n t e x t o f a s i m p l e n e a r e s t - m e a n - v e c t o r c l a s s i f i e r . I t s h o u l d be n o t e d t h a t f o r t h i s a p p l i c a t i o n , a m i s c i a s s i f i c a t i o n e v e n t (mc) o c c u r s when a f e a t u r e s p a c e p o i n t b e l o n g i n g t o c l a s s i a t o r i e n t a t i o n j i s c l a s s i f i e d as c l a s s k ; c l a s s i f i c a t i o n as c l a s s i a t an o r i e n t a t i o n o t h e r t h a n j i s n o t d e f i n e d as an e r r o r . The p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n f o r t h e d e s i g n s e t u n d e r t e s t ( a n d h e n c e t h e f e a t u r e t y p e and s e g m e n t a t i o n ) i s t h e n d e r i v e d f r o m t h e p r o b a b i l i t y o f m i s c i a s s i f y i n g any p o i n t b e l o n g i n g t o c l a s s i a t o r i e n t a t i o n j , P ( m c / i , j ) , as f o l 1 o w s : Pmc = I P ( m c / i ) * P ( i ) ( 3 . 1 ) i w h e r e P ( m c / i ) = I P ( m c / i , j ) * P ( j ) j 22 L a c k i n g a b a s i s f o r any o t h e r a s s u m p t i o n , i t i s a s s u m e d t h a t a l l c l a s s and o r i e n t a t i o n s a r e e q u i p r o b a b l e . E q u a t i o n 3 , 1 t h e n r e d u c e s t o : Pmc = 1 * z s P ( m c / i , j ) ( 3 . 2 ) n c * n 0 i j w h e r e n c - number o f c l a s s e s v j ne - number o f o r i e n t a t i o n s / c l a s s D e t e r m i n i n g t h e p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n P ( m c / i , j ) f o r e a c h p o i n t ( i , j ) i s n o t s t r a i g h t f o r w a r d f o r a c l a s s i f i e r i n v o l v i n g more t h a n two c l a s s e s . H o w e v e r , i t i s p o s s i b l e t o d e t e r m i n e an u p p e r l i m i t f r o m t h e f o l l o w i n g r e l a t i o n : P ( m c / i , j ) < I P ( d e c ' n = k / i , j ) ( 3 . 3 ) w h e r e P ( d e c ' n = k / i , j ) i s t h e p r o b a b i l i t y t h a t a member o f t h e c l a s s c e n t r e d a t t h e p o i n t ( i , j ) i s c l a s s i f i e d as b e l o n g i n g t o a n o t h e r c l a s s k i n a two c l a s s c a s e ; i . e . w h e r e m i s c i a s s i f i c a t i o n as c l a s s k i s t h e o n l y p o s s i b l e e r r o r . A v a l u e f o r t h i s p r o b a b i l i t y ( P ( d e c ' n = k / i , j )) i s n o t a v a i l a b l e w i t h o u t r e s o r t i n g t o a v e r y e x t e n s i v e c o m p u t a t i o n a l m e t h o d . H e n c e an a s s u m p t i o n i s made a b o u t t h e m a g n i t u d e o f t h i s p r o b a b i l i t y . I t i s a s s u m e d t h a t P ( d e c ' n = k / i , j ) c a n be a p p r o x i m a t e d by P ( d e c ' n = k , 1 / i , j ) w h e r e t h e p o i n t ( k , l ) i s t h e n e a r e s t p o i n t o f t h e c l a s s k t o t h e p o i n t ( i , j ) f o r any 1. I t s h o u l d be n o t e d t h a t P ( d e c ' n = k , 1 / i , j ) , i n t h e g e n e r a l c a s e , i s o n l y one c o m p o n e n t o f P ( d e c 1 n = k / i , j ) , a l b e i t t h e l a r g e s t . H e n c e t h e r e s u l t a n t v a l u e P m c ( E q n 3 . 7 ) i s o n l y an 23 a p p r o x i m a t i o n t o t h e p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n . T h u s , A P m c s h o u l d be c o n s i d e r e d o n l y as a r e l a t i v e p e r f o r m a n c e m e a s u r e , f r o m one m e t h o d t o a n o t h e r , r a t h e r t h a n an a b s o l u t e e s t i m a t e o f t h e p e r f o r m a n c e . G i v e n t h i s a s s u m p t i o n , c a l c u l a t i o n o f t h i s two c l a s s p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n i s s t r a i g h t f o r w a r d u n d e r t h e a s s u m p t i o n s d e r i v i n g f r o m t h e c o n s t r u c t i o n o f t h e d a t a s e t (AWGN, e q u a l v a r i a n c e v e c t o r s ) [ 1 5 ] . I t i n v o l v e s m e r e l y t h e d e t e r m i n a t i o n o f t h e w e i g h t e d d i s t a n c e ( d i , j , k ^ b e t w e e n t h e p o i n t i n q u e s t i o n ( m i , j ) and t h e n e a r e s t d e s i g n s e t p o i n t o f a n o t h e r c l a s s ( m | c } i ) . mi n i , j , k 1 m. , - m. - k , 1 - i , J s . . ' ( 3 . 4 ) T h e n P ( d e c ' n = k / i , j ) « P ( Z > d . . . / 2 ) 1 , J , K ( 3 . 5 ) w h e r e Z i s g o v e r n e d by a n o r m a l d i s t r i b u t i o n , mean=0 , v a r i a n c e - ! F o r t h i s c a s e t h e n P ( m c / i , j ) - I P ( Z > d . , . / 2 ) k * i 1 , J ' K ( 3 . 6 ) a n d P. ITIC * I I I P ( Z > d . . . / 2 ) ( 3 . 7 ) n c x n Q i j k v i 24 The r e s u l t s o f t h i s e v a l u a t i o n a r e p r e s e n t e d i n F i g u r e s 3 . 2 and 3 . 3 . The p e r f o r m a n c e t r e n d s c o n f i r m i n t u i t i v e e x p e c t a t i o n s . The m e a n - b a s e d f e a t u r e t y p e s ( F 3 , F 4 ) g e n e r a l l y p e r f o r m b e t t e r as t h e number o f s e g m e n t s a r e i n c r e a s e d and t h e y r e p r e s e n t t h e s i g n a l b e t t e r . T h i s t r e n d b r e a k s down f o r t h e mean d i f f e r e n c e f e a t u r e t y p e ( F 4 , F i g u r e s 3 . 2 ( d ) , 3 . 3 ( d ) ) a t l a r g e n u m b e r s o f s e g m e n t s s i n c e f o r t h e s e c a s e s t h e n o i s e r e d u c t i o n a s s o c i a t e d w i t h c o m p u t i n g t h e mean i s d e c r e a s e d . The F4 f e a t u r e t y p e i s more s e n s i t i v e t o t h i s e f f e c t due t o t h e i n c r e a s e i n e r r o r a s s o c i a t e d w i t h t h e d i f f e r e n c e o p e r a t i o n . The f e a t u r e t y p e s b a s e d on f i t t i n g t h e d a t a ( F 1 . F 2 ) e x h i b i t e d a d e g r a d a t i o n i n p e r f o r m a n c e as t h e s e g m e n t l e n g t h d e c r e a s e d . T h i s i s due t o t h e i n c r e a s e i n u n c e r t a i n t y i n e s t i m a t i n g t h e f e a t u r e s as t h e number o f p o i n t s i n t h e s e g m e n t d e c r e a s e s . No d e f i n i t i v e c o n c l u s i o n s c a n be made a t t h i s p o i n t on t h e r e s u l t s o f t h e s e t e s t s w i t h r e p e c t t o t h e r e l a t i v e p e r f o r m a n c e o f t h e two s e g m e n t a t i o n p l a n s , a l t h o u g h t h e one b a s e d on e q u a l a r e a a p p e a r s t o p r o v i d e r e s u l t s t h a t a r e somewha t s u p e r i o r f o r t h e m a j o r i t y o f c a s e s . o IO <0J o J 3 6 9 12 15 Number of segments (a) segment exponential time constant (F1) 3 6 9 12 Number of segments (b) segment linear slope (F2) — i 15 8 i No J — t M O ( 0 - o j I * ' t—'—•—1—*—•—r-3 6 9 12 Number of segments (c) segment mean (F3) 15 —I • • — l — • — ' — i r -3 6 9 12 Number of segments 15 (d) mean difference (F4) Figure 3.2: Estimated probability of misclass'rfication ( P m c ) for equal area segmentation. o to o to <0J O J *?0 o J 3 6 9 1 2 1 5 Number of segments (a) segment exponential time constant (F1) S i <0J o J - ? — • — • — I — • — ' — 1 — ' — ' — I — 3 6 9 12 Number of segments (c) segment mean (F3) 1 5 8 l <0J O J 3 6 9 1 2 Number of segments (b) segment linear slope (F2) —i 1 5 i » ' I ' i ' i 3 6 9 1 2 Number of segments (d) mean difference (F4) 1 5 A. Figure 3.3: Estimated probability of misclassification (P m c ) for equal length segmentation. 27 C h a p t e r 4: C l a s s i f i e r d e s i g n 4 . 1 C l a s s i f i e r T y p e The g o a l o f any p a t t e r n r e c o g n i t i o n s y s t e m i s t o e x t r a c t m e a n i n g f u l i n f o r m a t i o n . The i n f o r m a t i o n t h a t we r e q u i r e i n t h i s a p p l i c a t i o n i s t h e t y p e o f t h e o b j e c t , i n o r d e r t o a l l o w one t o d e a l w i t h t h e o b j e c t i n an a p p r o p r i a t e m a n n e r . The o r i e n t a t i o n o f t h e o b j e c t , w h i l e o f p e r i p h e r a l i n t e r e s t , i s n o t an i m p o r t a n t p a r a m e t e r f o r t h i s a p p l i c a t i o n . As n o t e d p r e v i o u s l y , t h e r e s p o n s e s i g n a t u r e s c h a n g e w i t h r e s p e c t t o c h a n g e s i n b o t h t h e o b j e c t t y p e and o r i e n t a t i o n . T h i s b e h a v i o r c a r r i e s o v e r i n t o t h e f e a t u r e s p a c e r e p r e s e n t a t i o n as w e l l . In some c o n t e x t s w h e r e one m i g h t r e q u i r e b o t h t h e t y p e and o r i e n t a t i o n i n f o r m a t i o n , t h i s v a r i a t i o n c o u l d be b e n i f i c i a l . H o w e v e r , t h i s i s n o t so f o r t h i s a p p l i c a t i o n . One i s f a c e d w i t h t h e p r o b l e m t h a t , e v e n i n t h e a b s e n c e o f n o i s e , t h e s i g n a t u r e s o f any c l a s s c o m p r i s e a c o n t i n u u m o f p o i n t s i n t h e f e a t u r e s p a c e c o r r e s p o n d i n g t o t h e i n f i n i t e v a r i a b i l i t y i n t h e o r i e n t a t i o n o f t h a t o b j e c t . A c o m p l e t e f e a t u r e d e s i g n s e t f o r a g i v e n o b j e c t w o u l d h e n c e f o r m a l o c u s t h r o u g h t h e f e a t u r e s p a c e r e p r e s e n t i n g t h e r e s p o n s e o f t h a t o b j e c t a t a l l o r i e n t a t i o n s i n a v e r t i c a l p l a n e ( 6 = 0 . , 9 0 ° ) . S i n c e one c a n n o t a c q u i r e an i n f i n i t e number o f p o i n t s t o d e f i n e t h i s t r a c k , one mus t c o n s i d e r a p p r o x i m a t i n g i t by some m e t h o d . One p o t e n t i a l s o l u t i o n i s t o a p p r o x i m a t e t h i s i n f i n i t e c o l l e c t i o n o f p o i n t s by a s m a l l s e q u e n c e o f p o i n t s r e p r e s e n t i n g t h e o b j e c t r e s p o n s e a t v a r i o u s a n g l e s , and t h e n t o u s e c o n v e n t i o n a l c l a s s i f i c a t i o n t e c h n i q u e s w i t h t h e s e p o i n t s as t h e d e s i g n s e t . 28 S e v e r e l i m i t a t i o n s e x i s t f o r t h i s a p p r o a c h however. As one can see i n t h e example c o o r d i n a t e v e c t o r p r o j e c t i o n i n F i g u r e 4.1, the d i s t a n c e i n the f e a t u r e space between p o i n t s w i t h i n a c l a s s measured 15° a p a r t can g r e a t l y e x c e e d the i n t e r - c l a s s d i s t a n c e . I t can a l s o be seen from t h i s f i g u r e t h a t a d e s i g n s e t c o n s i s t i n g of f e a t u r e s p a c e p o i n t s r e p r e s e n t i n g the o b j e c t r e s p o n s e a t 15° i n c r e m e n t s i n v e r t i c a l a n g l e would be i n a d e q u a t e f o r any c l a s s i f i e r b a s e d on any s i m p l e d i s t a n c e measure. The p o i n t (A) r e p r e s e n t i n g o b j e c t 1 a t an i n t e r m e d i a t e a n g l e (0=35°) i s c l e a r l y c l o s e r t o the d e s i g n s e t p o i n t r e p r e s e n t i n g o b j e c t 4 at 45° than any p o i n t i n t h e d e s i g n s e t of o b j e c t 1. Hence, any c l a s s i f i e r , such as a n e a r e s t n e i g h b o r [1] or minimum d i s t a n c e to mean [2] ( n e a r e s t mean v e c t o r ) , c o m p u t i n g a d i s t a n c e measure to d i s c r e t e d e s i g n s e t p o i n t s would be c l e a r l y u n w o r k a b l e u n l e s s a g r e a t number of p o i n t s were used to d e f i n e the d e s i g n s e t f o r t h e c l a s s . A l t e r n a t i v e l y , a b e t t e r method of a p p r o x i m a t i n g t h i s d e s i g n s e t t r a c k would be to f i t t h e c l a s s d e s i g n s e t p o i n t s w i t h some g e n e r a l f u n c t i o n . T h i s would a l l o w the use of a s i m p l e e x t e n s i o n of a n e a r e s t mean v e c t o r (NMV) c l a s s i f i e r i n which the d e f i n i t i o n of the c l a s s mean would be b r o a d e n e d t o i n c l u d e any p o i n t on the f u n c t i o n d e f i n e d f o r t h a t c l a s s . An i m p l e m e n t a t i o n of t h i s c l a s s i f i e r would th e n d e t e r m i n e the c l a s s of an a r b i t r a r y t e s t p o i n t by d e t e r m i n i n g the d i s t a n c e s from the t e s t p o i n t to the n e a r e s t p o i n t on each of t h e f u n c t i o n s d e f i n e d f o r the i n d i v i d u a l c l a s s e s and then c h o o s i n g the c l a s s f o r which t h i s d i s t a n c e was a minimum. T h i s l a t t e r a p p r o a c h r a i s e s c e r t a i n d i f f i c u l t i e s however. The f i t t i n g of a d e f i n i n g f u n c t i o n to the d e s i g n s e " p o i n t s i s i n i t s e l f a complex t a s k i n t h a t we have no 2250_ CM I— LU Eli 2150-ui oc 20501 90* 75' 2500 60' • 75* 80* A + 60* + 45* ± 45* 35* 0 0 STEEL STEEL A - 3 3 M M A - S 9 U U E « 3 . 6 E « 2 . 0 40 CU 50 CU + A t + 30* 2600 2700 J ro 2800 FEATURE ELEMENT 1 Figure 4.1: Feature variation with object and orientation. Feature type is segment mean. 30 t h e o r e t i c a l b a s i s on w h i c h t o e s t a b l i s h a mode l f o r t h e . f u n c t i o n f i t t e d t o t h e p o i n t s . H e n c e , d e f i n i n g t h e g o o d n e s s o f f i t f o r v a r i o u s p o s s i b l e f i t t i n g f u n c t i o n s w o u l d be d i f f i c u l t . The m a i n d i f f i c u l t y w i t h t h i s a p p r o a c h i s , h o w e v e r , t h a t i n t h e d e s i g n o f a n e a r e s t n e i g h b o r c l a s s i f i e r f o r i t , a t e s t p o i n t s h o u l d be c l a s s i f i e d a c c o r d i n g t o t h e c l a s s f o r m i n g t h e n e a r e s t t r a c k . Thus t h e m in imum d i s t a n c e f r o m t h e n - s p a c e c u r v e d e f i n e d by t h e f i t t e d f u n c t i o n t o t h e t e s t p o i n t mus t be c o m p u t e d . T h i s c o m p u t a t i o n i s n o t g e n e r a l l y a m e n a b l e t o d i r e c t s o l u t i o n and an i t e r a t i v e s o l u t i o n w o u l d h a v e to be i m p l e m e n t e d w i t h i t s i n h e r e n t d e p e n d e n c e on i n i t i a l g u e s s e s and p r o b l e m s o f e n s u r i n g c o n v e r g e n c e . H e n c e f o r t h e p u r p o s e s o f t h i s t h e s i s , t h e c o n c e p t o f u s i n g a g e n e r a l c u r v e i n t h e f e a t u r e s p a c e as t h e d e s i g n s e t f o r a c l a s s has been r e s t r i c t e d t o a p p r o x i m a t i n g t h e t r a c k by a s e r i e s o f l i n e s e g m e n t s c o n n e c t i n g e a c h two s u c c e s s i v e p o i n t s i n t h e d e s i g n s e t . T h i s a p p r o a c h i s a good a p p r o x i m a t i o n f o r t h i s a p p l i c a t i o n and i t a l l o w s f o r d i r e c t s o l u t i o n o f t h e min imum d i s t a n c e c a l c u l a t i o n w i t h m i n i m a l c o m p u t i o n a l c o m p l e x i t y . The d e s i g n s e t f o r e a c h c l a s s , d e t e r m i n e d by ne p o i n t s i n t h e f e a t u r e s p a c e , t h e n c o n s i s t s o f n e - 1 l i n e s e g m e n t s d e f i n e d , i n p a r a m e t r i c f o r m , as f o l l o w s : l i , j = c i y _ i , j + JEi.J >0<q<l • w h e r e V J J = mi,j + i - mi,j ( 4 . 1 ) l < i < n c , l < j < n e - l 31 From t h i s d e f i n i t i o n the d i s t a n c e from a t e s t p o i n t t to any l i n e segment d e f i n e d by .1 i , j i s t n e magnitude of the normal to the l i n e passing through the p o i n t _t. This i s given by: c . . • v . . where d. . =c. .- v. . with c. . = t-m v . . «v. . - 1,3 -i,3 , j i / j 11 j T h i s s o l u t i o n i s only v a l i d i f the i n t e r s e c t i o n p o i n t of the normal i s w i t h i n the l i m i t s of the l i n e segment ( i . e . , 0<q<l). Wherever t h i s i s not the case the minimum d i s t a n c e to that l i n e segment must be d e f i n e d , not as the d i s t a n c e to the l i n e , but as the d i s t a n c e to the nearest endpoint of that l i n e segment. T h i s provides f o r both the case of a p o i n t near a "convex" track segment (Figure 4.2,pt.B) and a l s o f o r a p o i n t beyond the endpoint of the track d e f i n e d (Figure 4.2, p t . A ) . Examples of each of these cases are shown i n F i g u r e 4.2 f o r a 2-dimensional case, where the d i s t a n c e to the c l a s s design set i s d r a t h e r than d" f o r both cases. The c l a s s determined by the c l a s s i f i e r i s then that c l a s s f o r which d i f j i s a minimum ( f o r any j ) . T h i s approach appears to provide an e x c e l l e n t approximation f o r t h i s a p p l i c a t i o n . Figure 4.3 shows the design set developed f o r o b j e c t 1 based on 15° increments i n o r i e n t a t i o n angle. T h i s i s o v e r l a i d with f e a t u r e space p o i n t s r e p r e s e n t i n g o b j e c t 1 at intermediate angles. The d e v i a t i o n s STEEL A - 3 3 U M E - 3. 6 4 0 C U X -1 L 2100 2200 2300 FEATURE ELEMENT 2 Figure 4.2: Distance calculation anomalies. 2400, 20001 i i . i I . i i i__J___< i 1 i . i . L_ 2500 2600 2700 2800 2900 3000 3100 FEATURE ELEMENT 1 Figure 4.3: Design set for object 1 compared to data for angles other than those used to form the design set. 34 seen in these points from the design set l ine segments are within the noise bounds for this data. Hence, the approximation would appear to be s u f f i c i e n t l y accurate that a c l a s s i f i e r implementation using this approach is v a l i d . 4.2 C l a s s i f i e r Distance Measure Having developed a concept def ining the design set one must s t i l l approach other decisions on the methods used for the precise implementation of the c l a s s i f i e r . One of these descis ions is to choose a measure of distance. The form of distance measure considered within this work is as fol lows; d. • = ( (t-m. , ) T C - 1 ( t - m . .) ] H (4.3) where the matrix C may take one of three common forms: i ) an ident i ty matrix. In this instance the distance measured is the Euclidean distance between the poi nts _t and mn'} j . i i ) a matrix with the diagonal terms equal to the components of the variance vector estimated. Terms off the diagonal are zero. The distance measure then computes a weighted distance in which the Euclidean distance in each dimension of the feature space is divided by the estimated standard deviation in that dimension. i i i ) an estimate of the covariance matrix. The distance measure would then be computing the Mahalanobis [16] distance between the points t^ and m-j t j . In this work, the signal to noise ra t io and hence the variance of the data can change with a change in depth or 35 o r i e n t a t i o n . T h e r e f o r e , i t i s not p o s s i b l e to e s t a b l i s h a v a r i a n c e ( o r c o v a r i a n c e ) f o r the d e s i g n s e t s t r u c t u r e , as i t i s i n t e n d e d to be i n v a r i a n t under changes i n o b j e c t d e p t h . The o n l y a v a i l a b l e e s t i m a t e of v a r i a n c e i s then t h a t of the t e s t d a t a r e c o r d to be c l a s s i f i e d . As i t i s not p o s s i b l e t o g e n e r a t e an e s t i m a t e of the c o v a r i a n c e m a t r i x from a s i n g l e r e c o r d , use of the M a h a l a n o b i s d i s t a n c e i s p r e c l u d e d and t h e s i m p l e r w e i g h t e d d i s t a n c e measure was used f o r t h i s work. No component of the w e i g h t i n g used i n t h i s measure i s due to p o s s i b l e e r r o r i n t h e d e s i g n s e t . T h i s i m p l i e s t h a t t h e e r r o r s i n the d e s i g n s e t a r e s i g n i f i c a n t l y l e s s than t h e e r r o r s i n the t e s t p o i n t . T h i s a s s u m p t i o n i s v a l i d at the p o i n t s i n the d e s i g n s e t d e f i n i n g the l i n e segments, as t h e s e p o i n t s a r e t a k e n from the a v e r a g e of f i v e r e s p o n s e s at the h i g h e s t s i g n a l to n o i s e r a t i o a v a i l a b l e . However, i t w i l l b r e ak down a t o t h e r p o i n t s a l o n g the f e a t u r e t r a c k where the l i n e a p p r o x i m a t i o n d e v i a t e s from the e x a c t l o c u s . T h i s e f f e c t can not be s i m p l y a v o i d e d however, as the w e i g h t i n g used must be u n i f o r m t h r o u g h o u t the c l a s s to a l l o w c o m p u t a t i o n of the minimum d i s t a n c e between the t e s t p o i n t and the l i n e segments. More complex w e i g h t i n g s y s t e m s [17] based on r e l a t i v e i n f o r m a t i o n c o n t e n t , f e a t u r e to f e a t u r e , were c o n s i d e r e d but were r e j e c t e d . In g e n e r a l , they r e q u i r e knowledge of t h e p r o b a b i l i t y of m i s c i a s s i f i c a t i o n a t a p o i n t i n the d e s i g n s e t , and i n t h i s i m p l e m e n t a t i o n t h a t would r e q u i r e a p r i o r i k nowledge of the o r i e n t a t i o n of the o b j e c t . One c o u l d e s t a b l i s h a w e i g h t by a v e r a g i n g the m i s c l a s s i f i c a t i o n p r o b a b i l i t i e s at each o r i e n t a t i o n i n c l u d e d i n the d e s i g n s e t : however, i t was f e l t t h a t the c o m p l e x i t y e n t a i l e d was n o t w a r r a n t e d . 36 C h a p t e r 5: E x p e r i m e n t s 5.1 I n t r o d u c t i o n T h r e e g o a l s were e s t a b l i s h e d f o r the e x p e r i m e n t s d e v e l o p e d f o r t h i s work. These were: i ) To v a l i d a t e the d e s i g n of the c l a s s i f i e r , w i t h p a r t i c u l a r r e f e r e n c e to t e s t i n g i t s a b i l i t y to c o r r e c t l y c l a s s i f y t e s t d a t a t a k e n a t o r i e n t a t i o n s o t h e r than t h o s e used i n f o r m i n g the d e s i g n s e t . i i ) To v a l i d a t e the t r e n d s a s s o c i a t e d w i t h t h e p r e d i c t e d p e r f o r m a n c e of the v a r i o u s f e a t u r e e x t r a c t i o n t e c h n i q u e s p r o p o s e d , and to d e t e r m i n e w h i c h t e c h n i q u e h o l d s the most p r o m i s e f o r f u r t h e r work. i i i ) To d e t e r m i n e the p o t e n t i a l f o r e x t e n d i n g t h e c l a s s i f i e r t o c l a s s i f y d a t a d e r i v e d from d e p t h s o t h e r than t h a t used i n f o r m i n g the d e s i g n s e t . The c r i t e r i a d e f i n i n g the p e r f o r m a n c e of t h e c l a s s i f i e r must a l s o be examined a t t h i s p o i n t . In c h a p t e r 3, an e s t i m a t e of the P m c was d e v e l o p e d which was a s i n g l e v a l u e f o r any g i v e n c o m b i n a t i o n of s e g m e n t a t i o n and f e a t u r e e x t r a c t i o n t e c h n i q u e s . T h i s s i n g l e v a l u e d c h a r a c t e r i z a t i o n i s p o s s i b l e o n l y due to the a s s u m p t i o n of a b s o l u t e knowledge o f t h e d e s i g n s e t p o i n t s and the a s s u m p t i o n of a t h e o r e t i c a l d i s t r i b u t i o n f o r the t e s t d a t a . In p r a c t i c e t h i s would i m p l y 37 t h a t t h e number o f r e c o r d s u s e d t o b o t h d e s i g n and t e s t t h e c l a s s i f i e r was i n f i n i t e . M o r e g e n e r a l l y , t h e p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n f o r a NMV c l a s s i f i e r i s a random v a r i a b l e w i t h a c e n t r a l v a l u e P j ^ c d e t e r m i n e d by t h e s t r u c t u r e o f t h e c l a s s i f i e r and t h e s t a t i s t i c a l d i s t r i b u t i o n o f members i n t h e c l a s s e s . T h e v a r i a n c e o f t h e random v a r i a b l e P m c i s d e t e r m i n e d by t h e number o f r e c o r d s u s e d t o f o r m t h e c l a s s m e a n - v e c t o r s and t h e d i s t r i b u t i o n a s s o c i a t e d w i t h e a c h c l a s s . T h i s ; d i s t r i b u t i o n o f P m c c a n be t h o u g h t o f as t h e r e f l e c t i o n o f t h e u n c e r t a i n t y i n e s t i m a t i n g t h e c l a s s means f r o m a f i n i t e number o f r e c o r d s and P^ c i s t h e l i m / i t v a l u e t o w h i c h t h e d i s t r i b u t i o n w o u l d t e n d as t h e number o f r e c o r d s u s e d t o f o r m t h e c l a s s means wen t t o i n f i n i t y . The p r o b a b i l i t y o f m i s c i a s s i f i c a t i o n a s s o c i a t e d w i t h any p a r t i c u l a r i m p l e m e n t a t i o n o f t h e c l a s s i f i e r , u s i n g a f i n i t e d e s i g n s e t , i s o n l y a s a m p l e ( P m c ) d rawn f r o m t h e d i s t r i b u t i o n P m c . I t s h o u l d t h e n be n o t e d t h a t t h e c l a s s i f i c a t i o n e r r o r ( E c = n o . o f e r r o r s / n o . o f t r i a l s ) o b s e r v e d on any s i n g l e run o f t h e c l a s s i f i e r , w i t h a f i n i t e t e s t s e t , i s o n l y an e s t i m a t e o f t h i s v a l u e , P m c . T h e c o n v e n t i o n a l m e t h o d o f c h a r a c t e r i z i n g P m c r e l i e s on t h e use o f l a r g e d e s i g n and t e s t s e t s . T h i s a l l o w s a c c u r a t e d e f i n i t i o n o f t h e c l a s s means and i m p l i e s t h a t t h e t e s t s e t w i l l r e f l e c t c l o s e l y t h e s t a t i s t i c a l d i s t r i b u t i o n o f t h e c l a s s p o p u l a t i o n . The c l a s s i f i e r i s t h e n r u n a g a i n s t t h e t e s t s e t and t h e r e s u l t i n g o b s e r v a t i o n o f E c i s p r e s u m e d t o be a good p r . t i m a t e o f P j £ c . 38 W h i l e a t f i r s t g l a n c e i t w o u l d seem t h a t t h e 518 r e c o r d s a v a i l a b l e i n t h e p r i m a r y d a t a s e t ( T a b l e 5 . 1 ) w o u l d a l l o w t h i s a p p r o a c h , t h i s i s n o t t h e c a s e . The d e s i g n o f t h e c l a s s i f i e r r e q u i r e s s i x l i n e s e g m e n t s t o d e f i n e a s i n g l e c l a s s , t w e n t y - f o u r t o d e f i n e a l l f o u r . T h i s a l l o w s an a v e r a g e o f ' o n l y 22 p o i n t s t o d e f i n e t h e d e s i g n s e t and t o t e s t t h e c l a s s i f i e r f o r e a c h r e g i o n a b o u t a l i n e s e g m e n t i n t h e d e s i g n s e t . G i v e n t h i s v e r y l i m i t e d d a t a s e t , t h e E c o b s e r v e d on any s i n g l e t e s t o f t h e c l a s s i f i e r i s u n l i k e l y t o a c c u r a t e l y c h a r a c t e r i z e i t s p e r f o r m a n c e i n a more g e n e r a l c a s e . The m e t h o d u s e d h e r e i n t o c o u n t e r t h i s l i m i t a t i o n i s t o r u n t h e c l a s s i f i e r s e v e r a l t i m e s , e a c h t i m e c h o o s i n g a d i f f e r e n t d e s i g n s e t f r o m w i t h i n t h e d a t a and o b s e r v i n g t h e r e s u l t i n g v a l u e o f E c . T h i s a p p r o a c h i s a r e d u c e d i m p l e m e n t a t i o n o f t h e h o l d o u t m e t h o d [ 1 8 , 6 ] i n w h i c h t h e c l a s s i f i e r i s e v a l u a t e d f o r a l l p o s s i b l e p a r t i t i o n s o f t h e d a t a s e t w i t h a g i v e n d e s i g n s e t s i z e . F o r t h i s m e t h o d t h e e r r o r p r o b a b i l i t y i s t h e n e s t i m a t e d as t h e a v e r a g e o f t h e r e s u l t s . A c o m p l e t e i m p l e m e n t a t i o n o f t h i s a p p r o a c h i s c o m p u t a t i o n a l l y p r o h i b i t i v e i n t h i s a p p l i c a t i o n as i t w o u l d i n v o l v e o v e r 250 r u n s o f t h e c l a s s i f i e r f o r e a c h t e s t . The number o f o b s e r v a t i o n s o f E c u s e d i n t h i s work i s s i x , t h e s p r e a d o f w h i c h i s c o n s i d e r e d t o c h a r a c t e r i z e P m c a d e q u a t e l y , f o r t h e c l a s s i f i e r s t r u c t u r e and t h e g i v e n d a t a s e t . T h i s number o f o b s e r v a t i o n s was; c h o s e n on t h e b a s i s t h a t , i f one w e r e t o assume i n d e p e n d e n c e , t h e p r o b a b i l i t y o f s i x s a m p l e s s p a n n i n g t h e mean w o u l d be 97% [ 1 4 ] . I t s h o u l d be n o t e d t h a t t h e v a l u e s o f E c o b s e r v e d f o r d i f f e r e n t c h o i c e s o f d e s i g n s e t c a n n o t be c o n s i d e r e d t o be e n t i r e l y i n d e p e n d e n t o f e a c h o t h e r as t h e y a l l d e r i v e f r o m 39 T a b l e 5 . 1 : P r i m a r y D a t a S e t O b j e c t D e p t h Number o f D e s c r i p t i o n I d e n t i f i e r ( c m . ) R e c o r d s 1 40 130 10 e a c h a t 9=0 , 1 5 , . . 9 0 ° 5 e a c h a t 9 = 5 , 1 0 , 2 0 , . . 8 5 ' 2 50 128 1 0 a e a c h a t 8 = 0 , 1 5 , . . 9 0 ° 5 D e a c h a t 9 = 5 , 1 0 , 2 0 , . . 8 5 ' 3 60 130 10 e a c h a t 9 = 0 , 1 5 , . . 9 0 ° 5 e a c h a t 9 = 5 , 1 0 , 2 0 , . . 8 5 4 50 130 10 e a c h a t 9 = 0 , 1 5 , . . 9 0 ° 5 e a c h a t 6 = 5 , 1 0 , 2 0 , . . 8 5 ' a) 9 o n l y a t 6=30° b) 4 o n l y a t 8=40° o T a b l e 5 . 2 : A d d i t i o n a l D a t a S e t O b j e c t I d e n t i f i e r D e p t h ( c m . ) Number o f R e c o r d s D e s c r i p t i o n 1 50 70 10 e a c h a t 8 = 0 , 1 5 , . . 9 0 ° 2 60 70 10 e a c h a t 9 = 0 , 1 5 , . . 9 0 ° 3 70 69 1 0 a e a c h a t 9 = 0 , 1 5 , . . 9 0 ° 4 60 68 1 0 D e a c h a t 6 = 0 , 1 5 , . . 9 0 ° a) 9 o n l y a t 8=30° b) 9 o n l y a t 9 = 7 5 , 9 0 ° 40 the same d a t a s e t . Hence c o n c l u s i o n s b a s e d on t h e s e r e s u l t s must, i n g e n e r a l , be r e s t r i c t e d to i n s t a n c e s where r e l a t i v e l y g r o s s t r e n d s are e v i d e n t . An e x c e p t i o n to t h i s l i m i t a t i o n e x i s t s f o r c a s e s where the same group of d e s i g n s e t s i s used i n a s e r i e s of t r i a l s . F o r t h i s i n s t a n c e , i t i s p o s s i b l e to " p a i r " the r e s u l t s of runs u s i n g the same d e s i g n s e t and r e d u c e the e f f e c t s of dependence by a n a l y z i n g the d i f f e r e n c e s i n p e r f o r m a n c e from t r i a l t o t r i a l . T h i s a l l o w s one to use c o n v e n t i o n a l s t a t i s t i c a l t e c h n i q u e s such as the W i l c o x o n rank sum t e s t ( A p p e n d i x 8) to d e t e r m i n e w h e t h e r the c l a s s i f i e r s t r u c t u r e s i n each t r i a l d i f f e r i n p e r f o r m a n c e , and i f so,-wh i c h i s s u p e r i o r . 5.2 O r i e n t a t i o n S e n s i t i v i t y T e s t The f i r s t t e s t of the c l a s s i f i e r used d a t a from one d e p t h f o r each o b j e c t ( T a b l e 5 . 1 ) . In g e n e r a l , t e n i n d e p e n d e n t r e c o r d s were a v a i l a b l e f o r each o b j e c t a t each of the s even v e r t i c a l a n g l e s used to d e v e l o p the d e s i g n s e t (0 , 15, 30 ,.. , 90) w i t h f i v e a d d i t i o n a l measurements f o r each i n t e r v e n i n g f i v e d egree i n c r e m e n t i n a n g l e . Of the t e n r e c o r d s at each of the d e s i g n s e t a n g l e s , f i v e were randomly c h o s e n and used to form the d e s i g n s e t w i t h the r e m a i n d e r c o n t r i b u t i n g to the t e s t s e t . T e s t 1 then c o n s i s t e d of s i x runs of the c l a s s i f i e r (one f o r each d i f f e r e n t c h o i c e of the d e s i g n s e t ) f o r each of the f o u r f e a t u r e e x t r a c t i o n t e c h n i q u e s p r e v i o u s l y d e t a i l e d . T h e s e runs were r e p e a t e d f o r each of the two d i f f e r e n t s e g m e n t a t i o n schemes and f o r t h r e e c h o i c e s i n the number of segments, namely t h r e e , s i x and t w e l v e . The r e s u l t s of t h i s t r i a l were then be used to d e t e r m i n e the v a l i d i t y of t h e .41 i p r e d i c t e d p e r f o r m a n c e t r e n d s , and to i s o l a t e the most p r o m i s i n g f e a t u r e e x t r a c t i o n t e c h n i q u e . The s u c c e s s of the c l a s s i f i e r a t h a n d l i n g the c o n t i n u o u s changes i n f e a t u r e p a r a m e t e r s due to changes i n the v e r t i c a l a n g l e of the o b j e c t was a l s o a s s e s s e d from t h i s e x p e r i m e n t . The r e s u l t s of t h i s t e s t are p r e s e n t e d i n s e c t i o n 6.1. 5.3 Depth S e n s i t i v i t y T e s t s T h r e e t e s t s were run to d e t e r m i n e the s e n s i t i v i t y o f th e c l a s s i f i e r to the r e l a t i v e l y minor v a r i a t i o n s i n s i g n a t u r e s o b s e r v e d w i t h a change i n depth of the o b j e c t . T h e s e t e s t s a l l made use of the f e a t u r e e x t r a c t i o n t e c h n i q u e j u d g e d to p r o v i d e the b e s t p e r f o r m a n c e i n the f i r s t t e s t . T h e s e t e s t s made use of an a d d i t i o n a l d a t a s e t ( T a b l e 5.2) which i n c l u d e s d a t a from the f o u r o b j e c t s a t a depth 10 cm. g r e a t e r t han t h a t used i n the f i r s t t e s t . However, t h i s d a t a s e t i n c l u d e d o n l y r e c o r d s t a k e n at the 15° i n c r e m e n t s i n v e r t i c a l a n g l e a s s o c i a t e d w i t h the d e s i g n s e t ( 0 , 1 5 , . . 9 0 ° ) . The r e s u l t s of t h i s s e r i e s of t r i a l s a r e i n c l u d e d as s e c t i o n s 6.2 and 6.3. The s e c o n d t r i a l ( T e s t 2) i n v o l v e s s i x runs of t h e c l a s s i f i e r u s i n g the d e s i g n s e t s d e v e l o p e d i n the f i r s t t e s t . However i n t h i s t r i a l , the t e s t s e t i s expanded to i n c l u d e t e s t p o i n t s d e r i v e d from both the d a t a s e t s such t h a t i t r e p r e s e n t s both d e p t h s . T h i s t r i a l d e t e r m i n e s the p e r f o r m a n c e o f the c l a s s i f i e r s t r u c t u r e a g a i n s t d a t a from d e p t h s o t h e r than t h o s e used to form the d e s i g n s e t . T e s t 3 a g a i n c o m p r i s e s s i x runs of the c l a s s i f i e r w i t i d a t a t a k e n from s o l e l y t h e a d d i t i o n a l d a t a s e t . T h a t i s , both 42 t h e d e s i g n and t e s t s e t a r e d e r i v e d f r o m d a t a t a k e n a t a d e p t h g r e a t e r t h a n t h a t u s e d i n t h e T e s t 1. A l t h o u g h t h i s i s n o t a c o m p l e t e t e s t due t o t h e l a c k o f d a t a a t v e r t i c a l a n g l e s n o t a s s o c i a t e d w i t h t h e d e s i g n s e t , t h i s p r e s e n t s t h e c l a s s i f i e r w i t h d a t a o f m a r k e d l y p o o r e r s i g n a l t o n o i s e t h a n t h a t o f t h e f i r s t t r i a l , and p r o v i d e s a r e f e r e n c e w i t h w h i c h t o c o m p a r e t h e r e s u l t s o f T e s t 2 . , T e s t 4 u s e s a t e s t s e t d e r i v e d f r o m b o t h d e p t h s as i n T e s t 2 ; h o w e v e r , i n t h i s i n s t a n c e t h e d e s i g n s e t i s d rawn f r o m t h e two d e p t h s r e p r e s e n t e d i n t h e t e s t s e t r a t h e r t h a n f r o m s i m p l y one d e p t h . F o r t h i s t e s t , t h e d e s i g n s e t c o n s i s t e d o f s i x r e c o r d s r a t h e r t h a n f i v e , w i t h t h r e e r e c o r d s b e i n g u s e d f r o m e a c h d e p t h . T h i s t r i a l i s d e s i g n e d t o d e t e r m i n e t h e p o t e n t i a l f o r r e d u c i n g any d e p t h s e n s i t i v i t y t h a t t h e c l a s s i f i e r may e x h i b i t , by i n c o r p o r a t i n g d a t a f r o m a l l d e p t h s o f i n t e r e s t i n t h e d e s i g n s e t . 43 C h a p t e r 6: R e s u l t s and E v a l u a t i o n 6.1 R e s u l t s of O r i e n t a t i o n S e n s i t i v i t y T e s t s The r e s u l t s of T e s t 1 a r e shown i n F i g u r e s 6.1 and 6.2. These f i g u r e s o v e r l a y the p r e d i c t e d v a l u e s of P m c d e v e l o p e d i n C h a p t e r 3 w i t h the range of the o b s e r v e d E c f o r the s e g m e n t a t i o n and f e a t u r e e x t r a c t i o n t e c h n i q u e s t e s t e d . The r e s u l t s c o n f i r m the p r e d i c t e d t r e n d s of c l a s s i f i e r p e r f o r m a n c e w i t h r e s p e c t to changes i n the number of segments and t y p e of f e a t u r e e x t r a c t e d . The mean-based f e a t u r e e x t r a c t i o n t e c h n i q u e s (F3.F4) c l e a r l y o u t p e r f o r m the o t h e r t e c h n i q u e s , e s p e c i a l l y f o r l a r g e r numbers of segments. A d d i t i o n a l l y , t h e s e r e s u l t s show t h a t s e g m e n t a t i o n based on e q u a l segment a r e a s p r o v i d e s b e t t e r r e s u l t s f o r the m a j o r i t y o f c a s e s . One s h o u l d note t h a t d i f f e r e n c e s between p r e d i c t e d and o b s e r v e d r e s u l t s are to be e x p e c t e d s i n c e the p r e d i c t i o n s were based on a c o m p a r i t i v e l y s i m p l e c l a s s i f i e r s t r u c t u r e . A d d i t i o n a l l y , the p r e d i c t i o n s were based on d a t a t h a t d i d n o t i n c o r p o r a t e o r i e n t a t i o n s o t h e r than t h o s e used to form the d e s i g n s e t and which had an a r t i f i c i a l l y e s t a b l i s h e d n o i s e c h a r a c t e r i s t i c . As one can see ( F i g u r e 6 . 1 ( a ) ) the b e s t r e s u l t s a p p e a r t o have been a c h i e v e d u s i n g the c o m b i n a t i o n of t w e l v e e q u a l a r e a segments w i t h the segment mean (F3) as the f e a t u r e e x t r a c t e d . T h i s o b s e r v a t i o n was c o n f i r m e d by a p p l i c a t i o n of the W i l c o x o n rank sum t e s t ( A p p e n d i x B) which i n d i c a t e s t h a t t h e p e r f o r m a n c e of t h i s c l a s s i f i e r i m p l e m e n t a t i o n i s s u p e r i o r to the o t h e r s i n the t r i a l ( c o n f i d e n c e l e v e l i s 9 8 % ) . Not o . o J o o J o 3 6 9 12 Number of segments — i 15 (a) segment exponential time constants (F1) o J o L U 0 3 6 9 12 Number of segments (c) segment mean (F3) 15 3 6 9 12 15 Number of segments (b) segment linear slopes (F2) I Observed range • Predicted value 3 6 9 12 Number of segments (d) mean difference (F4) 15 Figure 6.1: Classifier performance with equal area segmentation. o J Lift, -J o UJiO 3 6 9 12 15 Number of segments (a) segment exponential time constants (F1) 3 6 9 12 Number of segments 15 •—*. Left, "I o J . • I 3 6 9 12 Number of segments (b) segment linear slopes (F2) I Observed range • Predicted value 15 I — i r 1 1 3 6 9 12 Number of segments (d) mean difference (F4) tn 15 (c) segment mean (F3) Figure 6.2: Classifier performance with equal length segmentation. 46 o n l y were the o v e r a l l r e s u l t s f o r t h i s c a s e b e t t e r than f o r t h e o t h e r c a s e s , the range of the o b s e r v a t i o n s was the s m a l l e s t , i n d i c a t i n g a r e l a t i v e l y m i n o r s e n s i t i v i t y to d e s i g n s e t v a r i a t i o n . The c o n f u s i o n t a b l e s f o r the runs a s s o c i a t e d w i t h t h i s c a s e a r e p r e s e n t e d i n T a b l e 6.1. The e r r o r s o b s e r v e d a r e m a i n l y t h e r e s u l t of m i s c i a s s i f i c a t i o n of o b j e c t 2 as o b j e c t 1, a l t h o u g h even f o r t h i s o b j e c t the v a l u e s o f E c a r e s t i l l l e s s t h an 2.2%. A f u r t h e r breakdown of the e r r o r s i n c l a s s i f i c a t i o n o c c u r r i n g w i t h t h i s f e a t u r e e x t r a c t i o n t e c h n i q u e i s made i n T a b l e 6.2. T h i s t a b l e shows the a v e r a g e c l a s s i f i c a t i o n e r r o r o v e r a l l d e s i g n s e t s v e r s u s the v e r t i c a l a n g l e of the t e s t s e t o b j e c t s . No e r r o r s were o b s e r v e d f o r any of the a n g l e s from w h i c h d a t a was i n c o r p o r a t e d i n t o the d e s i g n s e t . The e r r o r s a t o t h e r a n g l e s a r e l a r g e l y due to m i s c i a s s i f i c a t i o n s o f o b j e c t 2 a t v e r t i c a l a n g l e s of 5 and 10 d e g r e e s . A l l of t h e s e m i s c i a s s i f i c a t i o n s t h a t were o b s e r v e d f o r o b j e c t 2 a t 5,10° were the r e s u l t of the c l a s s i f i e r m i s c i a s s i f y i n g the same two d a t a r e c o r d s i n each i n s t a n c e . T h i s c o i n c i d e n c e o f r e c o r d s was u n u s u a l and hence the d a t a f o r t h i s o b j e c t at t h e s e o r i e n t a t i o n s was f u r t h e r i n v e s t i g a t e d . A c l a s s mean f o r each c a s e was d e v e l o p e d and the d i s t a n c e , d-j , from each i n d i v i d u a l p o i n t to the c l a s s mean was computed u s i n g a w e i g h t e d d i s t a n c e measure as f o l l o w s . i i " m (6.1) As one can see i n T a b l e 6.3, the two p o i n t s i d e n t i f i e d as f a i l i n g i n the c l a s s i f i e r a re r e l a t i v e l y d i s t a n t from the c l a s s mean. W h i l e one c a n n o t d i s c a r d t h e s e p o i n t s on t h e 47 T a b l e 6 . 1 : C o n f u s i o n T a b l e s f o r T e s t 1 F e a t u r e : s e g m e n t mean ( F 3 ) S e g m e n t a t i o n : 12 e q u a l a r e a s e g m e n t s C o n d i t i o n s : D e s i g n and t e s t s e t s d rawn f r o m p r i m a r y d a t a \ D e c O b g \ 1 n s e t . 2 3 4 \ D e c 0 b j \ : ' n 1 2 3 4 1 95 0 0 0 1 94 1 0 0 2 2 91 0 0 2 2 91 0 0 3 0 0 95 0 3 0 0 95 0 4 0 0 1 94 4 0 0 1 94 a ) D e s i g n s e t 1: E c = 0 .79% b ) D e s i g n s e t 2 : E c = l . 06% 1 94 0 1 0 1 95 0 0 0 2 2 91 0 0 2 2 91 0 0 3 0 0 94 1 3 0 0 95 0 4 0 0 0 95 4 0 0 0 95 c) D e s i g n s e t 3 : .06% d ) D e s i g n s e t 4 : E c = 0 . 53% 1 95 0 0 0 1 94 1 0 0 2 2 91 0 0 2 2 91 0 0 3 0 0 95 0 3 0 0 95 0 4 0 0 1 94 4 0 0 1 94 e ) D e s i g n s e t 5 : E c = 0 .79% f ) D e s i g n s e t 6 : E c = l . 06% T a b l e 6 . 2 : D i s t r i b u t i o n o f F a i l u r e s w i t h A n g l e O b j e c t 1 2 1 2 6 6 3 1 4 3 1 0 5 10 15 20 25 30 35 40 45 E0 55 60 65 70 75 80 85 90 V e r t i c a l A n g l e ( e , ° ) N o t e : R e s u l t s f r o m a l l d e s i g n s e t s i n c l u d e d . ( T o t a l number o f t r i a l s = 2 2 6 8 ) 48 Tab le 6.3: D i s t ance to C l a s s Means f o r Objec t 2 Ob jec t Or i e n t a t i on Record D i s t ance to number number c l a s s mean 2 5° 1 2.34 2 2.70 3* 4.83 4 2.68 5 2.17 2 10° 1 2.93 2 2.54 3* 4.91 4 1.96 5 2.24 records caus ing c l a s s i f i e r f a i l u r e s f o r a l l des i gn sets (Test 1) . Tab le 6.4 Con fus ion Tab le s f o r Te s t 2 F e a t u r e : segment mean (F3) Segmentat ion : 12 equal area segments C o n d i t i o n s : Design sets drawn from pr imary data s e t . Tes t se t i n c l u d e s remainder of pr imary data set and whole of a d d i t i o n a l data s e t . v Dec 1 n v D e c ' n O b j \ 1 2 3 4 O b j \ 1 2 3 4 1 156 9 0 0 1 153 10 2 0 2 5 158 0 0 2 4 159 0 0 3 1 0 157 0 3 2 0 155 7 4 0 0 11 152 4 0 0 11 152 a ) D e s i g n set 1: E c = 4 .89% b)Des ign set 2: E c = 5. 50% 1 155 10 0 0 1 154 9 2 0 2 6 157 0 0 2 4 159 0 0 3 1 0 156 7 3 1 0 158 5 4 0 0 11 152 4 0 0 11 152 c ) D e s i g n set 3: E c = 5 .34% d)Des ign set 4: E c = 4. 89% 1 155 10 0 0 1 156 9 0 0 2 5 158 0 0 2 5 158 0 0 3 1 0 157 6 3 2 0 156 6 4 0 0 11 152 4 0 0 12 151 e) D e s i g n set 5: E c = 5 .34% f )Do s i gn set 6: E c = 4. 89% 4 9 b a s i s o f t h i s m e a s u r e , t h e y a r e c l e a r l y a n o m a l o u s a n d may h a v e b e e n t h e p r o d u c t o f an u n o b s e r v e d s y s t e m a t i c e r r o r i n t h e d a t a c o l l e c t i o n p r o c e s s . 6 . 2 R e s u l t s o f D e p t h S e n s i t i v i t y T e s t s T h e c o n f u s i o n t a b l e s f o r t h e r u n s d o n e w i t h t h e t e s t s e t d e r i v e d f r o m t w o d e p t h s a r e s h o w n i n T a b l e 6 . 4 . A s c a n be r e a d i l y s e e n , t h e p e r f o r m a n c e o f t h e c l a s s i f i e r i s d e g r a d e d by t h e i n c l u s i o n o f t h e e x t r a d a t a . F u r t h e r a n a l y s i s o f t h i s r e s u l t , a g a i n u s i n g t h e W i l c o x o n t e s t ( A p p e n d i x B ) , i n d i c a t e s t h a t t h i s d e g r a d a t i o n i s i n d e e d s i g n i f i c a n t a t a 98% c o n f i d e n c e l e v e l . A b r e a k d o w n o f t h e s e r e s u l t s i s s h o w n i n T a b l e 6 . 5 , w h i c h s h o w s t h e o b s e r v e d v a l u e o f E c f o r i n c r e m e n t s i n d e p t h v e r s u s t h e d i f f e r e n t o b j e c t t y p e s . Two f a c t o r s a r e s e e n a s c o n t r i b u t i n g t o t h e s e r e s u l t s . T h e f i r s t a n d m o s t o b v i o u s o f t h e s e i s t h a t t h e s i g n a l t o n o i s e r a t i o o f t h e a d d e d d a t a i s m a r k e d l y p o o r e r t h a n t h a t o f t h e o r i g i n a l d a t a b e c a u s e t h e a d d e d d a t a w a s t a k e n a t a g r e a t e r d e p t h . T h i s w o u l d l e a d t o g r e a t e r s p r e a d o f t h e d a t a i n t h e f e a t u r e s p a c e a n d w o u l d be e x p e c t e d t o c o n t r i b u t e t o p o o r e r r e s u l t s . T h e s e c o n d p o t e n t i a l c o n t r i b u t i n g f a c t o r i s t h a t s i g n a t u r e s o f t h e o b j e c t s a t t h e s a m e o r i e n t a t i o n , b u t d i f f e r e n t d e p t h s , may d i f f e r t o s u c h an e x t e n t t h a t a d e s i g n s e t b a s e d on o n l y o n e d e p t h w o u l d n o t be r e p r e s e n t a t i v e o f d a t a f r o m a n o t h e r d e p t h . T h e c o n t r i b u t i o n o f e a c h o f t h e s e f a c t o r s t o t h e m i s c i a s s i f i c a t i o n s o b s e r v e d i n t h i s t r i a l i s d i f f i c u l t t o d e t e r m i n e . H o w e v e r , T e s t 3 i s d e s i g n e d t o i n v e s t i g a t e t h e e x t e n t t o w h i c h t h e f i r s t f a c t o r c o n t r i b u t e s . T h e a p p r o a c h 50 T a b l e 6.5: E c v e r s u s O b j e c t Type and Depth f o r T e s t 2 Type E c (%) 1 0.53 13.80 2 2.15 4.05 3 0.18 10.63 4 0.70 15.44 O v e r a l l 0.88% 10.83% I n i t i a l S e c o n d d e p t h d e p t h N o t e : R e s u l t s f o r a l l d e s i g n s e t s i n c l u d e d . T a b l e 6.6: C o n f u s i o n T a b l e s f o r T e s t 3 F e a t u r e : segment mean (F3) S e g m e n t a t i o n : 12 equal a r e a segments C o n d i t i o n s : D e s i g n and t e s t s e t s drawn from a d d i t i o n a l d a t a \Dec O b j \ 'n 1 s e t . 2 3 4 \Dec Ob.iX 'n ^ 2 3 4 1 35 0 0 0 1 35 0 0 0 2 0 35 0 0 2 0 35 0 0 3 0 0 34 0 3 0 0 34 0 4 0 0 3 30 4 0 0 2 31 a ) D e s i g n s e t 1: E c = 2 . 19% b ) D e s i g n s e t 2: E c = l . 46% 1 35 0 0 0 1 35 0 0 0 2 0 35 0 0 2 0 35 0 0 3 0 0 34 0 3 0 0 33 1 4 0 0 1 32 4 0 0 1 32 c ) D e s i g n s e t 3: E c = 0 . 73% d ) D e s i g n s e t 4: E c = l . 46% 1 35 0 0 0 1 35 0 0 0 2 0 35 0 0 2 0 35 0 0 3 0 0 33 1 3 0 0 34 0 4 1 0 2 30 4 0 0 3 30 e) D e s i g n s e t 5: E C = 2 .92% f ) D e s i g n s e t 6: E c = 2. 19% 51 u n d e r t a k e n f o r t h i s t e s t i s t o u s e s o l e l y t h e d a t a f r o m , t h e g r e . a t e r d e p t h f o r b o t h t h e d e s i g n a n d t e s t s e t . T h i s r e m o v e s a n y p o s s i b l e . - , b i a s i n t h e d e s i g n s e t , d u e t o d e p t h s e n s i t i v i t y , a t t h e e x p e n s e o f g r e a t e r v a r i a b i l i t y d u e t o t h e i n c r e a s e d n o i s e . T h e r e s u l t s f o r t h i s t r i a l a r e s h o w n i n T a b l e 6 . 6 . W h i l e t h e r e s u l t s a r e n o t a s g o o d a s t h o s e f o r T e s t 1 , t h e y a r e s t i l l m a r k e d l y b e t t e r t h a n t h o s e f o r T e s t 2 . T h i s i n d i c a t e s t h a t w h i l e t h e n o r m a l i z e d s i g n a t u r e s o f t h e s a m e o b j e c t a t . d i f f e r e n t d e p t h s a r e v i r t u a l l y i n d i s t i n g u i s h a b l e t o t h e e y e , e n o u g h o f a d i f f e r e n c e e x i s t s t o i n t e r f e r e w i t h p r o p e r c l a s s i f i c a t i o n w h e n t h e d e s i g n s e t d a t a i s c o l l e c t e d a t a d i f f e r e n t d e p t h t h a n t h e t e s t d a t a . 6 . 3 E f f e c t s o f a M o d i f i e d D e s i g n S e t ( T e s t 4 ) O n e p o t e n t i a l s o l u t i o n t o t h e p r o b l e m o f d e p t h d e p e n d e n c e i n t h e d e s i g n s e t i s t o i n c o r p o r a t e d a t a f r o m b o t h d e p t h s i n t o t h e d e s i g n s e t . T h i s w i l l h a v e t h e e f f e c t o f s h i f t i n g t h e c l a s s m e a n f o r e a c h d e s i g n s e t p o i n t t o some i n t e r m e d i a t e p o i n t b e t w e e n t h e t w o c l a s s m e a n s f o r m e d f o r e a c h d e p t h . t a k e n i n d e p e n d e n t l y . W h i l e t h i s c a n be e x p e c t e d t o r e s u l t i n an i n f e r i o r c l a s s i f i e r f o r . d a t a t a k e n f r o m e i t h e r o f t h e d e p t h s , i f c o n s i d e r e d s e p a r a t e l y , i t may i m p r o v e r e s u l t s f o r t h e c o m b i n a t i o n o f d e p t h s . T h e f o u r t h t r i a l s e r i e s i m p l e m e n t e d t h i s a p p r o a c h w i t h r e s u l t s a s s h o w n i n t a b l e 6 . 7 a n d 6 . 8 . A n o t i c e a b l e i m p r o v e m e n t i s g a i n e d i n t h e o v e r a l l p e r f o r m a n c e o f t h e c l a s s i f i e r i n c o m p a r i s o n t o o n e u s i n g a d e s i g n s e t b a s e d on a s i n g l e d e p t i . C o m p a r i n g t a b l e s 6 . 5 a n d 6 . 8 , o n e c a n s e e t h a t 52 T a b l e 6 . 7 : C o n f u s i o n t a b l e s f o r T e s t 4 F e a t u r e : s e g m e n t mean ( F 3 ) S e g m e n t a t i o n : 12 e q u a l a r e a s e g m e n t s C o n d i t i o n s : D e s i g n and t e s t s e t s d rawn f r o m c o m b i n a t i o n o f p r i m a r y and a d d i t i o n a l d a t a s e t s . v De O b j \ c ' n 1 2 3 4 0 b , ] \ ic' n 1 2 3 4 1 164 0 1 0 1 161 4 0 0 2 3 160 0 0 2 5 158 0 0 3 2 0 152 10 3 2 0 156 6 4 0 0 10 153 4 0 0 9 154 a ) D e s i g n s e t 1: E C = 3 .97% b ) D e s i g n s e t 2 : E c = 3 . 97% 1 164 0 1 0 1 165 0 0 0 2 2 161 0 0 2 2 161 0 0 3 1 0 159 4 3 1 0 153 10 4 0 0 6 157 4 0 0 9 154 c ) D e s i g n s e t 3 : Ec = 2 . 14% d ) D e s i g n s e t 4 : E c = 3 . 36% 1 165 0 0 0 1 165 0 0 0 2 3 160 0 0 2 2 161 0 0 3 1 0 153 10 3 1 0 158 5 4 0 0 8 155 4 0 0 7 156 e ) D e s i gn s e t 5 : E C = 3 .36% f ) D e s i g n s e t 6 : E c = 2 . 29% T a b l e 6 . 8 : E c v e r s u s o b j e c t t y p e a n d d e p t h f o r T e s t 4 T y p e E C (%) 1 0 . 9 2 0 . 0 0 2 2 . 1 5 0 . 0 0 3 1 .07 1 5 . 9 7 4 1 . 8 4 1 3 . 1 2 O v e r a l l 1 .63% 7 . 1 7 % I n i t i a l S e c o n d d e p t h d e p t h N o t e : R e s u l t s f o r a l l d e s i g n s e t s i n c l u d e d . 53 t h e c l a s s i f i e r e r r o r i s r e d u c e d f o r the l a r g e r depth at t h e e x p e n s e o f t h e l e s s e r . The r e s u l t s a r e s i g n i f i c a n t l y worse t h a n t h o s e r e c o r d e d f o r e i t h e r of the i n s t a n c e s where t h e d e s i g n and t e s t s e t are from the same d e p t h . 6.4 E v a l u a t i o n o f R e s u l t s The r e s u l t s summarized above i n d i c a t e t h a t the f e a t u r e e x t r a c t i o n t e c h n i q u e and the c l a s s i f i e r d e v e l o p e d h e r e i n have e x c e l l e n t p o t e n t i a l as a s o l u t i o n to the p r o b l e m o f i d e n t i f i c a t i o n of b u r i e d m e t a l l i c o b j e c t s . The r e s u l t s of a p p l y i n g t h e c l a s s i f i e r t o d a t a from a s i n g l e d e p t h a r e e s p e c i a l l y e n c o u r a g i n g c o n s i d e r i n g the c o n t i n u o u s v a r i a t i o n i n / f e a t u r e s w i t h r e s p e c t to o b j e c t o r i e n t a t i o n . T h r e e i m p o r t a n t i s s u e s a r e r a i s e d by t h e s e r e s u l t s however. The f i r s t i s t h a t the o b j e c t r e s p o n s e s a r e c l e a r l y n o t a b s o l u t e l y depth i n v a r i a n t . The c l a s s i f i e r was a b l e t o a c h i e v e b e t t e r r e s u l t s a g a i n s t d a t a from a s i n g l e depth than a g a i n s t m u l t i p l e d e p t h s , even when the d e s i g n s e t i n c l u d e d i n f o r m a t i o n from a l l d e p t h s . T h i s d e g r a d a t i o n i n p e r f o r m a n c e was not d r a m a t i c , but was s t i l l g r e a t e r than would be a c c e p t a b l e i n an o p e r a t i o n a l s y s t e m . Hence w h i l e p r e c i s e d e p t h d e t e r m i n a t i o n i s not seen to be n e c e s s a r y , some p r e l i m i n a r y l o c a l i z a t i o n of o b j e c t depth on the o r d e r of *5 cm. would a p p e a r to be n e c e s s a r y f o r a s y s t e m based on t h e s e t e c h n i q u e s to f u n c t i o n a d e q u a t e l y . T h i s would a l l o w one to d e f i n e d i f f e r e n t d e s i g n s e t s f o r s e v e r a l d i f f e r e n t d e p t h s and t o use t h e most a p p r o p r i a t e one based on t h e e s t i m a t e d depth of t h e o b j e c t . W h i l e no e m p i r i c a l s y s t e m has been d e m o n s t r a t e d t o d e t e r m i n e depth t o t h i s a c c u r a c y , i t has been shown [5] t h a t f o r the s p h e r e model, such a d e t e r m i n a t i o n i s 54 possible using a pulse induction system with two receive c o i l s of d i f f e r i n g r a d i i . The second issue raised by these results relates to the adequacy of the f i t t i n g procedure used to approximate the locus of class means in the feature space. In the majority of cases, the li n e segment approximation resulted in excellent performance. However, the arbitrary d e f i n i t i o n of segment endpoints based on fixed changes in object orientation cannot be expected to be optimal as the rate of change of the feature space representation for a class does not follow a straight l i n e r e l a t i o n with the object angle (ref. Figures 4.1,4 . 3 ) . The last major issue is that of the s e n s i t i v i t y of the c l a s s i f i e r to the selection of the design set. The variation in result over d i f f e r i n g sets indicates that a much greater amount of data would have to be taken to accurately establish the class means at the design set points. This would be most especially true at greater depths where the signal to noise r a t i o of the responses is lower. Both of these l a t t e r issues r e f l e c t limitations in the available data set rather than li m i t a t i o n s in the technique. One can e a s i l y envisage an interactive system which would establish the number of segments and the endpoints to use in forming the class design d e f i n i t i o n by examining the discrete "curvature" of the succession of points developed in the feature space for that object as i t was rotated in a v e r t i c a l plane. As well, the number of records defining a design set point could be adjusted based on the actual signal to noise r a t i o of the data, such that the class mean for a point could be established to any desired accuracy. 55 C h a p t e r 7: C o n c l u s i o n s and Recommendations f o r F u r t h e r S t u d y  7.1 C o n c l u s i o n s T h i s work has d e m o n s t r a t e d the f e a s i b i l i t y o f i d e n t i f y i n g s t e e l s p h e r o i d s by i n t e r p r e t a t i o n of t h e i r e l e c t r o m a g n e t i c r e s p o n s e , g i v e n t h a t t h e y a r e from a known s e t . In the p r o c e s s , two o t h e r n o t a b l e r e s u l t s have emerged. An e f f e c t i v e f e a t u r e t y p e has been d e v e l o p e d f o r t h i s f o r m o f r e s p o n s e s i g n a t u r e ; t h i s f e a t u r e b e i n g the mean v a l u e of the samples w i t h i n segments of the r e s p o n s e which have r o u g h l y equal a r e a . W h i l e no s t a t e m e n t can be made about the a b s o l u t e m e r i t of t h i s f e a t u r e a g a i n s t a l l o t h e r p o s s i b l e f e a t u r e s , c e r t a i n l y i t s p e r f o r m a n c e r e l a t i v e to the o t h e r methods p r o p o s e d h e r e i n i s e x c e l l e n t . A d d i t i o n a l l y , i t has t h e d i s t i n c t a d v a n t a g e of e x c e p t i o n a l i m p l e m e n t a t i o n s i m p l i c i t y and of b e i n g amenable to i m p l e m e n t a t i o n i n e i t h e r s o f t w a r e or s p e c i a l p u r p o s e h a r d w a r e . The o t h e r n o t a b l e r e s u l t e s t a b l i s h e d h e r e i n has been t h e d e v e l o p m e n t of an e x t e n s i o n to the NMV t y p e of c l a s s i f i e r . The e x t e n s i o n g e n e r a l i z e s the c l a s s mean to t h e l o c u s of p o i n t s i n f e a t u r e s p a ce d e f i n i n g the c l a s s . T h i s a l l o w s c l a s s i f i c a t i o n of s i g n a t u r e s whose f e a t u r e s v a r y w i t h r e s p e c t to a p a r a m e t e r whose range i s c o n t i n u o u s ( w i t h i n some l i m i t s ) . The i m p l e m e n t a t i o n d e s c r i b e d h e r e i n l i m i t s t h i s a p p r o a c h , f o r c o m p u t a t i o n a l s i m p l i c i t y , t o a pi.ecewise l i n e a r a p p r o x i m a t i o n to the l o c u s . 56 I t s h o u l d be n o t e d t h a t i t would be s t r a i g h t f o r w a r d t o e x t e n d the c l a s s i f i e r c o n c e p t to e x t r a c t an e s t i m a t e of the c o n t i n u o u s p a r a m e t e r based on the p o i n t of the d e s i g n l o c u s t h a t most c l o s e l y a p p r o a c h e s the t e s t p o i n t . F o r example, i n the i m p l e m e n t a t i o n d e t a i l e d h e r e i n one must d e t e r m i n e the l i n e p a r a m e t e r , q, t o d e t e r m i n e i f the i n t e r s e c t i o n p o i n t of the normal t o the l i n e i s w i t h i n t h e l i n e segment. From t h i s p a r a m e t e r , and knowing the a n g l e s e l > e 2 c o r r e s p o n d i n g to the s t a r t and end p o i n t s of the l i n e segment, one c o u l d i n t e r p o l a t e an e s t i m a t e o f 9 as f o l l o w s : 9 = ( 6 2 - 8 i ) * q 7.2 Recommendations f o r F u r t h e r S t u d y S e v e r a l a s p e c t s of the work p r e s e n t e d h e r e i n r e q u i r e f u r t h e r s t u d y to d e t e r m i n e p o s s i b l e improvements and to a d e q u a t e l y d e l i n e a t e the c l a s s i f i e r p e r f o r m a n c e i n an u n c o n s t r a i n e d e n v i r o n m e n t . These i n c l u d e : i ) D e v elopment of a method to c h o o s e the d e s i g n s e t p o i n t s d e t e r m i n i n g the l i n e segments, such t h a t c l a s s i f i e r p e r f o r m a n c e i s o p t i m i z e d f o r any g i v e n number of segments. i i ) F u r t h e r t e s t s to d e t e r m i n e the e f f e c t on c l a s s i f i e r p e r f o r m a n c e of r e l a t i v e l y minor (±1..5 cm.) t e s t s e t depth v a r i a t i o n s f o r a g i v e n d e s i g n s e t . 57 i i i ) F u r t h e r t e s t s are a l s o r e q u i r e d to d e t e r m i n e t h e e f f e c t s of r e l a x i n g the a s s u m p t i o n s made f o r t h i s work. The e f f e c t s of i n c l u d i n g more complex a s s y m m e t r i c shapes i n t h e o b j e c t s e t s h o u l d be d e t e r m i n e d . A l s o of i n t e r e s t i s the e f f e c t of a h o r i z o n t a l p o s i t i o n i n g e r r o r i n d u c e d when the t e s t d a t a i s c o l l e c t e d . i v ) F i n a l l y , f u r t h e r t e s t s a r e r e q u i r e d to d e t e r m i n e t h e e f f e c t of m e a s u r i n g t h e r e p o n s e s i n e a r t h r a t h e r than i n a i r . W h i l e t h i s e f f e c t i s e x p e c t e d t o be minor i n e x t e n t due to the use of b a c k g r o u n d s u b t r a c t i o n i n the d a t a c o l l e c t i o n , l o c a l v a r i a t i o n s i n s o i l c o n d u c t i v i t y and p e r m e a b i l i t y c o u l d i n d u c e e f f e c t s t h a t would b e a r i n v e s t i g a t i o n . Many a s p e c t s of t h e s e r ecommendations a r e q u i t e s t r a i g h t f o r w a r d ; however, they must a l l a w a i t c o l l e c t i o n of a much l a r g e r d a t a s e t . 7 .3 F i n a l Remarks In o v e r a l l summary, the work d e t a i l e d h e r e i n d e v e l o p s a f e a s i b l e s o l u t i o n to the p r o b l e m w i t h i n t h e s t a t e d a s s u m p t i o n s . E x c e l l e n t r e s u l t s a r e a c h i e v e d f o r the s i n g l e d e p t h c a s e s . W h i l e one c a n n o t p r o j e c t the r e s u l t of the f u r t h e r work n e c e s s a r y to d e t e r m i n e the p e r f o r m a n c e i n an u n c o n s t r a i n e d e n v i r o n m e n t , the c o n c e p t p r o v i d e s e x c e p t i o n a l p r o m i s e f o r t h i s a p p l i c a t i o n . 58 A p p e n d i x A: E x p e r i m e n t a l A p p a r a t u s An o v e r v i e w of the d a t a c o l l e c t i o n s y s t e m d e v e l o p e d f o r t h i s r e s e a r c h i s shown i n F i g u r e A . l . The s e n s o r s y s t e m ( c o i l s , Tx and Rx) was l o c a t e d i n a s p e c i a l p u r p o s e l a b o r a t o r y . A f e a t u r e of t h i s l a b o r a t o r y i s t h a t i t i s c o n s t r u c t e d e n t i r e l y w i t h o u t m e t a l , so t h a t the r e s p o n s e s o f m e t al o b j e c t s can be measured i n i s o l a t i o n . O u t p u t s of the s e n s o r s y s t e m are t r a n s m i t t e d to a s e p a r a t e s u p p o r t b u i l d i n g (35 m. d i s t a n t ) which houses the d i g i t i z a t i o n s y s t e m and i t s a s s o c i a t e d computer. The d a t a i s t h e n t r a n s m i t t e d o v e r a s e r i a l d a t a l i n k to a s e c o n d computer f o r s t o r a g e and l a t e r p r o c e s s i n g . The d i g i t i z e r used i s a s p e c i a l p u r p o s e s y s t e m w h i c h a l l o w s s y n c h r o n o u s a v e r a g i n g of s u c c e s s i v e frames of d a t a to r e d u c e u n c o r r e l a t e d n o i s e . The s e n s o r system c o m p r i s e s c o - p l a n a r t r a n s m i t and r e c e i v e c o i l s . These c o i l s are mounted on a t e s t s t a n d as shown i n F i g u r e s A.2 and A.3. T h i s s t a n d a l l o w s c o n t r o l of d i s t a n c e from t h e c o i l s t o t h e o b j e c t by v a r y i n g t h e p o s i t i o n of the p l a t f o r m h o l d i n g the o b j e c t j i g . The o b j e c t j i g i s d e s i g n e d to a l l o w changes i n t h e o b j e c t o r i e n t a t i o n i n the v e r t i c a l p l a n e w h i l e m a i n t a i n i n g c o n s t a n t c o i l - o b j e c t d i s t a n c e . The e x p e r i m e n t a l p r o c e d u r e used i s as f o l l o w s : i ) A measurement of the b a c k g r o u n d r e s p o n s e (due t o c o i l i n t e r a c t i o n , e a r t h r e s p o n s e , e t c . ) i s t a k e n and d i g i t i z e d . i i ) The o b j e c t i s p l a c e d under the c o i l s ( a l o n g t h e Data Acquisition Facility High Speed A/D System Figure A.1: Data Collection System Data Analysis Facility VAX 11/780 CPU 1 Disk/Tape / I Storage I / Graphics \ \ ^ Display / ^ Tenriina! ~J i ~ . | Terminal | I I I co Figure A.2: Photograph of experimental apparatus. 61 Figure A.3: Detail of coils and object jig. 62 c o i l a x i s ) a t the c o r r e c t d i s t a n c e and o r i e n t a t i o n . i i i ) A s e q u e n c e of r e s p o n s e measurements (5 or 10) a r e t a k e n and d i g i t i z e d . i v ) The p r e v i o u s l y measured b a c k g r o u n d r e s p o n s e i s s u b t r a c t e d from the r e c o r d s and they a r e t h e n s t o r e d . v) The o b j e c t i s removed from the t e s t s t a n d and t h e s e q u e n c e i s r e p e a t e d A summary of the s y s t e m p a r a m e t e r s f o l l o w s ( s y m b o l s as d e f i n e d i n C h a p t e r 2 ) : N j - 27 t u r n s N - 100 t u r n s R j - 28 cm. R - 25 cm. I - ~2.6 Amperes T r a n s m i t t e r r e p e t i t i o n r a t e - 488.28 Hz. T r a n s m i t t e r duty c y c l e - 50% R e c e i v e r g a i n - 60 dB D i g i t i z a t i o n r a t e - 250 kHz. Number of frames a v e r a g e d - 500 Note: T r a n s m i t t e r r e p e t i t i o n r a t e i s based on m e a s u r i n g 256 r e s p o n s e p o i n t s i n the q u i e s c e n t p e r i o d of the t r a n s m i t t e r c y c l e . 63 Appendix B; Wilcoxon Rank Sum Test f o r Pai r e d Experiments Within t h i s work a t e s t i s r e q u i r e d ( S e c t i o n 6.3) to determine whether observed performance d i f f e r e n c e s between c l a s s i f i e r implementations are s t a t i s t i c a l l y s i g n i f i c a n t . While t h i s may seem s t r a i g h t f o r w a r d , two c o n s i d e r a t i o n s must be addressed f o r t h i s a p p l i c a t i o n . The f i r s t of these i s that no knowledge of the d i s t r i b u t i o n of P m c (or i t s estimate E c) can be assumed. Hence, many con v e n t i o n a l s t a t i s t i c a l t e s t s to determine i f d i f f e r e n c e s are s i g n i f i c a n t (Student's T, etc.) cannot be a p p l i e d . The second c o n s i d e r a t i o n i s that the method used to ga i n samples of E c (re-use of the data set with d i f f e r i n g p a r t i t i o n s of design and t e s t s e t s ) does not guarantee that the samples do not trend with these changes. The f i r s t c o n s i d e r a t i o n i s addressed by the use of the Wilcoxon Rank Sum t e s t [ l l ]. T h i s i s a non-parametric s t a t i s t i c a l t e s t i n which no assumptions are made about the d i s t r i b u t i o n s from which the samples are drawn. The n u l l hypothesis (H D) f o r the t e s t i s that the d i s t r i b u t i o n s from which the two sample sets are drawn are i d e n t i c a l . The a l t e r n a t e hypothesis (H a) i s that the median of one d i s t r i b u t i o n i s l e s s than that of the other. A p a i r e d experiment i s used to compensate f o r the p o s s i b l e trends of the r e s u l t s f o r a sequence of data set p a r t i t i o n i n g s (second c o n s i d e r a t i o n ) . T r i a l s using the same design set f o r two experiments are p a i r e d and the d i f f e r e n c e i n performance f o r a succesion of p a i r s i s then the data analyzed. The t e s t i s p r e d i c a t e d on the hypothesis that, f o r H D to hold , the number of p o s i t i v e and negative d i f f e r e n c e s between p a i r s of experimental r e s u l t s shoud be equal. 64 A d d i t i o n a l l y , i f the d i f f e r e n c e s a r e r a n k e d i n o r d e r o f a b s o l u t e v a l u e , t h e r a n k s o f p o s i t i v e and n e g a t i v e e l e m e n t s s h o u l d be random. To a s s e s s t h i s , t h e r a n k s o f a l l p o s i t i v e ( o r n e g a t i v e ) e l e m e n t s a r e summed. The p r o b a b i l i t y o f t h e rank sum e q u a l l i n g a p a r t i c u l a r v a l u e c an t h e n be d e t e r m i n e d by a p p l i c a t i o n o f b a s i c p r o b a b i l i t y t h e o r y [12 ]. T h i s c a l c u l a t i o n i s , i n g e n e r a l , q u i t e t e d i o u s ; however, t a b l e s g i v i n g c r i t i c a l v a l u e s o f rank sum a r e a v a i l a b l e [13 ]. The c a l c u l a t i o n s i n v o l v e d i n t h e two i n s t a n c e s f o r w h i c h t h i s t e s t i s i n v o k e d i n t h e t e x t f o l l o w as exampl e s . Example 1: For Test 1 - Determining the best combination of segmentation and feature extract ion technique. Case A -• 12 equal area segments, feature is segment mean (F3). Case B • • 12 equal area segments, feature is mean di fference (F4). Desi gn Result Result Di fference Rank Set under A under B B-A 1 .79 1.85 1.06 Sh 2 1.06 1.85 .79 2 3 1.06 1.85 .79 2 4 ,. .53 1.32 .79 2 5 .79 1.59 .80 4 6 1.06 2.12 1.06 Sh No negative elements are present, therefore the negative rank sum is 0. From tables the c r i t i c a l value of rank sum 65 (98% c o n f i d e n c e l e v e l , 6 t r i a l s ) i s 1 . H e n c e , H 0 i s r e j e c t e d ; i . e . t h e r e i s s i g n i f i c a n t d i f f e r e n c e b e t w e e n t h e r e s u l t s . E x a m p l e 2 : F o r T e s t 2 - D e t e r m i n i n g i f p e r f o r m a n c e d e g r a d e s w i t h a d d i t i o n o f d a t a , f r o m n o n - d e s i g n s e t d e p t h s , t o t h e t e s t s e t . C a s e A - T e s t s e t i s t h e p r i m a r y d a t a s e t ( T a b l e 5 . 1 ) w i t h t h e r e c o r d s d rawn t o f o r m t h e d e s i g n s e t r e m o v e d . C a s e B -• T e s t s e t as a b o v e , w i t h r e c o r d s f r o m t h e a d d i t i o n a l d a t a s e t ( T a b l e 5 . 2 ) i n c l u d e d . D e s i g n R e s u l t R e s u l t D i f f e r e n c e Rank S e t u n d e r A u n d e r B B - A 1 . 7 9 4 . 8 9 4 . 1 0 2 2 1 .06 5 . 5 0 4 . 4 4 5 3 1 . 0 6 5 . 3 4 4 . 2 6 3 4 . 5 3 4 . 8 9 4 . 3 6 4 5 . 7 9 5 . 34 4 . 55 6 6 1 .06 4 . 8 9 3 . 8 3 1 N e g a t i v e r ank sum i s 0 . H 0 i s r e j e c t e d as a b o v e ; p e r f o r m a n c e does d e g r a d e s i g n i f i c a n t l y w i t h t h e i n c l u s i o n o f d a t a f r o m n o n - d e s i g n s e t d e p t h s . 66 Re ferences [ I ] Chen, C .H . , S t a t i s t i c a l P a t t e rn R e c o g n i t i o n , Hayden, 1973, pp. 109-111 [2.] i b i d , p p . 21-24 [3] Y. D a s , J . E . McFee, and M.E. B e l l , " D e t e c t i o n and I d e n t i f i c a t i o n of Bur ied Ordnance by Magnetic and E l e c t r o m a g n e t i c Means", S u f f i e l d Report No. 283, Defence Research E s t a b l i s h m e n t S u f f i e l d , R a l s t o n , A l b e r t a , Canada, June 1981,pp. 25-29 [4] i b i d , p p . 29-33 [5] Y. Das, p r i v a t e communication [6] W.H. Highleyman, "The Design and A n a l y s i s of P a t t e r n R e c o g n i t i o n E x p e r i m e n t s " , B e l l Sy s t . Tech. J . , V o l . 41, March 1962 ,pp. 723-744 [7] R. Kemp, "A T h e o r e t i c a l Report f o r Pu l sed Eddy C u r r e n t Metal D e t e c t o r " , P les sy Company L t d . , Havant, U.K., March 1970 [8] J . E . McFee and Y. Das, "The D e t e c t i o n of Bur ied E x p l o s i v e O b j e c t s " , Can. J . of Remote Sens ing , V o l . 6, No. 2, December 1980 ,pp. 104-121 [9] i b i d , p . 108 [10] J . E . McFee and Y. Das, "Review of Unexploded Ordnance D e t e c t i o n Methods" , S u f f i e l d Report No. 292, Defence Research E s t ab l i shment S u f f i e l d , R a l s t o n , A l b e r t a , Canada, September 1981 [ I I ] W. Mendenhall and R.L. S c h e a f f e r , Mathemati ca l  S t a t i s t i c s With A p l i c a t i o n s , Duxbury, 1973. pp. 532-535 [12] i b i d , pp. 539-542 [13] i b i d , p. A43 [14] A . F . Mood, I n t r o d u c t i o n to the Theory of S t a t i s t i c s , M c G r a w - H i l l , 1950, pp. 387-189 67 [15] J . T . Tou and R.C. G o n z a l e z , P a t t e r n R e c o g n i t i o n P r i n c i p i e s , A d d i s o n - W e s l e y , 1974, pp.124-127 [16] i b i d , p. 87 [17] i b i d , pp. 263-269 [18] G.T. T o u s s a i n t , " B i b l i o g r a p h y on E s t i m a t i o n o f C l a s s i f i c a t i o n " , IEEE T r a n s , on I n f o r m a t i o n T h e o r y , V o l . IT- 2 0 , J u l y 1974, pp. 472-479 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0095648/manifest

Comment

Related Items