UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

A cooperative scheme for image understanding using multiple sources of information Glicksman, Jay 1982

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1983_A1 G55.pdf [ 21.94MB ]
Metadata
JSON: 831-1.0051869.json
JSON-LD: 831-1.0051869-ld.json
RDF/XML (Pretty): 831-1.0051869-rdf.xml
RDF/JSON: 831-1.0051869-rdf.json
Turtle: 831-1.0051869-turtle.txt
N-Triples: 831-1.0051869-rdf-ntriples.txt
Original Record: 831-1.0051869-source.json
Full Text
831-1.0051869-fulltext.txt
Citation
831-1.0051869.ris

Full Text

A COOPERATIVE SCHEME FOR IMAGE UNDERSTANDING USING MULTIPLE SOURCES OF INFORMATION by JAY GLICKSMAN B.Sc. The U n i v e r s i t y of Toronto, 1975 M.E. The U n i v e r s i t y of Utah, 1977 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES DEPARTMENT OF COMPUTER SCIENCE We accept t h i s t h e s i s as conforming to the r e q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA November 1982 © Jay Glicksman, 1982 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t of the requirements f o r an advanced degree at the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and study. I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying o f t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the head of my department or by h i s o r her r e p r e s e n t a t i v e s . I t i s understood t h a t copying or p u b l i c a t i o n of t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without my w r i t t e n p e r m i s s i o n . Department of Ct>fnPuT€i So(C/JC€ The U n i v e r s i t y of B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 DE-6 (3/81) A b s t r a c t One method of r e s o l v i n g the ambiguity inherent i n i n t e r -p r e t i n g images i s to add d i f f e r e n t sources of i n f o r m a t i o n . The m u l t i p l e i n f o r m a t i o n source paradigm emphasizes the a b i l i t y to u t i l i z e knowledge gained from one source that may not be present i n another. However, u t i l i z i n g d i s p a r a t e i n f o r m a t i o n may c r e a t e s i t u a t i o n s i n which data from d i f f e r e n t sources are incon-s i s t e n t . A schemata-based system has been developed that can take advantage of m u l t i p l e sources of i n f o r m a t i o n . Schemata are.com-bined i n t o a semantic network v i a the r e l a t i o n s decomposition, s p e c i a l i z a t i o n , i n s t a n c e of, and neighbour. C o n t r o l depends on the s t r u c t u r e of the e v o l v i n g network and a c y c l e of p e r c e p t i o n . Schemata cooperate by message p a s s i n g so that a t t e n t i o n can be d i r e c t e d where i t w i l l be most advantageous. T h i s system has been implemented to i n t e r p r e t a e r i a l photo-graphs of small urban scenes. Geographic f e a t u r e s are i d e n t i -f i e d u s i n g up to three i n f o r m a t i o n sources: the i n t e n s i t y image, a sketch map, and i n f o r m a t i o n p r o v i d e d by the user. The product i s a robust system where the accuracy of the r e s u l t s r e f l e c t s the q u a l i t y and amount of data p r o v i d e d . Images of s e v e r a l geo-graphic l o c a l e s are analyzed, and p o s i t i v e - r e s u l t s are r e p o r t e d . Table of Contents 1 I n t r o d u c t i o n 1 1 . 1 The Domain 1 1.2 Steps Toward a S o l u t i o n 3 1.3 The Scope of the T h e s i s 7 1.4 Reading Guide 8 2 M u l t i p l e Information Sources 10 2.1 A New Paradigm 10 2.1.1 Paradigms 10 2.1.2 The Current State of Computational V i s i o n 11 2.1.3 A New Focus 13 2.2 Information Sources 14 2.2.1 What i s an Information Source? 14 2.2.2 The Conjecture 16 2.2.3 I l l u s t r a t i v e Examples 17 2.3 Research that has Combined Information Sources 20 2.3.1 In A r t i f i c i a l I n t e l l i g e n c e 20 2.3.2 In Computational V i s i o n 23 2.4 Summary and Looking Ahead 26 3 A Framework of the F i e l d 29 3.1 Model-Based V i s u a l P e r c e p t i o n 29 3.1.1 D i f f e r e n t Types of Models 30 3.1.2 Geographic Models 34 3.2 P o s s i b i l i t i e s i n Matching Data to Models 41 3.3 Knowledge Repre s e n t a t i o n ( D e s c r i p t i v e Adequacy) 44 i i i i v 3.4 Knowledge Representation (Procedural Adequacy) 49 3.5 G r a c e f u l Degradation (Robustness) 51 4 How to Combine M u l t i p l e Information Sources 53 4.1 I n t r o d u c t i o n 53 4.2 A Schemata M a n i p u l a t i o n System 54 4.2.1 Schemata Have Four P a r t s 55 4.2.2 M a n i p u l a t i n g Schemata 60 4.3 I n t e r a c t i o n as a Source of Information 61 4.3.1 A c c e p t i n g and Accomodating Information from Users 63 4.4 A Cycl e of P e r c e p t i o n 64 4.4.1 I n t e r a c t i o n i n the Cycle of P e r c e p t i o n 67 4.5 Mixing Top-Down and Bottom-Up C o n t r o l 72 4.5.1 Top-Down and Bottom-Up C o n t r o l 72 4.5.2 Rela x i n g Thresholds 75 4.5.3 Mixed C o n t r o l S t r a t e g i e s 77 4.5.4 D i s t r i b u t e d AI, VLSI, and P a r a l l e l P r o c e s s i n g — 80 4.6 Using G l o b a l s to Remove Thresholds 81 4.7 C l u s t e r A n a l y s i s as an Inference Mechanism 84 4.7.1 A Method f o r C l u s t e r i n g 88 4.7.2 Gaussian Smoothing 90 4.7.3 Peak and Trough D e t e c t i o n 91 4.7.4 The S t a b i l i t y H e u r i s t i c 94 4.7.5 Exte n s i o n s to M u l t i v a r i a t e Data 97 4.8 Summary 104 5 The Implementation 106 5.1 I n t r o d u c t i o n 106 5.2 MAIDS--MISSEE Aids 108 V 5.3 The Input Information Sources 111 5.3.1 The D i g i t i z e d Image 111 5.3.1.1 Edge D e t e c t i o n 113 5.3.1.2 Region Merging 115 5.3.1.2.1 A C a t e g o r i z a t i o n f o r the Regions 120 5.3.2 The Sketch Map 123 5.3.3 The User 129 5.4 I n s t a n t i a t i n g Schemata 130 5.4.1 I n s t a n t i a t i o n D i r e c t l y from the Information Sources 132 5.4.1.1 I n s t a n t i a t i n g %bridge-1 with the A i d of a Sketch Map 132 5.4.1.2 I n s t a n t i a t i n g %bridge-2 from the I n t e n s i t y Image 139 5.4.2 B u i l d i n g up the H i e r a r c h i e s 144 5.4.3 Top-Down Attached Procedures 147 5.5 C l u s t e r A n a l y s i s Using Gaussian-Smoothed Histograms . 151 5.5.1 Determining R e c t a n g u l a r i t y 153 5.6 Control--The P r i o r i t y Queue, Messages, and Demons ... 158 6 R e s u l t s and E v a l u a t i o n 163 6.1 I n s t a n t i a t i n g Objects With the A i d of a Sketch Map .. 163 6.2 I n s t a n t i a t i n g Objects from the I n t e n s i t y Image Alone 177 6.3 B u i l d i n g Up H i e r a r c h i e s 190 6.4 Moving Up and Down the H i e r a r c h i e s 193 6.5 How User I n t e r a c t i o n Can I n f l u e n c e I n t e r p r e t a t i o n ... 200 6.6 D e s c r i p t i v e Adequacy 203 6.7 Pro c e d u r a l Adequacy 205 v i 6.7.1 S o l u t i o n s to Problems i n Matching Data to Models 207 6.8 Robustness 209 7 Summary and Co n c l u s i o n s 211 7.1 A Summary of What i s New and I n t e r e s t i n g 211 7.2 Open Issues 214 7.3 Future D i r e c t i o n s 215 7.4 Con c l u s i o n s 217 References 218 Appendix A 230 Appendix B 240 Appendix C 243 Appendix D 246 Appendix E 247 L i s t of Tables Table 4.1 M o d i f i c a t i o n and R e t r i e v a l f o r Schemata A t t r i b u t e s 60 Table 4.2 The Relevant S t a t i s t i c s f o r F i g u r e s 4.12-17 99 Table 5.1 I n t e r p r e t a t i o n s of the A s h c r o f t Chains 126 Table 5.2 O r i e n t a t i o n s and Confidence Values 155 Table 6.1 A s h c r o f t : Sketch Map to Image 174 Table 6.2 Houston: Sketch Map to Image 175 Table 6.3 Spences Bridge: Sketch Map to Image 175 Table 6.4 Spences Bridge West: Sketch Map to Image 176 Table 6.5 Spences Bridge E a s t : Sketch Map to Image 176 Table 6.6 Cranbrook: Sketch Map to Image 176 Table 6.7 A l l Images: I n s t a n t i a t i o n R e s u l t s 189 Table 6.8 The Number of Regions Searched 198 v i i L i s t of F i g u r e s F i g u r e 1.1 An A e r i a l Photograph 2 F i g u r e 1.2 A Sketch Map 5 F i g u r e 2.1 An Image of Noise 18 F i g u r e 2.2 An Object of I n t e r e s t 18 F i g u r e 2.3 A l e t t e r 19 F i g u r e 2.4 Some context 19 F i g u r e 3.1 A Model of a R i v e r 35 F i g u r e 3.2 A Sketch Map 37 .Figure 3.3 Mapsee2 Decomposition H i e r a r c h y 39 F i g u r e 3.4 Mapsee2 S p e c i a l i z a t i o n H i e r a r c h y f o r Geosystems 40 F i g u r e 3.5 P o s s i b i l i t i e s i n Matching Data to Models 42 F i g u r e 3.6 A Maya Instance: *BRIDGE-1 45 F i g u r e 3.7 An FRL Frame: SUPPLY 46 F i g u r e 4.1 A Schema: % r i v e r - 6 57 F i g u r e 4.2 A C y c l e of P e r c e p t i o n 66 F i g u r e 4.3 I n t e r a c t i o n i n a C y c l e of P e r c e p t i o n 68 F i g u r e 4.4 H i e r a r c h i e s i n a Geographic Domain 73 F i g u r e 4.5 Edge Segments Superimposed on Cranbrook 86 F i g u r e 4.6 A Histogram of Edge Segment O r i e n t a t i o n s 87 F i g u r e 4.7 An I d e a l i z e d , Continuous Histogram 88 F i g u r e 4.8 The Gaussian D i s t r i b u t i o n 92 F i g u r e 4.9 A Gaussian-Smoothed Histogram and D e r i v a t i v e s .. 93 F i g u r e 4.10 The E f f e c t s of o on Smoothing 95 F i g u r e 4.11 F i n d i n g the L a r g e s t S t a b l e Region of a Values . 96 v i i i ix F i g u r e 4.12 A 2-D Histogram of a B i v a r i a t e D i s t r i b u t i o n ... 98 F i g u r e 4.13 The Histogram Convolved with the Gaussian 100 F i g u r e 4.14 The Histogram Convolved with the L a p l a c i a n .... 101 F i g u r e 4.15 Zero C r o s s i n g s of the Second D e r i v a t i v e 102 F i g u r e 4.16 Determining a Using the S t a b i l i t y H e u r i s t i c ... 103 F i g u r e 4.17 Using Channels as D e c i s i o n Boundaries 105 F i g u r e 5.1 The MISSEE System 107 F i g u r e 5.2 A s h c r o f t , B r i t i s h Columbia 112 F i g u r e 5.3 A s h c r o f t : Edge D e t e c t i o n 114 F i g u r e 5.4 Ordering Edges in A s h c r o f t , a = 1.1 116 F i g u r e 5.5 O r d e r i n g Edges in A s h c r o f t , a = 2.2 . .' 117 F i g u r e 5.6 A s h c r o f t : Region Merging 119 F i g u r e 5.7 A s h c r o f t : Region C l a s s i f i c a t i o n 121 F i g u r e 5.8 A Sketch Map of A s h c r o f t 124 F i g u r e 5.9 A s h c r o f t with a Superimposed Sketch Map 125 F i g u r e 5.10 A Mapsee2 Decomposition Hierarchy of A s h c r o f t . 127 F i g u r e 5.11 The Generic Objects i n MISSEE 131 F i g u r e 5.12 The Stereotype %bridge Schema 133 F i g u r e 5.13 Three %curb Instances 137 F i g u r e 5.14 Instance %bridge-1 139 F i g u r e 5.15 A s h c r o f t : %bridge-1 Regions 140 F i g u r e 5.16 A s h c r o f t : %bridge-1 Edge Segments (Curbs) 141 F i g u r e 5.17 Instance %bridge-2 144 F i g u r e 5.18 B u i l d i n g up the Semantic Network 146 F i g u r e 5.19 An A s h c r o f t Instance H i e r a r c h y 148 F i g u r e 5.20 C l u s t e r i n g O r i e n t a t i o n s i n A s h c r o f t %road-4 ... 154 F i g u r e 5.21 A s h c r o f t : %road-4 Edges 156 X F i g u r e 5.22 C l u s t e r i n g O r i e n t a t i o n s along the Boundary .... 157 F i g u r e 5.23 The E x e c u t i o n C y c l e 161 F i g u r e 6.1 Houston, B r i t i s h Columbia 164 F i g u r e 6.2 Spences Bridge, B r i t i s h Columbia 165 F i g u r e 6.3 Cranbrook, B r i t i s h Columbia 166 F i g u r e 6.4 A s h c r o f t : Information Sources 167 F i g u r e 6.5 Houston: Information Sources 168 F i g u r e 6.6 Spences B r i d g e : Information Sources 169 F i g u r e 6.7 Spences Bridge West: Information Sources 170 F i g u r e 6.8 Spences Bridge E a s t : Information Sources 171 F i g u r e 6.9 Cranbrook: Information Sources 172 F i g u r e 6.10 A s h c r o f t : Sketch Map. %town-1 178 F i g u r e 6.11 A s h c r o f t : Sketch Map. R i v e r Regions 179 F i g u r e 6.12 A s h c r o f t : Sketch Map. Mountain Edges 180 F i g u r e 6.13 A s h c r o f t : Sketch Map. Roads 1-3, Edges 181 F i g u r e 6.14 A s h c r o f t : Sketch Map. Roads 4-6, Edges 182 F i g u r e 6.15 A s h c r o f t : Sketch Map. Roads 1-3, Regions 183 F i g u r e 6.16 A s h c r o f t : Sketch Map. %road-2 185 F i g u r e 6.17 A s h c r o f t : Image Alone. Four Roads 186 F i g u r e 6.18 A s h c r o f t : Image Alone. %road-12 187 F i g u r e 6.19 A s h c r o f t : Image Alone. S e v e r a l Urban Regions .. 188 F i g u r e 6.20 A s h c r o f t : Instance H i e r a r c h y 2 192 F i g u r e 6.21 A s h c r o f t : Instance H i e r a r c h y 3 194 F i g u r e 6.22 Houston: An Instance H i e r a r c h y 195 F i g u r e 6.23 Spences B r i d g e : An Instance H i e r a r c h y 196 Acknowledgments F i r s t and f o r e m o s t , I w i s h t o thank my s u p e r v i s o r , A l a n Mackworth, f o r h i s s u p p o r t ( b o t h i n t e l l e c t u a l and f i n a n c i a l ) , g u i d a n c e , and a d v i c e . I am a l s o i n d e b t e d t o him f o r w r i t i n g Mapsee and e x p o s i n g me t o t h e p o t e n t i a l f o r e x p l o i t i n g t h e c o n -s t r a i n t s i n h e r e n t i n models of g e o g r a p h i c o b j e c t s . I would a l s o l i k e t o a c k n o w l e d g e h i s p e r m i s s i o n t o use t h e d i a g r a m s t h a t a r e f o u n d i n F i g u r e s 3.3 and 3 .4. Thanks a r e a l s o e x t e n d e d t o t h e r e s t o f my c o m m i t t e e , Bob Woodham, R i c h a r d R o s e n b e r g , D a v i d K i r k p a t r i c k , and Anne T r e i s -man, who made h e l p f u l comments a l o n g t h e way t o t h i s f i n a l d o c u -ment . I am g r a t e f u l t o B i l l Havens, J a n M u l d e r , and A l a n f o r e x p l a n a t i o n s of t h e i n n e r w o r k i n g s o f Maya and Mapsee2, and f o r f i x i n g t h e bugs t h a t I f o u n d . J i m L i t t l e gave me a g r e a t d e a l of a s s i s t a n c e w i t h a l g o r i t h m s f o r l i n e g e n e r a l i z a t i o n , f i n d i n g c h a n n e l s and p a s s e s , and edge d e t e c t i o n u s i n g a S o b e l edge d e t e c t o r . The l a t t e r p r o g r am was s u p e r c e d e d by t h e M a r r -H i l d r e t h edge d e t e c t o r ( s o w i l l n o t be d e s c r i b e d h e r e i n ) but t h a t was n o t J i m ' s f a u l t . I w o u l d l i k e t o t h a n k my o f f i c e m a t e s f o r b e i n g i n t e r e s t e d i n what I was d o i n g and f o r n o t b e i n g v i s i o n a r i e s w h i c h f o r c e d me t o e x p l a i n my work i n E n g l i s h . I l e a r n e d a l o t a b o u t what I had done from t h o s e d i s c u s s i o n s w i t h Randy G o e b e l and Bob M e r c e r . Many members of t h e L a b o r a t o r y f o r C o m p u t a t i o n a l V i s i o n p r o v i d e d i n s p i r a t i o n a nd f e e d b a c k . E s p e c i a l l y h e l p f u l were J i m L i t t l e , J a n M u l d e r , Roger Browse, A l a n Mackworth, B i l l Havens, Bob Woodham, and Marc M a j k a . And i n a c l a s s a l l by h e r s e l f , Susan Suh. T h a n k s . x i CHAPTER 1 I n t r o d u c t i o n J_.J_. The Domain Remote sensing i s p e r c e p t i o n at a d i s t a n c e . L i k e computer v i s i o n , i t i s p r i m a r i l y concerned with the i d e n t i f i c a t i o n and r e p r e s e n t a t i o n of o b j e c t s i n the e x t e r n a l world. Computer-based remote sensing and computer v i s i o n both have as t h e i r usual source of i n f o r m a t i o n a two-dimensional a r r a y of i n t e n s i t y v a l u e s . U n l i k e the general v i s i o n problem, computer-based remote sensing cannot take advantage of many a d d i t i o n a l t h r e e -dimensional cues. Furthermore, whereas human v i s i o n (the simu-l a t i o n of which i s o f t e n t y p i f i e d as the v i s i o n problem) i s r e s -t r i c t e d to a narrow range of the ele c t r o m a g n e t i c spectrum, remote sensing o f t e n u t i l i z e s s e v e r a l d i f f e r e n t p o r t i o n s of the spectrum. F i g u r e 1.1 was taken from an a i r p l a n e f l y i n g over A s h c r o f t , B r i t i s h Columbia. People have no problem d i s c e r n i n g geographic e n t i t i e s such as the roads, r i v e r s , b u i l d i n g s , or rai l w a y t r a c k s . A e r i a l p h o t o - i n t e r p r e t a t i o n i s q u i t e u s e f u l f o r such tasks as land management, cartography, monitoring p o l l u t i o n , e r o s i o n , and urban growth, e s t i m a t i n g crop y i e l d , and t r a c k i n g the movement of s h i p s , c a r s , and the l i k e . 1 2 F i g u r e 1.1 An A e r i a l Photograph ( s c a l e reduced from o r i g i n a l 23 cm. x 23 cm. photograph) Chapter 1. I n t r o d u c t i o n 3 Computer-based a e r i a l p h o t o - i n t e r p r e t a t i o n i s a l s o a u s e f u l r e s t r i c t i o n of the machine v i s i o n problem because i t i s essen-t i a l l y two-dimensional and the number of o b j e c t s that are con-s i d e r e d i s manageable. However, as has been found to be gen-e r a l l y t rue i n A r t i f i c i a l I n t e l l i g e n c e , tasks that seem "easy" fo r people, l i k e language and v i s i o n , are the most d i f f i c u l t to do by machine. What s a l i e n t f e a t u r e s of an a e r i a l photograph such as F i g -ure 1.1 enable a person to i d e n t i f y i t s p a r t s ? C e r t a i n l y the i n t e n s i t i e s of the v a r i o u s r e g i o n s and where the i n t e n s i t i e s change are important. But that i s r a r e l y s u f f i c i e n t . Gen-e r a l l y , we must use our knowledge of the o b j e c t s t h e m s e l v e s — t h e i r shapes, r e l a t i v e s i z e s , and the c o n f i g u r a t i o n s of many o b j e c t s taken t o g e t h e r . T h i s d i s s e r t a t i o n d i s c u s s e s the use of knowledge of the o b j e c t s to be modelled in geographic scenes. I t aims to combine the i n f o r m a t i o n that can be e x t r a c t e d d i r e c t l y from images with data dependent on the domain of the task. Model knowledge can be used to o r g a n i z e the data and to c o n t r o l the process of i n t e r p r e t a t i o n . _.2. Steps Toward a S o l u t i o n The i n f o r m a t i o n found i n a s i n g l e , monochromatic image such as F i g u r e 1.1 i s o f t e n very ambiguous. The t r a n s i t i o n from a b r i g h t to a dark region might represent the boundary between a bridge and water, a road and a b u i l d i n g , the s u n l i t and shadowed Chapter 1. I n t r o d u c t i o n 4 p a r t s of a r i d g e , or many other p a i r s of o b j e c t s . While some a e r i a l photographs can be r e a d i l y i n t e r p r e t e d by a novice, oth-ers are so ambiguous that even experts may have d i f f i c u l t y i n d i s c o v e r i n g the c o r r e c t i n t e r p r e t a t i o n . One way to a i d the i n t e r p r e t a t i o n of an image, f o r both people and a computer program, i s to add more i n f o r m a t i o n . Maps (e.g. topographic and road maps) c o n t a i n many s t y l i z e d symbols which make i t e a s i e r to understand them. A sketch map i s a freehand drawing of the c r u c i a l f e a t u r e s of an a r e a . I t encodes a great d e a l of i n f o r m a t i o n i n a very sparse p i c t u r e (see F i g u r e 1.2 ) . E a s i l y drawn, i t can be used to a i d the i n t e r p r e t a t i o n of a c o r r e s p o n d i n g image. The work d e s c r i b e d i n t h i s document i s p r i m a r i l y concerned with the u s e f u l n e s s of combining d i f f e r e n t types of i n f o r m a t i o n and the mechanisms to do so. A t h i r d source of i n f o r m a t i o n that can be u t i l i z e d , besides the i n t e n s i t y image and a sketch map, i s the user who can provide g e n e r a l i n f o r m a t i o n (e.g. "The nomi-n a l s c a l e of the image i s 1:10000") or very s p e c i f i c i n f o r m a t i o n (e.g. "There i s a bridge i n the c e n t r e of the image"). A l s o , two f e a t u r e s that can be d e r i v e d from an i n t e n s i t y image are edges and r e g i o n s . They are g e n e r a l l y e x t r a c t e d i n d i f f e r e n t ways and can be used to emphasize d i f f e r e n t aspects of the image. A l l these d i f f e r e n t s o r t s of data must be r e l a t e d to the models of the o b j e c t s to be i d e n t i f i e d i n the scene. However, Chapter 1. I n t r o d u c t i o n 5 F i g u r e 1.2 A Sketch Map while i t i s hoped that a l l the evidence w i l l support the same c o n c l u s i o n , mismatches may occur between the data and the models. These d i f f i c u l t i e s must be handled in a c o n s i s t e n t and robust manner so that the accuracy of the r e s u l t s w i l l r e f l e c t the q u a l i t y of the a v a i l a b l e data. One way of accomodating d i s p a r a t e types of i n f o r m a t i o n i s to r e l a t e them a l l to the o b j e c t s themselves. Schemata, or frames, are a data s t r u c t u r e that can be used to s t o r e a l l the Chapter 1. I n t r o d u c t i o n 6 f a c t s p e r t i n e n t to a p a r t i c u l a r e n t i t y such as a road. Schemata c o n t a i n a d e c l a r a t i v e p a r t where in f o r m a t i o n d e r i v e d from an i n t e r p r e t a t i o n can be s t o r e d (e.g. "The edges t h a t correspond to t h i s road i n the image are 11, 12, and 13") and a p r o c e d u r a l p a r t that c o n t a i n s methods f o r u t i l i z i n g i n f o r m a t i o n sources to i n s t a n t i a t e the e x i s t e n c e of the o b j e c t (such as using the sketch map with the image, using the image a l o n e ) . Schemata can be c o n j o i n e d i n t o a network that r e f l e c t s t h e i r o r g a n i z a t i o n i n t o cohesive u n i t s i n the scene. For in s t a n c e , a network of roads and bri d g e s can be r e l a t e d as c o u l d a l l the e n t i t i e s that are pa r t of a landmass d e l i n e a t e d by a body of water. Agreement i n the i n t e r p r e t a t i o n of neighbouring i n s t a n c e s r a i s e s the co n f i d e n c e of the e n t i r e group. For exam-p l e , roads are o f t e n found next to b u i l d i n g s , not s h i p s . F u r t h -ermore, the e x i s t e n c e of some in s t a n c e s r a i s e s the e x p e c t a t i o n that there w i l l be other r e l a t e d i n s t a n c e s nearby. I d e n t i f i c a -t i o n of a bridge p o i n t s to the p o s s i b i l i t y of a r i v e r f l o w i n g under i t . Using model knowledge i n t h i s way i s an i s l a n d - d r i v e n c o n t r o l s t r a t e g y wherein the d i s c o v e r y of one o b j e c t leads to others nearby. T h i s group of o b j e c t s expands i n subsequent steps to encompass a l l of the r e l e v a n t p a r t s of the scene. I t i s c r u c i a l that the system be p r o c e d u r a l l y adequate so that o b j e c t s are not i n c o r r e c t l y i n s t a n t i a t e d . The procedures atta c h e d to schemata should not presuppose the e x i s t e n c e of an ob j e c t without f i n d i n g the proper support i n the data nor should they jump to c o n c l u s i o n s . T h i s i s i n t i m a t e l y r e l a t e d to the Chapter- 1. I n t r o d u c t i o n 7 d e s c r i p t i v e adequacy of the m o d e l s — a r e d e f i n i t i o n s s u f f i c i e n t to c o n f i r m or deny the e x i s t e n c e of the o b j e c t s using the a v a i l -a b l e sources of information? The d e s c r i p t i o n s must f i t i n t o a midrange, n e i t h e r too l i m i t e d nor too encompassing. In a d d i -t i o n , the d e s c r i p t i o n s must be s u i t a b l e f o r any p o s s i b l e a p p l i -c a t i o n s that are of i n t e r e s t (e.g. "How long are the bri d g e s i n an image"). j_.3_. The Scope of the T h e s i s S e v e r a l d i s p a r a t e i n f o r m a t i o n sources are combined to i n t e r p r e t a e r i a l photographs. Along with the d i g i t i z e d image, in f o r m a t i o n from sketch maps and the user i s made a v a i l a b l e . The i n t e r p r e t a t i o n r e s u l t s depend on the amount and q u a l i t y of the i n f o r m a t i o n p r o v i d e d . A new v a r i e t y of schemata has been developed along with a system f o r man i p u l a t i n g them. The schemata provide a con-s i s t e n t , uniform r e p o s i t o r y f o r i n f o r m a t i o n coming from d i f -f e r e n t i n f o r m a t i o n sources. Schemata c o n t a i n a t t a c h e d pro-cedures which c o n t r o l the i n t e r p r e t a t i o n process. As e n t i t i e s are d i s c o v e r e d i n the image they cause i n s t a n c e s of schemata to be c r e a t e d and l i n k e d i n t o a semantic network. T r a v e r s a l of the growing network enables c o n t r o l to be e i t h e r top-down or bottom-up, whichever i s most a p p r o p r i a t e . The nodes of the semantic network communicate by passing messages. The messages are suggestions f o r f u t u r e i n s t a n t i a -t i o n s and are placed on a p r i o r i t y queue. P r i o r i t i e s depend Chapter 1. I n t r o d u c t i o n 8 upon the c o n f i d e n c e values of the instances sending the messages and are determined dur i n g i n s t a n t i a t i o n by how w e l l the data f i t the models. P r o c e s s i n g proceeds in a c y c l i c manner f o l l o w i n g a c y c l e of p e r c e p t i o n . T h i s c y c l e p r o v i d e s convenient p o i n t s where i n t e r a c t i o n with the user can take p l a c e . S e v e r a l computational t o o l s are developed to a i d i n the i n s t a n t i a t i o n of geographic e n t i t i e s . C l u s t e r a n a l y s i s using Guassian-smoothed histograms i s an i n f e r e n c e mechanism f o r determining which data elements best f i t model d e f i n i t i o n s . A method of r e l a x i n g t h r e s h o l d s i s based on the used of model knowledge. In a d d i t i o n , many t h r e s h o l d s can be reduced to s i n -g l e values when model knowledge i s combined with g l o b a l sche-mata . In summary, t h i s d i s s e r t a t i o n d e s c r i b e s a v i s i o n system that can accomodate i n f o r m a t i o n from s e v e r a l sources. I t i s o b j e c t - o r i e n t e d and based around schemata t h a t form a semantic network of geographic o b j e c t s and i n s t a n c e s of those o b j e c t s . It u t i l i z e s the c u r r e n t context of i n s t a n t i a t e d e n t i t i e s to con-t r o l where f u t u r e e f f o r t should be d i r e c t e d . J_.4. Reading Guide The next three chapters correspond to the three p a r t s of the t i t l e of the t h e s i s i n r e v e r s e order. Chapter 2 i n t r o d u c e s the m u l t i p l e i n f o r m a t i o n sources paradigm as the u n d e r l y i n g hypothesis of t h i s work. Chapter 3 i s concerned with one aspect of image understanding: model-based v i s u a l p e r c e p t i o n . Previous Chapter 1. I n t r o d u c t i o n 9 r e l e v a n t r e s e a r c h i n the f i e l d i s o u t l i n e d . In a d d i t i o n , three c r i t e r i a f o r judging the success of a model-based v i s i o n system are d e s c r i b e d . Chapter 4 d i s c u s s e s the c o o p e r a t i v e scheme, a schema-based system that i s used to combine m u l t i p l e i n f o r m a t i o n sources. Each s e c t i o n of t h i s chapter d e s c r i b e s a component of the system. Chapter 5 d i s c u s s e s MISSEE, the implementation of the sys-tem d e s c r i b e d i n Chapter 4. R e s u l t s and e v a l u a t i o n of the sys-tem are the t o p i c of Chapter 6. F i n a l l y , Chapter 7 summarizes the work and p o i n t s to p o s s i b l e d i r e c t i o n s f u t u r e r e s e a r c h might proceed i n . Readers who are concerned with i s s u e s of knowledge r e p r e s e n t a t i o n and not v i s i o n may wish to read only those p a r t s of the d i s s e r t a t i o n concerned with schemata. The r e l e v a n t sec-t i o n s are 3.3, 3.4, 4.2, 4.5.3, 5.2, 5.4.2, 5.6, 6.6, 6.7, and Appendix A. Chapter 1. I n t r o d u c t i o n CHAPTER . 2 M u l t i p l e Information Sources 2.]_. A New Paradigm 2.]_.]_. Paradigms In h i s i n f l u e n t i a l book, "The S t r u c t u r e of S c i e n t i f i c Revo-l u t i o n s " (Kuhn, 1970), Thomas Kuhn prese n t s a theory to account for the means by which s c i e n c e progresses i n a p a r t i c u l a r f i e l d . "Normal s c i e n c e " g e n e r a l l y r e v o l v e s around the n o t i o n of a "paradigm". Paradigms "provide models from which s p r i n g p a r t i c u -l a r l y coherent t r a d i t i o n s of s c i e n t i f i c r e s e a r c h " (p. 10). Where paradigms e x i s t (e.g. Ptolemaic astronomy) advances i n the f i e l d c o n s i s t of s o r t i n g out the problems not s o l v e d by the work that d e f i n e s the paradigm. I t i s important to note that the paradigm i n f l u e n c e s which problems are s i g n i f i c a n t . Thus s c i e n t i s t s not working w i t h i n the paradigm are not c o n s i d e r e d to be working on meaningful problems by the adherents of the c u r r e n t l y dominant paradigm. Paradigms can be adopted i n two ways. An e x i s t i n g paradigm can be supplanted by a new theory, the " s o - c a l l e d r e v o l u t i o n " , which r e d e f i n e s l e g i t i m a t e problems and rese a r c h methods (e.g. the Copernican " r e v o l u t i o n " which r e p l a c e d Ptolemaic astronomy). [The new s c i e n t i f i c ] achievement [must be] s u f f i c i e n t l y unprecedented to a t t r a c t an enduring group of adherents away from competing modes of s c i e n t i f i c a c t i v i t y . Simul-10 11 taneously, i t [must be] s u f f i c i e n t l y open-ended to leave a l l s o r t s of problems f o r the r e d e f i n e d group of p r a c t i -t i o n e r s to r e s o l v e , (p. 10) Paradigms can a l s o emerge i n a young s c i e n c e where none are c u r r e n t l y e s t a b l i s h e d . In pre-paradigmatic s c i e n c e , there are no g e n e r a l l y accepted views which a c t as a focus f o r r e s e a r c h . Rather, there are many small groups of p r a c t i t i o n e r s with t h e i r own views and methodologies. In t h i s s t a t e , s c i e n c e proceeds more s l o w l y : . . . when the i n d i v i d u a l s c i e n t i s t can take a paradigm fo r granted, he need no lo n g e r , i n h i s major works, a t -tempt to b u i l d h i s f i e l d anew, s t a r t i n g from f i r s t p r i n -c i p l e s and j u s t i f y i n g the use of each concept i n t r o -duced. That can be l e f t to the w r i t e r of textbooks, (pp. 19-20) Only when a paradigm has been e s t a b l i s h e d can progress become cumulative so that one can stand on the shoulders of g i a n t s (as Newton wrote) r a t h e r than on t h e i r t o e s . 2.J_.2. The Current State of Computational V i s i o n A r t i f i c i a l I n t e l l i g e n c e , i n g e n e r a l , and Computational V i s i o n , i n p a r t i c u l a r , are very young sciences' without a c l e a r l y r e c o g n i z e d c e n t r a l paradigm. For e x a c t l y t h i s reason, some peo-p l e q u e s t i o n whether they are even s c i e n c e s at a l l . Compared to p h y s i c s , the lack of u n i f y i n g p r i n c i p l e s i s noteworthy[1]. [ 1 ] U n f o r t u n a t e l y there are many o l d e r d i s c i p l i n e s that have the same d i f f i c u l t y . T h i s leads some to q u e s t i o n whether Kuhn 1s theory i s a p p l i c a b l e to e v e r y t h i n g that i s l a b e l l e d s c i e n c e . I t leads others to qu e s t i o n whether the l a b e l " s c i e n c e " i s always a p p r o p r i a t e l y a p p l i e d . Chapter 2. M u l t i p l e Information Sources 1 2 While s e v e r a l textbooks on A r t i f i c i a l I n t e l l i g e n c e have been w r i t t e n ( N i l s s o n , 1971 and 1980; Winston, 1977), only r e c e n t l y has a textbook on computational v i s i o n appeared ( B a l l a r d and Brown, 1982). While there has never been a g e n e r a l l y accepted paradigm i n computer v i s i o n , there have been s e v e r a l u n i f y i n g models that have extended beyond the l a b o r a t o r y of t h e i r o r i g i n a t o r . The l a b e l l i n g of l i n e drawings i n the blocks world has had a long h i s t o r y (Huffman, 1971; Clowes, 1971; Kanade, 1981) as a means of determining which s u r f a c e s belong to which d i s t i n c t o b j e c t s . The work of David Marr and h i s group on the p r i m a l sketch and i t s use i n e a r l y v i s i o n (Marr, 1976 and 1982) has i n s p i r e d much f u r t h e r r e s e a r c h . The p r i m a l sketch i s designed to be a r i c h , i n t e r m e d i a t e r e p r e s e n t a t i o n and i s modelled on what takes p l a c e i n b i o l o g i c a l v i s i o n s y s t e m s — y e t another paradigm. In a pre-paradigmatic f i e l d , a l l problems seem e q u a l l y i n t e r e s t i n g . In computational v i s i o n , some of the problems that have r e c e i v e d a t t e n t i o n i n c l u d e the d e t e c t i o n of i n t e n s i t y d i s c o n t i n u i t i e s (edges) i n an image ( S h i r a i , 1975; Davis, 1975), d e t e c t i o n of homogeneous regions i n an image (Yakimovsky and Feldman, 1973; Kanade, 1980), the o r i e n t a t i o n of a s u r f a c e ("shape from" shading: Horn, 1975; s t e r e o : Grimson, 1981; con-t o u r : W i t k i n , 1981; t e x t u r e : Render, 1980; motion: Ullman, 1978), the three- d i m e n s i o n a l r e p r e s e n t a t i o n of o b j e c t s (Brooks, 1981), and the mo d e l l i n g of what might appear i n scenes (Sloan and Bajcsy, 1977). Chapter 2. M u l t i p l e Information Sources 1 3 2 ' 1 « 3 . A New Focus The major hypothesis of the d i s s e r t a t i o n concerns m u l t i p l e i n f o r m a t i o n sources. As w i l l be seen in l a t e r s e c t i o n s , other r e s e a r c h has u t i l i z e d m u l t i p l e i n f o r m a t i o n sources, but only as a secondary f a c e t of i t s s o l u t i o n to the problem at hand. The proposal i s to make the noti o n of m u l t i p l e i n f o r m a t i o n sources primary, and to r e l a t e the other a s p e c t s of problem s o l u t i o n to i t . In t h i s sense the use of m u l t i p l e i n f o r m a t i o n sources i s being presented as a new paradigm. I t i s not presumed that t h i s i s a major paradigm that p r o v i d e s coherence f o r the whole f i e l d [ 2 ] . Rather, the paradigm generates a problem domain ( i . e . how to best manipulate and c o o r d i n a t e the i n f o r m a t i o n sources) that can p r o v i d e a focus f o r r e s e a r c h . The r e s t of t h i s chapter w i l l t r y to p r o v i d e a c o n v i n c i n g case fo r u s i n g m u l t i p l e i n f o r m a t i o n sources as a focus of a t t e n -t i o n . Subsequently, the t h e s i s w i l l d i s c u s s the problems of d e a l i n g with i n f o r m a t i o n sources, both i n d i v i d u a l l y and .in con-c e r t , i n l i g h t of the the paradigm. One s o l u t i o n to those prob-lems w i l l be addressed i n d e t a i l . Thus the o v e r a l l ( s i m p l i f i e d ) t h e o r e t i c a l t h r u s t of the t h e s i s i s as f o l l o w s : major hypothesis--one u s e f u l s o l u t i o n to the problem of understanding [ 2 ] I t i s c l e a r that a c r u c i a l aspect of p e r c e p t i o n would be missing from t h i s paradigm because i t i s p o s s i b l e , f o r people at l e a s t , to form a percept from a s i n g l e i n f o r m a t i o n source such as a photograph. Chapter 2. M u l t i p l e Information Sources 14 images i s to use m u l t i p l e i n f o r m a t i o n sources (Chapter 2); secondary h y p o t h e s i s — u s e f u l t o o l s that allow one to take advan-tage of m u l t i p l e i n f o r m a t i o n sources i n c l u d e schema-based languages, user i n t e r a c t i o n , and combined top-down and bottom-up c o n t r o l (Chapter 4). A p a r t i c u l a r implementation of those t o o l s (Chapter 5) demonstrates t h e i r u s e f u l n e s s i n the geographic domain of understanding a e r i a l photographs. 2.2. Information Sources 2_.2_.j_. What i s an I nf ormat ion Source? An i n f o r m a t i o n source i s a set of input data. I t can e i t h e r be what i s i n i t i a l l y input to the system or an i n t e r m e d i -ate r e s u l t that becomes a v a i l a b l e to l a t e r stages of p r o c e s s i n g . Standard i n f o r m a t i o n sources (henceforth sometimes a b b r e v i a t e d IS) i n computational v i s i o n are d i g i t i z e d images (a two-dimensional a r r a y of i n t e n s i t y values b e a r i n g some r e l a t i o n to the amount of l i g h t r e f l e c t e d from o b j e c t s i n the scene in view) and l i n e drawings ( v e r t i c e s and the l i n e s between them forming some i d e a l r e p r e s e n t a t i o n of the edges of s u r f a c e s of o b j e c t s i n the scene). The IS i s the input to a v i s i o n system and the out-put i s some symbolic r e p r e s e n t a t i o n of the scene. There i s , of course, a range of symbolic r e p r e s e n t a t i o n s that can be handled. A d i g i t i z e d image i s at the low end of the s c a l e . I t i s an i c o n i c r e p r e s e n t a t i o n of the scene s i n c e there i s a d i r e c t geometric mapping between them. A l i n e drawing i s more a b s t r a c t because i t only captures one aspect of the image, Chapter 2. M u l t i p l e Information Sources 15 namely, the edges of s u r f a c e s . One can progress f u r t h e r up the s c a l e to a r e p r e s e n t a t i o n of o b j e c t s and f i n a l l y to s p e c i f i c o b j e c t s such as c h a i r s and t a b l e s . The task of an image under-standing system i s to take an IS represented somewhere in that range and produce a "more a b s t r a c t " output f u r t h e r up the s c a l e . Besides t h i s range of how a b s t r a c t a r e p r e s e n t a t i o n i s , there e x i s t s an orthogonal dimension based on the type of i n f o r -mation t h a t can be found i n an i n f o r m a t i o n source. T h i s i s i l l u s t r a t e d c l e a r l y i n the work done by Barrow and Tenenbaum on i n t r i n s i c images (Barrow and Tenenbaum, 1978). T h e i r b a s i c model c o n s i s t s of a stack of r e g i s t e r e d a r r a y s which are i c o n i c r e p r e s e n t a t i o n s of the image. Each i n d i v i d u a l a r r a y c o n t a i n s a d i f f e r e n t kind of i n f o r m a t i o n : i n t e n s i t y , i l l u m i n a t i o n , r e f l e c -tance, o r i e n t a t i o n , and d i s t a n c e . In t h e i r system there i s only one i n p u t , the i n t e n s i t y image; the other r e p r e s e n t a t i o n s are b u i l t up from i t through an i t e r a t i v e r e l a x a t i o n p r o c e s s . Thus the n o n - i n t e n s i t y a r r a y s are both IS's and the f i n a l output r e p r e s e n t a t i o n . T h i s i s an example of how i n f o r m a t i o n sources can cooperate with each other. S e v e r a l of the other types of i n f o r m a t i o n used in computer v i s i o n i n c l u d e c o l o u r images ( u s u a l l y represented as three i n t e n s i t y images--for red, green, and b l u e ) , m u l t i s p e c t r a l images[3], range data which g i v e s d i s t a n c e d i r e c t l y , d i g i t a l [ 3 ] T h i s i s a g e n e r a l i z a t i o n of c o l o u r images where c e r t a i n bands i n the e l e c t r o m a g n e t i c spectrum are sampled. The most com-mon examples of t h i s are i n f r a - r e d photography and s a t e l l i t e im-agery such as Landsat which has 4 MSS bands. Chapter 2. M u l t i p l e Information Sources 16 t e r r a i n models ( y i e l d i n g the h e i g h t s of p o i n t s i n the scene), sketch maps ( g i v i n g the t o p o l o g i c a l r e l a t i o n s h i p s between sym-b o l i c o b j e c t s ) , and geographic data bases ( c o n t a i n i n g the l o c a -t i o n and c h a r a c t e r i s t i c s of landmarks). 2.2_.2. The Conjecture The goal of image understanding i s to determine the mapping from what i s a c t u a l l y sensed, the image, to the scene of i n t e r e s t . U n f o r t u n a t e l y , the problem i s v a s t l y u nderconstrained i n g e n e r a l . The f a c t o r s r e s u l t i n g from p r o p e r t i e s of the o b j e c t s , shape and s u r f a c e m a t e r i a l , cannot be separated from each other or from f a c t o r s a r i s i n g from i l l u m i n a t i o n , shadows, viewing d i r e c t i o n , path phenomena, or sensor noise (Woodham, 1981). Images are ambiguous and can be the r e s u l t of an i n f i n -i t e number of s c e n e s [ 4 ] . The hypothesis i s t h a t , M u l t i p l e i n f o r m a t i o n sources are u s e f u l i n r e s o l v i n g am-b i g u i t i e s . Or, simply, More i s b e t t e r . T h i s viewpoint i s widely h e l d by many v i s i o n r e s e a r c h e r s as w i l l be seen in the d i s c u s s i o n of systems that take advantage of [4 ] T h i s i s the op p o s i t e of human v i s i o n i n which images usu-a l l y seem to be o v e r c o n s t r a i n e d and not ambiguous at a l l . Chapter 2. M u l t i p l e Information Sources 1 7 m u l t i p l e IS's. There i s a st r o n g e r c o n j e c t u r e t h a t can be made, however, that has not been addressed b e f o r e . T h i s r e l a t e s to the n o t i o n of d i f f e r e n t types of i n f o r m a t i o n . The more d i s p a r a t e that the i n f o r m a t i o n sources combined are, the b e t t e r able they are to r e s o l v e a m b i g u i t i e s . Or, r e s t a t e d , The more d i f f e r e n t , the b e t t e r . T h i s i s based on the i n t u i t i o n that s i m i l a r aspects of the scene w i l l be encoded i n d i f f e r e n t ways by the d i f f e r e n t i n f o r -mation sources. One thus has more chance of d i s c o v e r i n g what i s important i n the i n t e r p r e t a t i o n . Some examples may c l a r i f y these i d e a s . 2.2.3. I l l u s t r a t i v e Examples If one s t a r t s with an i n t e n s i t y image, then the simplest a d d i t i o n a l i n f o r m a t i o n source t o add would be another i n t e n s i t y image (more i s b e t t e r ) . To o b t a i n something i n t e r e s t i n g , one would l i k e to vary at l e a s t one parameter of the imaging pro-c e s s . Changing the viewpoint a l l o w s one to perform s t e r e o p s i s which r e s u l t s i n a depth map (Grimson, 1981,'.Baker and B i n f o r d , 1981). Changing the p o s i t i o n of the source of i l l u m i n a t i o n a l l o w s one to take advantage of a type of s t e r e o c a l l e d pho-t o m e t r i c s t e r e o (Woodham, 1980a). If the time parameter i s a l t e r e d , one i s able to determine shape from motion (Ullman, Chapter 2. M u l t i p l e Information Sources 18 1 978) . A simple example w i l l show how people use motion to good advantage. Consider the image of random dots i n F i g u r e 2.1. C l e a r l y , there i s nothing of i n t e r e s t i n t h i s image. Now con-s i d e r F i g u r e 2.2 below to be on a t r a n s p a r e n t o v e r l a y r e s t i n g on top of F i g u r e 2.1. The square would not stand out. of the n o i s e . However, i f one moved the o v e r l a y around, the square would s p r i n g i n t o view. Stop the motion and the square becomes i n d i s -t i n g u i s h a b l e from the background. If one had two i n t e n s i t y images separated in time, then a simple s u b t r a c t i o n would r e v e a l the square i n t h i s example ( A n s t i s , 1970). F i g u r e 2.1 An Image of Noise Fi g u r e 2.2 An Object of I n t e r e s t Chapter 2. M u l t i p l e Information Sources 19 I f the square had been a d i f f e r e n t c o l o u r , then i t would have been apparent, to people, r i g h t away. In t h i s case, one can imagine two i n t e n s i t y a r r a y s , one f o r the c o l o u r of the n o i s e , which i s u n i n t e r e s t i n g , and one f o r the c o l o u r of the square, which i s i n t e r e s t i n g . In a l l of the above, the IS's have been the s a m e - - i n t e n s i t y images. Now c o n s i d e r the advantages of using d i f f e r e n t kinds of i n f o r m a t i o n . F i g u r e 2.3 c o n t a i n s a l i n e drawing that r e p r e s e n t s a l e t t e r drawn h a l f way between an "A" and an "H". Now t h i s image i s d e l i b e r a t e l y ambiguous and no other s i m i l a r image would be of any h e l p . However, what i f context was added as an i n f o r -mation source? The e x p e c t a t i o n s c r e a t e d i n F i g u r e 2.4a would probably cause us to conclude that the l e t t e r was an "H". S i m i -l a r l y , i f the context was as p r o v i d e d i n F i g u r e 2.4b then the F i g u r e 2.3 A l e t t e r LETTER_BEFORE = T LETTER AFTER = E LETTER_BEFORE = C LETTER AFTER = T a b F i g u r e 2.4 Some context Chapter 2. M u l t i p l e Information Sources 20 l e t t e r i s l i k e l y an "A" ( N e i s s e r , 1967). . The Stroop e f f e c t i s another example of how d i f f e r e n t types of i n f o r m a t i o n can i n f l u e n c e p e r c e p t i o n ( N e i s s e r , 1976). Stroop had s u b j e c t s read words which were the names of c o l o u r s (e.g. "brown"). When the l e t t e r s of a word were drawn in a d i f f e r e n t c o l o u r than the word represented (e.g. "brown" drawn in green i n k ) , then s u b j e c t s had more d i f f i c u l t y , and a s i g n i f i c a n t l y slower r e a c t i o n time, than when the names and c o l o u r s c o i n c i d e d . Here, then, i s a case of where people are compelled to use mul-t i p l e IS's even though i t degrades t h e i r performance of a task. 2.3_. Research that has Combined I nf ormat ion Sources 2^3_.J_. I_n A r t i f i c i a l I n t e l l i g e n c e The u s e f u l n e s s of combining a second source of i n f o r m a t i o n with a problem s o l v e r was r e c o g n i z e d i n the e a r l y days of Al r e s e a r c h . G e l e r t n e r ' s work on p r o v i n g geometry theorems was g r e a t l y a i d e d by the i n f o r m a t i o n provided i n a diagram which r e p r e s e n t s the problem[5] ( G e l e r t n e r , 1963). When using a diagram to r u l e out impossible hypotheses, h i s program was a b l e to prune 995 successors out of 1000, on the average, at each node i n the search t r e e . Information sources are d i r e c t l y r e l a t e d to knowledge sources as they were d e f i n e d i n the HEARSAY II speech under-[5]Much i n the same way that high school students are. Chapter 2. M u l t i p l e Information Sources 21 standing p r o j e c t (Erman et a l . , 1980). In HEARSAY each knowledge source i s viewed as "an agent which embodies the knowledge of i t s area and which can take a c t i o n s based on that knowledge" (Erman and Le s s e r , 1975, p. 483). T h i s i s s i m i l a r to the ACTOR philosophy (Hewitt et a l . , 1973) of a community of experts work-ing c o o p e r a t i v e l y to so l v e problems. However, whereas ACTORs seem best s u i t e d to an o b j e c t - o r i e n t e d s o l u t i o n (see Chapters 3.4 and 4), knowledge sources were implemented as p r o d u c t i o n r u l e s . Information sources are r e l a t e d to knowledge sources i n that each IS r e q u i r e s a t l e a s t one KS to d e a l with i t . Thus IS's and KS's can be seen to be the two s i d e s of the same c o i n . The major d i f f e r e n c e between HEARSAY and t h i s work i s that the former c o n c e n t r a t e s on m u l t i p l e KS's while the l a t t e r d e a l s with m u l t i p l e IS's. There i s only one input IS i n HEARSAY--the speech waveform. The other IS's e x i s t as int e r m e d i a t e output that r e s i d e together on the blackboard and p r o v i d e context f o r subsequent stages of p r o c e s s i n g , together with the r e l e v a n t i n f o r m a t i o n r e t r i e v e d from the Long Term Memory. Desp i t e t h a t , major p h i l o s o p h i c a l d i f f e r e n c e , there are s e v e r a l methodological s i m i l a r i t i e s between HEARSAY and the sys-tem d e s c r i b e d i n Chapters 4 and 5. They both attempt to keep knowledge modular ( i n pr o d u c t i o n r u l e s on the one hand; schemata on the o t h e r ) . They both e x p l o i t a h i e r a r c h i c a l r e p r e s e n t a t i o n (the blackboard; the semantic network). Feedback c o n t r o l i s used i n both systems (hypothesize and t e s t ; a c y c l e of Chapter 2. M u l t i p l e Information Sources 22 p e r c e p t i o n ) . They both s t r i v e to achieve g r a c e f u l d e g r a d a t i o n when the i n f o r m a t i o n i s poor. Knowledge "sources have been used e x t e n s i v e l y i n Al systems, c u s u a l l y when i t i s u s e f u l to d i s t i n g u i s h the types of knowledge that can be a p p l i e d to the problem at hand. New knowledge sources can be i n t e g r a t e d i n t o DENDRAL because i t i s r e q u i r e d that they encode t h e i r knowledge i n the form of p r o d u c t i o n r u l e s which a l l communicate in terms of a common graph language. Data a c q u i r e d from new types of instruments turned out to be very u s e f u l . . . . the impact of b r i n g i n g j u s t one a d d i t i o n a l source of knowledge to bear on a problem can be s t a r t l i n g . In one d i f f i c u l t (but not u n u s u a l l y d i f f i c u l t ) mass spec-trum a n a l y s i s problem, the program using i t s mass spec-trometry knowledge alone would have generated an impos-s i b l y l a r g e set of p l a u s i b l e candidates (over 1.25 m i l -l i o n ! ) . Our e n g i n e e r i n g response to t h i s was to add another source of data and knowledge, proton NMR. The a d d i t i o n of a simple i n t e r p r e t i v e theory of t h i s NMR data, from which the program c o u l d i n f e r a few a d d i t i o n -a l c o n s t r a i n t s , reduced the set of p l a u s i b l e candidates to one, the r i g h t s t r u c t u r e ! T h i s was not ,an i s o l a t e d r e s u l t but showed up dozens of times i n subsequent ana-l y s e s . (Feigenbaum, 1977, p. 1020) S e v e r a l r e s e a r c h e r s have argued for the use of d i s p a r a t e types of knowledge. Brown and Burton t a l k of the "synergism obtained by f o c u s i n g the d i v e r s e c a p a b i l i t i e s of the p r o c e d u r a l s p e c i a l i s t s " (Burton and Brown, 1975, p. 312). Davis has emphasized t h e i r importance (Davis, 1980a) and even r a i s e d t h e i r use to a p r i n c i p l e : A f o u r t h pr inc i p l e - - e x p l o i t i n g redundancy — i s n i c e l y i l -Chapter 2. M u l t i p l e Information Sources 23 l u s t r a t e d by work on HEARSAY (Erman et a l . , 1980) that i l l u s t r a t e d how redundancy can be a remedy f o r incom-p l e t e and in e x a c t knowledge. The t r i c k i s t o f i n d m u l t i -p l e o v e r l a p p i n g sources of knowledge with d i f f e r e n t areas of s t r e n g t h and d i f f e r e n t shortcomings. P r o p e r l y used, the e n t i r e c o l l e c t i o n of knowledge sources can be a good deal more robust than any one of them taken a l o n e . (Davis, 1982, pp. 6-7) 2.3.2. In Computational V i s i o n As i n the l a r g e r f i e l d of AI, s e v e r a l r e s e a r c h e r s have argued f o r b r i n g i n g " d i v e r s e sources of knowledge together" (Hanson and Riseman, 1978, p. 316; see a l s o Flinchbaugh and Chandrasekaran, 1981, p. 391; and W i t k i n , 1981, p. 44). R u s s e l l has claimed that h i s system "must be able to work over a wide range of inputs from many sources" ( R u s s e l l , 1979, p. 179). He does not explore the consequences of that b e l i e f . P r evious work that combines more than one i n t e n s i t y image to take advantage of ste r e o or motion has a l r e a d y been men-t i o n e d . Another c l a s s i c use of s e v e r a l i n t e n s i t y images i s found i n p a t t e r n r e c o g n i t i o n where c l a s s i f i c a t i o n can be done by m u l t i d i m e n s i o n a l c l u s t e r a n a l y s i s (see H a l l , 1979, Chapter 8). Each dimension r e p r e s e n t s a d i f f e r e n t i n t e n s i t y image which i s d e r i v e d by sampling from a d i s t i n c t s p e c t r a l band. Using d i f -f e r e n t s p e c t r a l bands to f i n d d i s t i n g u i s h e d f e a t u r e s works because some p r o p e r t i e s stand out i n c e r t a i n bands more than i n o t h e r s . For example, the boundary between land and water i s q u i t e c l e a r i n imagery from band 7 of Landsat (near i n f r a - r e d ) . T h i s methodology works when the t e r r a i n i s f l a t , but i s not Chapter 2. M u l t i p l e Information Sources 24 e f f e c t i v e when the s u r f a c e slope causes o c c l u s i o n s and shadows. For more complicated scenes, d i g i t a l t e r r a i n models can be combined e f f e c t i v e l y with i n t e n s i t y images (Horn and Bachman, 1978; Woodham, 1980b). A DTM records the e l e v a t i o n of a l l of the p o i n t s i n the scene. If the i l l u m i n a t i o n d i r e c t i o n i s known, as i n Landsat imagery, then i t i s p o s s i b l e to generate a s y n t h e t i c image from the DTM, making some assumptions concerning the s u r f a c e m a t e r i a l of the scene o b j e c t s p l u s some other imag-ing parameters. By r e g i s t e r i n g the s y n t h e t i c image to a r e a l remotely sensed image, one can account f o r the e f f e c t s of slope and a low i l l u m i n a t i o n angle. At t h i s p o i n t , the d i f f e r e n c e between the two images should i d e a l l y correspond to the charac-t e r i s t i c s of s u r f a c e cover. In mountainous t e r r a i n ' t h i s t e c h -nique produces good r e s u l t s but work conti n u e s so that more aspe c t s of the imaging process may be understood. That i s , e i t h e r more knowledge or more IS's are r e q u i r e d . Work has been done at SRI to combine a map data base with a d i g i t i z e d image (Tenenbaum et a l . , 1978; B o l l e s et a l . , 1979). As they e x p l a i n , Map knowledge can provide important c o n s t r a i n t s on where to look i n an image, what to look f o r , and how to i n t e r -p r e t what i s seen. Such c o n s t r a i n t s , p r o p e r l y e x p l o i t e d , permit the e x t r a c t i o n of complex i n f o r m a t i o n without ex-t e n s i v e computation. (Tenenbaum et a l . , 1978, p. 1) The map data base c o n t a i n s t h r e e - d i m e n s i o n a l l o c a t i o n s and shapes of landmarks and monitoring s t a t i o n s . I t can be indexed by l o c a t i o n , e n t i t y name, and e n t i t y type. SRI r e s e a r c h e r s have Chapter 2. M u l t i p l e Information Sources 25 produced a working system that demonstrates the f e a s i b i l i t y of u s i n g t h i s technique i n s e v e r a l a p p l i c a t i o n s , such as measuring the volume of water i n a r e s e r v o i r , counting r a i l w a y boxcars, c o u n t i n g s h i p s i n a harbour, and t r a c k i n g roads. In c o n t r a s t to the other systems examined, the map data base i s a very a b s t r a c t r e p r e s e n t a t i o n of i n f o r m a t i o n and very d i f f e r e n t i n kind from an i n t e n s i t y image. T h i s work c l e a r l y shows that i t i s p o s s i b l e to combine d i s p a r a t e types of i n f o r m a t i o n to good advantage. A method of u t i l i z i n g d i f f e r e n t road o p e r a t o r s has a l s o o r i g i n a t e d at SRI ( F i s c h l e r et a l . , 1981). I t uses one input IS, but many op e r a t o r s can be used to produce d i f f e r e n t i n t e r n a l images c o n t a i n i n g the l i k e l i h o o d that each p i x e l i s part of a road. The g l o b a l f i g u r e s of merit are then o p t i m i z e d to produce the "best" road image. T h i s program i s d i s c u s s e d f u r t h e r i n Sect ion 3.1.2. Di s p a r a t e sources of i n f o r m a t i o n can be combined in systems that use e v i d e n t i a l reasoning i f the model knowledge has been s u i t a b l y encoded (Wesley and Hanson, 1982; Garvey et a l . , 1981). T h i s has been done in the VISIONS system u s i n g a dependency graph and a model of b e l i e f that manipulates both support and p l a u s i b i l i t y . I t has the a b i l i t y t o e f f e c t i v e l y handle informa-t i o n that i s imprecise, u n c e r t a i n , and i n e x a c t . There are other examples of model-based v i s i o n systems that w i l l be d e s c r i b e d i n S e c t i o n 3.1. Where the model i s e x p l i c i t l y d e f i n e d , i t can be c o n s i d e r e d an i n f o r m a t i o n source. In other Chapter 2. M u l t i p l e Information Sources 26 cases the model knowledge i s compiled i n t o the procedures that c a r r y out the i n t e r p r e t a t i o n . I t i s o f t e n d i f f i c u l t to draw a l i n e between these cases. 2._. Summary and Looking Ahead T h i s chapter has l a i d the framework for the t h e s i s . The n o t i o n of using m u l t i p l e i n f o r m a t i o n sources w i l l s t r o n g l y i n f l u e n c e the d e s c r i p t i o n s i n the next chapter o u t l i n i n g the c u r r e n t r e l e v a n t p u r s u i t s i n the f i e l d s of computer v i s i o n and knowledge r e p r e s e n t a t i o n . Moreover, t h i s n o t i o n p r o v i d e s the r a t i o n a l e f o r the proposed s o l u t i o n s to the problem of how to best u t i l i z e i n f o r m a t i o n sources. Information sources have been d e s c r i b e d as having three a s p e c t s . F i r s t , an IS can be input data that i s fed to the sys-tem from the o u t s i d e or i t can be an intermediate r e s u l t that i s a v a i l a b l e to l a t e r processes or i t e r a t i o n s . Second, IS's w i l l vary over a wide range with r e s p e c t to how a b s t r a c t they a r e : from i c o n i c r e p r e s e n t a t i o n s a l l the way to "names" l i k e " c h a i r " . F i n a l l y , in a dimension that can be orthogonal to the previous d i s t i n c t i o n , i n f o r m a t i o n sources can encode d i f f e r e n t types of i n f o r m a t i o n . In accordance with the two c o n j e c t u r e s of S e c t i o n 2.2.2 (more and d i f f e r e n t are b e t t e r ) emphasis in l a t e r chapters w i l l be on using s e v e r a l input IS's c o n t a i n i n g d i f f e r e n t types of i n f o r m a t i o n and v a r y i n g g r e a t l y i n t h e i r symbolic nature. Evidence f o r the e f f i c a c y of combining m u l t i p l e IS's was presented i n t h i s chapter. S e v e r a l advantages were Chapter 2. M u l t i p l e Information Sources 27 demonstrated, and claimed, f o r t h e i r use. Primary i s the a b i l -i t y to r e s o l v e a m b i g u i t i e s . Where i n f o r m a t i o n does not e x i s t i n one IS or i s not c o n c l u s i v e i t may be found, or be r e t r i e v a b l e , i n another. Furthermore, combining m u l t i p l e IS's allows one to b e t t e r deal with inexact and incomplete knowledge. I f the "experts" that know about the IS's can cooperate, then together they can s o l v e problems that as i n d i v i d u a l s they c o u l d not. Besides s u f f i c i e n c y reasons f o r combining IS's (the problem can be s o l v e d with them), there are a l s o e f f i c i e n c y reasons (the problem can be s o l v e d more q u i c k l y ) . For example, there are many "expensive" l o c a l o p e r a t i o n s to be c a r r i e d out on i n t e n s i t y images, such as c o n v o l u t i o n f o r edge d e t e c t i o n . I f a second IS, such as a map data base or sketch map, can i s o l a t e " i n t e r e s t i n g " areas i n the image, then these l o c a l o p e r a t i o n s need not be c a r -r i e d out over an e n t i r e image. Such a focus of a t t e n t i o n mechanism can r e s u l t i n c o n s i d e r a b l e savings i n computation. There are a l s o p o t e n t i a l disadvantages i n t r y i n g to combine d i s p a r a t e types of knowledge. F i r s t , there i s the correspon-dence problem. I t i s sometimes necessary to r e g i s t e r represen-t a t i o n s of the image to each other, as i n s t e r e o p s i s , before any u s e f u l i n f o r m a t i o n can be d e r i v e d . More g e n e r a l l y , one must d e a l with the problem of c o n f l i c t i n g i n f o r m a t i o n where one KS may d e r i v e a f a c t c o n t r a d i c t o r y to what was d i s c o v e r e d by a KS from a d i f f e r e n t IS. Chapter 2. M u l t i p l e Information Sources 28 The concepts of ambiguity, incompleteness, and incon-s i s t e n c y w i l l be made more s p e c i f i c i n the next chapter. Chapter 2. M u l t i p l e Information Sources CHAPTER 3 A Framework of the F i e l d 3.J_. Model-Based V i s u a l Percept ion Computer v i s i o n systems can roughly be d i v i d e d i n t o two types. Domain independent v i s i o n ( a l s o known as image-based, low l e v e l , or e a r l y ) i s concerned with c a p t u r i n g as much i n f o r -mation as p o s s i b l e from an image without recourse to knowledge of the u n d e r l y i n g scene. Domain dependent v i s i o n ( a l s o known as model-based or high l e v e l ) attempts to i d e n t i f y o b j e c t s i n the scene and make use of t h e i r semantics to a i d the i n t e r p r e t a t i o n . Domain independent v i s i o n a p p l i e s to g e n e r a l , i n t r i n s i c p r o p e r t i e s of images (Zucker et a l . , 1975). C e r t a i n assumptions regarding the images are usually" made to make the problems more t r a c t a b l e [ 1 ]. These systems g e n e r a l l y deal with such image f e a t u r e s as edges, s u r f a c e s , and r e f l e c t a n c e . T h i s work i s concerned with domain dependent or model-based v i s i o n . Model-based v i s i o n a p p l i e s to scenes c o n t a i n i n g c e r t a i n types of o b j e c t s such as cubes, c h a i r s , t a b l e s , roads, c a r s , t r e e s , and houses. Only those o b j e c t s can be "re c o g n i z e d " . Domain dependent systems can take advantage of knowledge con-c e r n i n g the scene to i n f l u e n c e the process of i n t e r p r e t a t i o n . [1]Sometimes these assumptions r e l a t e to what can appear i n the scene, making them model-dependent i f not model-based. 29 30 For example, horses never have more than four l e g s . Nonethe-l e s s , model-based systems must c o n t a i n some i n t e r f a c e to an image-based component to p r o v i d e the a p p r o p r i a t e data from the image. 3.J_.J_. D i f f e r e n t Types of Models The f i r s t v i s i o n system to use simple models was developed by Roberts (Roberts, 1965). The "world" c o n s i s t e d of a cube, a r e c t a n g u l a r wedge, a hexagonal prism, and combinations t h e r e o f . The t h r e e - d i m e n s i o n a l model, encoded in terms of i t s v e r t i c e s and l i n e s , was matched to the two-dimensional image v i a h e u r i s -t i c search. Furt h e r work i n the b l o c k s world domain allowed f o r more o b j e c t s to be modelled (e.g. F a l k , 1972, with 9 o b j e c t s ; see Mackworth, 1976, f o r a review). More complex three-dimensional o b j e c t s can be modelled as g e n e r a l i z e d c y l i n d e r s (Agin and Duda, 1973; Brooks et a l . , 1979) or g e n e r a l i z e d cones (Marr and N i s h i h a r a , 1978). Having models of o b j e c t s l e t s one take advantage of the known s p a t i a l r e l a t i o n s h i p s among them. K e l l y ' s h e u r i s t i c p l a n -ning system ( K e l l y , 1971) used knowledge of fa c e s to h e l p l o c a t e the other prominent f e a t u r e s (eyes, nose, mouth). F i s c h l e r and E l s c h l a g e r extended the idea to i n c l u d e the r e l a t i o n s h i p s among a l l the f e a t u r e s of the face ( F i s c h l e r and E l s c h l a g e r , 1973). In the ACRONYM system (Brooks et a l . , 1979; Brooks, 1981) a p r e d i c t i o n graph i s generated from the o b j e c t models of such Chapter .3. A Framework of the F i e l d 31 t h i n g s as machine p a r t s and a i r c r a f t . The c o n s t r a i n t s embodied i n the graph r e s t r i c t the p o s s i b i l i t i e s when a n a l y z i n g the image. In B o l l e s ' s work on v e r i f i c a t i o n v i s i o n ( B o l l e s , 1975), models of the o b j e c t s (machine p a r t s ) and the scene enable the system to run almost t o t a l l y top-down. When one has complete models of the o b j e c t s , as i n i n s p e c t i o n t a s k s , the models can d i c t a t e what should be looked f o r . Model knowledge can be captured i n v a r i o u s ways. Freuder d i s t i n g u i s h e s between ge n e r a l knowledge about the domain and p a r t i c u l a r knowledge about the image (Freuder, 1976). His " a c t i v e " knowledge modules were procedures embedded in~a t r e e s t r u c t u r e r e f l e c t i n g the c o n t r o l regime used f o r image i n t e r p r e -t a t i o n of hammers. The model can be represented e x p l i c i t l y i n the form of t u p l e s i n a r e l a t i o n a l data base (Levine and Shaheen, 1981), i n terms of frames (Sakai et a l . , 1976), i n a semantic network (Bajcsy and Lieberman, 1974), or i n p r o d u c t i o n r u l e s (Bajcsy and J o s h i , 1978). The model knowledge can a l s o be i m p l i c i t l y encoded i n a p r o c e d u r a l system ( S h i r a i , 1975) or using fuzzy set theory ( J a i n and Haynes, 1982). The input i n f o r m a t i o n source u t i l i z e d by the above systems i s g e n e r a l l y an i n t e n s i t y image. Other IS's have been used such as l i n e drawings (of houses: Mulder, 1979; of human body p a r t s : Browse, 1982). Kuipers represented people's map knowledge and Chapter 3. A Framework of the F i e l d 32 how they use i t to move about i n urban s e t t i n g s (Kuipers, 1977). Two tasks are p r e v a l e n t i n systems d e a l i n g with the f i r s t stages of v i s u a l p r o c e s s i n g . Model knowledge has been a p p l i e d to both edge d e t e c t i o n ( S h i r a i , 1975; Yachida et a l . , 1979) and s e g m e n t a t i o n — a l s o known as r e g i o n merging (Yakimovsky and F e l d -man, 1973; Tenenbaum, 1973; Tenenbaum and Barrow, 1976; Barrow and Tenenbaum, 1976; Hanson and Riseman, 1978a; Weymouth, 1981). Few comprehensive systems have encompassed s e v e r a l aspects of v i s i o n . The VISIONS system, developed at the U n i v e r s i t y of Massachusetts at Amherst (Hanson and Riseman, 1978a and 1978b; see Weymouth, 1981, f o r some more recent work) i s a l a r g e , comprehensive system f o r understanding n a t u r a l scenes composed of houses, t r e e s , c a r s , and other o b j e c t s . There i s a s i n g l e input IS (a c o l o u r i n t e n s i t y image) and a p l e t h o r a of i n t e r n a l i n f o r m a t i o n sources. Two major r e p o s i t o r i e s of knowledge are the short-term memory ( p a r t i c u l a r knowledge) and the long-term memory (general knowledge), both of which are h i e r a r c h i c a l and based on l e v e l s of a b s t r a c t i o n . The l e v e l s go from v e r t i c e s through segments, r e g i o n s , s u r f a c e s , volumes, and o b j e c t s to schemata. The schemata are c e n t r a l to the VISIONS system. They are used to represent both o b j e c t s and scenes. They p r o v i d e i n f o r -mation r e q u i r e d to b u i l d up the volume (or su r f a c e ) d e s c r i p t i o n of what i s represented i n an image. F i n a l l y , schemata guide the i n s t a n t i a t i o n of hypotheses d u r i n g i n t e r p r e t a t i o n . The system Chapter 3. A Framework of the F i e l d 33 f o l l o w s a hypothesize and t e s t mode of c o n t r o l (focus-expand-v e r i f y ) which, depending on the knowledge sources i n v o l v e d , c o u l d e x h i b i t e i t h e r bottom-up or top-down c o n t r o l . The ACRONYM v i s i o n system has been developed at S t a n f o r d (Brooks et a l . , 1979; Brooks, 1981a and 1981b). O b j e c t s , modelled in three dimensions as g e n e r a l i z e d cones, and t h e i r r e l a t i o n s h i p s are represented i n an " o b j e c t " graph. The " r e s -t r i c t i o n " graph c o n t a i n s c o n s t r a i n t s on the s p a t i a l r e l a t i o n -s h i p s between o b j e c t s . From these r e p r e s e n t a t i o n s , a " p r e d i c -t i o n " graph i s generated to hypothesize o b j e c t to image f e a t u r e matches. "Observation" and " i n t e r p r e t a t i o n " graphs are b u i l t d u r i n g the a n a l y s i s of an image. The input IS i s taken from the r e s u l t s of a l i n e f i n d e r . From t h i s , two-dimensional ribbons are found. Using a con-s t r a i n t m a n i p u l a t i o n system, these ribbons are matched a g a i n s t models i n the r e s t r i c t i o n graph. As the i n t e r p r e t a t i o n graph i s c o n s t r u c t e d , more s p e c i f i c models can be deduced (e.g. an L1011 i s a type of wide-bodied j e t ) . An important aspect of these v i s i o n systems i s the matter of program c o n t r o l . Most domain independent systems tend to operate in a bottom-up, l i n e a r f a s h i o n beginning with the image and working towards the p o s s i b l e o b j e c t s i n i t . Domain depen-dent systems g e n e r a l l y have a wider range of c o n t r o l s t r a t e g i e s , but most c o n t a i n e i t h e r a top-down component or a feedback loop of some s o r t . Such systems have the problem of d e c i d i n g which Chapter 3. A Framework of the F i e l d 34 o b j e c t to hypothesize (known as the chicken and egg problem: Mackworth, 1977b; or the p a r s i n g problem: Palmer, 1975), but once determined, these programs can make e f f i c i e n t use of o b j e c t - s p e c i f i c h e u r i s t i c s to pl a n t h e i r s t r a t e g i e s . A p r i o r i e x p e c t a t i o n s d e r i v e d from the models as w e l l as the context r e s u l t i n g from p r e v i o u s i n t e r p r e t a t i o n both p r o v i d e important c l u e s (and cues) to guide the understanding of images. Geographic Models The domain of t h i s d i s s e r t a t i o n concerns models of o b j e c t s that might be seen i n remotely sensed images. Objects one might t r y to d i s t i n g u i s h i n a e r i a l or s a t e l l i t e imagery i n c l u d e roads, r i v e r s , b r i d g e s , houses, mountains, and f i e l d s . Work has been done to study the c o g n i t i v e a s p e c t s of how people a c q u i r e i n f o r -mation from maps (Thorndyke, 1979) and s e v e r a l computational v i s i o n systems have been developed which use such knowledge. S e v e r a l p r o j e c t s at the U n i v e r s i t y of Pennsylvania have u t i l i z e d models of geographic e n t i t i e s . In one (Bajcsy and T a v a k o l i , 1973), models e x i s t f o r water, land, r i v e r s , l a k e s , b r i d g e s , and i s l a n d s . F i g u r e 3.1 shows the d e s c r i p t i o n of a r i v e r (Bajcsy and T a v a k o l i , 1973, p. 2A-57). T h i s simple model c o n t a i n s three of the four major p r o p e r t i e s of a ge n e r i c model: a l i s t of p r o p e r t i e s which d e f i n e the model (e.g. a r i v e r has t e x t u r e ) ; r e s t r i c t i o n s on those p r o p e r t i e s (e.g. t e x t u r e i s homogeneous); and r e l a t i o n s h i p s between o b j e c t s (e.g. s p a t i a l r e l a t i o n s h i p to b r i d g e : below). The l a s t aspect of a model i s a Chapter 3. A Framework of the F i e l d 35 R i v e r s : Gray value of the water Texture: homogeneous Boundaries: open C o n t r a s t : l a r g e S p a t i a l r e l a t i o n s h i p s to b r i d g e : below T o p o l o g i c a l r e l a t i o n s h i p s : continuous S p a t i a l r e l a t i o n s h i p s to l a n d : surrounded by F i g u r e 3.1 A Model of a River mechanism f o r determining i f the f e a t u r e s of a s p e c i f i c image match those i n the model d e s c r i p t i o n . In t h i s system these are d e r i v e d p r o c e d u r a l l y . In a s i m i l a r system, models of roads were used (Bajcsy and T a v a k o l i , 1976; see a l s o T a v a k o l i and Rosen-f e l d , 1980). S e v e r a l other r e s e a r c h e r s have developed systems to f i n d roads. S c i e n t i s t s at SRI have developed a method of using s e v e r a l types of i n f o r m a t i o n about roads. The approach i s based on a new paradigm f o r combining l o c a l i n f o r m a t i o n from m u l t i p l e and p o s s i b l y incommen-surat e , sources, i n c l u d i n g v a r i o u s l i n e and edge detec -t i o n o p e r a t o r s , map knowledge about the l i k e l y path of roads through an image, and generic knowledge about roads (e.g., c o n n e c t i v i t y , and width c o n s t r a i n t s ) . ( F i s c h l e r et a l . , 1981, p. 201) They d i v i d e a l l t h e i r i n f o r m a t i o n sources i n t o two c l a s s e s : Type I, where almost no f a l s e road elements are accepted, and Type I I , where p o s s i b l e f a l s e elements are accepted but l o c a l , r e l e v a n t parameters are a c c u r a t e l y measured. The IS's are a p p l i e d i n d i f f e r e n t ways and a l l the r e s u l t s are optimized f o r maximum l i k e l i h o o d scores (using assumptions of c o n t i n u i t y and Chapter 3. A Framework of the F i e l d 36 width) to d e r i v e the "best" t r a c k . A requirement f o r the IS's i s that the operator that produces them must a l s o supply a numerical l i k e l i h o o d estimate f o r each p o i n t i n the a r r a y . The m u l t i p l e IS's are combined by reducing them a l l to a common r e p r e s e n t a t i o n p l u s ad hoc m o d i f i c a t i o n s i n the o p t i m i z a t i o n procedure. Work was a l s o done at SRI on map-guided i n t e r p r e t a t i o n . I t was d e s c r i b e d i n S e c t i o n 2.3.2. S e l f r i d g e uses "appearance" models f o r e n t i t i e s such as b u i l d i n g s , roads, and shadows ( S e l f r i d g e and Sloan, 1981). A c r e d i t a b l e f e a t u r e of h i s system i s the use of adaptive opera-t o r s which allow parameters to vary when t r y i n g t o f i n d r e g i o n s having c e r t a i n areas, c o n t r a s t , e t c . ( c f . S e c t i o n 4.5.2 f o r a mechanism t h a t accomplishes s i m i l a r tasks.) Nagao and h i s coworkers have d e s c r i b e d a v i s i o n system f o r i n t e r p r e t i n g a e r i a l photographs modelled a f t e r HEARSAY-II (Nagao et a l . , 1978 and 1979). Common data i s a v a i l a b l e on a b l a c k -board and the knowledge i s encoded i n pr o d u c t i o n r u l e s . Object models f o r such items as crop f i e l d s , bare s o i l , grass l a n d , roads, and r i v e r s are not s t o r e d e x p l i c i t l y but are a v a i l a b l e p r o c e d u r a l l y as p a r t of "model-driven subsystems". I n t e r p r e t a -t i o n proceeds from ge n e r a l p r o p e r t i e s ( r e g i o n segmentation) to more s p e c i f i c ones ( s e l e c t i o n of cue regions) and a feedback loop allows the context to a i d i n the c l a s s i f i c a t i o n of ambigu-ous r e g i o n s . Chapter 3. A Framework of the F i e l d 37 Many maps use a b s t r a c t symbols to represent o b j e c t s , such as • f o r a b u i l d i n g , — j — | — | — | — | — f o r a r a i l w a y . A l i n e drawing c o n t a i n i n g only such s t y l i z e d symbols i s a sketch map. With only a l i t t l e p r a c t i c e , one can form a c o g n i t i v e map based only on the in f o r m a t i o n found in a sketch map (see F i g u r e 3.2). Mackworth developed a program to i n t e r p r e t sketch maps c a l l e d Mapsee (Mackworth, 1976c; see B a l l a r d et a l . , 1978 for a d i f f e r e n t use of sketch maps i n v i s i o n ) . Model knowledge i s found i n a t a b l e which i s indexed by p i c t u r e cues ( f e a t u r e s of l i n e drawings such as TEE, OBTUSE L, and FREE END). The cues are ambiguous but network c o n s i s t e n c y (Mackworth, 1977a) i s used F i g u r e 3.2 A Sketch Map Chapter 3. A Framework of the F i e l d 38 to reduce the l a b e l s e t s . T h i s i s done by t a k i n g advantage of the r e l a t i o n s h i p s between o b j e c t s (e.g. r i v e r s flow under br i d g e s ) that were used to b u i l d the t a b l e . A second system to s o l v e the same problem--Mapsee2--has a l s o been developed (Havens and Mackworth, 1980; Mackworth and Havens, 1981). T h i s system i s schema-based and w r i t t e n i n the language Maya (Havens, 1978). Schemata e x i s t f o r a l l the e l e -ments of the domain from p o i n t s through chains to roads, r i v e r s , r i v e r systems, and so on up to "world". A s s o c i a t e d with the schemata (which w i l l be d i s c u s s e d i n more d e t a i l i n S e c t i o n 3.3) are procedures f o r i n s t a n t i a t i n g them p l u s a mechanism to c a r r y on when that i n s t a n t i a t i o n i s completed. Another a d d i t i o n i n Mapsee2 i s the use of decomposition and s p e c i a l i z a t i o n h i e r a r -c h i e s to organize the schemata i n t o a coherent network. The complete decomposition h i e r a r c h y i s shown in F i g u r e 3.3 and the s p e c i a l i z a t i o n h i e r a r c h y that e x i s t s under geosystems can be found i n F i g u r e 3.4. R e c o g n i t i o n begins by c r e a t i n g h y p o t h e t i c a l i n s t a n c e s f o r e n t i t i e s that might correspond to each c h a i n . These are r e s -t r i c t e d i n some cases by the shapes of the c h a i n s . Thus, shore-l i n e s must be c l o s e d curves, towns must be blobs, mountains i n v e r t e d V s . However, many shapes are ambiguous and can gen-e r a t e m u l t i p l e i n s t a n c e s . The newly c r e a t e d i n s t a n c e s t r y to f i n d a schema in the decomposition h i e r a r c h y to complete to but the i n s t a n c e s must be c o n s i s t e n t with s p e c i a l i z a t i o n s of the schema. For example, roads must be next to r e g i o n s l a b e l l e d as Chapter 3. A Framework of the F i e l d F i g u r e 3.3 Mapsee2 Decomposition H i e r a r c h y Chapter 3. A Framework of the F i e l d 40 F i g u r e 3.4 Mapsee2 S p e c i a l i z a t i o n H i e r a r c h y f o r Geosystems land. I n c o n s i s t e n t i n s t a n c e s , such as a r i v e r going over a bridge, are pruned. H y p o t h e t i c a l , mutually e x c l u s i v e i n s t a n c e s form separate s p e c i a l i z a t i o n s . Unconnected i n s t a n c e s cause the c r e a t i o n of new higher l e v e l i n s t a n c e s , such as road systems. T h i s c o n t i n u e s f o r a l l the p o s s i b l e i n t e r p r e t a t i o n s f o r each c h a i n . How might t h i s work be extended? Mackworth s t a t e s , . . . i n the long run, . . . understanding [of LANDSAT images] would proceed more s u c c e s s f u l l y i f programs were able to accept a d v i c e , in the form of sketch maps, about Chapter 3. A Framework of the F i e l d 41 the geography u n d e r l y i n g the image. (Mackworth, 1978, p. 55) T h i s d i s s e r t a t i o n i s concerned with such a system. 3.2. P o s s i b i l i t i e s i n Matching Data to Models Image i n t e r p r e t a t i o n can be c o n s i d e r e d a mapping between models and data. I f the e v a l u a t i o n i s top-down then the models form the domain and data elements the range. The s i t u a t i o n i s reversed f o r bottom-up e v a l u a t i o n . Besides the problem of i n c o r r e c t i n t e r p r e t a t i o n , r e l a t i n g the wrong model with a datum, the r e l a t i o n s may not always be one to one and onto. F i g u r e 3.5 i l l u s t r a t e s the f i v e p o s s i b i l i t i e s that can occur when matching elements from the image (data) to models of scene o b j e c t s . Example elements i n an IS d e r i v e d from an input image i n c l u d e edges ( a c t u a l l y i n t e n s i t y d i s c o n t i n u i t i e s ) and regions (where i n t e n s i t i e s are homogeneous). Some scene o b j e c t s that might be modelled i n c l u d e roads, r i v e r s , and b r i d g e s . I d e a l l y , one would l i k e to d i s c o v e r a one to one r e l a t i o n -s h i p between a datum and a model. So, i f r i v e r s expect to be matched with a r e g i o n from the image, i t would be best i f one p a r t i c u l a r r e g i o n , say region-12, can be i d e n t i f i e d as a r i v e r ( a r b i t r a r i l y numbered, as r i v e r - 2 , to keep i t d i s t i n g u i s h a b l e ) . An i n c o n s i s t e n t s i t u a t i o n e x i s t s where more than one datum s a t i s f i e s the requirements of a model. A l t e r n a t i v e l y , t h i s i s a s i t u a t i o n where there i s evidence both f o r and a g a i n s t an Chapter 3. A Framework of the F i e l d F i g u r e 3 .5 P o s s i b i l i t i e s i n Matching Data to Models Chapter 3. A Framework of the F i e l d 43 i n t e r p r e t a t i o n . With regard to combining m u l t i p l e i n f o r m a t i o n sources, the data elements need not come from the same IS. T h i s can i n f a c t cause great d i f f i c u l t y i f the data from d i f f e r e n t IS's c o n f l i c t rather than i n agree. On the other hand, i f no data f u l f i l l the requirements of a model, then the model i s u n s a t i s f i e d . Here m u l t i p l e i n f o r m a t i o n sources can be an advantage, s i n c e there are more p o s s i b i l i t i e s to f i n d the data that w i l l s a t i s f y the model. Ambiguity i s g e n e r a l l y the g r e a t e s t problem f a c i n g computer v i s i o n systems. I t e x i s t s whether the system i s domain depen-dent or independent. Image f e a t u r e s are ambiguous when they can be l i n k e d with more than one model. Model-based systems can take advantage of more g l o b a l i n f o r m a t i o n such as the s p a t i a l o r g a n i z a t i o n of f e a t u r e s so that s i n g l e image elements are not the only f a c t s a v a i l a b l e . F i n a l l y , there can be data elements f o r which no mapping to a scene element can be found. T h i s can occur f o r two reasons: e i t h e r the " c o r r e c t " model has not been represented i n the sys-tem, or the model e x i s t s but i s too r i g i d l y d e f i n e d to accept t h i s " s p e c i a l case". U n t i l v i s i o n systems i n c l u d e modules capa-ble of a b s t r a c t ion and l e a r n i n g , t h i s w i l l remain a d i f f i c u l t y encountered i n model-based v i s i o n . One c r i t e r i o n f o r e v a l u a t i n g model-based v i s i o n systems i s how we l l they handle the four problems d e s c r i b e d above. In par-t i c u l a r , the problems of i n c o n s i s t e n c y and u n s a t i s f i a b i l i t y are Chapter 3. A Framework of the F i e l d 44 germane. These problems w i l l be examined again with regard to the s o l u t i o n s that have been d e v i s e d ( S e c t i o n 6.7.1). 3^.3. Knowledge Representation ( D e s c r i p t i v e Adequacy) Knowledge r e p r e s e n t a t i o n schemes l i e i n a continuum from t o t a l l y d e c l a r a t i v e to t o t a l l y p r o c e d u r a l . D e c l a r a t i v e s t r u c -t u r e s make knowledge e x p l i c i t and a v a i l a b l e f o r m o d i f i c a t i o n . P r o c e d u r a l systems are e a s i e r to c o n t r o l and can make reasoning more d i r e c t . Schema-based r e p r e s e n t a t i o n s are a compromise between the two extremes. Schemata ( B a r t l e t t , 1932) or frames (Minsky, 1975) (or s c r i p t s , s c e n a r i o s , u n i t s , templates, e tc.) are a means by which a s s o c i a t e d f a c t s can be r e l a t e d i n a convenient way. F u r t h e r -more, they can encode the methods by which the f a c t s may be u t i l i z e d . A l s o , schemata that are r e l a t e d i n an e x p l i c i t way, i n a semantic network, allow i n f e r e n c e s to be drawn from the manner i n which they are connected. Since they c o n t a i n both a d e c l a r a t i v e p a r t (the f a c t s and the network l i n k s ) and a pro-c e d u r a l part (methods f o r man i p u l a t i n g f a c t s ) , schemata can take advantage of the str e n g t h s of each. There are many v a r i e t i e s of schemata that have been developed. T h e i r merits can be judged on how w e l l they encode the knowledge r e q u i r e d to c a r r y out the tasks f o r which they have been designed. Questions of d e s c r i p t i v e adequacy i n c l u d e : how e x p l i c i t are the f a c t s ? how unif o r m l y are they encoded? can facts, be m o d i f i e d as new i n f o r m a t i o n i s di s c o v e r e d ? how modular Chapter 3. A Framework of the F i e l d 45 are i n d i v i d u a l schemata? how completely and c o r r e c t l y does the r e p r e s e n t a t i o n capture our i n t u i t i o n s ? See (Barrow and Tenen-baum, 1975) f o r some other c o n s i d e r a t i o n s . F i g u r e 3.6 shows a Maya (Havens, 1978) schema f o r a bridge i n Mapsee2. T h i s i s an in s t a n c e of a p a r t i c u l a r b r i d g e . Maya d i s t i n g u i s h e s p a r t i c u l a r from general knowledge through the use of i n s t a n c e s and o b j e c t s r e s p e c t i v e l y . The ins t a n c e r e l a t i o n -s h i p i s the only l i n k between schemata that has a s p e c i a l mean-ing to the i n t e r p r e t e r . - F a c t s are encoded as a t t r i b u t e name/value p a i r s . Note that values can be S-expressions, p o i n t e r s to other schemata (e.g. *CHAIN-5), or combinations of the two. Maya a l s o supports tuplebases i n which p a t t e r n -invoked, schema-specific procedures may be s t o r e d . s i d e l : *CHAIN-3 s i d e l - d e s c : ((57 . 33) (0.886914 . 0.461934) 54.1202) sid e 2 : side2-desc r e g i o n s : C / l a b e l s : Q/models: *CHAIN-5 ((69 . 62) (-0.918062 . -0.396436) 47.918062) (*REGION-1 *REGION-2 *REGION-3 *REGION"4 *REGION~5) ((*CHAIN-3 . *BRIDGE) (*CHAIN-5 . *BRIDGE)) ((*RIVER-SYSTEM *RIVER-SYSTEM-1) (*ROAD-SYSTEM *ROAD~SYSTEM-1)) instance-> *BRIDGE F i g u r e 3.6 A Maya Instance: *BRIDGE-1 Chapter 3. A Framework of the F i e l d 4 6 A sample FRL frame (Roberts and G o l d s t e i n , 1977a and 1977b) i s d i s p l a y e d i n F i g u r e 3.7 (from Rosenberg, 1977). . FRL frames have more p a r t s to an a t t r i b u t e , c a l l e d a s l o t , than Maya does. Besides the name and value, there can be a d e f a u l t value and a REQUIRE p r e d i c a t e which r e s t r i c t s the value s that may f i l l the SUPPLY AKO domain o l d - c a r r y o v e r product ion imports t o t a l - s u p p l y use domestic exports t o t a l - u s e c a r r y o v e r date $VALUE $DEFAULT $1F-ADDED $1F-REMOVED $DEFAULT $IF-ADDED $IF-REMOVED $DEFAULT $IF-ADDED $IF-REMOVED $DEFAULT $DEFAULT $DEFAULT $IF-ADDED $IF-REMOVED $DEFAULT $IF-ADDED $IF-REMOVED $DEFAULT $DEFAULT $DEFAULT $DEFAULT t h i n g c a r r y o v e r , p r o d u c t i o n , imports t o t a l - s u p p l y , c a r r y o v e r t o t a l - s u p p l y , c a r r y o v e r 0 t o t a l - s u p p l y , c a r r y o v e r t o t a l - s u p p l y , c a r r y o v e r 0 t o t a l - s u p p l y , c a r r y o v e r • t o t a l - s u p p l y , c a r r y o v e r 0 t o t a l - s u p p l y domestic, exports c a r r y o v e r , t o t a l - u s e c a r r y o v e r , t o t a l - u s e 0 c a r r y o v e r , t o t a l - u s e c a r r y o v e r , t o t a l - u s e 0 t o t a l - u s e c a r r y o v e r nowc F i g u r e 3.7 An FRL Frame: SUPPLY Chapter 3. A Framework of the F i e l d 47 s l o t . There are a l s o IF-ADDED and IF-REMOVED forms which are e v a l u a t e d whenever a value i s added or removed from the s l o t i n q u e s t i o n . These demons can be used to spread the e f f e c t of changes to the data base. The one d i s t i n g u i s h e d s l o t type i n FRL i s the AKO l i n k (a kind of) and i t s i n v e r s e , INSTANCE. These l i n k s form a s p e c i a l -i z a t i o n or subset h i e r a r c h y which can be used f o r p r o p e r t y i n h e r i t a n c e . However, there are no p a r t i c u l a r r e s t r i c t i o n s on how the AKO l i n k s are to be used (see Brachman, 1982, f o r an e x p l a n a t i o n of how many d i f f e r e n t n o t i o n s have been represented with AKO l i n k s ) . A l s o , there i s no c l e a r d i s t i n c t i o n between g e n e r a l or s t e r e o t y p i c a l knowledge and p a r t i c u l a r knowledge. The UNIT package ( S t e f i k , 1979; Smith and F r i e d l a n d , 1980) a l s o has s l o t s as the b a s i c a t t r i b u t e i n u n i t s (frames). S l o t s are composed of names, va l u e s , d e f a u l t s , datatypes (the type of the value such as i n t e g e r or s t r i n g ; not to be confused with value r e s t r i c t i o n s that are l i k e r e q u i r e c l a u s e s i n FRL), and i n h e r i t a n c e r o l e s . As with FRL there i s one i n h e r i t a n c e h i e r a r -chy. However, the values that can be i n h e r i t e d are determined by t h e i r i n h e r i t a n c e r o l e . T h i s p r o v i d e s another type of s t r u c -t u r e to the knowledge. Besides p r o p e r t i e s , s l o t s can have other d e f i n i t i o n a l r o l e s which in e f f e c t form other h i e r a r c h i e s . In p a r t i c u l a r , there i s the p a r t - o f r o l e , and i t s i n v e r s e , s u p e r - u n i t , which forms a decomposition h i e r a r c h y , and, a " r e l a t i o n " r o l e which p r o v i d e s a Chapter 3. A Framework of the F i e l d 48 general mechanism for d e s c r i b i n g b i n a r y r e l a t i o n s h i p s . Pro-cedures can be attached to u n i t s , s l o t s , and datatypes; messages can be passed between u n i t s or s l o t s . These mechanisms have been used s u c c e s s f u l l y i n the b u i l d i n g of expert systems ( S t e f i k , 1980). PSN i s a r e p r e s e n t a t i o n language b u i l t from the b a s i c e n t i -t i e s of c l a s s and b i n a r y r e l a t i o n s (Levesque and Mylopoulos, 1979). A c l a s s i s a c o l l e c t i o n of o b j e c t s which share common p r o p e r t i e s . A r e l a t i o n i s a mapping between two c l a s s e s . Employing simple basic o p e r a t i o n s on c l a s s e s and r e l a t i o n s ( c r e a t e / a s s e r t , destroy, f e t c h , and t e s t ) , the PSN developers b u i l t IS-A ( s p e c i a l i z a t i o n ) and PART-OF h i e r a r c h i e s — b o t h of which can take part i n the i n h e r i t a n c e of p r o p e r t i e s . There i s a l s o the n o t i o n of a metaclass--a c l a s s whose in s t a n c e s are a l s o c l a s s e s - - t h a t can have p r o p e r t i e s not possessed by those i n s t a n c e s . For example, the Student metaclass may have the pro-p e r t y Average-Age that would not make sense i n the Person i n s t a n c e . Numerous other frame-based r e p r e s e n t a t i o n languages have been developed. Some knowledge r e p r e s e n t a t i o n languages l i k e KRL (Bobrow and Winograd, 1977) have developed e l a b o r a t e concep-t u a l i z a t i o n s and imply a c e r t a i n model of memory. Others, l i k e Maya, can be viewed as ext e n s i o n s to L i s p that allow one to access knowledge more d i r e c t l y . Chapter 3. A Framework of the F i e l d 49 3.4. Knowledge Representation ( P r o c e d u r a l Adequacy) . . . a data s t r u c t u r e i s not knowledge, any more than an e n c y c l o p e d i a i s knowledge. We can say, m e t a p h o r i c a l -l y , t h a t a book i s a source of knowledge, but without a reader, the book i s j u s t ink on paper. S i m i l a r l y , we of t e n t a l k of the 1 i s t - a n d - p o i n t e r data s t r u c t u r e s i n an Al database as knowledge per se, when we r e a l l y mean that they represent f a c t s or r u l e s when used by a c e r -t a i n program to behave i n a knowledgeable way. (Barr and Feigenbaum, 1981, p. 143) Schemata i n a semantic network can be c o n s i d e r e d c o l l e c t i v e l y as a data base which i s manipulated by separate programs. Con-v e r s e l y , procedures a t t a c h e d to the schemata can c o n t r o l the process of i n t e r p r e t a t i o n . The l a t t e r case i s an example of o b j e c t - o r i e n t e d programming. Programming i n an o b j e c t - o r i e n t e d language, of which S m a l l t a l k (Robson and Goldberg, 1981) i s a prime example, con-s i s t s of d e f i n i n g o b j e c t s and how they w i l l respond to messages sent by other o b j e c t s . Schemata are o b j e c t s and most schema-based languages allow messages to be sent between schemata (e.g. Maya), s l o t s w i t h i n schemata (e.g. FRL), or both (e.g. UNITs and KRL). In an image understanding system, a message might come from a r i v e r schema, saying , "A r i v e r has been found i n the image, i t i s r i v e r - 6 " . The a p p r o p r i a t e response might be that the r i v e r system schema checks i f r i v e r - 6 should be added to an e x i s t i n g r i v e r - s y s t e m i n s t a n c e . Questions of p r o c e d u r a l adequacy r e l e v a n t to o b j e c t -o r i e n t e d systems i n c l u d e : how e f f e c t i v e are the c o n t r o l s t r u c -t u r e s f o r e f f e c t i n g i n t e r p r e t a t i o n s ? how convenient i s i t to Chapter 3. A Framework of the F i e l d 50 "program" with them? how w e l l can procedures att a c h e d to d i f -f e r e n t schemata cooperate? how schema-specific must the mes-sages be? Mapsee2 i s a schema-based image understanding system (Havens and Mackworth, 1980; Mackworth and Havens, 1981) which i s w r i t t e n i n Maya (Havens, 1978). In Maya, one of the a t t r i -butes a schema may have i s a p o i n t e r to a database, c a l l e d a tuplebase, of methods. Methods ( f u n c t i o n s ) are r e t r i e v e d and v a r i a b l e s are bound to e s t a b l i s h the c u r r e n t context by p a t t e r n matching. Hence each schema can have s e v e r a l l o c a l f u n c t i o n s from which the a p p r o p r i a t e one i n the c u r r e n t context w i l l be chosen. There i s a l s o a b a c k t r a c k i n g mechanism so that one can access the other f u n c t i o n s i f the f i r s t one does not succeed. Mapsee2 c u r r e n t l y i n t e r p r e t s sketch maps i n an almost s t r i c t l y bottom-up f a s h i o n . Each chain ( l i n e ) i s a p o s s i b l y ambiguous cue f o r s e v e r a l schemata such as roads, r i v e r s , towns, brid g e s i d e s , or o t h e r s . The a p p r o p r i a t e schema-specific pro-cedures examine the c h a i n and determine whether i t supports c r e a t i n g an i n s t a n c e , p o s s i b l y h y p o t h e t i c a l , to represent that e n t i t y . If so, that procedure i s suspended and the f u n c t i o n s a s s o c i a t e d with higher l e v e l o b j e c t s , such as road systems, r i v e r systems, or geosystems, are i n i t i a t e d to b u i l d up the decomposition h i e r a r c h y . Chapter 3. A Framework of the F i e l d 51 3_.5_. G r a c e f u l Degradation (Robustness) A computer program i s s a i d to be robust or d i s p l a y g r a c e f u l degradation i f i t s performance d i m i n i s h e s g r a d u a l l y when the c o n d i t i o n s f o r which i t was designed are not met. Marr d e f i n e s the c o n d i t i o n s f o r a robust program to be that "degrading data w i l l not prevent one from d e l i v e r i n g at l e a s t some of the answer" (Marr, 1976, p. 486). I f an ungrammatical sentence i s input to a n a t u r a l language understanding program, then a robust system w i l l s t i l l be able to analyze p a r t s of the se n t e n c e [ 2 ] . A system t h a t does not degrade g r a c e f u l l y might j u s t go i n t o an i n f i n i t e loop because i t "expected" a part of the sentence that was not t h e r e . G r a c e f u l degradation i s very important i n i n t e r a c t i v e sys-tems. Both the user and the system must have ways of r e c o v e r i n g when communication i s not complete. T h i s has been c a l l e d " g r a c e f u l i n t e r a c t i o n " by workers on the Spice p r o j e c t at CMU (Hayes and Reddy, 1979; Hayes et a l . , 1981). They have i d e n t i -f i e d s e v e r a l components of g r a c e f u l i n t e r a c t i o n , i n c l u d i n g focus t r a c k i n g (keeping t r a c k of what the c o n v e r s a t i o n i s about), i d e n t i f i c a t i o n from d e s c r i p t i o n (matching o b j e c t s represented i n t e r n a l l y from e x t e r n a l d e s c r i p t i o n s ) , an e x p l a n a t i o n f a c i l i t y (what the system i s doing and why), and p e r s o n a l i z a t i o n ( a d j u s t -ing to the p r e f e r e n c e s of the u s e r - - c f . S e c t i o n 4.3). [2]People are very robust understanders of n a t u r a l language. Chapter 3. A Framework of the F i e l d 52 G r a c e f u l degradation i s a l s o r e l e v a n t to combining informa-t i o n sources. I f an IS i s removed, can the system u t i l i z e the remaining source(s) to advantage, or does i t f a i l completely? As an extreme example, c o n s i d e r s t e r e o p s i s and i t s two IS's. If one IS i s removed, then i t i s impossible to c a l c u l a t e depth d i r e c t l y from a monocular image. Hence, i n terms of i t s input IS's, s t e r e o systems can not degrade g r a c e f u l l y when only one IS remains. In systems combining d i f f e r e n t types of input i n f o r m a t i o n sources, more robust s o l u t i o n s can be developed. If one i s using an " a i d " IS to h e l p i n t e r p r e t an i n t e n s i t y image (a map data base, DTM, sketch map) then a robust system w i l l s t i l l be a b l e to f u n c t i o n with only the o r i g i n a l image. One would not expect the r e s u l t s to be as good without the a d d i t i o n a l IS; how-ever, c o r r e c t p a r t i a l r e s u l t s are almost always b e t t e r than nothing at a l l . Thus, while robustness i s a g e n e r a l l y advanta-geous property f o r a l l computer programs, i t can be p a r t i c u l a r l y d e s i r a b l e i n i n t e r a c t i v e systems and those attempting to combine i n f o r m a t i o n sources. Chapter 3. A Framework of the F i e l d CHAPTER 4 How to Combine M u l t i p l e Information Sources 4_.J_. I ntroduct ion E a r l i e r in t h i s t h e s i s a case was begun f o r the u s e f u l n e s s of combining m u l t i p l e i n f o r m a t i o n sources. Taking that paradigm as a s t a r t i n g p o i n t , there i s s t i l l the q u e s t i o n of how best to combine IS's so that they may be used i n image i n t e r p r e t a t i o n . T h i s chapter i s concerned with one s o l u t i o n to that problem. The major hypothesis of t h i s d i s s e r t a t i o n i s "Combining i n f o r m a t i o n sources i s u s e f u l " ; the minor hypothesis i s that the methods d i s c u s s e d i n t h i s chapter are a way of r e a l i z i n g that g o a l . T h i s chapter may a l s o be thought of as d e s c r i b i n g t o o l s which can be assembled (see Chapter 5 ) to form a v i s i o n system that can e x p l o i t d i f f e r e n t types of i n p u t s . The t o o l s d e s c r i b e d below can be d i v i d e d i n t o three c a t e g o r i e s corresponding, to the l a s t s e c t i o n s of Chapter 3 . D e s c r i p t i v e adequacy i s the m o t i v a t i o n f o r c r e a t i n g a new type of schema and a l i b r a r y of f u n c t i o n s f o r schemata manipulation ( S e c t i o n 4 . 2 ) . Information that comes d i r e c t l y from the user can be very u s e f u l . The means by which user i n f o r m a t i o n can be encoded i s d e s c r i b e d ( 4 . 3 ) . P r o c e d u r a l adequacy i s p r i m a r i l y the r e s u l t of using a c y c l e of p e r c e p t i o n as the b a s i c c o n t r o l regime ( 4 . 4 ) . The 5 3 54 c y c l e c o n t a i n s convenient p o i n t s f o r i n t e r a c t i o n with the user (4.4.1) and can be used to generate top-down, bottom-up, or a mixed c o n t r o l s t r a t e g y (4.5). S e v e r a l techniques have been d e v i s e d to promote g r a c e f u l d e g r a d a t i o n . Using context permits t h r e s h o l d s to be r e l a x e d so that expected o b j e c t s can be i d e n t i f i e d with weaker evidence (4.5.2) . I s l a n d - d r i v e n c o n t r o l causes a t t e n t i o n to be given to the most r e l i a b l e hypotheses (4.5.3). R e l a t i n g t h r e s h o l d s to g l o b a l schemata connects m o d e l - s p e c i f i c knowledge with more r e l e v a n t scene i n f o r m a t i o n (4.6). F i n a l l y , an i n f e r e n c e mechan-ism based on c l u s t e r i n g p r o v i d e s a f l e x i b l e , incremental means of c a t e g o r i z i n g f e a t u r e s d e r i v e d from i n f o r m a t i o n sources (4.7). Chapters 4 and 5 are c l o s e l y l i n k e d . Chapter 4 i n t r o d u c e s t o o l s that can be a p p l i e d to u t i l i z i n g m u l t i p l e i n f o r m a t i o n sources i n p e r c e p t i o n . Chapter 5 d e s c r i b e s three s p e c i f i c i n f o r m a t i o n sources and the a c t u a l implementation of a system which can analyze them u s i n g i n s t a n c e s of those t o o l s . j4.2. A Schemata M a n i p u l a t i o n System Few image understanding systems have used frames or sche-mata as a r e p r e s e n t a t i o n medium. Most of the schema-based languages d e s c r i b e d i n S e c t i o n 3.3 have been a p p l i e d to tasks i n n a t u r a l language understanding and expert systems. From an AI p e r s p e c t i v e , one would hope that one r e p r e s e n t a t i o n scheme would s u f f i c e f o r a l l domains;- however, there are d i f f e r e n c e s in domains that can have an e f f e c t on how i n f o r m a t i o n can best be Chapter 4. Combining Information Sources 55 represented. In p a r t i c u l a r , knowledge d e r i v e d from images i s very t e n t a -t i v e and h y p o t h e t i c a l , u n l i k e the words input to an NLU program. T h i s knowledge must be d i s t i n g u i s h e d from generic knowledge and p r o v i s i o n must be made to remove h y p o t h e t i c a l i n s t a n t i a t i o n s that prove to be i n c o r r e c t . Another d i f f e r e n c e i s t h a t , with v i s u a l knowledge, the decomposition h i e r a r c h y o f t e n turns out to be at l e a s t as important as s p e c i a l i z a t i o n . Thus, a s i n g l e d i s -t i n g u i s h e d r e l a t i o n a l s t r u c t u r e between schemata i s i n s u f f i -c i e n t . F i n a l l y , the understanding of r e a l imagery r e q u i r e s a great d e a l of p r o c e d u r a l f l e x i b i l i t y . Scenes can vary so much from image to image t h a t uniform, s y n t a c t i c methods have not u s u a l l y met with much success. Maya i s a schema-based language that has been used i n image understanding. Maya pr o v i d e s only a simple data s t r u c t u r e to encode knowledge--attribute name/value p a i r s . The d e s c r i p t i o n that f o l l o w s shows how Maya can be extended i n ways that are sometimes s i m i l a r to the other more complex knowledge represen-t a t i o n languages, but o f t e n d i f f e r i n accordance with the requirements of image understanding as d e s c r i b e d above. The r e s u l t i s a more d i f f e r e n t i a t e d data s t r u c t u r e (Glicksman, 1982) . 4_._2.j_. Schemata Have Four Part s Schemata are made up of four d i s t i n c t types of a t t r i b u t e s . The VALUE type i s used to s t o r e the f a c t s that d e f i n e the Chapter 4. Combining Information Sources 56 represented o b j e c t (e.g. the "area" of r i v e r - 6 i s 76 p i x e l s ) . It i s an extension of Maya's a t t r i b u t e type to i n c l u d e FRL's s l o t f a c e t s and more. Although a value can be any L i s p S-e x p r e s s i o n , g e n e r a l l y i t w i l l be a number, a l i s t of numbers, or a s t r i n g . VALUEd types p r o v i d e a s l o t t h a t can be f i l l e d with a v a l u e . As i n FRL, the s l o t can have exp r e s s i o n s a s s o c i a t e d with i t , such as a d e f a u l t or the r e q u i r e d p r o p e r t i e s of the s l o t f i l l e r . Furthermore, f u n c t i o n s can be i n i t i a t e d when the value i s added, removed, or needed. U n l i k e FRL, a f u n c t i o n a l s o e x i s t s which i s invoked when the value i s m o d i f i e d . T h i s f l e x i -b i l i t y i s used to maintain the c o n s i s t e n c y of the i n t e r p r e t a t i o n and to spread the e f f e c t s of changes to the s l o t s . The f i n a l m o d i f i e r that can be a s s o c i a t e d with a VALUEd type i s c o n f i d e n c e . T h i s allows one to a s s o c i a t e a r e l i a b i l i t y f a c t o r with the components of a schema that can be used to c a l -c u l a t e the o v e r a l l CONFIDENCE of the schema. I t i s u s e f u l i n image understanding where the e x i s t e n c e of o b j e c t s i s not an a l l or nothing p r o p o s i t i o n (see Davis and Rosenfeld, 1981, f o r a review of r e l a x a t i o n methods). Confidence i s a l s o r e l a t e d to the i s s u e of g r a c e f u l degradation s i n c e the system should have l e s s c o n f i d e n c e i n i t s r e s u l t s when presented with poor data ( J a i n and Haynes, 1982). F i g u r e 4.1 c o n t a i n s an example of a schema having these a t t r i b u t e types. Chapter 4. Combining Information Sources 57 le n g t h width area i n t e n s i t y aio—> ako—> neighbours-CONFIDENCE CONF-ALG PROCEDURES VALUE: 46 DEFAULT: n i l REQUIRED: ( g r e a t e r p % v a l 0) IF-ADDED: (and (sgetv %name 'width 'n) (sputv %name 'area (times % v a l width))) IF-MODIFIED: (and (sgetv %name 'width 'n) (sputv %name 'area (times % v a l width))) (sputv %name 'area n i l ) (and (sgetv %name 'width) (sgetv %name 'area) (quotient area width)) 37 IF-REMOVED IF-NEEDED: CONFIDENCE: VALUE: REQUIRED: IF-ADDED: CONFIDENCE: VALUE: REQUIRED: IF-NEEDED: CONFIDENCE: 6 DEFAULT: n i l (g r e a t e r p % v a l 0) . . . s i m i l a r to l e n g t h . . . 48 276 DEFAULT: n i l (g r e a t e r p % v a l 0) (and (sgetv %name 'length) (sgetv %name 'width) (times l e n g t h width)) 42 n i l VALUE: 102 DEFAULT: REQUIRED: (and (g r e a t e r p % v a l -1) ( l e s s p % v a l 257)) IF-ADDED: ( m o d i f y - i n t e r p r e t a t i o n - r a n g e % v a l ) IF-MODIFIED: ( m o d i f y - i n t e r p r e t a t i o n - r a n g e % v a l ) IF-REMOVED: ( m o d i f y - i n t e r p r e t a t i o n - r a n g e % v a l ) (qu o t i e n t ( s u m - a l l - p i x e l - v a l u e s %name) (sgetv %name 'area)) 26 IF-NEEDED: CONFIDENCE: % r i v e r apo—> %river-system-2 (%transportat ioh-system-1 %waterbody-3) -> (%bridge-7 % r i v e r - 8 ) 43 ( c l u s t e r (sgetv %name ' i n t e n s i t y ) r i v - i n t e n s i t i e s ) BOTTOM-UP—> river-bottom-up TOP-DOWN—> river-top-down F i g u r e 4.1 A Schema: % r i v e r - 6 Chapter 4. Combining Information Sources 58 The schemata are c o n t a i n e d i n a semantic network (or knowledge base), with r e l a t i o n s h i p s i n d i c a t e d by L I N K S — t h e second d i s t i n g u i s h e d p a r t of a schema. Any number of b i n a r y r e l a t i o n s can be formed, and the system maintains the p o i n t e r s and t h e i r i n v e r s e s and allows f o r s e v e r a l forms of automatic t r a v e r s a l of the network. LINKs can be used with VALUEd types to generate pr o p e r t y i n h e r i t a n c e , u s u a l l y down a s p e c i a l i z a t i o n h i e r a r c h y . Knowledge sharing of t h i s kind can o f t e n compensate f o r incomplete data. Some r e l a t i o n s induce h i e r a r c h i e s which c r e a t e i n t e r e s t i n g p o s s i b i l i t i e s f o r c o n t r o l s t r a t e g i e s (see Rosenthal and Bajcsy, 1978). One prominent b u i l t - i n r e l a t i o n i s INSTANCE/AIO[1] which as i n Maya i s used to separate g e n e r a l or s t e r e o t y p i c o b j e c t s from t h e i r r e a l i z a t i o n s i n a scene. Instances can be used to explore s e v e r a l p o s s i b l e i n t e r p r e t a t i o n s without any commitment to t h e i r u l t i m a t e e x i s t e n c e . Other r e l a t i o n s that would be g e n e r a l l y used are SPECIALIZES-TO/AKO to form a s p e c i a l i z a t i o n or subset h i e r a r c h y and DECOMPOSES-TO/APO to form a decomposition or sub-pa r t h i e r a r c h y (see F i g u r e 4.1). The CONFIDENCE a t t r i b u t e i s a number r e p r e s e n t i n g the o v e r a l l confidence one might have in a schema at a given time c o n s i d e r i n g the a v a i l a b l e i n f o r m a t i o n . The CONFIDENCE value i s a v a i l a b l e to a g l o b a l scheduler and can be used to encourage [ 1 ] L i n k s are r e f e r r e d to as l i n k / i n v e r s e . AIO stands f o r An Instance Of; AKO f o r A Kind Of; and, APO f o r A Part Of. Chapter 4. Combining Information Sources 59 e v a l u a t i o n of the most promising i n t e r p r e t a t i o n s . A s s o c i a t e d with t h i s a t t r i b u t e i s a s c h e m a -specific a l g o -rithm to modify the confidence value whenever new VALUES are found f o r t h i s o b j e c t or i t s components. Many " b e l i e f systems" r e l y on uniform f u n c t i o n s f o r c a l c u l a t i n g r e l i a b i l i t y . For example, MYCIN uses the minimum c e r t a i n t y f a c t o r of i t s antecedents ( S h o r t l i f f e and Buchanan, 1975; see Goebel, 1977, for a review of other systems which use the maximum or average v a l u e s ) . Using an a l g o r i t h m p r o v i d e s the f l e x i b i l i t y to con-s i d e r the d i f f i c u l t y and importance of completing the i n t e r p r e -t a t i o n of an object as w e l l as the p r o b a b i l i t y of i t s e x i s t e n c e . I t a l l o w s f o r schema-specific knowledge to be used to weight the importance of the VALUEd types as w e l l as components and spe-c i a l i z a t i o n s of a schema when c a l c u l a t i n g c o n f i d e n c e . The f o u r t h d i s t i n g u i s h e d p a r t of a schema i s i t s p r o c e d u r a l attachments. Besides the procedures that can be a t t a c h e d to a t t r i b u t e s (VALUEd and CERTAINTY t y p e s ) , a s s o c i a t e d with a schema there are g e n e r a l l y procedures which are used to i n s t a n -t i a t e them ( i . e . f i l l i n t h e i r s l o t s ) and c o n t r o l the i n t e r p r e -t a t i o n . These would i n c l u d e procedures to invoke the schema as a model (top-down) or to account f o r data (bottom-up). The pro-cedures would a l s o t y p i c a l l y send out messages which would cause other schemata to become " c u r r e n t " and other procedures to be e v a l u a t e d . Chapter 4. Combining Information Sources 60 4.2.2. M a n i p u l a t i n g Schemata Numerous f u n c t i o n s are a v a i l a b l e to manipulate schemata so that the i n f o r m a t i o n they encode can be u t i l i z e d . Appendix A d e s c r i b e s the f u n c t i o n s a v a i l a b l e i n the MAIDS system ( S e c t i o n 5.2) which i s a p a r t i c u l a r implementation of the r e p r e s e n t a t i o n scheme d e s c r i b e d i n the l a s t s e c t i o n . E s s e n t i a l f u n c t i o n s i n c l u d e those f o r c r e a t i n g schemata ( s c r e a t e — f o r stereotype o b j e c t s ; snewi — f o r h y p o t h e t i c a l i n s t a n c e s ) and d e s t r o y i n g them (seraseo and s e r a s e i ) , p l u s one fo r merging the a t t r i b u t e s of two in s t a n c e s i n t o one (smerge). A l s o there are b a s i c f u n c t i o n s f o r modifying and a c c e s s i n g the knowledge base. Table 4.1 l i s t s the f u n c t i o n s a v a i l a b l e f o r each of the a t t r i b u t e types. In a d d i t i o n to these standard f u n c t i o n s , there are s e v e r a l more that add to the u s e f u l n e s s of the schemata. Sgetv can be used to f e t c h d e f a u l t s and/or val u e s along a s p e c i f i e d LINK which can be used f o r pr o p e r t y i n h e r i t a n c e . One can determine the s h o r t e s t path between two schemata along a c e r t a i n l i n k ADD MODIFY REMOVE FETCH any a t t r i b u t e sputa sputa sgeta VALUEd sputv sputv sremovev sgetv LINKs saddl s a d d l sremovel s g e t l CONFIDENCE sputc sputc sgetc s p u t c l Table 4.1 M o d i f i c a t i o n and R e t r i e v a l f o r Schemata A t t r i b u t e s Chapter 4. Combining Information Sources 61 ( s l i n k ? ) , o r d i s c o v e r w h e t h e r a p a t h e x i s t s b e t w e e n two s c h e m a t a ( s a n y l i n k ? ) . The s e m a n t i c n e t w o r k c a n be made c o n s i s t e n t by a d d i n g m i s s i n g b a c k p o i n t e r s a n d r e m o v i n g s u p e r f l u o u s L I N K s ( s c o n s i s t ) . G i v e n a t y p e , s u c h a s L I N K , i t i s p o s s i b l e t o d e t e r m i n e a l l t h e a t t r i b u t e s i n a s chema t h a t a r e o f t h a t t y p e ( s a t t r ) . C o n -v e r s e l y , i f one knows t h e name o f an a t t r i b u t e , t h e n i t s t y p e c a n be f o u n d ( s a t t r t y p e ) . A l s o , s e v e r a l p r e t t y p r i n t i n g f u n c -t i o n s e x i s t t o show t h e c o m p o n e n t s o f s c h e m a t a ( s p r i n t ) o r t h e b r a n c h e s o f t h e s e m a n t i c n e t w o r k ( s p r i n t n a n d s s p r i n t n ) . The d a t a s t r u c t u r e s f o r s c h e m a t a r e p r e s e n t a t i o n p l u s t h e m a n i p u l a t i o n f u n c t i o n s p r o v i d e b u i l d i n g b l o c k s f o r a v i s i o n s y s -tem t h a t c a n c o m b i n e m u l t i p l e I S ' s . T h e i r g r e a t e s t a s s e t s a r e m o d u l a r i t y , t h e f l e x i b i l i t y t o e n c a p s u l a t e d i v e r s e i n t e r p r e t a -t i o n s t r a t e g i e s , a n d t h e i n t e r r e l a t i o n s h i p s among s c h e m a t a w h i c h e n a b l e s t h e m t o c o o p e r a t e . 4.3_. I n t e r a c t i o n a s a S o u r c e o f I n f o r m a t i o n M o s t c o m p u t e r v i s i o n s y s t e m s h a v e n o t a t t e m p t e d t o u s e a d v i c e f r o m a human u s e r . W h e r e a s t o t a l l y a u t o m a t e d s y s t e m s a r e t h e u l t i m a t e g o a l i n t h i s f i e l d , i t i s o f t e n t h e c a s e t h a t m i n i m a l human i n t e r v e n t i o n may p r o v i d e t h e c r u c i a l i n f o r m a t i o n t h a t w i l l e n a b l e i n t e r p r e t a t i o n t o s u c c e e d . F u r t h e r m o r e , i f a t r u l y s y m b i o t i c r e l a t i o n s h i p c a n be f o r m e d b e t w e e n man a n d m a c h i n e t h e n i t w o u l d be p o s s i b l e f o r more d i f f i c u l t p r o b l e m s t o be s o l v e d t h r o u g h c o o p e r a t i o n t h a n c o u l d be h a n d l e d by man o r C h a p t e r 4 . C o m b i n i n g I n f o r m a t i o n S o u r c e s 62 machine alon e . Even though the s u b j e c t has not been d e a l t with i n great d e t a i l , s e v e r a l r e s e a r c h e r s have advocated the u s e f u l n e s s of human i n t e r a c t i o n i n both v i s i o n systems (Barrow and Tenenbaum, 1975; Agin and Duda, 1975; Barrow and Tenenbaum, 1976; Stockman, 1978) and expert systems ( N i i et a l . , 1982). One of the reasons was w e l l d e s c r i b e d as f o l l o w s : The s u b s t a n t i a l amount of ad hoc world knowledge r e -q u i r e d t o plan p e r c e p t u a l s t r a t e g i e s i s most reasonably a c q u i r e d i n an incremental f a s h i o n . The system should thus be designed to request a d d i t i o n a l i n f o r m a t i o n from a user at times of f a i l u r e , i n d e c i s i o n , or on encounter-ing a new o b j e c t and to i n c o r p o r a t e t h i s i n f o r m a t i o n im-mediately i n a r e v i s e d s t r a t e g y . (Tenenbaum, 1973, pp. 8-9) Human i n t e r a c t i o n would be most u s e f u l i n c u r r e n t v i s i o n systems to r e s o l v e problems due to ambiguity or i n d e c i s i o n . If a v i s i o n program has reduced the number of p o s s i b i l i t i e s f o r an i n t e r p r e t a t i o n down to a small number, then very l i t t l e " i n f o r -mation" i s r e q u i r e d from the user to r e s o l v e the d i f f i c u l t y . Furthermore, i n t e r a c t i o n can be u s e f u l i n combating the other problems of model-based v i s i o n (as o u t l i n e d i n S e c t i o n 3.2): i n c o n s i s t e n c y , u n s a t i s f i a b i l i t y , and incompleteness. One of the c r i t e r i a f o r the success of a computer v i s i o n system i s that i t d i s p l a y g r a c e f u l degradation ( S e c t i o n 3.5). Hence, i f the user chooses not to p a r t i c i p a t e i n the i n t e r p r e t a -t i o n process, the system should s t i l l be robust enough to con-t i n u e without the a d d i t i o n a l guidance. The more u s e f u l Chapter 4. Combining Information Sources 63 i n f o r m a t i o n the user chooses to p r o v i d e , the b e t t e r the r e s u l t s one would expect. If the user wishes to input i n f o r m a t i o n , i t would be u s e f u l f o r a v i s i o n system to accept i t . An i n t e r a c t i v e v i s i o n system should only be making requests of the user at times when c r i t i c a l d e c i s i o n s must be made. How-ever, how i s the system to know what the user c o n s i d e r s c r i t i -c a l ? One way i s f o r the system to have a user model that has knowledge of the pr e f e r e n c e s and h a b i t s of the i n d i v i d u a l s t h a t use the system (Rich, 1979a and 1979b). Then the system can r e f e r to the model i n the context of the c u r r e n t d e c i s i o n when determining i f the user should be q u e r i e d and how. 4.3.J_. Accept ing and Accomodat ing Information from Users Man-machine communication i s best achieved when the com-puter i s a b l e to r e l a t e i n a n a t u r a l manner. Thus n a t u r a l language i s to be p r e f e r r e d over obscure programming languages. In image understanding where p i c t o r i a l i n f o r m a t i o n i s so impor-t a n t , graphic input and output becomes c r i t i c a l . Graphic input u s u a l l y c o n s i s t s of e i t h e r p o i n t i n g at p a r t s of an image or drawing. Sketch maps can be c o n s i d e r e d a type of i n t e r a c t i o n between the user and the v i s i o n system. They d e p i c t c e r t a i n standard symbols which are a b s t r a c t i o n s of what might appear i n a scene ( c f . S e c t i o n 3.1.2). They c o n t a i n i n f o r m a t i o n concerning what i s in a scene as w e l l as the t o p o l o g i c a l r e l a t i o n s h i p s among the e n t i t i e s . Chapter 4. Combining Information Sources 64 More d i r e c t i n f o r m a t i o n might be input to a v i s i o n system v i a t e x t . Whereas a sketch map r e t a i n s some a n a l o g i c p r o p e r t i e s of the i n t e n s i t y image, d i r e c t input i s g e n e r a l l y not i c o n i c . I t i s a l s o u s u a l l y much more a b s t r a c t than an i n t e n s i t y image. Examples i n c l u d e s p e c i f y i n g the e x i s t e n c e of an o b j e c t , i t s l o c a t i o n , or i t s c h a r a c t e r i s t i c s . To be able to accomodate d i s p a r a t e types of knowledge r e q u i r e s a method of r e p r e s e n t a t i o n independent of the nature of the images. The schemata d e s c r i b e d i n the l a s t s e c t i o n (4.2) p r o v i d e a f l e x i b l e , modular r e p r e s e n t a t i o n scheme that i s able to s t o r e and u t i l i z e a d v i c e from a user. Attached procedures can provide the i n t e r f a c e between man and machine so that s l o t s can be f i l l e d with v a l u e s d i r e c t l y . These values can then become pa r t of the context which i n f l u e n c e s subsequent pro c e s s -ing . In i n t e r a c t i v e systems there i s a l s o the q u e s t i o n of what i n f o r m a t i o n to p r o v i d e to a user, when i t i s a p p r o p r i a t e to d i s p l a y i t , and, c o n v e r s e l y , what and when in f o r m a t i o n should be r e c e i v e d . The next s e c t i o n on a c y c l e of p e r c e p t i o n i n d i c a t e s three p l a c e s i n the feedback loop where i n t e r a c t i o n can occur. 4.4. A Cycle of Percept ion C o g n i t i v e p s y c h o l o g i s t s have examined the r o l e of schemata as c o l l e c t i o n s of s t r u c t u r e s and processes that both accept per-c e p t u a l i n f o r m a t i o n and d i r e c t movements and e x p l o r a t o r y a c t i v i -t i e s ( N e i s s e r , 1976). N e i s s e r d e s c r i b e s how they might be used Chapter 4. Combining Information Sources 65 as f o l l o w s : In my view, the c o g n i t i v e s t r u c t u r e s c r u c i a l f o r v i s i o n are the a n t i c i p a t o r y schemata that prepare the p e r c e i v e r to accept c e r t a i n kinds of i n f o r m a t i o n r a t h e r than oth-ers and thus c o n t r o l the a c t i v i t y of l o o k i n g . Because we can see only what we know how to look f o r , i t i s these schemata (together with the i n f o r m a t i o n a c t u a l l y a v a i l a b l e ) that determine what w i l l be p e r c e i v e d . Per-c e p t i o n i s indeed a c o n s t r u c t i v e process . . . At each moment the p e r c e i v e r i s c o n s t r u c t i n g a n t i c i p a t i o n s of c e r t a i n kinds of i n f o r m a t i o n , that'enable him to accept i t as i t becomes a v a i l a b l e . Often he must a c t i v e l y ex-p l o r e . . . These e x p l o r a t i o n s are d i r e c t e d by the a n t i -c i p a t o r y schemata, which are plans f o r p e r c e p t u a l a c t i o n as w e l l as readiness f o r p a r t i c u l a r kinds of o p t i c a l s t r u c t u r e . The outcome of the e x p l o r a t i o n s - - t h e i n f o r -mation p i c k e d up--modifies the o r i g i n a l schema. Thus mo d i f i e d i t d i r e c t s f u r t h e r e x p l o r a t i o n and becomes ready f o r more i n f o r m a t i o n . ( N e i s s e r , 1976, pp. 20-21) N e i s s e r c a l l s t h i s c o n t i n u a l process the p e r c e p t u a l c y c l e . A s l i g h t l y m o d i f i e d v e r s i o n can be seen in F i g u r e 4.2. A s i m i l a r feedback loop has been proposed s p e c i f i c a l l y f o r model-based computer v i s i o n (Mackworth, 1978). I t s four stages are cue d i s c o v e r y ( o b j e c t ) , model i n v o c a t i o n (schema invoca-t i o n ) , model v e r i f i c a t i o n (schema i n s t a n t i a t i o n ) , and model e l a -b o r a t i o n ( e x p l o r a t i o n ) [ 2 ] . Mackworth's "paradigm" i s c a l l e d the c y c l e of p e r c e p t i o n . Depending on where the c y c l e i s entered, one w i l l observe d i f f e r e n t modes of o p e r a t i o n which correspond to the t r a d i t i o n a l methods of c o n t r o l i n computer s c i e n c e . If the c y c l e begins at [2]What has been l a b e l l e d "schema i n v o c a t i o n and i n s t a n t i a -t i o n " i n F i g u r e 4.2 i s simply c a l l e d "schema" i n N e i s s e r ' s per-c e p t u a l . c y c l e . Chapter 4. Combining Information Sources 66 SCHEMA INVOCATION AND INSTANTIATION Figure 4.2 A Cycle of Perception the "object" stage then bottom-up, or a l i n e a r stage model, can r e s u l t . Cues from the image derived from objects in the scene cause schemata (models) to be invoked and i n s t a n t i a t e d . In a l i n e a r stage model, processing would stop at t h i s point: the invoked model i s a c l a s s i f i c a t i o n . However, an i t e r a t i v e pro-cess would use the information just obtained to d i r e c t further exploration in the image and the cycle would continue. If the cycle i s entered at the stage of schema invocation, then top-down behaviour would r e s u l t . The model knowledge con-tained in the schema w i l l be used to hypothesize the existence of . objects in the image. There w i l l also be information con-Chapter 4. Combining Information Sources 67 c e r n i n g how and where to o b t a i n the i n f o r m a t i o n i n the image that w i l l v e r i f y the hypotheses. So, the e x p l o r a t i o n module w i l l cause the image to be sampled to f i n d the a p p r o p r i a t e i n f o r m a t i o n . Depending on what i s found i n the image, the schema w i l l be s u i t a b l y m o d i f i e d . The i n s t a n t i a t e d schema can then request more in f o r m a t i o n to c o n f i r m i t s e x i s t e n c e or i t can use the knowledge gained to h e l p hypothesize the whereabouts of other o b j e c t s . In t h i s way, the c y c l e of p e r c e p t i o n promotes c o o p e r a t i o n among the schemata. As o b j e c t s are i n s t a n t i a t e d they communi-cate to other schemata to give advice and/or to b u i l d the seman-t i c network. T h i s i n turn i n i t i a t e s another loop around the c y c l e as a new schema i s i n s t a n t i a t e d . The schemata work as i n d i v i d u a l s ( v i a attached procedures) and together ( v i a mes-sages) to b r i n g about the i n t e r p r e t a t i o n of p a r t i c u l a r informa-t i o n sources. The c y c l e of p e r c e p t i o n i s s i m i l a r to other feedback para-digms such as hypothesize and t e s t (e.g. Erman and L e s s e r , 1975). I t i s p a r t i c u l a r l y w e l l s u i t e d to a schema-based system because i t i d e n t i f i e s the n a t u r a l p l a c e s where schemata can move i n t o and out of the focus of a t t e n t i o n . The consequences of t h i s are d i s c u s s e d i n the f o l l o w i n g s e c t i o n (4.5). 4.4_.J_. I n t e r a c t i o n i n the C y c l e of P e r c e p t i o n The c y c l e of p e r c e p t i o n provides- convenient p o i n t s f o r com-munication with the user. At each node in the c y c l e there i s a Chapter 4. Combining Information Sources 68 p a r t i c u l a r , u s e f u l kind of i n f o r m a t i o n that can be exchanged. These inputs and outputs are shown i n Figu r e 4 .3 and are d e s c r i b e d below. OUTPUTS GLOBAL: When a schema i s invoked, the model type (e.g. r i v e r ) INPUT: GLOBAL what the user knows about the image OUTPUT: GLOBAL what schema has been invoked and why SCHEMA INVOCATION AND INSTANTIATION OUTPUT: LOCAL what f e a t u r e s have been d i scovered samples INPUT: PRIORITIES where to search and what f o r INPUT: LOCAL what the user sees i n the c\ image t N EXPLORATION v OUTPUT: PLANNING what the schema or a d v i s o r want to look f o r Fi g u r e 4 .3 I n t e r a c t i o n i n a Cycle of Per c e p t i o n "i ( Chapter 4 . Combining Information Sources 69 can be d e s c r i b e d f o r the user so that he/she knows what the c u r r e n t focus of a t t e n t i o n i s . A c u r i o u s user would o f t e n l i k e to know the reasons behind that c h o i c e . Furthermore, as the schema i s i n s t a n t i a t e d - - b y having i t s s l o t s f i l l e d with v a l u e s — t h e ' i n f o r m a t i o n can be output (e.g. the "area" of the r i v e r i s 276 p i x e l s ) . LOCAL: The i n f o r m a t i o n d e r i v e d from the image can be output when the image i s sampled. I t can be i n the form of v e r i f i c a t i o n s of hypotheses d i r e c t e d by the schema (e.g. r i v e r s have low i n t e n s i t y v a l u e s : the average i n t e n s i t y of t h i s region i s 102: so succeed), f a c t s concerning the image f e a t u r e s that may be r e l e v a n t to the schema (e.g. the long a x i s of the region i s 46 u n i t s ) , or data that can cause a s h i f t i n a t t e n t i o n (e.g. one of the neighbouring regions has a high average i n t e n s i t y ; i t can be a b r i d g e that the r i v e r i s f l o w i n g under). PLANNING: T h i s i s a l i s t of p o s s i b i l i t i e s f o r f u r t h e r process-i n g . As they are e x e c u t i n g , the schema-specific procedures can make i n f e r e n c e s about other models that should be invoked. They w i l l then send a message to a l e r t the a p p r o p r i a t e schema. Those messages p l u s the user's input make up the p o s s i b i l i t i e s l i s t . The s c h e m a - s p e cific procedures w i l l g ive up c o n t r o l f o r one of three reasons. The procedure might succeed and the c u r r e n t schema w i l l become pa r t of the semantic network. I t might f a i l , i n which case the h y p o t h e t i c a l schema w i l l be des-t r o y e d . Or i t might suspend to wait f o r more i n f o r m a t i o n . Chapter 4. Combining Information Sources 70 A t t e n t i o n would then normally s h i f t to the f i r s t e n t ry i n the p o s s i b i l i t i e s l i s t . The p o s s i b i l i t i e s l i s t i s a p r i o r i t y queue. I t s e n t r i e s are ranked by the schema that sent the message. The value given by the schema w i l l g e n e r a l l y be a f a c t o r of i t s own con-f i d e n c e ( c f . S e c t i o n 4.2.1) and the c e r t a i n t y of the i n f e r e n c e that caused the message to be sent. The rankings are shown to the user so that he/she can see how p r o c e s s i n g w i l l c o ntinue and can a s s e r t h i s / h e r p r i o r i t i e s by r e a r r a n g i n g the queue which i s done v i a p r i o r i t i e s i n p u t . INPUTS GLOBAL: The user can enter general or s p e c i f i c i n f o r m a t i o n about the scene. General data i n c l u d e s parameters that a f f e c t a l l l e v e l s of p r o c e s s i n g , f o r example, the s c a l e of the a e r i a l photograph or the l o c a t i o n of the sun. S p e c i f i c i n f o r m a t i o n p e r t a i n s to s p e c i f i c o b j e c t s . For i n s t a n c e , one can i n d i c a t e that there i s a road i n the p i c t u r e . T h i s i n f o r m a t i o n can cause e n t r i e s to be added to the p o s s i b i l i t i e s l i s t or might cause g l o b a l schemata or v a r i a b l e s to be m o d i f i e d (see S e c t i o n 4.6) . LOCAL: More s p e c i f i c i n f o r m a t i o n can a l s o be i n t r o d u c e d concern-ing the i n t e r p r e t a t i o n that i s t a k i n g p l a c e . T h i s might take the form of an i n t e r a c t i o n with a s p e c i f i c r o u t i n e . For exam-p l e , System: "Is the bank of the r i v e r edge 12 ( d i s p l a y e d i n red) or edge 26 ( b l u e ) ? " The user would p o i n t at h i s / h e r c h o i c e . Or the user might j u s t input what he/she sees i n the Chapter 4. Combining Information Sources 71 image: region 72 i s a r i v e r . In the former case, the data would be used to i n f l u e n c e how a schema was i n s t a n t i a t e d ; i n the l a t t e r , an entry on the p o s s i b i l i t i e s l i s t may be added or m o d i f i e d . PRIORITIES: The user can i n f l u e n c e subsequent p r o c e s s i n g by modifying the p r i o r i t y queue. Rearranging the e n t r i e s changes the search and i n s t a n t i a t i o n p r i o r i t i e s of o b j e c t s . As an example, i f one i s i n t e r e s t e d i n b r i d g e s , move a l l the e n t r i e s r e l a t i n g to b r i d g e s to the head of the queue. Objects that are of no i n t e r e s t or that are p e r c e i v e d to be f a l l a c i o u s can be removed from the queue. By a l t e r i n g the parameters w i t h i n the e n t r i e s , one can modify the i n t e r p r e t a t i o n c o n t e x t . T h i s would i n c l u d e changing the l o c a t i o n i n the image where the o b j e c t w i l l be sought. The c y c l e of p e r c e p t i o n i s w e l l - s u i t e d to a schema-based, o b j e c t - o r i e n t e d v i s i o n system. I t allows feedback from p r e v i -o u s l y i n s t a n t i a t e d schemata to p r o v i d e a more informed context f o r subsequent p r o c e s s i n g . A l s o , i n t e r a c t i o n i s f a c i l i t a t e d by having c l e a r l y d i s t i n g u i s h e d p o i n t s i n the c y c l e where d i f f e r e n t types of i n f o r m a t i o n can be exchanged. The next s e c t i o n d e s c r i b e s a complex c o n t r o l s t r a t e g y which, because of the c y c l e of p e r c e p t i o n , can take advantage of both ge n e r a l and p a r t i c u l a r knowledge. Chapter 4. Combining Information Sources 72 4.5. Mixing Top-Down and Bottom-Up C o n t r o l Besides the c y c l e of p e r c e p t i o n , c o n t r o l i s s t r o n g l y i n f l u -enced by the h i e r a r c h i c a l r e l a t i o n s h i p s between schemata. In an o b j e c t - o r i e n t e d system, the connections between the o b j e c t s determine which p a i r s of schemata can most p r o f i t a b l y communi-c a t e . Messages can be sent from any schema to another; however, part of the goal of i n t e r p r e t a t i o n i s to j o i n a l l of the sche-mata i n t o a u n i f i e d network n e c e s s i t a t i n g s i g n i f i c a n t communica-t i o n between neighbouring nodes. The two important h i e r a r c h i e s i n terms of c o n t r o l are spe-c i a l i z a t i o n and decomposition. An example where these have been i n t e r t w i n e d i s found i n F i g u r e 4.4 (modified from a c l a s s i f i c a -t i o n i n L i l l e s a n d and K i e f e r , 1979, and the decomposition and s p e c i a l i z a t i o n h i e r a r c h i e s in Mapsee2). T h i s r e p r e s e n t s only general or s t e r e o t y p i c knowledge. A separate but r e l a t e d i n s t a n c e v e r s i o n of t h i s graph would be c r e a t e d d u r i n g the i n t e r p r e t a t i o n of an image. 4.5.j_. Top-Down and Bottom-Up C o n t r o l In a graphic form, the standard c o n t r o l paradigms are apparent. Top-down, model-driven behaviour w i l l be observed when schema-specific r o u t i n e s respond to messages from "above" in the h i e r a r c h i e s . Bottom-up, d a t a - d r i v e n c o n t r o l comes from below, s i n c e the o b j e c t s at the "bottom" of the graph (runway, road, r i v e r ) can be c o n s i d e r e d to be the most p r i m i t i v e , " c l o s e s t " to the data. The data (the i n t e n s i t y image and the Chapter 4. Combining Information Sources F i g u r e 4 .4 H i e r a r c h i e s i n a Geographic Domain Chapter 4. Combining Information Sources 74 sketch map but not n e c e s s a r i l y that coming from the user) can be thought of as being another l a y e r "below" the d e p i c t e d schemata. The r e l a t i o n s h i p i s not s p e c i a l i z a t i o n or decomposition but some type of mapping. The d i r e c t i o n s i n the graph make sense because the graph has been d i s p l a y e d with the schemata ordered from the bottom by i n c r e a s i n g g e n e r a l i t y and composition. Top-down c o n t r o l can take two forms. The f i r s t i s l i k e top-down p a r s i n g : e.g. i f one wants to e s t a b l i s h a geosystem then f i n d e i t h e r a landmass or a waterbody (G —> L | W). Model knowledge i s used at each stage i n d e c i d i n g how to move down the h i e r a r c h i e s u n t i l a bottom l e v e l schema ( t e r m i n a l symbol) i s reached. I f i t can be i n s t a n t i a t e d from the IS's then the h i e r a r c h i e s that have been b u i l t up can remain i n p l a c e . Other-wise, c o n t r o l must back up to a ch o i c e p o i n t and a d i f f e r e n t branch must be taken. The other type of top-down c o n t r o l can take e f f e c t only a f t e r part of the i n t e r p r e t a t i o n has taken p l a c e . Previous r e s u l t s h elp form the context of the c u r r e n t stage of proc e s s -i n g . The context p l u s the model knowledge of the schema are used to produce a more e f f i c i e n t s e a r c h . For example, i f i t has al r e a d y been e s t a b l i s h e d that a bri d g e e x i s t s , then i f c o n t r o l c u r r e n t l y r e s i d e s i n the road system schema, a good s t r a t e g y would be to look f o r a road going over the b r i d g e . Since the l o c a t i o n and o r i e n t a t i o n of the brid g e are known, t h i s g r e a t l y c o n s t r a i n s the r e s u l t i n g search f o r evidence of a road. Chapter 4. Combining Information Sources 75 4_.5.2. Rela x i n g Thresholds T h i s second type of top-down, or e x p e c t a t i o n - d r i v e n , con-t r o l can a l s o be used to r e l a x t h r e s h o l d s . Thresholds are those c u t o f f p o i n t s that schema-specific r o u t i n e s must use to d e t e r -mine whether the f e a t u r e s i n an IS are s u f f i c i e n t to i n s t a n t i a t e the o b j e c t . For example, the i n t e n s i t i e s of p i x e l s r e p r e s e n t i n g water are u s u a l l y l e s s than those corresponding to land. A t h r e s h o l d f o r water i s an i n t e n s i t y value that c l a s s i f i e s as water a l l p i x e l s having lower i n t e n s i t i e s . However, i t i s gen-e r a l l y true that i f one t r i e s to f i n d an i n t e n s i t y value that d i v i d e s a l l p i x e l s i n t o two c a t e g o r i e s (water and l a n d ) , there w i l l always be some p i x e l s that are c l a s s i f i e d i n c o r r e c t l y . The s i t u a t i o n i s not desperate i f one c o n s i d e r s a s i n g l e c l a s s i f i c a t i o n , such as water. I t i s o f t e n the case that a r e s -t r i c t e d range df i n t e n s i t y values can be found such that any p i x e l having an i n t e n s i t y i n the range can r e l i a b l y be c l a s s i -f i e d as w a t e r [ 3 ] . While t h i s i m p l i e s t h a t non-water o b j e c t s w i l l not have i n t e n s i t i e s i n that range, there w i l l be p i x e l s corresponding to water that are m i s c l a s s i f i e d . But i f the schema-spec ific r o u t i n e f o r a waterbody i s e x e c u t i n g , perhaps at the lake node, in a context where water i s expected to be found, then the t h r e s h o l d can be r e l a x e d ( i . e . the range extended) so [ 3 ] T h i s w i l l not always be the case in complicated images c o n t a i n i n g shadows, c l o u d s , l i g h t sources, e t c . but then, the r e l i a b i l i t y would be low. A l s o , s i n g l e p i x e l v a l u e s are prone to s e v e r a l types of e r r o r s , so that groups of p i x e l s are p r e f e r a b l e — c f . S e c t i o n 5.3.1.2. Chapter 4. Combining Information Sources 76 that more of the image can be c o r r e c t l y c l a s s i f i e d . In t h i s way, one s t a r t s from a c o n s e r v a t i v e p o s i t i o n t h at w i l l only accept f e a t u r e v a l u e s that are r e l i a b l e i n d i c a t o r s of the presence of an object (the p r i n c i p l e of l e a s t commitment--Marr, 1976). As the e x p e c t a t i o n s r i s e f o r e x i s t e n c e of the o b j e c t , the t h r e s h o l d s can be r e l a x e d so that l e s s r e l i a b l e f e a t u r e values w i l l s t i l l v e r i f y i t s presence. The c u r r e n t con-t e x t p l u s model knowledge determine how much the t h r e s h o l d s should be r e l a x e d . Returning to the example, i f the range of p o s s i b l e p i x e l i n t e n s i t i e s i s [0,255] and the c l a s s i f i c a t i o n s f o r p i x e l s i n c l u d e water, road, f i e l d , urban, and mountain, then there might be a range [20,50] such that a l l p i x e l s having values i n t h i s range are water. That i s , no p i x e l s to be c l a s s i f i e d as road w i l l have an i n t e n s i t y v a l u e s i n t h i s range. Another range [0,75] would i n c l u d e a l l p i x e l s to be c l a s s i f i e d as water but might i n c l u d e other c a t e g o r i e s as w e l l . Thus i f there i s no a p r i o r i knowledge of the p i x e l s being c o n s i d e r e d , use 20 and 50 as d e c i s i o n boundaries. I f there i s a high e x p e c t a t i o n that a p i x e l r e p r e s e n t s water then t e s t whether i t i s between 0 and 75. Intermediate e x p e c t a t i o n s can prompt the use of values somewhere between these extremes, such as 10 and 62. More f o r m a l l y , i f C i s the set of a l l p o s s i b l e c l a s s i f i c a -t i o n s (CO, C1, . . ., Cm), then l e t Cy be the c l a s s i f i c a t i o n that i s sought, and Cn a l l the other c a t e g o r i e s (C - Cy). Then, Chapter 4. Combining Information Sources 77 one s t a r t s with two ranges: [a,b]—no f e a t u r e corresponding to elements i n Cn has a value i n t h i s range. [ a ' , b * ] — a l l f e a t u r e s c o r r e s p o n d i n g to Cy have values i n t h i s range. a' < a & b < b' C a l l the i n t e r v a l of acceptance, I, where 1(0) = [a,b] 1(1) = [a',b'] I(e) = [ f ( e , a , a ' ) , f ( e , b , b ' ) ] and e i s the e x p e c t a t i o n that the f e a t u r e e x i s t s (0 < e < 1). f i s a f u n c t i o n that r e t u r n s a number that i s used as a t h r e s h o l d . For example, i t can be a s t e p f u n c t i o n , ~ f(e,x,y) = i f 0.00 < e < 0.33 -> x 0.33 < e < 0.67 -> (x + y)/2 0.67 < e < 1.00 -> y l i n e a r , f ( e,x,y) = x + e * (y - x ) ) , e x p o n e n t i a l , or something more complex. 4.5.3_. Mixed C o n t r o l S t r a t e g i e s Since both top-down and bottom-up c o n t r o l can take p l a c e , what determines which w i l l be a p p l i e d and when? A s i m i l a r ques-t i o n p e r t a i n s to where the c y c l e of p e r c e p t i o n i s to be entered. I n i t i a l l y t h i s l a r g e l y depends on the amount of user d i r e c t i o n . Chapter 4. Combining Information Sources 78 The user can s p e c i f y any schema(ta) as the s t a r t i n g p o i n t fo r i n t e r p r e t a t i o n . If he/she s p e c i f i e s a schema that can be i n s t a n t i a t e d d i r e c t l y from the data p l u s some r e l e v a n t informa-t i o n l i k e a l o c a t i o n , then bottom-up p r o c e s s i n g can s t a r t from that p o i n t . Otherwise, the schema-specific top-down r o u t i n e w i l l be i n i t i a t e d . T h i s procedure w i l l send out an a p p r o p r i a t e message to some schema lower down i n the h i e r a r c h y which w i l l propagate u n t i l a schema can be i n s t a n t i a t e d bottom-up from the data. T h i s communication does not n e c e s s a r i l y c r e a t e i n s t a n c e s of the h i g h e r - l e v e l schemata. That i s , a schema-specific rou-t i n e might be evaluated, but i f i t passes c o n t r o l to another r o u t i n e without f i l l i n g i n i t s s l o t s , the data s t r u c t u r e i t s e l f might not remain. If the user does not provide any s t a r t i n g p l a c e i n the net-work a d e f a u l t node i s used. Because i t repr e s e n t s g e n e r a l knowledge, one high up i n the h i e r a r c h i e s i s the most a p p l i c a -b l e . I t would send out model-based top-down messages as i n the case above i n which the user s e l e c t s the s t a r t i n g p o i n t . Once a schema i s i n s t a n t i a t e d from the data by having i t s s l o t s f i l l e d then two types of messages can be sent. F i r s t , the schema w i l l g e n e r a l l y t r a n s m i t a bottom-up request to h i g h e r -l e v e l schemata to d i s c o v e r where i t should f i t i n t o the semantic network. For example, a newly i n s t a n t i a t e d road w i l l send a message to "road system" which looks f o r a compatible e x i s t i n g road-system instance to j o i n . If one does not e x i s t , i t w i l l be c r e a t e d . Second, the schema may send out suggestions of other Chapter 4. Combining Information Sources 79 p o s s i b l e i n t e r p r e t a t i o n s . While the road schema was being i n s t a n t i a t e d , a dark, r e c t a n g u l a r region would be a candidate f o r a c a r , so the l o c a t i o n of the region can be sent to the car schema. These suggestions c o u l d go i n any d i r e c t i o n i n the h i e r a r c h y - - u p , down, or l a t e r a l l y . Once p r o c e s s i n g has passed the f i r s t few stages, there w i l l g e n e r a l l y be s e v e r a l p o s s i b i l i t i e s f o r f u r t h e r p r o c e s s i n g . These are p l a c e d i n a p r i o r i t y queue which can be m o d i f i e d by the user ( S e c t i o n 4.4.1). These p o s s i b i l i t i e s i n c l u d e a mixture of both top-down and bottom-up messages[4]. C o n t r o l w i l l move up and down the h i e r a r c h i e s and at each node more messages w i l l be spawned to induce more p r o c e s s i n g . T h i s type of p r o c e s s i n g i s n e i t h e r top-down nor bottom-up. It takes advantage of the r e l a t i o n s between schemata which organiz e s them i n t o a graph (see a l s o C o l l i n s and L o f t u s , 1975; B o l l e s , 1979). A message sent from a schema may be going up the h i e r a r c h y (bottom-up), down i t (top-down), l a t e r a l l y a c r o s s , l a t e r a l l y a c r o s s and down, or l a t e r a l l y a c r o s s and up. There i s no name f o r the l a t t e r three modes; the c l o s e s t might be h e t e r a r c h i c a l (Winston, 1972). The s t r a t e g y i s b e s t - f i r s t or i s l a n d - d r i v e n with the rankings on the p r i o r i t y queue used to determine the order of search. Since hypotheses can be [4]As w i l l be seen in Chapter 5, there are two types of bottom-up e n t r i e s , one d e a l i n g with the i n t e n s i t y image alone and another that uses the r e s u l t s of sketch map i n t e r p r e t a t i o n in c o n j u n c t i o n with the image. Chapter 4. Combining Information Sources 80 r e t r a c t e d , i t i s more akin to non-monotonic reasoning than most v i s i o n systems which u s u a l l y f o l l o w the p r i n c i p l e of l e a s t com-mitment (Marr, 1976). I t i s a l s o d e s c r i b e d w e l l and i s indepen-d e n t l y motivated by the c y c l e of p e r c e p t i o n . 4.5.4. D i s t r i b u t e d A l , VLSI, and P a r a l l e l P r o c e s s i n g D i s t r i b u t e d Al i s a q u i t e new subset of Al that d e a l s with a group of " d i s t i n c t , c o o p e r a t i n g problem s o l v e r s " (Davis, 1980b, p. 42; t h i s report summarizes the v a r i o u s a t t i t u d e s toward the f i e l d ) . The system being d e s c r i b e d f i t s i n t o t h i s model. Each schema can be c o n s i d e r e d a problem s o l v e r that knows how to r e c o g n i z e the o b j e c t being modelled. Coupled with recent work in VLSI, e x c i t i n g p o s s i b i l i t i e s i n p a r a l l e l p r o c e s s i n g can be imagined. If each schema, or b e t t e r yet each i n s t a n c e of a schema, had access to i t s own p r o c e s s o r , then much of the i n t e r p r e t a t i o n process can proceed in p a r a l l e l . Instead of messages being queued, they would be sent to a c t i v e , executing r o u t i n e s . I f the number of p r o c e s s o r s was l i m i t e d , the p r i o r i t y rankings can be used to determine which schemata would r e c e i v e a t t e n t i o n . P a r a l l e l p r o c e s s i n g as d e s c r i b e d would not i n c r e a s e the t h e o r e t i c a l power of a v i s i o n system only i t s e f f i c i e n c y . That i s , i t would not be able to i n t e r p r e t images that were more pr o b l e m a t i c . However, such a system would make i t more con-venient f o r schemata to communicate while t h e y are a c t i v e . T h i s i n c r e a s e d c o o p e r a t i o n might c r e a t e a s i t u a t i o n where the Chapter 4. Combining Information Sources 81 i n t e r p r e t a t i o n p o t e n t i a l i s i n c r e a s e d . 4.6. Using G l o b a l s to Remove Thresholds Thresholds were d e s c r i b e d i n S e c t i o n 4.5.2 as a necessary e v i l that w i l l o f t e n be found i n schema-specific i n s t a n t i a t i o n r o u t i n e s . T h i s s e c t i o n d e s c r i b e s a method for reducing the number of magic (ad hoc) numbers that must be used as t h r e s -h o l d s . Take a chain ( l i n e ) from a sketch map that has been i n t e r -p r e t e d by sketch map a n a l y s i s and c o n s i d e r i t as a r i v e r . If the sketch map i s known to be approximately r e g i s t e r e d to an i n t e n s i t y image of the same scene, then one can use the l o c a t i o n of the sketch map c h a i n to search f o r a r i v e r i n the image. However, a chain has no t h i c k n e s s and a r i v e r does; the c h a i n only approximates the r i v e r ' s l o c a t i o n . So, one would probably want to c r e a t e an e l l i p s e or r e c t a n g l e around the c h a i n and search w i t h i n the corresponding region i n the i n t e n s i t y image. The l e n g t h of the r e c t a n g l e (or major a x i s of the e l l i p s e ) can be d e f i n e d as the end p o i n t s of the c h a i n . However, the width (or minor a x i s ) would have to be an ad hoc number or t h r e s h o l d chosen to r e f l e c t what i s known about the widths of r i v e r s that are l i k e l y to appear in the images to be i n t e r p r e -t a t e d . S i m i l a r t h r e s h o l d s are r e q u i r e d i n almost a l l matches between f e a t u r e s from d i f f e r e n t i n f o r m a t i o n sources because the correspondence w i l l always be approximate. Chapter 4. Combining Information Sources 82 A l l of the t h r e s h o l d s based on s i z e , such as r i v e r width, are r e l a t e d to the s c a l e or r e s o l u t i o n of the image i n q u e s t i o n . If one knows the a c t u a l s i z e of each p i x e l , or a range of s i z e s , then r e a l - w o r l d knowledge of r i v e r s can be used to generate the t h r e s h o l d . For example, i f the r e s o l u t i o n of the image i s 20 f e e t per p i x e l and r i v e r s are g e n e r a l l y between 5 and 200 f e e t wide, then the r e c t a n g l e should be between .25 and 10 p i x e l s wide. In the s i t u a t i o n above, one would choose the l a r g e r value (10) to be sure to i n c l u d e enough of the image where the r i v e r might be. .However, in many cases, one would use the p r i n c i p l e of l e a s t commitment and choose the smaller v a l u e . Higher expec-t a t i o n s allow one to r e l a x the t h r e s h o l d s to the l a r g e r value as was d e s c r i b e d i n S e c t i o n 4.5.2. In the terminology d e s c r i b e d i n that s e c t i o n , f o r t h i s example the ranges are [a,b] = [0,.25] and [a',b'] = [0,10]. Threshold ranges that can be r e l a x e d are a p p l i c a b l e even i f ad hoc v a l u e s known only to a schema-specific r o u t i n e , such as r i v e r , are used. The advantage of t y i n g them to the s c a l e of the image i s that s c a l e i s a g l o b a l value that holds f o r a l l of the schemata in the image. Hence, i t s value can be f a c t o r e d out of the schema-specific r o u t i n e s . For the above example, the r e f e r e n c e might be to " 2 0 0 / r e s o l u t i o n " , where r e s o l u t i o n i s a g l o b a l v a r i a b l e . If a l l such s i z e - r e l a t e d t h r e s h o l d s are t r a n s l a t e d i n t o a f a c t o r of the s c a l e , then a l a r g e number of t h r e s h o l d s are a l l reduced to one value p l u s the f a c t o r s , which Chapter 4. Combining Information Sources 83 are based on model knowledge. T h i s i s not a case of " p u t t i n g a l l of one's eggs i n a s i n -g l e basket", however, i f the p r i n c i p l e of g r a c e f u l degradation i s a p p l i e d . The schema-specific r o u t i n e s can take i n t o account the c e r t a i n t y of the value f o r r e s o l u t i o n whenever i t i s used, as i n g e n e r a t i n g t h r e s h o l d f a c t o r s . Then, the q u a l i t y of that value would determine how good the r e s u l t s were. Such a value's use can be f a c i l i t a t e d i f s c a l e i s represented as a schema, s i n c e schemata have a s l o t f o r CONFIDENCE p l u s a mechanism f o r c a l c u l a t i n g i t ( S e c t i o n 4.2). In a d d i t i o n , schemata have pro-cedures a s s o c i a t e d with them to respond when the value i s added, m o d i f i e d , or removed. There i s a l s o a d e f a u l t value that can be used i f no value i s a v a i l a b l e . A l s o , there are atta c h e d pro-cedures that can attempt to c a l c u l a t e the value when i t does not yet e x i s t . Thus i n s t e a d of " 2 0 0 / r e s o l u t i o n " one might r e f e r to "200/(sgetv ' s c a l e ' f e e t p e r p i x e l 'yes 'one ' d e f a u l t ) " i n the MAIDS system. There are two ways to o b t a i n the value of a g l o b a l such as s c a l e . I t can be entered by the user as g l o b a l input ( c f . Sec-t i o n 4.4.1) s i n c e s c a l e i s o f t e n p r o v i d e d with remotely sensed imagery. Or, i t can be i n f e r r e d from the image plus model knowledge. Before, the t h r e s h o l d range was used to h e l p i n s t a n -t i a t e an o b j e c t . However, once the object i s i n s t a n t i a t e d , perhaps by some other t e s t or other IS, i t s known s i z e can be used to l i m i t the p o s s i b l e range of s c a l e v a l u e s — b y m u l t i p l y i n g the s i z e by the r e c i p r o c a l of the f a c t o r s . T h i s value can be Chapter 4. Combining Information Sources 84 m o d i f i e d and improved, i . e . r e s t r i c t e d , as i n t e r p r e t a t i o n proceeds. A mechanism f o r c a r r y i n g i t out i s found in the next s e c t i o n . These g l o b a l values are somewhat unusual schemata s i n c e they are not part of the semantic network of o b j e c t s . They are s i m i l a r to g l o b a l v a r i a b l e s i n programming languages which d i f f e r from the l o c a l v a r i a b l e s that are l e x i c a l l y ordered by procedure d e c l a r a t i o n s i n s t a t i c a l l y - s c o p e d languages. S c a l e i s a u s e f u l g l o b a l i n reducing s i z e - r e l a t e d t h r e s -h o l d s . Many angular t h r e s h o l d s can be r e l a t e d to the azimuth of the sun, e s p e c i a l l y when shadows are i n v o l v e d . T h i s value i s o f t e n a v a i l a b l e with s a t e l l i t e imagery. When d e a l i n g with t h r e e - d i m e n s i o n a l models, the e l e v a t i o n of the sun i s a l s o use-f u l . 4.2- C l u s t e r A n a l y s i s as an Inference Mechanism In S e c t i o n 4.5.2, the s i m i l a r i t y of p i x e l i n t e n s i t y v a l u e s was used to make an i n f e r e n c e about the c l a s s i f i c a t i o n of those e n t i t i e s i n t o water and l a n d : namely, s i m i l a r o b j e c t s i n the scene have s i m i l a r f e a t u r e v a l u e s . F u r t h e r , i f the f e a t u r e space i s s e l e c t e d a p p r o p r i a t e l y , these f e a t u r e values can be used to d i s c r i m i n a t e d i f f e r e n t o b j e c t s because they w i l l have d i f f e r e n t c l u s t e r s of v a l u e s . I f s i m i l a r , i n d i v i d u a l f e a t u r e s in the image are grouped ( c l u s t e r e d ) , then the corresponding image p o i n t s are a l l p a r t of some meaningful c l a s s . D e r i v e d f e a t u r e s of compound o b j e c t s , such as the o r i e n t a t i o n of edge Chapter 4. Combining Information Sources 85 segments, are even more u s e f u l than p i x e l i n t e n s i t y v a l u e s . Histograms i n the proper f e a t u r e space make c l u s t e r i n g apparent. F i g u r e 4.5 shows the edge segments found i n an image of Cranbrook, B r i t i s h Columbia, which c o n t a i n s a number of roads. F i g u r e 4.6 i s the one-dimensional histogram showing the number of segments having a given o r i e n t a t i o n . There are two s i g n i f i c a n t s e c t i o n s i n t h i s histogram: A corresponds to northeast-southwest roads, and B to northwest-southeast roads. Note however, that the histogram i s very rough and that d i v i -s i o n s between these s e c t i o n s are o f t e n d i f f i c u l t to determine. A smoother diagram would be p r e f e r a b l e . The method whereby image components are t r a n s l a t e d i n t o a f e a t u r e space that can be used i n d i s c r i m i n a t i n g s i g n i f i c a n t o b j e c t s i s known as the Hough t r a n s f o r m a t i o n . I t was o r i g i n a l l y used to group edge p o i n t s i n t o l i n e segments by s u c c e s s i v e l y c l u s t e r i n g one-dimensional histograms of e i t h e r slopes and i n t e r c e p t s (y = mx + b) or angles and d i s t a n c e s (p = xcosfl + ysin0) (Duda and Hart, 1972; O'Gorman and Clowes, 1973; Dudani and Luk, 1977). More r e c e n t l y , B a l l a r d and h i s group have a p p l i e d the technique to the a n a l y s i s of i n t r i n s i c image data ( B a l l a r d , 1981; B a l l a r d and Sabbah, 1981). They used m u l t i -dimensional histograms i n a v a r i e t y of a p p l i c a t i o n s to determine model-to-image t r a n s f o r m a t i o n parameters, s u r f a c e o r i e n t a t i o n , o p t i c a l flow, e t c . Chapter 4. Combining Information Sources 86 F i g u r e 4.5 Edge Segments Superimposed on Cranbrook Chapter 4. Combining Information Sources 35 30 25 Number of 20 Edge Segments (a = 3.0) 1 5 1 0 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * B * * * * * * * * * A * * * ** * ** * ** * ** * ** * ** ** ** ** ** ** ** ** * ** * ** * ** * ** * * * * * ** * * * * * ** * * * * * * ** * * * * * * * * * * * * * * * * * * * * *** * * ***** * **** * * ***** * ****** * * ***** * ****** * * * ***** ** ** ** ***** ** ** ** ***** *** ** ** ***** *** *********** *** ************ *** **************** ***************** 511223344556677889911111111111111111 05050 5050505050 50 50011223344 5566778 05050505050505050 O r i e n t a t i o n i n Degrees F i g u r e 4.6 A Histogram of Edge Segment O r i e n t a t i o n s Chapter 4. Combining Information Sources 88 C l u s t e r i n g i s a method of unsupervised, nonparametric p a t -t e r n r e c o g n i t i o n . When using histograms, the gen e r a l s t r a t e g y uses peaks f o r the modes of c a t e g o r i e s and troughs as d e c i s i o n boundaries to c l a s s i f y v a l u e s . Using image i n t e n s i t i e s as f e a t u r e s , segmentation can be done i n t h i s manner (Eigen et a l . , 1974; Ohlander, 1975; Schacter et a l . , 1976; Ohta et a l . , 1978; Tomita, 1981). The method d e s c r i b e d below f o l l o w s from t h i s work. I t s major advance i s the use of Gaussian smoothing to f a c i l i t a t e peak and trough s e l e c t i o n . 4.7.K A Method f o r C l u s t e r i n g Consider the f u n c t i o n i n F i g u r e 4.7 as an i d e a l i z e d , con-tinuous histogram. I n t u i t i v e l y , t h i s r e p r e s e n t s two c l u s t e r s with modes at peaks A and C and a d i v i s i o n at the trough, B. I f A Number of Data Points Feature F i g u r e 4.7 An I d e a l i z e d , Continuous Histogram Chapter 4. Combining Information Sources 89 we regard the histogram as a f u n c t i o n , then n o t i c e that p o i n t s A, B, and C are the only ones ( i g n o r i n g the ends) where the slope of the tangent to the f u n c t i o n (or, e q u i v a l e n t l y , the f i r s t d e r i v a t i v e ) i s 0 . Hence, we can f i n d both the modes and d e c i s i o n boundaries by l o o k i n g f o r zeros, or zero c r o s s i n g s i n the d i s c r e t e case, i n the f i r s t d e r i v a t i v e of the histogram of a given f e a t u r e . T h i s d i s c u s s i o n presumes that a histogram can be d e r i v e d from the data. F i r s t , i t i s c l e a r t h at n-dimensional histograms can be formed fo r m u l t i v a r i a t e samples. Each f e a t u r e would represent a d i f f e r e n t dimension. Although, f o r c l a r i t y , the a n a l y s i s w i l l be e x p l a i n e d using the one-dimensional case, i t holds i n higher dimensions. An example of the two-dimensional case w i l l be given i n S e c t i o n 4 . 7 . 5 . The range of f e a t u r e values can be q u i t e l a r g e . However, as long as a l l the values are f i n i t e , or i f a l l the i n f i n i t e v a l u e s can be put i n t o a f i n i t e number of d i s t i n c t buckets, then a histogram can be d e r i v e d . If there are many empty buckets then a s u i t a b l e t r a n s f o r m a t i o n (e.g. l o g a r i t h m i c ) can reduce the space to a more manageable s i z e or i n d i r e c t a d d r e s s i n g can be used f o r a computer implementation. Histograms, by t h e i r nature, are d i s c r e t e , with buckets or b i n s c o n t a i n i n g a l l the values in a c e r t a i n range. Thus i f the data are r e a l - v a l u e d , bucket boundaries must be s e l e c t e d : i f they are too wide, then too many data p o i n t s w i l l be put i n t o Chapter 4 . Combining Information Sources 90 each bucket and the histogram w i l l be too smooth; i f they are too narrow, the histogram w i l l be too rough, or can even f l a t t e n out, d e s t r o y i n g the modes. These c o n d i t i o n s correspond to l a r g e b i a s and l a r g e v a r i a n c e , r e s p e c t i v e l y . The f i r s t case i s a problem that i s e a s i l y avoided by scanning the data and making sure that no bucket r e c e i v e s more than a small percentage of the t o t a l number of samples. The l a t t e r case can be avoided i f there i s some smoothing or grouping of the buckets. 4.2-2. Gaussian Smoothing Many histograms of r e a l - w o r l d data have a very n o i s y appearance, l e a d i n g to the g e n e r a t i o n of many t i n y l o c a l peaks. For t h i s reason, many r e s e a r c h e r s have employed smoothing t e c h -niques before peak s e l e c t i o n . In most cases, t h i s has taken the form of l o c a l a v e r a g i n g . I t has been accomplished e i t h e r by v a r y i n g the b i n width (Eigen et a l . , 1974; Leboucher and Lowitz, 1978) or by a r i t h m e t i c averaging under a moving window ( S c o t t , 1979; Dudani and Luk, 1977). Uniform averaging i s , i n e f f e c t , a low pass f i l t e r . How-ever, f o l l o w i n g Marr and H i l d r e t h (Marr and H i l d r e t h , 1980; H i l -d r e t h , 1980), t h i s f i l t e r i s not b a n d - l i m i t e d i n the frequency domain. They show that a Gaussian, i . e . normal, f u n c t i o n leads to the smoothing f i l t e r that i s optimal with respect to the t r a d e o f f between s p a t i a l l o c a l i z a t i o n and m a i n t a i n i n g a small v a r i a n c e in the frequency domain. Chapter 4. Combining Information Sources 91 The one-dimensional Gaussian f u n c t i o n , G(x) = 1/ /2~r7ar e x p ( - x 2 / 2 a 2 ) i s d e p i c t e d i n F i g u r e 4.8a. Since the s c a l e f a c t e r 1/ \Z2~TFOT i s i r r e l e v a n t t o the f o l l o w i n g , i t w i l l be omitted. There i s one r e q u i r e d parameter, a, which must be s u p p l i e d before the f u n c t i o n can be c a l c u l a t e d . A h e u r i s t i c w i l l be pro-v i d e d in S e c t i o n 4.7.4 f o r d i v i n i n g good values f o r o. 4.7.3_. Peak and Trough D e t e c t i o n Once the histogram i s smoothed, c l u s t e r s can be separated by t a k i n g d e r i v a t i v e s of the histogram f u n c t i o n . S p e c i f i c a l l y , the zero c r o s s i n g s of the f i r s t d e r i v a t i v e w i l l i n d i c a t e the peaks and troughs and the second d e r i v a t i v e can be used to d i s -t i n g u i s h one from the other. F i g u r e 4.9a shows an example histogram and F i g u r e 4.9b shows the r e s u l t of c o n v o l v i n g i t with a Gaussian f i l t e r with a = 5. The f i r s t and second d e r i v a t i v e s are shown in F i g u r e 4.9c and d. The zero c r o s s i n g s i n F i g u r e 4.9c show the p o i n t s of i n t e r e s t . C2 and C4 are the modes of the c l u s t e r s and C1, C3, and C5 are the d e c i s i o n boundaries or c u t o f f p o i n t s f o r member-sh i p i n those c l u s t e r s . The second d e r i v a t i v e s i n F i g u r e 4.9d show that peaks have a corresponding negative value i n the second d e r i v a t i v e and troughs have a p o s i t i v e v a l u e . Chapter 4. Combining Information Sources 9 2 F i g u r e 4.8 The Gaussian D i s t r i b u t i o n a. G(x) The Gaussian f u n c t i o n b. G'(x) The f i r s t d e r i v a t i v e of the Gaussian f u n c t i o n c. G''(x) The second d e r i v a t i v e of the Gaussian f u n c t i o n Chapter 4. Combining Information Sources F i g u r e 4.9 A Gaussian-Smoothed Histogram and D e r i v a t i v e s a. Histogram b. Histogram convolved with the Gaussian f u n c t i o n c. Histogram convolved with the f i r s t d e r i v a t i v e of the Gaussian f u n c t i o n d. Histogram convolved with the second d e r i v a t i v e of the Gaussian f u n c t i o n Chapter 4. Combining Information Sources 94 Thus, f i n d i n g the zero c r o s s i n g s i n f ( x ) = d[G(x) * H(x)] dx i s the e q u i v a l e n t of peak and trough s e l e c t i o n where G i s the Gaussian, H i s the histogram f u n c t i o n , and * i s the c o n v o l u t i o n o p e r a t o r . By the d e r i v a t i v e r u l e f o r c o n v o l u t i o n s , f ( x ) = dG(x) * H(x) dx where, dG(x) = -x e x p ( - x 2 / 2 a 2 ) dx and exp stands f o r e r a i s e d to a power. S i m i l a r l y , the second d e r i v a t i v e of the Gaussian f u n c t i o n i s d 2G(x) = ( x 2 - a 2 ) e x p ( - x 2 / 2 a 2 ) . dx These are p l o t t e d i n F i g u r e s 4.8b and 4.8c. 4.7.4. The S t a b i l i t y H e u r i s t i c The method d e s c r i b e d f o r edge d e t e c t i o n i s dependent on the amount of smoothing. T h i s i s determined by the value chosen f o r a. A l a r g e a w i l l cause many p o i n t s to be merged together, hence fewer peaks and fewer c l u s t e r s . Conversely f o r small o there are p o t e n t i a l l y as many c l u s t e r s as data p o i n t s . Chapter 4. Combining Information Sources 95 These c o n d i t i o n s are shown in F i g u r e 4.10, f o r the same data used f o r F i g u r e 4.9. The value a = 10 i n F i g u r e 4.10a leads to only a s i n g l e peak, whereas a = 1 i n F i g u r e 4.10b r e s u l t s i n 4 peaks. Eigen et a l . (1974) and P o s t a i r e and Vasseur (1981) both d e s c r i b e c l u s t e r i n g methods that depend on a q u a n t i z a t i o n f a c t o r ( t h e i r 8 and M r e s p e c t i v e l y ) . As i s the case with a, the value f o r the q u a n t i z a t i o n f a c t o r determines the number of r e s u l t i n g c l u s t e r s . By p l o t t i n g the number of components or c l u s t e r s b. a = 1 F i g u r e 4.10 The E f f e c t s of a on Smoothing a. a = 10 b. a = 1 Chapter 4. Combining Information Sources 96 versus 6 (or M), they d i s c o v e r e d a region of s t a b i l i t y — t h e l a r g e s t range of 8 (or M) v a l u e s where the number of c l u s t e r s remained c o n s t a n t . P o s t a i r e and Vasseur found that t h i s ' value of M was very c l o s e to the optimal Bayes minimum e r r o r r a t e . T h i s h e u r i s t i c works very w e l l f o r the parameter a. F i g u r e 4.11 shows the r e s u l t of v a r i o u s a's on the number of c l u s t e r s f o r the example histogram of F i g u r e 4.9. The longest s t a b l e r e g i o n extends from o equal to 2 to 8. The midpoint of t h i s r e g i o n (a = 5) i s the value used i n the- Gaussian smoothing of F i g u r e 4.8 and 4.9. There i s one immediate d i f f i c u l t y i n v o l v e d with using t h i s h e u r i s t i c that concerns l a r g e v a l u e s of o. As one i n c r e a s e s a, CO CO ZD LJ 11 f\l O LU CD 6 7 8 SJGMfl 10 11 12 ]3 F i g u r e 4.11 F i n d i n g the Largest S t a b l e Region of a Values Chapter 4. Combining Information Sources 97 e v e n t u a l l y there w i l l be only one c l u s t e r l e f t . A l l l a r g e r a's w i l l a l s o r e s u l t i n one c l u s t e r . Consequently, there w i l l always be an i n f i n i t e range of a's l e a d i n g to one c l u s t e r . I f there i s no a p r i o r i way to f i x an upper l i m i t to the value of a, one must ignore one c l u s t e r as a v a l i d r e s u l t of the h e u r i s -t i c . _4_.2.5- Extensions to M u l t i v a r i a t e Data As one goes from u n i v a r i a t e to m u l t i v a r i a t e data spaces the b a s i c technique remains the same: smoothing and d i f f e r e n t i a t i o n are accomplished by c o n v o l u t i o n , then the modes and d e c i s i o n boundaries are found, although t h i s becomes more complex. F i n a l l y , the s t a b i l i t y h e u r i s t i c i s a p p l i e d . A b i v a r i a t e exam-pl e w i l l show how the g e n e r a l i z a t i o n takes p l a c e . F i g u r e 4.12 shows a two-dimensional example with f i v e c l u s -t e r s c r e a t e d by a pseudo-random number generator. Table 4.2 shows the means, c o v a r i a n c e s , and number of samples i n each c l u s t e r . The f i r s t s tep i n the procedure i s to convolve t h i s h i s t o -gram with a s u i t a b l e f i l t e r . The two-dimensional Gaussian func-t i o n (without the s c a l e f a c t o r ) i s G(x,y) = e x p ( - ( x 2 + y 2 ) / 2 a 2 ) . To a v o i d d i r e c t i o n a l e f f e c t s , one would l i k e to use non-d i r e c t i o n a l d e r i v a t i v e s . There i s no n o n - d i r e c t i o n a l f i r s t d e r i v a t i v e ; however, f o r the second d e r i v a t i v e , we can use the Chapter 4. Combining Information Sources 98 F i g u r e 4.12 A 2-D Histogram of a B i v a r i a t e D i s t r i b u t i o n Chapter 4. Combining Information Sources 99 CLUSTER MEAN COVARIANCE MATRIX SAMPLE SIZE 2 3 4 5 [20 60] [60 60] [20 40] [60 40] [40 20] fo r a l l c l u s t e r s 9 0 0 9 20 20 1 0 30 20 Table 4.2 The Relevant S t a t i s t i c s for F i g u r e s 4.12-17 Laplac i a n : 9 2G(x,y) + 3 2G(x,y) = 3x 2 9y 2 V 2G(x,y) = ( x 2 + y 2 - 2 a 2 ) e x p ( - ( x 2 + y 2 ) / 2 a 2 ) . The r e s u l t s of c o n v o l v i n g the histogram with these two f i l t e r s with a = 4.5 can be seen i n F i g u r e 4.13 and 4.14. Although there are no f i r s t d e r i v a t i v e zero c r o s s i n g s , the number of c l u s t e r s can be found from the Gaussian-smoothed image. A s m a l l , l o c a l operator can be used to f i n d the l o c a l maxima i n the image. Each peak corresponds to a c l u s t e r . F i g -ure 4.15 shows the zero c r o s s i n g s of the second d e r i v a t i v e func-t i o n along with the maxima of the smoothed image (the +'s). As i n the one-dimensional case, the number of modes depends on the amount of smoothing. The number of peaks can be used with the s t a b i l i t y h e u r i s t i c to f i n d a good c h o i c e f o r a. T h i s i s done i n F i g u r e 4.16, which shows how a value of o = 4.5 g i v e s 5 c l u s t e r s . Chapter 4. Combining Information Sources F i g u r e 4.13 The Histogram Convolved with the Gaussian f u n c t i o n , where a = 4.5 Chapter 4. Combining Information Sources F i g u r e 4.14 The Histogram Convolved with the L a p l a c i a n of the Gaussian f u n c t i o n Chapter 4. Combining Information Sources 1 02 F i g u r e 4.15 Zero C r o s s i n g s of the Second D e r i v a t i v e F i n d i n g the d e c i s i o n boundaries i s not as simple i n two dimensions. The second d e r i v a t i v e zero c r o s s i n g s . c o r r e s p o n d to i n f l e c t i o n p o i n t s on the o r i g i n a l smoothed c o n v e x i t i e s . I f the d i s t r i b u t i o n s are normal and r o t a t i o n a l l y symmetric, 67 percent of a l l the p o i n t s i n the c l u s t e r w i l l f a l l i n t o the p o r t i o n where the second d e r i v a t i v e i s n e g a t i v e . Hence the d e c i s i o n boundaries w i l l be somewhere between these negative second Chapter 4. Combining Information Sources 103 .00 CO !*•' CO ID . _,<o CJ u_io' O 03 21 "1 " I I 1 1 1 1 1 1 1 1 1 1 2 3 4 5 i l G H f l 8 9 1 0 1 1 1 2 1 3 F i g u r e 4.16 Determining o Using the S t a b i l i t y H e u r i s t i c d e r i v a t i v e regions (see F i g u r e 4.15). One can grow the zero c r o s s i n g boundaries f o r a l l the c l u s t e r s u n t i l they meet, then use the r e s u l t i n g l i n e s as d e c i s i o n boundaries. One a l t e r n a t i v e scheme i s to use some form of nearest neighbour technique f o r p o i n t s i n regions having a p o s i t i v e second d e r i v a t i v e . The prototype f o r each c l u s t e r would be i t s peak. S i m i l a r l y , a maximum l i k e l i h o o d technique might use the height of the peak as the a p r i o r i p r o b a b i l i t y f o r a c l u s t e r . Chapter 4. Combining Information Sources 1 04 Another a l t e r n a t i v e i s to t r e a t the smoothed histogram as a thr e e - d i m e n s i o n a l topographic s u r f a c e . I f the c l u s t e r s are thought of as peaks, then the d e c i s i o n boundaries w i l l f a l l a l o ng the channels. Using s u i t a b l e d e f i n i t i o n s , channels and r i d g e s , which c r o s s at passes, form complementary networks (Fowler and L i t t l e , 1980). The channel network f o r t h i s example i s found i n F i g u r e 4.17. While the nature of the problem becomes more d i f f i c u l t i n two dimensions, the b a s i c technique s t i l l a p p l i e s . The exten-s i o n to three and higher dimensions can a l s o be made, but at the c o s t of g r e a t e r complexity and computation. .4.8. Summary T h i s chapter has i n t r o d u c e d a number of t o o l s which can be put together i n t o a v i s i o n system. Such a system, c a l l e d MIS-SEE, i s the sub j e c t of the next chapter. The techniques d e s c r i b e d i n the i n t r o d u c t i o n are designed to achieve the goals of d e s c r i p t i v e and p r o c e d u r a l adequacy and g r a c e f u l d e g r a d a t i o n . How w e l l those c r i t e r i a have been met w i l l be d i s c u s s e d i n Chapter 6 which i n c l u d e s the r e s u l t s of running MISSEE on s e v e r a l examples. Chapter 4. Combining Information Sources x xxxxxxx x x x x x x x x x P x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x xxxxxx P xxxxx xxxxxx xxxxxxx x X ' HH x + x ### X X #### # X X HHHHH *## X X HHHHHH HHHHH X X HHHHHHHH HHHHHHH x X HHHHHHHHH HHHHHHHHHH X X HHHHHHHHHHH HHHHHHHHHHHHH X HHHHHHHHHHHHH HHHHHHHHHHHHHH HHHHHHHHHHHHHHH HHHHHHHHHHHHHHH HHHHHHHHHHHHHHH HHHHHHHHHHHHHHH HHHHHHHHUHHHHHHH HHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHH* HHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHH HHHHHHHH HHHHHHHHHHHHHHH H HHHHHHH HHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHH F i g u r e 4.17 Using Channels as D e c i s i o n Boundaries Legend + Peak - P i t P Pass x Channel # Zero value of the Surface Chapter 4. Combining Information Sources CHAPTER 5 The Implementation 5_._1_. I n t r o d u c t i o n The MISSEE ( M u l t i p l e Information Source SEE) system i s an implementation of many of the ideas d e s c r i b e d i n the p r e v i o u s chapter. F i g u r e 5.1 shows an overview of the components of MIS-SEE. The core of the system i s a semantic network of schemata which both c o n t r o l s the i n t e r p r e t a t i o n process and b u i l d s the r e s u l t i n g r e p r e s e n t a t i o n . The schemata are c r e a t e d and u t i l i z e d by the schemata manipulation subsystem ( c a l l e d MAIDS--MISSEE A i d s ) . The three input i n f o r m a t i o n sources that feed i n t o MIS-SEE are a d i g i t i z e d image, a sketch map, and the user. The system has gone through two i n c a r n a t i o n s . O r i g i n a l l y i t was w r i t t e n i n M u l t i l i s p (Koomen, 1980) and P a s c a l and ran on. an Amdahl 470. I t c u r r e n t l y e x i s t s i n a Unix environment on a VAX 11/780 u t i l i z i n g Franz L i s p and C. MISSEE i s not designed to be an end product. I t was w r i t -ten to t e s t the ideas of the previous chapter i n a domain that was r e s t r i c t e d , yet r i c h enough so that a l l of the problems i n model-based i n t e r p r e t a t i o n can a r i s e ( c f . S e c t i o n 3.2). While not e v e r y t h i n g proposed in Chapter 4 has been implemented, a subset demonstrating the power of the key ideas has been. The f o l l o w i n g s e c t i o n s w i l l d e s c r i b e what has been implemented and 1 06 MAPSEE2 ZERO CROSSING EDGE DETECTION / \ 1 \ 1 \ i \ REGION MERGING MAIDS A Schemata Manipulation System MISSEE (Multiple Information Source SEE) -Schemata for -Geographic Features -Roads, Rivers, etc. -Global Variables -Procedures for Schemata In s t a n t i a t i o n -Top Down -Bottom Up -From the Image alone -From a Sketch Map -Control Strategy  AN INFERENCE MECHANISM Stat i st i c a l Cluster Analysis sources of A information V data programs and schemata - data — control o other systems F i g u r e 5.1 The MISSEE System Chap te r 5. The Imp l emen ta t i on 108 p o i n t out what has not. 5.2. MAIDS--MISSEE Aids MAIDS i s an implementation of the schemata manipulation system d e s c r i b e d i n S e c t i o n 4.2. W r i t t e n i n Franz L i s p as an extension of Maya (Havens, 1978), i t makes a v a i l a b l e a number of L i s p f u n c t i o n s f o r the c r e a t i o n , m o d i f i c a t i o n , m a n i p u l a t i o n , and output of schemata. A user manual d e s c r i b i n g the use of these f u n c t i o n s i s found i n Appendix A. While MAIDS i s a general purpose system f o r h a n d l i n g sche-mata, MISSEE c o n t a i n s s p e c i f i c schemata in the geographic domain that are u s e f u l i n a n a l y z i n g a e r i a l photographs. These i n c l u d e such o b j e c t s as roads, r i v e r s , b r i d g e s , road systems, mountain ranges, and geosystems. The system c o n t a i n s s t e r e o t y p i c , gen-e r a l i n f o r m a t i o n i n the form of s l o t s , or a t t r i b u t e s , which d e f i n e the o b j e c t and p o s s i b l e d e f a u l t v a l u e s , and a t t a c h e d pro-cedures which f i l l those s l o t s d u r i n g i n s t a n t i a t i o n - - s e e s e c t i o n 5.4. General o b j e c t s , such as road, are d i s t i n g u i s h e d from par-t i c u l a r i n s t a n t i a t i o n s of them (road-4) v i a the INSTANCE l i n k and i t s i n v e r s e , AIO, an i n s t a n c e o f . The i n s t a n c e s are c r e a t e d d u r i n g i n s t a n t i a t i o n of a p a r t i c u l a r image, and, being hypothet-i c a l , may be destroyed i f i n s u f f i c i e n t or c o n t r a r y evidence i s found r e g a r d i n g the e n t i t y . O b j e c t s , being part of general knowledge, are r a r e l y , i f ever, destroyed. Chapter 5. The Implementation 109 Two of the major o r g a n i z i n g r e l a t i o n s h i p s w i t h i n both o b j e c t s and i n s t a n c e s are s p e c i a l i z a t i o n and decomposition. S p e c i a l i z a t i o n (SPECIALIZES-TO/AKO) and decomposition (DECOMPOSES-TO/APO) form h i e r a r c h i e s that are a n a t u r a l way to o r g a n i z e knowledge. The p r e v i o u s three r e l a t i o n s are a l l p a r t of the standard MAIDS system because of t h e i r general u s e f u l n e s s in schemata o r g a n i z a t i o n . A f o u r t h r e l a t i o n , "neighbours", ( i t s i n v e r s e i s a l s o neighbours) has been added s p e c i f i c a l l y f o r the MISSEE sys-tem. I t i s used to denote s p a t i a l congruency i n a p a r t i c u l a r scene. Thus i f i t i s found that r i v e r - 4 flows under bridge-2, then r i v e r - 4 w i l l i n c l u d e a NEIGHBOUR LINK to bridge-2 and v i c e v e r s a . T h i s f a c t i s used to determine which schemata w i l l be p a r t of higher l e v e l i n s t a n c e s i n the s p e c i a l i z a t i o n and decom-p o s i t i o n h i e r a r c h i e s . That i s , r i v e r - 4 and bridge-2 w i l l be p a r t of the same r i v e r system ins t a n c e ( f o r example, r i v e r -system- 1 ) . The s p e c i a l i z a t i o n and decomposition r e l a t i o n s form h i e r a r -c h i e s that are present i n both ge n e r a l and p a r t i c u l a r knowledge. H i e r a r c h i e s d e f i n e an up-down p o s i t i o n i n g of e n t i t i e s . The i n s t a n c e and neighbour r e l a t i o n s do not form h i e r a r c h i e s and only r e l a t e i n s t a n c e s ( p a r t i c u l a r knowledge). They can be thought of as d e f i n i n g the l e f t - r i g h t p o s i t i o n i n g of schemata. Thus the i n i t i a l MISSEE system c o n t a i n s generic o b j e c t s formed i n t o s p e c i a l i z a t i o n and decomposition h i e r a r c h i e s . Dur-Chapter 5. The Implementation 110 ing i n s t a n t i a t i o n , those schemata s p e c i f i c to the image being i n t e r p r e t e d are represented i n a p a r a l l e l network of i n s t a n c e s and r e l a t e d by AIO LINKS to the g e n e r a l o b j e c t s . A d d i t i o n a l l y , t h i s semantic network i s o r g a n i z e d by s p e c i a l i z a t i o n and decom-p o s i t i o n and uses the neighbour r e l a t i o n to determine which i n s t a n c e s should be grouped as p a r t s or kinds of higher l e v e l e n t i t i e s . CONFIDENCE values are used i n a very s i m p l i s t i c manner in MISSEE, again not t a k i n g f u l l advantage of the MAIDS c a p a b i l i -t i e s . The r e s u l t s of i n t e r p r e t a t i o n cause some minor v a r i a t i o n s in the r e l i a b i l i t y of i n s t a n t i a t e d schemata, but the b a s i c pre-cedence i s as f o l l o w s : schemata i n s t a n t i a t e d with the a i d of the sketch map are given more credence than those i n s t a n t i a t e d from the i n t e n s i t y image alone. And, higher l e v e l schemata (that i s , higher i n e i t h e r the decomposition or s p e c i a l i z a t i o n h i e r a r -c h i e s ) are given higher c o n f i d e n c e than schemata lower down. Information p r o v i d e d by the user i s g e n e r a l l y accompanied by a c o n f i d e n c e v a l u e . T h i s o r d e r i n g i s done to ensure that the sys-tem w i l l e x h i b i t g r a c e f u l d e g r a d a t i o n . Procedures are attached to s l o t s , the CONFIDENCE v a l u e s , and the schemata as a whole. In MISSEE, schemata have two types of a t t a c h e d procedures: top-down--to i n t e r p r e t messages r e c e i v e d from above; and, b o t t o m - u p — e i t h e r to respond to a message from below or to i n t e r a c t d i r e c t l y with an IS. There are two types of bottom-up procedures f o r h a n d l i n g input data: those concerned with the i n t e n s i t y image alone and those that can use the Chapter 5. The Implementation 111 r e s u l t s of sketch map a n a l y s i s to guide i n t e r p r e t a t i o n . A t h i r d p o s s i b i l i t y that was not implemented i n MISSEE (due to a l a c k of i n t e r a c t i v e g r a p h i c s c a p a b i l i t i e s ) i s a set of procedures that i n t e r a c t d i r e c t l y with the user at t h i s l e v e l . As w i l l be seen in S e c t i o n s 5.3 and 5.6, the user i n t e r f a c e occurs elsewhere. 5.3_« The Input Information Sources The goal of MISSEE i s to i n t e r p r e t d i g i t i z e d , monochromatic a e r i a l photographs of small urban scenes. Thus the primary IS i s an i n t e n s i t y image d e r i v e d from a photograph, such as F i g u r e 5.2 of A s h c r o f t , B r i t i s h Columbia. T h i s photograph i s a p o r t i o n of F i g u r e 1.1. Two other input IS's are a v a i l a b l e to h e l p with the i n t e r p r e t a t i o n of the i n t e n s i t y image: a sketch map and d i r e c t guidance from the user. T h i s s e c t i o n d e s c r i b e s the i n f o r m a t i o n s p e c i f i c to the three IS's. 5_._3._1_. The D i g i t i z e d Image Most work on segmenting images has c o n c e n t r a t e d on e i t h e r edge d e t e c t i o n or region merging. While they may be c o n s i d e r e d dual r e p r e s e n t a t i o n s (edges are based on i n t e n s i t y d i s c o n t i n u i -t i e s whereas regions are based on the homogeneity of i n t e n s i -t i e s ) , each emphasizes d i f f e r e n t aspects of the image. Edges p e r t a i n to the l o c a t i o n and o r i e n t a t i o n of s u r f a c e boundaries while regions are more r e l e v a n t to s u r f a c e shape and connec-t i v i t y . In accordance with the hypothesis of t h i s d i s s e r t a t i o n (the more d i f f e r e n t , the b e t t e r ) , both types of i n f o r m a t i o n from Chapter 5. The Implementation 112 F i g u r e 5.2 A s h c r o f t , B r i t i s h Columbia Chapter 5. The Implementation 1 1 3 the d i g i t i z e d image are made a v a i I a b l e [ 1 ] . 5 . 3_._1_._l_. Edge D e t e c t i o n A M a r r - H i l d r e t h edge d e t e c t o r was implemented that f i n d s z e r o - c r o s s i n g s of the L a p l a c i a n of the Gaussian of the i n t e n s i t y image (Marr and H i l d r e t h , 1980; H i l d r e t h , 1980). The p a r t i c u -l a r s of the implementation are d e s c r i b e d in Appendix B. The end r e s u l t of the edge d e t e c t i o n process i s a " l i n e image" that corresponds to the i n t e n s i t y image except that i n s t e a d of i n t e n s i t i e s , each p i x e l c o n t a i n s a value s p e c i f y i n g whether or not i t i s an edge p o i n t . F i g u r e s 5.3b and 5.3c show the edge segments o v e r l a i d on the image of A s h c r o f t f o r o values of 1.1 and 2.2 r e s p e c t i v e l y . o i s r e l a t e d to the amount of d e t a i l recovered by the edge d e t e c t o r : the smaller the a v a l u e , the g r e a t e r the r e s o l u t i o n . F u r t h e r p r o c e s s i n g i s c a r r i e d out on each edge segment to generate a v e r s i o n of the raw p r i m a l sketch (Marr and H i l d r e t h , 1980). For each edge segment, i n f o r m a t i o n i s s t o r e d about i t s l o c a t i o n , l e n g t h , and o r i e n t a t i o n . A f i n a l v a l u e , which we s h a l l c a l l c o n t r a s t , i n d i c a t e s how l a r g e the d i s c o n t i n u i t y i n i n t e n s i t i e s on e i t h e r s i d e of the edge i s . I t i s a f i r s t d e r i v a t i v e taken from the o r i g i n a l image. L1]See (Nevatia and P r i c e , 1978 and 1982) and (Nazif and Levine, 1982) f o r other systems that combine edge and region i n -formation. Chapter 5. The Implementation 114 F i g u r e 5.3 A s h c r o f t : Edge D e t e c t i o n a. (upper l e f t ) A s h c r o f t b. (lower l e f t ) edges from o = 1.1 c. (lower r i g h t ) edges from a = 2.2 Chapter 5. The Implementation 1 15 The z e r o - c r o s s i n g edge d e t e c t o r responds to a l l edges i n an image r e g a r d l e s s of t h e i r s i g n i f i c a n c e . C o n t r a s t and l e n g t h v a l u e s . ( a l o n g with model knowledge) are u s e f u l i n o r d e r i n g the edge segments. F i g u r e s 5.4 and 5.5 again show edge segments d e r i v e d from A s h c r o f t using d i f f e r e n t a v a l u e s . Each s u b f i g u r e shows the edges ranked by a d i f f e r e n t c r i t e r i o n . The order i s i n d i c a t e d by the c o l o u r of the segment: b r i g h t blue to dark blue i s the highest rank, f o l l o w e d by b r i g h t to dark red. It i s e v i -dent that no s i n g l e c r i t e r i o n ranks a l l the " s i g n i f i c a n t " edges at the top. Model knowledge can be used, however, to determine the proper c r i t e r i a f o r i n d i v i d u a l o b j e c t s . 5 _ . _ 3 . R e g i o n Merging Region merging i s accomplished by a method d e s c r i b e d by Freuder c a l l e d a f f i n i t y (Freuder, 1976). B a s i c a l l y i t merges reg i o n s whose average i n t e n s i t i e s are most s i m i l a r . D e t a i l s can be found i n Appendix C. Although q u i t e a simple a l g o r i t h m , i t produces r e s u l t s reasonable i n appearance. I t s most prominent f a u l t i s that no concern i s p a i d to the shape or l e n g t h of com-mon boundaries between regions which leads to some " i n c o r r e c t " merges (see B r i c e and Fennema, 1970). Segmentation based on simple, u n r e l i a b l e techniques was used because 1) even the best segmentation techniques render l e s s than p e r f e c t r e s u l t s , 2) i t shows how c o o p e r a t i v e i n t e r p r e t a t i o n i s powerful enough to over-come the d e f i c i e n c i e s of a simple segmentation technique, and 3) i t i s easy to implement. Chapter 5. The Implementation 1 16 F i g u r e 5.4 Ordering Edges in A s h c r o f t , a = 1.1 a. (upper r i g h t ) ordered by l e n g t h b. (lower l e f t ) ordered by c o n t r a s t c. (lower r i g h t ) ordered by c o n t r a s t * l e n g t h Chapter 5. The Implementation 117 H I L L EC s 7 HST LENGTH F i g u r e 5.5 Ordering Edges in A s h c r o f t , a = 2.2 a. (upper r i g h t ) ordered by l e n g t h b. (lower l e f t ) ordered by c o n t r a s t c. (lower r i g h t ) ordered by c o n t r a s t * l e n g t h Chapter 5. The Implementation 1 18 F i g u r e 5.6 shows the r e s u l t s of region merging on the Ash-c r o f t image stopped when there are 172 and 75 regions l e f t . The process can e i t h e r be stopped by the user or by the a l g o r i t h m when there are no p o t e n t i a l merges l e f t . In t h i s case, i t was stopped by the user. The r e s u l t of t h i s process i s a "region image" where each p i x e l i n d i c a t e s to which r e g i o n i t belongs. As with edges, p o s t - p r o c e s s i n g i s c a r r i e d out to d e r i v e p r i m a l s k e t c h - l i k e i n f o r m a t i o n f o r each r e g i o n . The l o c a t i o n of a perimeter p o i n t i s s t o r e d , along with the l e n g t h of the perim-e t e r and the area of the r e g i o n . The average i n t e n s i t y of a l l the p i x e l s i n the region i s c a l c u l a t e d as part of the a f f i n i t y measure. F i n a l l y , the l i s t of a l l neighbouring regions i s made e x p l i c i t . Both edge d e t e c t i o n and region merging are l o c a l o p e r a t i o n s that can be c a r r i e d out on subimages of any s i z e . Both are f a i r l y expensive c o m p u t a t i o n a l l y . The former i n v o l v e s convolu-t i o n and the l a t t e r many i t e r a t i o n s : one f o r each stage of merg-in g . If model knowledge can be u t i l i z e d e f f e c t i v e l y , i t may be p o s s i b l e to l i m i t the p a r t s of the image where edge d e t e c t i o n and/or region merging must be performed. For example, i f i t i s known or d i s c o v e r e d that a l a r g e p o r t i o n of a p a r t i c u l a r image c o n t a i n s no f e a t u r e s of i n t e r e s t , then i t can be ignored. Or, small s e c t i o n s of the image may be searched i n order of l i k e l i -hood f o r an o b j e c t of i n t e r e s t and when i t i s found the search could terminate. The only new (but not n e c e s s a r i l y major) prob-lem t h i s s t r a t e g y e n t a i l s i s how to "put together" the Chapter 5. The Implementation 119 F i g u r e 5.6 A s h c r o f t : Region Merging a. (upper l e f t ) A s h c r o f t b. (lower l e f t ) 172 r e g i o n s c. (lower r i g h t ) 75 r e g i o n s Chapter 5. The Implementation 120 i n f o r m a t i o n of two contiguous subimages at t h e i r mutual boun-dary. T h i s methodology can i n c r e a s e the e f f i c i e n c y of the o v e r a l l i n t e r p r e t a t i o n g r e a t l y i f only a small p o r t i o n of the whole image needs to be examined. On the other hand, i t would not i n c r e a s e the problem s o l v i n g power of the system. I t has not been u t i l i z e d i n the c u r r e n t implementation because of the d i f -f i c u l t y i n c o o r d i n a t i n g the v a r i o u s programs and data. For example, the r e g i o n merging process i s t i e d to a s p e c i f i c output d e v i c e (so the user can decide when to stop the merging) that was not connected to the computer used to run MISSEE. 5. 2 A C a t e g o r i z a t i o n f o r the Regions A crude c l a s s i f i c a t i o n of the regions can be made to pro-v i d e a d d i t i o n a l i n f o r m a t i o n to the system. T h i s c l a s s i f i c a t i o n can be c o n s i d e r e d another type of user input although i t would be p r e f e r a b l e to use an automatic method such as was d e s c r i b e d i n S e c t i o n 4.7. F i g u r e 5.7 shows one such c a t e g o r i z a t i o n f o r the 75 r e g i o n s d i s c o v e r e d f o r the A s h c r o f t image. Th i s c l a s s i f i c a t i o n i s very crude because i t has only four c a t e g o r i e s and two of them are ambiguous ( u r b a n / h i l l s and road/mountain). Furthermore, i t has i n a c c u r a c i e s due to some i n c o r r e c t l y l a b e l l e d r e g i ons (e.g. some urban regions have been l a b e l l e d "water" and v i c e v e r s a ) . Thus, the c a t e g o r i z a t i o n does not solve the problem of i n t e r p r e t a t i o n ; i t i s only another type of a v a i l a b l e i n f o r m a t i o n — t h e "INTERPRET" box i n F i g u r e 5.1. Chapter 5. The Implementation 121 F i g u r e 5.7 A s h c r o f t : Region C l a s s i f i c a t i o n a. (upper l e f t ) A s h c r o f t b. (lower l e f t ) 75 regions c. (lower r i g h t ) c l a s s i f i e d r e g i o ns Chapter 5. The Implementation 1 22 The c l a s s i f i c a t i o n i s based s o l e l y on the average i n t e n s i -t i e s of the r e g i o n s . There are s e v e r a l ways that the d e c i s i o n boundaries can be d e r i v e d : 1) v a l u e s that were found to work f o r a d i f f e r e n t image can be used, 2) the boundaries can be moved on an image d i s p l a y (with f u n c t i o n mapping) u n t i l one "sees" where a reasonable d i v i s i o n p o i n t should be, or, 3) as was a c t u a l l y done, a t r a i n i n g set of regions and t h e i r c a t e g o r i e s can be used to f i x the boundaries f o r a l l the r e g i o n s . Even though i t does not e x a c t l y f i t the d e f i n i t i o n i n Sec-t i o n 4.5.2, the c l a s s i f i c a t i o n shown i n F i g u r e 5.7 corresponds ( f o r each category) to the c o n s e r v a t i v e range [ a , b ] . The mismatch from the d e f i n i t i o n a r i s e s because there w i l l be some regions i n Cn having an average i n t e n s i t y in the range, such as the urban ' regions c l a s s i f i e d as water. Nonetheless, s i n c e the schemata take the crudeness of the i n t e r p r e t a t i o n i n t o account, and r e q u i r e other s u p p o r t i v e evidence f o r i n s t a n t i a t i o n , i t i s a c c e p t a b l e . In MISSEE, a second s e r i e s of o v e r l a p p i n g ranges i s used, c o r r e s p o n d i n g to [ a * , b ' ] , that ensures that a l l r egions w i l l be i n c l u d e d i n the proper category range. The i n t e r v a l of acceptance r e l i e s on a two s t e p f u n c t i o n , such t h a t with no a p r i o r i i n f o r m a t i o n the r e s t r i c t e d range, [ a , b ] , i s used, while with some reasonable e x p e c t a t i o n f o r the c a t e g o r i z a t i o n , the l a r g e r range, [ a ' , b ' ] , i s employed. Even though the c l a s s i f i c a t i o n of regions i s crude and i n a c c u r a t e , i t represents a r e a l i s t i c r e s u l t that might come from any automatic c l a s s i f i e r . S t a t i s t i c a l p a t t e r n r e c o g n i t i o n Chapter 5. The Implementation 123 techniques are i n s u f f i c i e n t f o r i n t e r p r e t i n g complicated images. More knowledge i s r e q u i r e d . 5.3.2. The Sketch Map Sketch maps are analyzed by Mapsee2 (Havens and Mackworth, 1980), which l i k e MAIDS i s w r i t t e n i n Maya. Input i s a s e r i e s of moves/draws (or p l o t s / g o t o s ) which t r a c e the curves of the ske t c h . Output i s a decomposition h i e r a r c h y of i n s t a n c e s of geographic o b j e c t s ranging from p o i n t s , l i n e s , and chains to roads, r i v e r s , mountains, mountain ranges, road systems, geosys-tems, and the world. S p e c i f i c s can be found i n Appendix D. F i g u r e 5.8 shows a sketch map of the A s h c r o f t area. I t i s a simple task to draw t h i s over an image d i s p l a y e d on a s u i t a b l e d e v i c e ( i n t h i s case a Comtal V i s i o n I ) . F i g u r e 5.9 shows t h i s combination. The p o i n t s i n the sketch map can then be grouped i n t o l i n e s and made a v a i l a b l e to Mapsee2. Mapsee2 r e s u l t s that are made a v a i l a b l e to MISSEE are a number of Maya schemata ( c f . S e c t i o n 3.3). P o i n t s i n the sketch curves are j o i n e d i n t o l i n k s and then c h a i n s . Chains are gen-e r a l i z e d (Appendix B) i n t o a t r e e of l i n e s r e p r e s e n t i n g a b i n a r y breakdown of the curves i n t o l i n e a r segments. Chains are g e n e r a l l y ambiguous cues f o r o b j e c t s such as roads, r i v e r s , and mountains[2]. The shapes of the chains and [2]The exception are c l o s e d c h a i n s which are unambiguous cues f o r shores and blobs which map to towns. Chapter 5. The Implementation A' 2 A , 3 /\,4 F i g u r e 5.8 A Sketch Map of A s h c r o f t t h e i r s p a t i a l r e l a t i o n s h i p s are used to b u i l d aggregate s t r u c -t u r e s , such as road systems, r i v e r systems, mountain ranges, and the l i k e and to remove some of the l a b e l s f o r the c h a i n s . A f t e r a l l the c h a i n s have been examined, though, some of the l a b e l s w i l l remain ambiguous, which r e s u l t s i n a p o s s i b l y l a r g e number Chapter 5. The Implementation 125 F i g u r e 5.9 A s h c r o f t with a Superimposed Sketch Map Chapter 5. The Implementation 1 26 of h y p o t h e t i c a l i n s t a n c e s . Table 5.1 shows a l l the l a b e l s f o r the cha i n s i n the Ash-c r o f t sketch map. The cha i n numbers correspond to the values i n F i g u r e 5.8. The " i n t e r p r e t a t i o n " column l i s t s the intended i n t e r p r e t a t i o n f i r s t . These i n s t a n c e s of geographic o b j e c t s along with the corresponding c h a i n s , l i n e s , and p o i n t s which s t o r e t h e i r l o c a t i o n s are the major source of sketch map i n f o r -mation used by MISSEE. The i n s t a n c e s are embedded i n a decomposition h i e r a r c h y that groups r e l a t e d o b j e c t s . F i g u r e 5.10 shows the h i e r a r c h y b u i l t from the "intended" i n t e r p r e t a t i o n s only. Higher l e v e l model knowledge about s p a t i a l c o n n e c t i v i t y c o u l d be used by MIS-SEE. U n f o r t u n a t e l y , when MISSEE was developed, the then c u r r e n t v e r s i o n of Mapsee2 d i d not c o l l e c t enough of t h i s i n f o r m a t i o n . HAIN LINKS LINES INTERPRETATIONS 1 1 - 31 1 - 73 *road-6 * r i v e r - 6 2 32 - 56 74 1 46 *road-4 * r i v e r - 4 3 57 - 101 147 239 *bridge-1 ( f i r s t s i d e ) 4 102 - 1 04 240 244 *town-1 5 105 - 1 52 245 339 *bridge-1 (second side) 6 153 - 292 340 648 *road-5 * r i v e r - 5 7 293 - 313 649 701 *road-7 * r i v e r - 7 8 314 - 332 702 752 *road-8 * r i v e r - 8 9 333 - 406 753 899 * r i v e r - 1 *road-1 1 0 407 - 41 1 900 908 *road-3 * r i v e r - 3 1 1 412 - 439 909 991 * r i v e r - 2 *road-2 1 2 440 - 475 992 1 060 *mountain-1 *road-9 * r i v e r - 9 1 3 476 - 508 1 061 1 1 25 *mountain-2 *r o a d - l 0 * r i v e r - l 0 1 4 509 - 530 1 126 1 168 *mountain-3 *road-11 * r i v e r - 1 1 Table 5.1 I n t e r p r e t a t i o n s of the A s h c r o f t Chains Chapter 5. The Implementation 127 Figure 5 .10 A Mapsee2 Decomposition Hierarchy of Ashcroft Sketch Map Instances Chapter 5. The Implementation 128 In p a r t i c u l a r , the connection between two roads or a road and a town were d i s c o v e r e d only i f the end of the road c h a i n i s near the other c h a i n . A l s o , i f a road went over a b r i d g e or a r i v e r went under a b r i d g e , that f a c t was not recorded. Use of t h i s type of i n f o r m a t i o n would become very important i f one were to r e l a x the c o n d i t i o n that the sketch map be r e g i s t e r e d to the image. Maya schemata are l e s s s t r u c t u r e d than MAIDS schemata. However, MAIDS schemata are upwards compatible with those used by Mapsee2. The schema-specific r o u t i n e s that use i n f o r m a t i o n from sketch maps use only the p r i m i t i v e form of the r e t r i e v a l funct i o n s . D i g i t i z e d images c o n t a i n i n f o r m a t i o n that i s d i f f e r e n t i n kind from that i n sketch maps. Some in f o r m a t i o n d e r i v e d from images can be u s e f u l i n i n t e r p r e t i n g sketch maps. For i n s t a n c e , a c l o s e d curve i n a sketch map can be unambiguously l a b e l l e d a shore. However, t h i s g i v e s no i n d i c a t i o n whether i t i s a lake (water i n s i d e ) or an i s l a n d (water o u t s i d e ) . If the l o c a t i o n of the s h o r e l i n e can guide the search f o r the a p p r o p r i a t e f e a t u r e s i n the i n t e n s i t y image, then a simple p i x e l c l a s s i f i c a t i o n scheme, as was d e s c r i b e d i n the l a s t s e c t i o n , w i l l g e n e r a l l y s u f f i c e to determine which region corresponds to water and which to l a n d . While Mapsee2 and MISSEE do not cooperate i n t h i s way at t h i s time, the p o t e n t i a l e x i s t s f o r the i n t e r p r e t a t i o n of each knowledge source to provide u s e f u l i n f o r m a t i o n to a i d the i n t e r p r e t a t i o n of the other source. Chapter 5. The Implementation 129 5.3.3. The User The user can i n f l u e n c e s e v e r a l aspects of the i n t e r p r e t a -t i o n p r o c e s s . He/She can p r o v i d e c e r t a i n types of i n f o r m a t i o n to the system and can determine when and how i t i s to be used. In a d d i t i o n , he/she can modify the i n t e r a c t i v e environment to allow d i f f e r i n g amounts of i n f o r m a t i o n to be d i s p l a y e d . The user's means of p r o v i d i n g i n f o r m a t i o n and i n f l u e n c i n g the i n t e r p r e t a t i o n i s v i a the p r i o r i t y queue used i n the c y c l e of p e r c e p t i o n . The exact means by which t h i s i s accomplished i s d e s c r i b e d i n S e c t i o n 5.6. U n f o r t u n a t e l y , the i n t e r f a c e i s r a t h e r " u n n a t u r a l " i n t h a t the user must know the format of i n t e r n a l messages and, while graphic output i s used, there i s no means for r e c e i v i n g graphic i n p u t - - o t h e r than the sketch map. T h i s i s due to the l a c k of i n t e r a c t i v e hardware on our c u r r e n t system. S e c t i o n 4.4.1 d e s c r i b e d the t h r e e types of input a v a i l a b l e to a user. G l o b a l input p e r t a i n s to the image as a whole and i n c l u d e s the s e t t i n g of g l o b a l v a l u e s such as the s c a l e of the image ( c f . S e c t i o n 4.6). In a d d i t i o n , g l o b a l v a r i a b l e s are used to determine the output a v a i l a b l e to the user. Depending on what hardware i s a c c e s s i b l e , the user can d i r e c t images that i l l u s t r a t e the i n s t a n t i a t i o n p rocess to the d e s i r e d d e v i c e . T h i s f e a t u r e i s represented by the "images" c l o u d i n F i g u r e 5.1. A l s o , a l l the output statements t h a t are d i r e c t e d to the p r o t o -c o l c o n t a i n a number which must be l e s s than a g l o b a l p r i n t l e v e l Chapter 5. The Implementation 130 v a r i a b l e before they are a c t u a l l y p r i n t e d . Hence, the output can be very s e l e c t i v e or very complete depending on the needs of the user. In MISSEE, the g l o b a l i n f o r m a t i o n , "there i s an X i n the image" i s e q u i v a l e n t to the p r i o r i t i e s input statement, " f i n d an X i n the image". General requests to f i n d s p e c i f i c o b j e c t s , such as a road or a r i v e r system, can be made by adding the a p p r o p r i a t e message to the p r i o r i t y queue. In a d d i t i o n , one can change the p r i o r i t i e s of the i n t e r p r e t a t i o n by r e a r r a n g i n g the elements i n the queue. By a l t e r i n g the parameters of the mes-sage, one can change the context f o r i n t e r p r e t a t i o n and i n t r o -duce more l o c a l i n f o r m a t i o n to the system. L o c a l input i s concerned with what s p e c i f i c f e a t u r e s the user sees in the image and where they are. Along with an o b j e c t (e.g. road), the context of a c o r r e s p o n d i n g item i n the sketch map (e.g. *road~7) p r o v i d e s u s e f u l i n f o r m a t i o n . A d d i t i o n a l l y , the l o c a t i o n of the o b j e c t can be s p e c i f i e d as e i t h e r a region in the i n t e n s i t y image or a p o s i t i o n . While t h i s g i v e s the user the a b i l i t y to s p e c i f y e v e r y t h i n g that i s important i n the image, but s t i l l a l l o w i n g MISSEE to do u s e f u l bookkeeping chores such as c a l c u l a t i n g the l e n g t h of a r i v e r , i t should be r e i t e r a t e d t h a t , i n a robust system, i t i s not necessary. 5.4. I n s t a n t i a t i n g Schemata MISSEE i n t e r p r e t s images of small urban areas. F i g u r e 5.11 shows the models used i n t h e i r decomposition and s p e c i a l i z a t i o n Chapter 5. The Implementation 131 h i e r a r c h i e s . While t h i s i s only a small p o r t i o n of the domain e x h i b i t e d i n F i g u r e 4.4, i t i s s u f f i c i e n t to c l a s s i f y the scenes that w i l l be used. T h i s s e c t i o n d e s c r i b e s how MISSEE i n s t a n -t i a t e s the schemata i n t h i s graph from the i n f o r m a t i o n found i n p a r t i c u l a r images and sketch maps. F i g u r e 5.11 The Generic Objects i n MISSEE Chapter 5. The Implementation 1 32 5.4_.j_. I n s t a n t i a t i o n D i r e c t l y from the Information Sources The f i v e "bottom l e v e l " schemata—town, mountain, road, b r i d g e , and river--must d i r e c t l y i n t e r a c t with one or two IS's when they are i n s t a n t i a t e d bottom-up[3]. T h i s i s g e n e r a l l y where the i n t e r f a c e occurs when p r o c e s s i n g f i r s t s t a r t s . The p a r t i c u l a r o b j e c t that i s f i r s t s e l e c t e d may be chosen by the user or from general model knowledge. For example, the landmass schema "knows" that road systems are e a s i e r to f i n d than moun-t a i n ranges. The l o c a t i o n i n the image may again be s p e c i f i e d by the user or by model knowledge. For example, the l a r g e s t r e g ion i n the image can be examined f i r s t . F i g u r e 5.12 i s the generic MISSEE schema f o r a br i d g e . I n s t a n t i a t i o n of t h i s schema i n v o l v e s f i l l i n g i n i t s s l o t s by f i n d i n g the a p p r o p r i a t e f e a t u r e s i n the IS's. There are two bottom-up procedures a t t a c h e d to b r i d g e : one that r e q u i r e s a sketch map r e g i s t e r e d to the image and one that d e a l s only with the d i g i t i z e d image. The f o l l o w i n g two su b s e c t i o n s i l l u s t r a t e ( i n pseudo-Lisp) how these procedures work (Glicksman, 1982). _5._4.J_._j_. I nstant i a t ing %br idge-1 with the A i d of a Sketch Map T h i s s e c t i o n c o n s i s t s of b r i e f e x c e r p t s of "pseudo-code" (preceded by "|") and e x p l a n a t i o n s of what i s accomplished. Responses from the program are indented and preceded by ">". [3]Ridges and curbs are c r e a t e d as part of t h i s process, they are not i n s t a n t i a t e d independently. Chapter 5. The Implementation 1 33 sketchmapi tem: v a l u e : n i l % c o n f i d e n c e : n i l o r d e r l i s t : neighbourregions: shadowregions: ro a d r e g i o n s : a-part-of—> decomposes-to—> instances—> neighbours—> % i f - a d d e d : (prog n i l ( p r i n t l b "value added to sketch map item = " % v a l ) ) %if-removed: (prog n i l ( p r i n t l b "value removed from sketch map item = " % v a l ) ) % i f - m o d i f i e d : (prog n i l ( p r i n t l b "value m o d i f i e d in sketch map item = " % v a l ) ) value: n i l v a l u e : n i l v a l u e : n i l value: n i l % c o n f i d e n c e : n i l % c o n f i d e n c e : n i l % c o n f i d e n c e : n i l % c o n f i d e n c e : n i l (%road-system (%curb) n i l n i l ;r iver-system) c o n f i d e n c e : n i l c o n f - a l g : (prog ( v a l 1st) (setq v a l (cond ((sgetv %name 'sketchmapitem 'n) 50.0) (t 25.0))) (se t q 1st (sgeta %name 'decomposes-to)) (cond ( ( n u l l 1 s t ) ( r e t u r n v a l ) ) ((atom 1 s t ) ( r e t u r n (plus v a l (times 0.2 (sgetc 1 s t ) ) ) ) ) (t ( r e t u r n (plus v a l (times 0.2 (quotient (apply 'plus (mapear '(lambda (n) (sgetc n)) 1st ) ) ( l e n g t h 1 s t ) ) ) ) ) ) ) ) F i g u r e 5.12 The Stereotype % b r i d g e Schema Schemata manipulation f u n c t i o n s w i l l be u n d e r l i n e d . I f *bridge-1 has been s e l e c t e d from the sketch map a n a l y s i s r e s u l t s ( c f . F i g u r e 3.6) to a i d i n the i n s t a n t i a t i o n of a b r i d g e , then Chapter 5. The Implementation 134 the v a r i a b l e SKETCHMAPITEM w i l l be bound to i t when the pro-cedure i s entered. The b a s i s of the r o u t i n e i s the f o l l o w i n g : The r o u t i n e searches f o r regions i n the a p p r o p r i a t e area that can be i n t e r p r e t e d as road or shadow. Then edges are found i n the same area which can d e f i n e the s i d e s of the bridge and p o s s i b l y a shadow. 1. i f (SKETCHMAPSCHEMA = (sgetv '%bridge 'sketchmapitem 'yes 'one ' ( i n s t a n c e s ) ) ( r e t u r n ' a l r e a d y e x i s t s ) ) T h i s i s a check to determine whether t h i s schema has been exam-ined b e f o r e . Sgetv searches from %bridge down, the " i n s t a n c e s " LINKs l o o k i n g f o r one in s t a n c e schema c o n t a i n i n g SKETCH-MAPSCHEMA as i t s sketch map item. 2. | Inst = (snewi '%bridge) > c r e a t e a new brid g e i n s t a n c e : %bridge-1 p1 = (car (sgeta SKETCHMAPSCHEMA 'side1-desc)) p2 = (car (sgeta SKETCHMAPSCHEMA 'side2-desc)) > p i = (57 . 33) > p2 = (69 . 62) Since Mapsee2 schemata are u n d i f f e r e n t i a t e d , sgeta r e t u r n s the value of s l o t s . p1 and p2 are the l o c a t i o n s of the mid-p o i n t s of the bridge s i d e s . 3. | r e g l i s t = ( p o i n t s t r i p s p1 p2 % r e g m a t f i l e (quotient 15.0 (sgetv ' % s c a l e ' f e e t p e r p i x e l 'no ' v a l u e o r d e f a u l t 'one))) > regions l i s t = (130 1186 1629 9 1750 2018) P o i n t s t r i p s searches f o r a l l the re g i o n s (generated from a region-merging alg o r i t h m ) in % r e g m a t f i l e that are enclosed i n a r e c t a n g u l a r s t r i p whose co r n e r s are 15 f e e t Chapter 5. The Implementation 1 35 from p1 and p2. 4. o r d e r l i s t = ( o r d i n t e r p r e g l i s t Inst) > order l i s t = (Other Shadow Bridge Bridge Other) O r d e r i n t e r p i n t e r p r e t s the regions and determines i f t h e i r i n t e r p r e t a t i o n s are c o n s i s t e n t with regions around and over a b r i d g e . I n t e r p r e t a t i o n s i n c l u d e ROAD, WATER, URBAN, SHADOW, MOUNTAIN, and HILLS. The a p p r o p r i a t e regions are added to the "%bridge-1" schema in the s l o t s "shadowregions", "roadre-gions", and "neighbourregions". The order l i s t shows the order of r e g i o n s from p1 to p2. In t h i s case they are Other, Shadow, and B r i d g e . 5. | (sputv Inst ' o r d e r l i s t o r d e r l i s t ) Put the value of o r d e r l i s t i n the VALUEd type of the same name in I n s t . 6. | (sputc Inst ' o r d e r l i s t 100) Put the c o n f i d e n c e of o r d e r l i s t at 100 (the maximum). 7. | e d g e l i s t = ( p o i n t s t r i p s p1 p2 %edgematfile ( q u o t i e n t 15.0 (sgetv ' % s c a l e ' f e e t p e r p i x e l 'no ' v a l u e o r d e f a u l t 'one))) > edges l i s t = (88 11 205 206 241 242) F i n d a l l the edge segments in the same r e c t a n g u l a r s t r i p that was searched f o r r e g i o n s . Edge segments come from groups of z e r o - c r o s s i n g s of the L a p l a c i a n of the Gaussian a p p l i e d to the image. 8. i f ( n u l l e d g l i s t ) then ( s f a i l - m o d e l I n s t ) If there are no edge segments then t h i s model can not be c o r r e c t . So, remove i t and any of i t s descendants from the graph. Chapter 5. The Implementation 136 9. i f (notwithinrange edgeangle b r i d g e o r i e n t a t i o n ) then (remove edge e d g l i s t ) > edges l i s t = (88 11 242) Remove a l l edge segments that are not s i m i l a r to the o r i e n t a -t i o n of the bridge i n the sketch map, which i s 67 degrees. 10. f o r a l l i i n e d g e l i s t (addto e d g e l i s t ( l i n e n e i g h b o u r s i -1.0 35 % e d g e f i l e ) ) > edges l i s t = ((88 89)(10 11 12 13 14 15) (242 243)) Expand a l l the edge segments i n e d g e l i s t to i n c l u d e seg-ments out of the r e c t a n g u l a r s t r i p that are connected to and i n a s i m i l a r o r i e n t a t i o n as e x i s t i n g edge segments. 11. ( f o r e a c h group of edges i i n e d g e l i s t ( l i n e s t a t s i ) ) L i n e s t a t s c a l c u l a t e s the l e n g t h of the edge segments, the average c o n t r a s t a c r o s s the edge, and the maximum con-t r a s t . 12. | (matchregionstoedges o r d e r l i s t e d g e l i s t I n s t ) Match the re g i o n s from r e g i o n l i s t with the edges i n e d g e l i s t . One r e s u l t of t h i s i s the c r e a t i o n of new in s t a n c e s of the %curb schema. Each %curb schema c o n t a i n s the data f o r one edge of the b r i d g e . A s p e c i a l type of %curb i s reserved f o r shadows. A f t e r t h i s r o u t i n e has executed, the schemata shown i n F i g u r e 5.13 w i l l e x i s t . The conf i d e n c e a l g o r i t h m i s executed a f t e r the l a s t value s l o t i s f i l l e d by a c a l l to sputc with the a p p r o p r i a t e f l a g s e t . 13. | r i v e r r e g = (sgetv ' % r i v e r 'regions 'yes 'valueonly ' a l l ' ( i n s t a n c e s ) ) > r i v e r r e g = ( ( % r i v e r - 1 130 1058 943 874) (%riv e r - 2 1750 2018 2095)) f o r a l l r e g i o n s r in neighbourregions i f r i s i n r i v e r r e g then (saddl Inst 'neighbours R i v e r ) ) Chapter 5. The Implementation 1 37 avgstr e n g t h : v a l u e : 3.5 %conf idence 75 maxstrength: v a l u e : 5 %conf idence 75 l e n g t h : v a l u e : 20 %conf idence 1 00 angle s : v a l u e : (248 214) %conf idence (80) edgesegs: v a l u e : (88 89) %conf idence 1 00 type: v a l u e : shadow %conf idence 1 00 a-part-of-> %bridge-1 c o n f - a l g : (prog n i l ( r e t u r n (quot i e n t (apply 'add (cons (sgetc %name 'length) (cons (sgetc %name 'maxstrength) (sgetc %name ' a n g l e s ) ) ) ) (add 2 (l e n g t h (sgetc %name ' a n g l e s ) ) ) ) ) ) c o n f i d e n c e : 85 5.13a %curb-1 ###################################### avgstre n g t h : v a l u e : 96.33 %conf idence 100 maxstrength: v a l u e : 134 %conf idence 1 00 le n g t h : v a l u e : 54 %conf idence 1 00 ang l e s : v a l u e : (90 57 79 45 90 27) %conf idence (80) edgesegs: v a l u e : (10 11 12 13 14 15) %conf idence 1 00 type: v a l u e : road %conf idence 1 00 a-part-of—> %br idge-1 conf idence: 93 5.13b %curb-2 ###################################### a v g s t r e n g t h : v a l u e : 18.0 %conf idence 1 00 maxstrength: v a l u e : 22 %conf idence 1 00 l e n g t h : v a l u e : 20 %conf idence 1 00 angle s : v a l u e : (228 270) %conf idence (100) edgesegs: v a l u e : (242 243) %conf idence 1 00 type: v a l u e : road %conf idence 1 00 a-part-of—> %bridge-1 c o n f i d e n c e : 93 5.13c %curb-3 F i g u r e 5.13 Three %curb Instances Chapter 5. The Implementation 1 38 e l s e (addtoqueue 'td ' % r i v e r (sgetc Inst) ( l i s t 'region r ) ) ) (adddemon ( l i s t 'demonaddlink '%bridge 'regions r Inst 'neighbours)) > neighbours = (%river-1 % r i v e r - 2 ) Search through a l l the e x i s t i n g r i v e r schemata f o r the regions they c o n t a i n . I f they match the neighbouring regions of the b r i d g e , add l i n k s to the r i v e r s from the bridge and v i c e v e r s a . If not, send a message to the % r i v e r schema to t r y to i n s t a n t i a t e i t with the a p p r o p r i a t e r e g i o n s . I f the % r i v e r schema i s subsequently i n s t a n t i a t e d , a demon w i l l e s t a b l i s h the neighbours l i n k between i t and %b r i d g e - 1 . 14. | roadreg = (sgetv '%road 'regions 'yes 'valueonly ' a l l ' ( i n s t a n c e s ) ) > roadreg = n i l f o r a l l r egions r i n roadregions i f r i s i n roadreg then (saddl Inst 'neighbours Road)) e l s e (addtoqueue 'td '%road (sgetc I n s t ) ( l i s t 'region r ) ) ) (adddemon ( l i s t 'demonaddlink '%bridge 'regions r Inst 'neighbours)) > message added to QUEUE: (td %road 68 (region 1629)) > DEMON i n i t i a t e d : demonaddlink Repeat the a c t i o n s of 13 f o r the road that passes over the b r i d g e . 15. (sputv I n s t 'sketchmapitem SKETCHMAPSCHEMA) (sputc Inst 'sketchmapitem 100 t) > value added to sketchmapitem = *bridge-1 I n s t a n t i a t i o n has succeeded so add SKETCHMAPSCHEMA as the sketch map item f o r t h i s schema. Set i t s c o n f i d e n c e to the max-imum and propagate that f a c t . F i g u r e 5.14 d i s p l a y s the schema for %bridge-1 at t h i s stage. 16. |(addtoqueue 'bu ' % r i v e r - s y s t e m (plus 5 (sgetc I n s t ) ) ( l i s t I n s t ) ) (addtoqueue 'bu '%road-system (plus 5 (sgetc I n s t ) ) Chapter 5. The Implementation 139 sketchmapitem: va l u e : *bridge-1 % c o n f i d e n c e : 100 o r d e r l i s t : v a l u e : (Other Shadow Bridge Bridge Other) % c o n f i d e n c e : 100 neighbourregions: v a l u e : (2018 1750 9 130) % c o n f i d e n c e : 100 shadowregions: va l u e : (1186) %c o n f i d e n c e : 100 roadregions: value: (1629) %c o n f i d e n c e : 100 a-part-of—> n i l decomposes-to-> (%curb-1 %curb-2 %curb-3) neighbours—> (%river-1 % r i v e r - 2 ) c o n f i d e n c e : 68 F i g u r e 5.14 Instance % b r i d g e - l ( l i s t I n s t ) ) > messages added to QUEUE: (bu % r i v e r - s y s t e m 73 ( % b r i d g e - l ) ) (bu %road-system 73 ( % b r i d g e - l ) ) Send messages to the two higher l e v e l schemata that b r i d g e s are p a r t o f . %bridge-1 w i l l e i t h e r f i n d e x i s t i n g road and r i v e r s y s -tem i n s t a n c e s to become part of, or w i l l cause them to be c r e a t e d . F i g u r e s 5.15 and 5.16 show g r a p h i c a l l y the regions and edges ( r e s p e c t i v e l y ) that were used to i n s t a n t i a t e % b r i d g e - 1 . _5._4._1_._2. I n s t a n t i a t i n g %bridge-2 from the I n t e n s i t y Image Lik e the l a s t s e c t i o n , t h i s s e c t i o n c o n t a i n s pseudo-code and e x p l a n a t i o n s of how one can i n s t a n t i a t e a bridge from the i n t e n s i t y image alone . The argument the procedure enters with, Chapter 5. The Implementation 140 F i g u r e 5.15 A s h c r o f t : %bridge-1 Regions Chapter 5. The Implementation F i g u r e 5.16 A s h c r o f t : %bridge-1 Edge Segments (Curbs) Chapter 5. The Implementation 1 42 i f i t e x i s t s , i s a number denoting a region to be searched f o r the b r i d g e . The r o u t i n e f i n d s regions that can have the i n t e r p r e t a t i o n WATER surrounding a region with the i n t e r p r e t a t i o n ROAD. The p o s i t i o n s midway along the common boundaries are used to f i x the edges of the p o s s i b l e b r i d g e . 1.I i f #args = 1 then reg = (arg 1 ) i f (or (not (memq 'ROAD ( r e g i o n i n t e r p r e t r e g ) ) ) (regioncheck reg '%bridge)) then ( r e t u r n n i l ) e l s e go to 3. Whenever there i s an argument, i t i s a s p e c i f i c r e g ion in the image to be used. One of i t s p o s s i b l e i n t e r p r e t a t i o n s ( c f . Sec-t i o n 5.3.1.2.1) must be ROAD and i t should not have been p r e v i -o u s l y used to attempt bridge i n s t a n t i a t i o n . 2.I minsize = (quotient (times 50 50) (square (sgetv ' % s c a l e ' f e e t p e r p i x e l 'no ' v a l u e o r d e f a u l t 'one))) reg = ( n e x t l a r g e s t r e g i o n '%bridgearea minsize 'ROAD) i f (regioncheck reg '%bridge)) then go to 2. i f ( n u l l reg) then ( r e t u r n n i l ) > r e g i o n = 18 area = 1695 If there i s no region s p e c i f i e d , f i n d the l a r g e s t region in the image that can have ROAD as one of i t s i n t e r p r e t a t i o n s and has not been t r i e d b e f o r e . Stop s e a r c h i n g when the area of the regi o n i s l e s s than 2500 square f e e t . 3. neighb = (neighbourregions reg) > neighbours = (19 9 1359 738) F i n d a l l the regions that share a common boundary with reg. Chapter 5. The Implementation 1 43 f o r a l l r egions n l i n neighb i f (memq 'WATER ( r e g i o n i n t e r p r e t n l ) ) then n2 = (moveacross (commonboundary reg n l ) ) i f (memq 'WATER ( r e g i o n i n t e r p r e t n2)) then go to 5. i f (#args = 1) then ( r e t u r n n i l ) e l s e go to 2. > neighbour 1 = 1359 > no proper neighbour p a i r s f o r region 18 from 2. > region = 1629 area = 1330 from 3. > neighbours = (1186 130 9 2131 1750 2201 2293 and 13 others) from 4. > neighbourl = 130 > neighbourl = 1750 > neighbour2 = 130 shadow region = 1186 Check each region i n neighb to see i f i t can be i n t e r p r e t e d as WATER. If so, f i n d the midpoint of the common boundary between the two r e g i o n s . Then move back a c r o s s reg (1 p i x e l at a time) i n the d i r e c t i o n p e r p e n d i c u l a r to the boundary of n1 and reg. If the f i r s t new region that i s reached can be i n t e r p r e t e d as SHADOW, s t o r e t h i s f a c t and keep going i n the same d i r e c t i o n . I f the new region can be i n t e r p r e t e d as WATER, then a good p a i r of bounding regions has been found. If no s u i t a b l e p a i r of neighbours i s found i n neighb, then i f reg was given as an argu-ment, g i v e up, e l s e go back and r e s e t reg to the next l a r g e s t r e g i o n . 5. Inst = (snewi '%bridge) > c r e a t e of new bridge i n s t a n c e : %bridge-2 Only now do we have enough con f i d e n c e and i n f o r m a t i o n to c r e a t e an i n s t a n c e f o r the b r i d g e . P r o c e s s i n g continues much as i t d i d in the l a s t s e c t i o n when the sketch map was used. p1, p2, and o r d e r l i s t are determined from the r e s u l t s of the "commonboun-dary" and "moveacross" f u n c t i o n s . From step 5 on, i n s t a n t i a t i o n Chapter 5. The Implementation 1 44 co n t i n u e s almost i d e n t i c a l l y . For t h i s p a r t i c u l a r image, the r e s u l t s from both procedures are s i m i l a r . Note, however, that the second r o u t i n e i s both more expensive c o m p u t a t i o n a l l y and more prone to missing the f e a t u r e s necessary to e s t a b l i s h the e x i s t e n c e of the b r i d g e . F i g u r e 5.17 shows the f i n a l schema for % b r i d g e - 2 . The curb schemata are i d e n t i c a l to those i n F i g u r e 5.13. 5.4_.__. B u i l d i n g up the H i e r a r c h i e s In S e c t i o n 5.4.1, i t was shown how two of the four r e l a -t i o n s h i p s among schemata are formed. Instances are c r e a t e d when there i s s u f f i c i e n t evidence to support the h y p o t h e t i c a l e x i s t e n c e of a geographic o b j e c t . New evidence, or a lack sketchmapi tern: o r d e r l i s t : neighbourregions shadowregions: ro a d r e g i o n s : a-part-of—> decomposes-to—> neighbours—> conf idence: v a l u e : n i l %conf idence: n i l v a l u e : (Other Shadow Bridge Bridge Other) %conf idence: 100 v a l u e : (1750 130) % c o n f i d e n c e : 100 v a l u e : (1186) % c o n f i d e n c e : 100 v a l u e : (1629) % c o n f i d e n c e : 100 n i l (%curb-4 %curb-5 %curb-6) (%river-1 % r i v e r - 2 ) 43 F i g u r e 5.17 Instance %bridge-2 Chapter 5. The Implementation 1 45 th e r e o f , may cause the inst a n c e to be destr o y e d . LINKs to the ge n e r i c o b j e c t s are i m p l i c i t l y c r e a t e d and destroyed with the i n s t a n c e . I n s t a n t i a t i o n p r o v i d e s i n f o r m a t i o n t h a t i s used to determine the neighbour r e l a t i o n s h i p s . When bri d g e s are i n s t a n -t i a t e d , the region going over the bridge belongs to a road and at l e a s t two re g i o n s beside i t belong to r i v e r s . In S e c t i o n 5.4.1.1 i t was assumed that two r i v e r s had a l r e a d y been i n s t a n -t i a t e d and because they shared r e g i o n s i n common with % b r i d g e - 1 , the neighbour LINK was formed (step 13). I t was assumed that the road had not yet been e s t a b l i s h e d so no LINK c o u l d be formed (step 14). However, i f and when the road a s s o c i a t e d with that r e g i o n was i n s t a n t i a t e d , a demon would add the LINKs between the two schemata. A l s o present i n S e c t i o n 5.4.1.1 were the messages that would cause the s p e c i a l i z a t i o n and decomposition h i e r a r c h i e s to be b u i l t up (step 16). When higher l e v e l schemata such as road system and r i v e r system are entered from below, they t r y to f i t new in s t a n c e s i n t o the e x i s t i n g semantic network or b u i l d on new a d d i t i o n s to the network i t s e l f . T h i s communication r e q u i r e s schemata to have knowledge of what i s above them i n the h i e r a r -c h i e s , but i f demons were used (see S e c t i o n 5.6) i t would be p o s s i b l e f o r that knowledge to e x i s t i n the higher l e v e l sche-mata alone . A s t r a i g h t f o r w a r d a l g o r i t h m f o r b u i l d i n g up the h i e r a r c h i e s i s shown i n F i g u r e 5.18. There are three b a s i c p o s s i b i l i t i e s . 1) If the new in s t a n c e i s not s p a t i a l l y near any e x i s t i n g Chapter 5. The Implementation 1 46 HIGH—the type of high l e v e l schema being joined NEW — t h e new instance to be put into the hiera r c h i e s INSTANCES = a l l current instances of HIGH ^Foreach I in INSTANCES done Cny Neighbours in" Common Between Components of J and NEW? N = N + 1 F = I LINK NEW to F MERGE I Into F -> done N > 0" -> done -Create new Instance of HIGH: G -LINK NEW to G -Send Message Up the Appropriate Hierarchy(s) to Continue Building -> done F i g u r e 5.18 B u i l d i n g up the Semantic Network Chapter 5. The Implementation 147 elements belonging to a higher l e v e l schema i t c r e a t e s a new instance of that schema and sends another message up the a p p r o p r i a t e h i e r a r c h i e s . 2) I f the new i n s t a n c e i s near an e l e -ment of one higher l e v e l node i t becomes a new descendant of that node. 3) If i t shares neighbours with more than one node then the higher l e v e l i n s t a n c e s are a l l merged i n t o one new schema which a l s o becomes the f a t h e r of the new i n s t a n c e . T h i s a l g o r i t h m i s g e n e r a l l y a p p l i c a b l e and r e q u i r e s only that each higher l e v e l i n s t a n c e provide the type of LINKS i t wishes to e s t a b l i s h with i t s ancestors and descendants in the graph. An example of a semantic network of i n s t a n c e s i s shown i n F i g u r e 5.19. The generic o b j e c t s and the AIO l i n k s have not been drawn s i n c e i t i s c l e a r from the names used f o r i n s t a n c e s what the correspondence would be. The other three types of LINKs are a l l i n d i c a t e d . T h i s network would r e s u l t i f a l l of the "intended" sketch map e n t i t i e s i n A s h c r o f t were i n s t a n t i a t e d from the i n t e n s i t y image. 5.4.3. Top-Down Attached Procedures In the l a s t two s u b s e c t i o n s i t was shown how bottom-up pro-cedures are used to i n s t a n t i a t e schemata and to b u i l d up the semantic network.. Top-down procedures are used to d i r e c t a t t e n -t i o n to schemata whose i n s t a n t i a t i o n would most p r o f i t a b l y advance the i n t e r p r e t a t i o n . S e c t i o n 4.5.1 d e s c r i b e d two types of top-down c o n t r o l . The f i r s t type a p p l i e s model knowledge i n the absence of other Chapter 5. The Implementation 148 F i g u r e 5.19 An A s h c r o f t Instance H i e r a r c h y Chapter 5. The Implementation 149 i n f o r m a t i o n . F o r e x a m p l e , i f t h e u s e r r e q u e s t s , " F i n d a g e o s y s -t e m " , t h e n t h e g e o s y s t e m t o p - d o w n p r o c e d u r e w i l l d e c i d e w h e t h e r t o s e a r c h f o r a l a n d m a s s o r a w a t e r b o d y ( i t s two s p e c i a l i z a -t i o n s ) b a s e d on a p r i o r i k n o w l e d g e o f w h i c h i s e a s i e r a n d more r e l i a b l y e s t a b l i s h e d . A l l o f t h e h i g h e r l e v e l s c h e m a t a c o n t a i n k n o w l e d g e o f t h i s s o r t t h a t moves c o n t r o l down t h e h i e r a r c h i e s . W h i l e i t w o u l d be p o s s i b l e t o jump s e v e r a l l e v e l s , w h i c h w o u l d be more e f f i c i e n t , t h i s h a s n o t been d o n e b e c a u s e o f m o d u l a r i t y . C o n s e q u e n t l y , e a c h schema o n l y n e e d s t o know a b o u t t h o s e s c h e m a t a i t c o n n e c t s t o d i r e c t l y . The f i v e b o t t o m l e v e l s c h e m a t a ( r o a d , m o u n t a i n , e t c . ) h a v e t h e k n o w l e d g e t h a t t h e y a r e a t t h e b o t t o m o f t h e h i e r a r c h i e s a n d t h e i r t o p - d o w n p r o c e d u r e s g i v e c o n t r o l t o t h e b o t t o m - u p r o u t i n e s . I n t h e a b s e n c e o f o t h e r i n f o r m a t i o n , t h e t o p - d o w n p r o c e d u r e s a l s o t r y t o f i n d a s k e t c h map i t e m o r a r e g i o n t h a t t h e a p p r o p r i a t e b o t t o m - u p p r o c e d u r e c a n be a p p l i e d t o . N o t e t h a t t h e s e t o p - d o w n r o u t i n e s o n l y r e d i r e c t c o n t r o l a n d do n o t a c t u a l l y i n s t a n t i a t e t h e i r s c h e m a t a . The s e c o n d t y p e o f t o p - d o w n c o n t r o l t a k e s p l a c e when t h e c u r r e n t c o n t e x t i s a v a i l a b l e t o c o m b i n e w i t h m o d e l k n o w l e d g e . T h u s , i f t h e u s e r a g a i n r e q u e s t s , " f i n d a g e o s y s t e m " , a n d t h e r e a l r e a d y i s an i n s t a n t i a t e d w a t e r b o d y , t h e g e o s y s t e m m i g h t d e c i d e t o i n s t a n t i a t e a l a n d m a s s . S o , t h e s c h e m a t a s t i l l d i r e c t c o n -t r o l down t h e h i e r a r c h i e s , b u t do i t i n a more i n f o r m e d m a n n e r . C h a p t e r 5. The I m p l e m e n t a t i o n 1 50 It i s a l s o p o s s i b l e f o r top-down r o u t i n e s to i n s t a n t i a t e schemata d i r e c t l y . For example, in the case where the only i n s t a n c e s i n r i v e r - s y s t e m s were two r i v e r s , the r i v e r system procedure might search i n the area between the r i v e r s , i f they were a s u i t a b l e d i s t a n c e a p a r t , f o r a b r i d g e s e p a r a t i n g them. If the s u p p o r t i n g f e a t u r e s were found, i t c o u l d i n s t a n t i a t e the b r i d g e d i r e c t l y and merge the three o b j e c t s i n t o one r i v e r -system. T h i s type of procedure has not been implemented, but a s i m i l a r e f f i c i e n c y gain r e s u l t s from the use of l a t e r a l mes-sages . S e c t i o n 5.4.1.1 i n d i c a t e s that when %bridge~1 was i n s t a n -t i a t e d , a l a t e r a l message was sent i n step 14 to the road schema. When schemata are i n s t a n t i a t e d , some of the knowledge they gain can be used to h e l p i n s t a n t i a t e other schemata. In t h i s case, the bridge procedure "knows" that one of i t s regions should be a road (and o t h e r s , r i v e r s ) and can send a s p e c i f i c message to that schema. The r o l e of the top-down r o u t i n e s f o r the bottom l e v e l schemata then becomes one of making sure the c o n t e x t u a l i n f o r m a t i o n ( i n t h i s case the r e g i o n number) i s presented i n the proper form to the bottom-up procedure. T h i s i s a l s o necessary i f the i n f o r m a t i o n has come from the user. For example, he/she might s p e c i f y that an i n s t a n c e from the sketch map a n a l y s i s use an exact p o s i t i o n . While t h i s system was not designed to be a p r o d u c t i o n sys-tem, e f f i c i e n c y q u e s t i o n s are s t i l l important. One c o u l d t r y to i n s t a n t i a t e every region i n the image, or worse yet, every Chapter 5. The Implementation 151 p o i n t , and every curve i n the sketch map with every model. Con-t r o l would be very simple and s i m i l a r r e s u l t s would ensue. How-ever, f o r anything other than t r i v i a l images t h i s would prove u n f e a s i b l e , whereas the methods d e s c r i b e d h e r e i n p r o v i d e g r e a t e r f l e x i b i l i t y and p r o c e d u r a l adequacy. 5.5. C l u s t e r A n a l y s i s Using Gaussian-Smoothed Histograms A one-dimensional v e r s i o n of the c l u s t e r i n g method d e s c r i b e d i n S e c t i o n 4.7 has been implemented and used i n s e v e r a l p l a c e s i n MISSEE. It has been s u c c e s s f u l l y employed to determine the f e a t u r e s that should be used to f i l l s l o t s d u r i n g i n s t a n t i a t i o n and to r e j e c t the e x i s t e n c e of h y p o t h e t i c a l sche-mata. The method has been a p p l i e d both i n an incremental and a s t a t i c f a s h i o n . The o r i e n t a t i o n of edges along the s i d e s of a road and of the r i d g e s i n mountains are incremental cases. Determining the r e c t a n g u l a r i t y of a region i s s t a t i c . The r i d g e s i n a mountain range a t l e a s t l o c a l l y tend to run in the same d i r e c t i o n . The o r i e n t a t i o n s of edge segments can be used to check the c o n s i s t e n c y of that v a l u e . O r i e n t a t i o n v a r i e s over 180 degrees ( i g n o r i n g the b r i g h t / d a r k d i r e c t i o n a c r o s s the edge) and i s c y c l i c , that i s , 0 and 180 degrees represent the same o r i e n t a t i o n . S e v e r a l edge segments are d i s c o v e r e d at a time, because they are c l o s e to a cha i n i n the sketch map, f o r example, and t h e i r o r i e n t a t i o n s are added to the histogram buck-e t s . The a l g o r i t h m i s executed a f t e r each new c o l l e c t i o n of Chapter 5. The Implementation 152 o r i e n t a t i o n s i s pr o v i d e d to determine i f the new r i d g e i s con-s i s t e n t with the others found so f a r . Of course, the new i n f o r -mation may a l t e r the conf i d e n c e of the o l d values (see below) which may r e s u l t i n the co n f i d e n c e value of an e x i s t i n g r i d g e to be r e c a l c u l a t e d . T h i s r e c a l c u l a t i o n a f f e c t s the co n f i d e n c e of mountains, and so on up the h i e r a r c h y . A f t e r the o r i e n t a t i o n histogram i s broken i n t o c l u s t e r s , the f o l l o w i n g i n f o r m a t i o n i s returned f o r each o r i e n t a t i o n v a l u e : the number of samples i n i t s c l u s t e r ( n ), the mean of i t s c l u s t e r (m), and the v a r i a n c e of i t s c l u s t e r ( v ) . Then f o r each o r i e n t a t i o n v a l u e , a con f i d e n c e l e v e l i s c a l c u l a t e d . The func-t i o n used i n MISSEE i s C(o) = 1 OOn N 50n N 50n * (2 -N o - m| ) 3v i f v. = 0 i f Io - mI > 3v otherwi se where o i s the o r i e n t a t i o n and N i s the t o t a l number of v a l u e s . T h i s f u n c t i o n responds both to i n t e r and i n t r a - c l u s t e r d i f f e r -ences. C l u s t e r s with many members w i l l have a l a r g e r value of n/N so each element w i l l have a higher c o n f i d e n c e v a l u e . Within c l u s t e r s , a f u n c t i o n v a r y i n g from .5 to 1.0 r a i s e s the c o n f i -dence val u e s of elements whose f e a t u r e value i s s i m i l a r to the mean of the c l u s t e r . Each co n f i d e n c e value i s s c a l e d to range Chapter 5. The Implementation 153 from 0 to 100. T h i s value i s then returned as the c o n f i d e n c e f o r the angles i n a l l r i d g e s being c o n s i d e r e d f o r i n c l u s i o n as w e l l as those r i d g e s p r e v i o u s l y i n s t a n t i a t e d . The o r i e n t a t i o n of l i n e segments that can be part of roads i s handled i n a manner s i m i l a r to r i d g e s . T h i s makes the assumptions that the,roads are s t r a i g h t (as i s u s u a l l y the c a s e ) . Roads that curve (such as the road represented by c h a i n 6 i n the A s h c r o f t sketch map) are not i n c l u d e d in the c l u s t e r -i n g ! ^ ] . Since roads i n urban areas are o f t e n found i n g r i d p a t -t e r n s one can a l s o c l u s t e r a l l the s t r a i g h t roads together u s i n g o r i e n t a t i o n modulo 90 degrees. F i g u r e 5.20 shows the r e s u l t s of c l u s t e r i n g f o r %road-4 (corresponding to c h a i n 1 i n the sketch map). Table 5.2 shows the c o n f i d e n c e values that are c a l c u l a t e d from these r e s u l t s . Since edges 139, 385, 383, 392, and 68 are in the minor c l u s t e r they have a much lower c o n f i d e n c e . F i g u r e 5.21 shows the edge segments that form %road-4. _5.5.__. Determining R e c t a n g u l a r i t y If one i s assuming that roads are l a i d out i n a r e c t a n g u l a r g r i d , then the urban regions ( c i t y b locks) that are bounded by roads should be r e c t a n g u l a r . A l s o , small s e c t i o n s of r i v e r s w i l l a l s o appear to be r e c t a n g u l a r i f the s c a l e i s l a r g e . [4]Curvature i s e a s i l y determined from the g e n e r a l i z e d l i n e r e p r e s e n t a t i o n - - s e e Appendix B. Chapter 5;. The Implementation 1 54 a 12.0 11.0 10.0 9.0 8.0 7.0 6.0 5.0 4.0 3.0 2.0 # c l u s t e r s 2 2 3 3 4 6 7 7 9 10 11 best a * c l u s t e r r i g h t e n d numsamples mean v a r i a n c e 1 2 92 151 22 5 19.455 639.066 126.400 98.640 a = 12; number of c l u s t e r s = 2 * * * * ****** * * * * * * * * * * * * * * * * * 0 1 2 3 4 5 6 7 8 9 0 0 0 . 0 0 0 0 0 0 1 1 1 1 1 1 0 1 2 3 4 5 0 0 0 0 0 0 1 1 1 6 7 8 0 0 0 1 st o r i e n t a t i o n F i g u r e 5.20 C l u s t e r i n g O r i e n t a t i o n s i n A s h c r o f t %road-4 1st. the r e s u l t of c o n v o l u t i o n with the f i r s t d e r i v a t i v e ( p o s i t i v e or n e g a t i v e ) . 2nd. the r e s u l t of c o n v o l u t i o n with the second d e r i v a t i v e . Hough transforms have p r e v i o u s l y been used to f i n d bounding r e c t a n g l e s (Sloan, 1982). However, Sloan used "dot product space", the extent of the p r o j e c t i o n of the boundary p o i n t s , whereas we w i l l use o r i e n t a t i o n space. In t h i s method, one moves along the boundary p o i n t s of the r e g i o n and c a l c u l a t e s the o r i e n t a t i o n of the l i n e j o i n i n g every p a i r of p o i n t s a s p e c i f i c d i s t a n c e (d) apart to form a histogram of the o r i e n t a t i o n s . By v a r y i n g both o and d, a two-dimensional t a b l e can be produced whose elements are the r e s u l t i n g number of c l u s t e r s . The c e n t r e Chapter 5. The Implementation 155 ANGLE EDGE CURB CONFIDENCE 1 1 71 7 81.301 1 4 394 2 81.365 1 4 69 7 81.365 18 387 1 81.450 22 359 3 81.427 23 59 8 81.406 27 1 36 6 81.321 37 384 1 81.108 45 382 1 80.938 45 367 5 80.938 45 1 38 6 80.938 45 70 7 80.938 63 393 2 80.556 72 72 7 80.364 109 139 6 17.974 1 24 385 1 18.443 127 383 1 18.499 1 36 392 2 18.218 1 36 68 7 18.218 1 54 361 4 78.622 1 58 364 5 78.537 1 75 1 35 6 78.176 180 386 1 78.069 180 366 5 78.069 180 1 37 6 78.069 180 360 3 78.069 180 58 8 78.069 Table 5.2 O r i e n t a t i o n s and Confidence Values of mass of the l a r g e s t r e gion i n the c h a r t where the number of c l u s t e r s i s constant i n d i c a t e s a "good" choice of parameters. If the f i g u r e i s r e c t a n g u l a r , the h e u r i s t i c should r e s u l t i n two c l u s t e r s whose means are 90 degrees a p a r t . T h i s i s i l l u s t r a t e d i n F i g u r e 5.22 f o r a diamond-shaped r e c t a n g l e . T h i s method works f o r other polygonal shapes too. For example, a t r i a n g l e w i l l r e s u l t i n three c l u s t e r s . Chapter 5. The Implementation 156 F i g u r e 5.21 A s h c r o f t : %road-4 Edges Chapter 5. The Implementation ( 2 0 , 2 0 ) ( 1 0 , 1 0 ) ( 1 0 , 3 0 ) ( 0 , 2 0 ) a d i s t 15 1 4 1 3 1 2 1 1 10 9 8 7 6 5 4 3 2 3 2 2 2 2 2 2 2 6 6 6 6 6 6 6 4 2 2 2 3 4 4 4 4 4 4 8 8 8 8 5 2 2 2 2 2 2 4 6 6 6 6 10 10 10 6 2 2 2 2 2 2 4 4 8 8 8 8 12 1 2 7 2 2 2 2 2 2 4 6 6 1 0 10 1 2 1 4 8 2 2 2 2 2 2 2 2 4 8 8 1 2 1 2 1 6 9 2 2 2 2 2 3 3 3 3 5 10 14 1 4 18 a = 12; d i s t a n c e = 7; number o f c l u s t e r s = 2 c l u s t e r r i g h t e n d numsamples mean v a r i a n c e 1 2 * * * * 91 0 * * * * * * * * 20 20 4 5 . 0 136.0 386 .8 386 .8 * * * * * * * * * * * * * * * * * * * * * * * * 1 1 1 1 1 1 3 4 5 6 7 8 0 0 0 0 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 1 0 0 1 1 0 o r i e n t a t i o n F i g u r e 5.22 C l u s t e r i n g O r i e n t a t i o n s a l o n g t h e B o u n d a r y o f a R e c t a n g l e C h a p t e r 5. The I m p l e m e n t a t i o n 158 T h i s technique has been used i n MISSEE as a p r e d i c a t e to determine i f a region i n the image i s roughly r e c t a n g u l a r . The longest s t a b l e s e c t i o n i n the c h a r t with two c l u s t e r s i s found and the d i f f e r e n c e of the c l u s t e r means i s c a l c u l a t e d . I f i t i s in the range of (80,110) degrees then the p r e d i c a t e r e t u r n s t r u e . I f no s t a b l e s e c t i o n can be found or the d i f f e r e n c e i n means i s not c l o s e enough to 90 then f a l s e i s ret u r n e d . T h i s p r e d i c a t e i s used by the system i n two ways. Regions that are i n s t a n t i a t e d as " c i t y b l o c k s " i n towns from the image alone must have the property of r e c t a n g u l a r i t y or they are r e j e c t e d . And, i f the region corresponding to a r i v e r i s r e c -t a n g u l a r , then the confidence of that r i v e r i s r a i s e d s l i g h t l y . While c l u s t e r i n g i s used as an i n f e r e n c e mechanism i n s e v e r a l p l a c e s w i t h i n MISSEE, there are other p o s s i b i l i t i e s that c o u l d be f r u i t f u l l y e x p l o r e d . In p a r t i c u l a r , attempts have not been made to i n f e r g l o b a l values such as s c a l e from the r e s u l t s of image i n t e r p r e t a t i o n , as d i s c u s s e d i n Chapter 4. 5.6. C o n t r o l — T h e P r i o r i t y Queue, Messages, and Demons C o n t r o l i s e x e r c i s e d through the c y c l e of p e r c e p t i o n ( c f . S e c t i o n 4.4) by means of a g l o b a l p r i o r i t y queue. Each entry on the queue des i g n a t e s a schema, how i t i s to be entered, and a p o s s i b l e context f o r the e v a l u a t i o n of one of i t s atta c h e d pro-cedures. T h i s i s analogous to, but more l i m i t e d than, p a t t e r n -d i r e c t e d i n v o c a t i o n i n Maya and a c h i e v e s a s i m i l a r m o d u l a r i t y . Chapter 5. The Implementation 159 Messages are sent from schemata to r e d i r e c t a t t e n t i o n to other schemata as the r e s u l t s of i n t e r p r e t a t i o n become a v a i l -a b l e . The user can a l s o i n i t i a t e messages e i t h e r to impart new i n f o r m a t i o n or to make known h i s / h e r requirements. A message i s made up of four p a r t s : 1) the name of the ge n e r i c schema i t i s d i r e c t e d t o, 2) how the schema i s being entered (top-down or bottom-up), 3) a p r i o r i t y number, and 4) parameters that are used to e s t a b l i s h the context when the a p p r o p r i a t e procedure a t t a c h e d to the schema i s e v a l u a t e d . Consequently, the sender of the message (a schema or the user) must "know" two aspects of the r e c e i v e r . The name of the r e c i p i e n t schema must be known as w e l l as where i t s i t s i n the h i e r a r c h y r e l a t i v e to the sender. L a t e r a l messages are always sent top-down. A p r i o r i t y value i s i n c l u d e d to rank the message. The user can pr o v i d e a value to r e f l e c t the importance of h i s / h e r message r e l a t i v e to those a l r e a d y i n the queue. A schema w i l l g e n e r a l l y use i t s CONFI-DENCE value p l u s or minus some small number to r e f l e c t the importance of the message. F i n a l l y , some in f o r m a t i o n u s e f u l to the procedure, such as l o c a t i o n , may be p r o v i d e d . A f u n c t i o n , addtoqueue, adds messages to the queue; see S e c t i o n 5.4.1.1 f o r an example. Sometimes i t i s necessary to modify p a r t s of messages that are on the queue. For example, S e c t i o n 5.4.2 d e s c r i b e d how schemata i n the h i e r a r c h i e s were sometimes merged when a newly i n s t a n t i a t e d schema r e v e a l e d that they were s p a t i a l l y contiguous. I f there were any messages whose parameter l i s t i n c l u d e d the name of the ins t a n c e whose Chapter 5. The Implementation 160 i d e n t i t y was submerged, then the new name would r e p l a c e i t on the queue as w e l l . T h i s s i t u a t i o n w i l l o f t e n a r i s e when the h i e r a r c h i e s are i n i t i a l l y being formed. Another g l o b a l f a c i l i t y i s the demon l i s t . Demons, func-t i o n s e s t a b l i s h e d by atta c h e d procedures, are a c t i v a t e d when some f u t u r e c o n d i t i o n becomes t r u e . A f t e r every attached pro-cedure i s executed, each demon i s eval u a t e d i f i t s i n i t i a t i o n c o n d i t i o n s are t r u e . Normally, a f t e r a demon has run i t w i l l remove i t s e l f from the l i s t . Demons are used i n MISSEE i n c o n j u n c t i o n with l a t e r a l mes-sages. While i n s t a n t i a t i n g a b r i d g e , a message may be sent to suggest a region that might correspond to a r i v e r . At the same time, a demon w i l l be set up to wait f o r the r i v e r to a c t u a l l y be i n s t a n t i a t e d . I f i t i s , the demon w i l l c r e a t e a neighbour l i n k between the two i n s t a n c e s and v a n i s h . The demon's a c t i o n saves the r i v e r schema from having to search a l l of i t s neigh-bouring regions f o r one corres p o n d i n g to a brid g e i n s t a n c e . S e c t i o n 5.4.1.1 i n c l u d e s an example of a demon being i n i t i a t e d ; Appendix E c o n t a i n s an example of one being executed. The execution c y c l e i s shown i n F i g u r e 5.23. Note how i n t e r a c t i o n with the user can be e a s i l y c o n t r o l l e d and v a r i e d . The g l o b a l v a r i a b l e % t e r s e i s used as the major switch to d e t e r -mine whether i n t e r a c t i o n should take p l a c e . In t e r s e mode, the user i s not q u e r i e d f o r any i n i t i a l i n f o r m a t i o n . Rather, the d e f a u l t message ( t d geosystem 100 n i l ) i s sent. A d d i t i o n a l l y , Chapter 5. The Implementation 161 I n i t i a t e QUEUE -By User -By D e f a u l t Remove F i r s t E n t r y From QUEUE and Ex e c u t e A p p r o p r i a t e P r o c e d u r e I n s t a n t i a t e Schema E x e c u t i n g P r o c e d u r e Sends out Messages Add P o s s i b i l i t i e s -5) E x e c u t e DEMONS done 1) S a t i s f i e d ? -2) Escape t o L i s p -Change G l o b a l V a l u e s -Examine the Schemata 3) M o d i f y QUEUE -Add Item - D e l e t e -Rearrange -> done F i g u r e 5.23 The Execution C y c l e Chapter 5. The Implementation 162 the user i s not prompted f o r i n f o r m a t i o n as i n t e r p r e t a t i o n proceeds. On the other hand, i f t e r s e mode i s o f f , then d u r i n g each c y c l e the user has a chance to query the system f o r d e t a i l e d i n f o r m a t i o n (over and above what i s p r o v i d e d on the d i s p l a y and i n the p r o t o c o l ) and to i n f l u e n c e p r o c e s s i n g i n v a r i o u s ways[5]. A new c o n t r o l loop i s entered that permits the user to escape to L i s p (to change g l o b a l v a l u e s such as the % p r i n t l e v e l , examine schemata, i n t e r a c t with the environment, e t c . ) or modify the p r i o r i t y queue. When he/she i s f i n i s h e d , perhaps without doing a n y t h i n g , then e i t h e r e x ecution i s resumed or the c y c l e i s broken. Examples of i n t e r a c t i o n w i t h i n the exe-c u t i o n c y c l e can be found i n Appendix E. T h i s e xecution c y c l e appears to be q u i t e d i f f e r e n t from the c y c l e of p e r c e p t i o n ( F i g u r e 4.2). However, i t c o n t r o l s i n t e r p r e t a t i o n i n the same manner and each aspect of the c y c l e of p e r c e p t i o n can be found somewhere in the execution loop. Both provide f o r the t i m e l y i n s t a n t i a t i o n of r e l e v a n t schemata and maintain a focus of a t t e n t i o n . [5]0ne might wonder how the user can get out of t e r s e mode when no i n t e r a c t i o n i s t a k i n g p l a c e . In a L i s p environment, i t i s p o s s i b l e to break the e x e c u t i o n , execute a command such as ( s e t q % t e r s e n i l ) , and then continue the execution where i t l e f t o f f . Chapter 5. The Implementation CHAPTER 6 R e s u l t s and E v a l u a t i o n MISSEE has been t e s t e d with 6 images of small urban a r e a s . The r e s u l t s i n d i c a t e the u s e f u l n e s s of the ideas p r e v i o u s l y d e s c r i b e d . T h i s chapter summarizes the r e s u l t s and i n d i c a t e s how they meet the c r i t e r i a of success d e s c r i b e d i n Chapter 3. The A s h c r o f t image and some r e s u l t s of i t s i n t e r p r e t a t i o n have been used i n p r e v i o u s chapters f o r i l l u s t r a t i v e purposes. F i g u r e s 6 . 1 , 6 .2 , and 6.3 show the other three images that have been used: Houston, Spences B r i d g e , and Cranbrook, a l l of which are i n B r i t i s h Columbia. Two subimages of the Spences Bridge photograph have been e x t r a c t e d to show that MISSEE i s robust with r e s p e c t to s c a l e d i f f e r e n c e s - - f o u r to one i n t h i s case. The e x t r a c t e d images w i l l be c a l l e d Spences Bridge West and Spences Bridge East to d i s t i n g u i s h them. The i n f o r m a t i o n sources f o r a l l the images can be found i n F i g u r e s 6.4 through 6.9 . 6.J_. I n s t a n t i a t i n g Objects With the A i d of a Sketch Map Except i n the case where the user i s c o n t r o l l i n g every step of the i n s t a n t i a t i o n process, the best r e s u l t s should appear when a sketch map i s used i n a d d i t i o n to the i n t e n s i t y image. Of course, only those o b j e c t s represented in the sketch map w i l l r e c e i v e guidance, but s i n c e the user probably i n c l u d e d a l l of 1 63 164 F i g u r e 6.1 Houston, B r i t i s h Columbia Chapter 6. R e s u l t s and E v a l u a t i o n F i g u r e 6.2 Spences Bridge, B r i t i s h Columbia Chapter 6. R e s u l t s and E v a l u a t i o n 166 F i g u r e 6 . 3 Cranbrook, B r i t i s h Columbia Chapter 6. R e s u l t s and E v a l u a t i o n 167 F i g u r e 6.4 A s h c r o f t : Information Sources Chapter 6. R e s u l t s and E v a l u a t i o n F i g u r e 6.5 Houston: Information Sources Chapter 6. R e s u l t s and E v a l u a t i o n 169 Chapter F i g u r e 6.6 Spences Bridge: 6. R e s u l t s and E v a l u a t i o n Informat ion Sources 170 F i g u r e 6.7 Spences Bridge West: Information Sources Chapter 6. R e s u l t s and E v a l u a t i o n 171 F i g u r e 6.8 Spences Bridge E a s t : Chapter 6. R e s u l t s and E v a l u a t i o n Information Sources F i g u r e 6.9 Cranbrook: Information Sources Chapter 6. R e s u l t s and E v a l u a t i o n 173 the items of i n t e r e s t , that should not be a l i m i t a t i o n . Two types of e r r o r s can r e s u l t when the procedure a s s o c i -ated with a schema that knows how to take advantage of sketch map i n s t a n c e s t r i e s to f i n d a c o r r e s p o n d i n g element i n the d i g i -t i z e d image. R e c a l l that c h a i n s i n the sketch map may be ambi-guously i n t e r p r e t e d so that there i s one "intended" i n t e r p r e t a -t i o n ( i . e . as the person who drew i t intended) and ( p o s s i b l y ) s e v e r a l that are "unintended". Thus two erroneous s i t u a t i o n s can o c c u r : 1) the intended model c o u l d not be i n s t a n t i a t e d i n the image, or 2) an unintended one c o u l d . In the terms used i n S e c t i o n 3.2, these would g e n e r a l l y be cases of u n s a t i s f i a b i l i t y and ambiguity, r e s p e c t i v e l y . The p o s s i b l e i n t e r p r e t a t i o n s of the A s h c r o f t sketch map were shown in Table 5.1. Table 6.1 d i v i d e s the i n s t a n c e s i n t o those that represent the i n t e r p r e t a t i o n intended by the person who drew the sketch map and those that were produced because of a m b i g u i t i e s i n the shapes of the c h a i n s . The t a b l e a l s o r e v e a l s which of the sketch map i n s t a n c e s were able to guide the i n s t a n -t i a t i o n of t h e i r c o u n t e r p a r t s i n the image. They are p r i n t e d i n upper case while i n s t a n c e s t h a t were unable to f i n d support i n the image are i n lower case. Thus, i d e a l l y , a l l the intended i n t e r p r e t a t i o n s i n the l e f t column would be i n upper case l e t t e r s and unintended i n t e r p r e t a t i o n s would be p r i n t e d i n lower case. Chapter 6. R e s u l t s and E v a l u a t i o n 174 Two e r r o r s of the second type come from c h a i n s 12 and 13 whose road schemata were ab l e to f i n d agreement i n the image. T h i s i s not s u r p r i s i n g s i n c e roads and mountains both search f o r b r i g h t regions and s t r a i g h t edges ( r i d g e s or c u r b s ) . If the r i d g e s happen to p a i r o f f at a c e r t a i n d i s t a n c e , then they w i l l be mistaken f o r roads. However, chains that correspond to roads w i l l very r a r e l y have the sketch map i n t e r p r e t a t i o n of "moun-t a i n " , so that a h e u r i s t i c can be used to remove those "unin-tended" road i n t e r p r e t a t i o n s . Table 6.2 summarizes the s i t u a t i o n f o r Houston. Here there i s an e r r o r of the f i r s t type. *road-5 has f a i l e d to f i n d the proper f e a t u r e s to be i n s t a n t i a t e d i n the image as a r o a d [ l ] . T h i s s i t u a t i o n a r i s e s because the region merger has j o i n e d the CHAIN INTENDED INTERPRETATION UNINTENDED INTERPRETATION(S) 3,5 *BRIDGE-1 4 *TOWN-1 9 *RIVER-1 *road-1 1 1 *RIVER-2 *road-2 1 0 *ROAD-3 * r i v e r - 3 2 *ROAD-4 * r i v e r - 4 6 *ROAD-5 * r i v e r - 5 1 *ROAD-6 * r i v e r - 6 7 *ROAD-7 * r i v e r - 7 8 *ROAD-8 * r i v e r - 8 1 2 *MOUNTAIN-1 *R0AD-9, * r i v e r -1 3 *M0UNTAIN-2 *R0AD-10, * r i v e r -1 4 *M0UNTAIN-3 *road-11, * r i v e r -Table 6.1 A s h c r o f t : Sketch Map to Image [1]Chain-5 i s the one i n the very top r i g h t of the sketch map. Chapter 6. R e s u l t s and E v a l u a t i o n 175 re g i o n c o r r e s p o n d i n g t o the road with the one next t o i t and the r e s u l t i n g average i n t e n s i t y has dropped below the t h r e s h o l d f o r acceptance. S i m i l a r r e s u l t s are shown f o r Spences B r i d g e , Spences Bridge West, Spences Bridge East, and Cranbrook i n Tables 6.3 through 6.6. In summary, out of 100 sketch map i n s t a n c e s there were 3 e r r o r s of the f i r s t type and 6 e r r o r s i n the second CHAIN INTENDED UNINTENDED INTERPRETATION INTERPRETATION(S) 2,4 *BRIDGE-1 8 *RIVER-1 *road-1 9 *RIVER-2 *road-2 1 *ROAD-3 * r i v e r - 3 6 *ROAD-4 * r i v e r - 4 5 *road-5 * r i v e r - 5 3 *ROAD-6 * r i v e r - 6 7 *ROAD-7 *RIVER-7, *bridge Table 6.2 Houston: Sketch Map to Image CHAIN INTENDED UNINTENDED INTERPRETATION INTERPRETATION(S) 1 3 *town-1 1 2 *ROAD-1 9 *ROAD-2 1 1 *ROAD-3 10 *ROAD-4 5,6 *BRIDGE-1 2 *RIVER-5 3 *RIVER-6 7,8 *BRIDGE-2 4 *RIVER-7 1 4 *MOUNTAIN-1 *ROAD-8, *RIVER-8 1 5 *MOUNTAIN-2 *road-9, * r i v e r - 9 1 6 *MOUNTAIN-3 * r o a d - l 0 , *RIVER-Table 6.3 Spences Bridge: Sketch Map to Image Chapter 6. R e s u l t s and E v a l u a t i o n 176 CHAIN INTENDED UNINTENDED INTERPRETATION INTERPRETATION(S) 4,5 *BRIDGE-1 2 *RIVER-1 3 *RIVER-2 10 *TOWN-1 9 *ROAD-3 6 *ROAD-4 7 *ROAD-5 * r i v e r - 5 8 *ROAD-6 * r i v e r - 6 Table 6.4 Spences Bridge West: Sketch Map to Image CHAIN INTENDED INTERPRETATION UNINTENDED INTERPRETATION(S) 4,5 *BRIDGE-1 2 *RIVER-1 3 *RIVER-2 6 *ROAD-3 7 *ROAD-4 * r i v e r - 4 8 *MOUNTAIN-1 * r i v e r - 5 , *road-5 9 *mountain-2 * r i v e r - 6 , *road-6 Table 6.5 Spences Bridge E a s t : Sketch Map to Image CHAIN INTENDED UNINTENDED INTERPRETATION INTERPRETATION(S) 5 *ROAD-1 * r i v e r - 1 4 *ROAD-2 * r i v e r - 2 8 *ROAD-3 * r i v e r - 3 6 *ROAD-4 * r i v e r - 4 9 *ROAD-5 * r i v e r - 5 3 *ROAD-6 * r i v e r - 6 2 *ROAD-7 * r i v e r - 7 7 *ROAD-8 * r i v e r - 8 Table 6.6 Cranbrook: Sketch Map to Image category. O v e r a l l , t h i s i s a 9% e r r o r r a t e . Chapter 6. R e s u l t s and E v a l u a t i o n 1 77 V i s u a l evidence a l s o i n d i c a t e s that a p p r o p r i a t e image f e a t u r e s were matched with MISSEE i n s t a n c e s . F i g u r e s 5.15 and 5.16 d i s p l a y e d the regions and edges that were found f o r the bridge i n the A s h c r o f t image. The f o l l o w i n g f i g u r e s show s i m i -l a r r e s u l t s f o r the other "bottom l e v e l " o b j e c t s i n the A s h c r o f t image. Regions are the prominent f e a t u r e s f o r towns ( F i g u r e 6.10) and r i v e r s (6.11). Mountains are a s s o c i a t e d with edges (6.12). Roads, l i k e b r i d g e s , map to both edges (6.13 and 6.14) and regions (6.15). The region f o r roads 4, 5, and 6 i s the same as f o r roads 1 and 2. 6.2. I n s t a n t i a t ing O b j e c t s from the I n t e n s i t y Image Alone Geographic o b j e c t s can be i n s t a n t i a t e d with some success from j u s t the d i g i t i z e d image. However, the s i t u a t i o n i s more ambiguous without the a i d of a sketch map and enough evidence i s o f t e n d i s c o v e r e d to i n s t a n t i a t e many e n t i t i e s with low c o n f i -dence . Roads and b r i d g e s are c o m p u t a t i o n a l l y much harder to f i n d u s i ng images alone because the r e g i o n s that correspond to them have g e n e r a l l y merged together many i n d i v i d u a l roads. Having found a s u i t a b l e r e g i o n , the program searches f o r bounding water regions f o r bridges (an example i s i n S e c t i o n 5.4.1.2) or s u i t -a b ly spaced p a r a l l e l edges fo r roads. With roads, t h i s tends to f i n d c i t y b l o c k s , from one i n t e r s e c t i o n to another, r a t h e r than continuous s t r e e t s . Thus, besides being l e s s e f f i c i e n t , the r e s u l t s may be d i f f e r e n t even when they are c o r r e c t . Chapter 6. R e s u l t s and E v a l u a t i o n 178 F i g u r e 6.10 A s h c r o f t : Sketch Map. %town-1 Chapter 6. R e s u l t s and E v a l u a t i o n 179 F i g u r e 6.11 A s h c r o f t : S k e t c h Map. R i v e r R e gions Chapter 6. R e s u l t s and E v a l u a t i o n 180 F i g u r e 6.12 A s h c r o f t : Sketch Map. Mountain Edges Chapter 6. R e s u l t s and E v a l u a t i o n F i g u r e 6.13 A s h c r o f t : Sketch Map. Roads 1-3, Edges Chapter 6. R e s u l t s and E v a l u a t i o n F i g u r e 6.14 A s h c r o f t : Sketch Map. Roads 4-6, Edges Chapter 6. R e s u l t s and E v a l u a t i o n 183 Fi g u r e 6.15 A s h c r o f t : Sketch Map. Roads 1-3, Regions Chapter 6. R e s u l t s and E v a l u a t i o n 184 There i s a s l i g h t mismatch i n the semantics of "town" as found i n sketch maps versus the images analyzed by MISSEE. To Mapsee, a town i s j u s t t h a t — t h e whole town—and roads are a l l i n t e r u r b a n thoroughfares. In the images that are being examined here, one i s l o o k i n g i n s i d e the town at roads, r i v e r s , e t c . Thus the sketch map "blob" has been used to represent "down-town", or the core of the c i t y . That i s a r a t h e r a r b i t r a r y d e n o t a t i o n , however, and when c o n s i d e r i n g images alone, town r e f e r s to areas that are urban (near roads, c o n t a i n i n g b u i l d -i n g s ) . Thus one would expect to see only one or two towns i n the sketch map, but s e v e r a l i n the image. The next few f i g u r e s compare the r e s u l t s of an i n t e r p r e t a -t i o n using the image alone and one with the a i d of a sketch map. %road-2 ( F i g u r e 6.16) was i n s t a n t i a t e d with the a i d of the Ash-c r o f t sketch map and c o n t a i n s 19 edges. Four roads i n s t a n t i a t e d from the image alone ( F i g u r e 6.17) are r e q u i r e d to o b t a i n s i m i -l a r r e s u l t s . However, sometimes the r e s u l t s from the two t e c h -niques are s i m i l a r as i s shown in F i g u r e s 5.21, where %road-4 i s i n s t a n t i a t e d with the a i d of the sketch map, and 6.18, where %ro a d - l 2 i s i n s t a n t i a t e d from the image alone. The one r e g i o n c o r r e s p o n d i n g to downtown i n the A s h c r o f t sketch map was shown in F i g u r e 6.10. That r e g i o n was not a c c e p t a b l e when the image alone was examined because i t i s not r e c t a n g u l a r . T h i s i s i l l u s t r a t e d i n F i g u r e 6.19 which c o n t a i n s s e v e r a l urban r e g i o n s . The r e s u l t s f o r the i n t e r p r e t a t i o n of the 5 bottom l e v e l o b j e c t s from the images alone are summarized i n Table 6.7. Chapter 6. R e s u l t s and E v a l u a t i o n 185 F i g u r e 6.16 A s h c r o f t : Sketch Map. %road~2 Chapter 6. R e s u l t s and E v a l u a t i o n 186 F i g u r e 6.17 A s h c r o f t : Image Alone. Four Roads Chapter 6. R e s u l t s and E v a l u a t i o n 187 F i g u r e 6.18 A s h c r o f t : Image Alone. %road-12 Chapter 6. R e s u l t s and E v a l u a t i o n 188 F i g u r e 6.19 A s h c r o f t : Image Alone. S e v e r a l Urban Regions Chapter 6. R e s u l t s and E v a l u a t i o n 189 BRIDGE MOUNTAIN RIVER ROAD TOWN A s h c r o f t CORRECT 1 3 2 30 7 INCORRECT 0 2 0 1 1 5 OMITTED 0 2 0 1 1 Houston CORRECT 1 0 2 20 26 INCORRECT 1 0 0 1 1 OMITTED 0 0 0 5 5 Spences Br idge CORRECT 2 3 2 51 4 INCORRECT 10 0 3 14 0 OMITTED 0 1 0 0 1 Spences Bridge West CORRECT 0 4 2 81 2 INCORRECT 7 0 2 0 1 OMITTED 1 0 0 0 0 Spences Bridge East CORRECT 0 0 2 1 2 4 INCORRECT 1 0 0 2 0 OMITTED 1 2 0 0 0 Cranbrook CORRECT 0 0 0 7 6 INCORRECT 0 0 0 0 0 OMITTED 0 0 0 4 1 Table 6.7 A l l Images: I n s t a n t i a t i o n R e s u l t s There are again two types of e r r o r s : i n s t a n c e s that have been i n s t a n t i a t e d t h a t should not have been ( c a l l e d i n c o r r e c t ) ; and, o b j e c t s i n the image that d i d not have the r i g h t f e a t u r e s to cause i n s t a n t i a t i o n ( o m i t t e d ) . The former e r r o r i s an i n c o r r e c t match between model and data (as opposed to a numeric mismatch between the domain and the range); the l a t t e r e r r o r i s one of Chapter 6. R e s u l t s and E v a l u a t i o n 190 incompleteness. O v e r a l l , the e r r o r r a t e f o r i n t e r p r e t a t i o n from the image alone i s (#INCORRECT + #OMITTED) / #TOTAL = 24%. As expected, t h i s i s higher that the e r r o r r a t e when a sketch map i s a l s o used ( 9 % ) . 6.3_. B u i l d i n q Up H i e r a r c h i e s When the f i r s t bottom l e v e l schemata are i n s t a n t i a t e d , they cause ( v i a messages) the e n t i r e h i e r a r c h y above them to be c r e a t e d . As more schemata are i n s t a n t i a t e d ( p o s s i b l y using the r e s u l t s of e a r l i e r p r o c e s s i n g ) , they e i t h e r f i t d i r e c t l y i n t o the e x i s t i n g network, or cause the c r e a t i o n of a subgraph that can. T h i s entry i n t o the network i s mediated by the neighbour r e l a t i o n which determines which scene elements are l o c a t e d next to each o t h e r . The neighbour r e l a t i o n s h i p s are e s t a b l i s h e d e i t h e r by demons or by the schema-specific r o u t i n e s a f t e r the e x i s t e n c e of a new in s t a n c e has been confirmed. Each o b j e c t has i n f o r m a t i o n about i t s p o s s i b l e neighbours and ways to e s t a b l i s h the l i n k . G e n e r a l l y , i t i n v o l v e s f i n d i n g common edges or regions or neigh-bouring regions i n the a p p r o p r i a t e i n s t a n c e s . When the l i n k i s e s t a b l i s h e d , the inv e r s e r e l a t i o n ( a l s o "neighbour", s i n c e the r e l a t i o n i s r e f l e x i v e ) i s a l s o c r e a t e d . Neighbouring i n s t a n c e s of the same type w i l l have a common parent node up at l e a s t one of the h i e r a r c h i e s . Sometimes a new Chapter 6. R e s u l t s and E v a l u a t i o n 191 i n s t a n c e w i l l i n d i c a t e that i t r e l a t e s two or more groups. In that case, the parent nodes are a l l c o l l a p s e d i n t o one of the nodes. The r e s u l t s of t h i s h i e r a r c h y b u i l d i n g are based on whether the neighbour r e l a t i o n i s e s t a b l i s h e d . When i t i s p r o p e r l y formed, the h i e r a r c h i e s w i l l always be b u i l t up c o r r e c t l y . I f some neighbour r e l a t i o n s h i p s are not found but some of the l i n k s are redundant, the h i e r a r c h i e s may s t i l l be c o r r e c t l y formed. However, i f enough c r u c i a l l i n k s are m i s s i n g , then the h i e r a r -c h i e s w i l l be s p l i t i n t o separate (sub)graphs. The formation of neighbour l i n k s can s u f f e r from two types of e r r o r s : 1) an i n c o r r e c t l i n k can be made between non-neighbouring schemata, or 2) the l i n k might not be found. In MISSEE, the former e r r o r has never o c c u r r e d and the l a t t e r only r a r e l y . Furthermore, as was j u s t e x p l a i n e d , e r r o r s of omission are l e s s problematic because neighbour l i n k s may be redundant. Hence, i t depends on which schemata were a c t u a l l y i n s t a n t i a t e d (and the v a l u e s used to f i l l t h e i r s l o t s ) , which i n turn depends on the user's s t r a t e g y . F i g u r e 5.19 showed the r e l a t i o n s (except i n s t a n c e of) that would be e s t a b l i s h e d i f a l l of the A s h c r o f t sketch map i n s t a n c e s were used to a i d i n i n t e r p r e t i n g the image. There are no e r r o r s i n t h i s graph. F i g u r e 6.20 shows what would happen i f only the A s h c r o f t image was used. Only the c o r r e c t i n s t a n t i a t i o n s are i n c l u d e d and s e v e r a l of the i n t e r v e n i n g f e a t u r e s r e q u i r e d to Chapter 6. R e s u l t s and E v a l u a t i o n 192 F i g u r e 6.20 A s h c r o f t : Instance H i e r a r c h y 2 Chapter 6. R e s u l t s and E v a l u a t i o n 193 e s t a b l i s h neighbour l i n k s are m i s s i n g ( e s p e c i a l l y f o r r o a d s ) . Thus the graph i s fragmented i n t o 13 p i e c e s more than the one in F i g u r e 5.19. Appendix E c o n t a i n s a p r o t o c o l that shows a s e s s i o n with the A s h c r o f t image where the user i n t e r a c t s with the system. A l l types of schema-specific r o u t i n e s are employed. The r e s u l t -ing h i e r a r c h y i s shown in F i g u r e 6.21. There are no e r r o r s i n t h i s graph which c o n t a i n s fewer i n s t a n c e s than F i g u r e 5.19, only because the user terminated p r o c e s s i n g with p o s s i b i l i t i e s remaining. S i m i l a r r e s u l t s have been ob t a i n e d f o r the other images. F i g u r e s 6.22 and 6.23 show the r e l a t i o n s h i p s f o r Houston and Spences Bridge that were d e r i v e d a u t o m a t i c a l l y using the sketch map to guide i n t e r p r e t a t i o n . 6.4. Moving Up and Down the H i e r a r c h i e s The goal of the c y c l e of p e r c e p t i o n i s to use the r e s u l t s of i n t e r p r e t a t i o n to guide f u r t h e r p r o c e s s i n g so that scene knowledge i s accumulated in an i n t e l l i g e n t manner. Instead of randomly s e a r c h i n g the image f o r i n t e r e s t i n g f e a t u r e s , the focus of a t t e n t i o n s h i f t s i n a d i r e c t i o n determined by model knowledge and newly a c q u i r e d i n f o r m a t i o n . In an i s l a n d - d r i v e n approach where a l l of the o b j e c t s are s p a t i a l l y connected, i t becomes necessary to search only f o r the f e a t u r e s that w i l l i n s t a n t i a t e the f i r s t item. Chapter 6. R e s u l t s and E v a l u a t i o n 194 F i g u r e 6.21 A s h c r o f t : Instance H i e r a r c h y 3 Chapter 6. R e s u l t s and E v a l u a t i o n 195 Chapter 6. F i g u r e 6.22 R e s u l t s and Houston: An E v a l u a t ion Instance H i e r a r c h y 196 F i g u r e 6.23 Spences B r i d g e : An Instance H i e r a r c h y Chapter 6. R e s u l t s and E v a l u a t i o n 1 97 When a sketch map IS i s a v a i l a b l e , i t i s not even necessary to search ( s p a t i a l l y ) f o r the i n i t i a l o b j e c t . In f a c t , the o r g a n i z a t i o n of the sketch map schemata in t h e i r own semantic network might preclude the requirement of having to do i t again f o r the image. No search would be necessary only i f the chains in the sketch map were unambiguously i n s t a n t i a t e d and i f the user was not i n t e r e s t e d i n any e n t i t i e s that might appear in the image but not i n the sketch map. While the l a t t e r w i l l o f t e n be t r u e , the former w i l l seldom be. So, to s t a r t the c y c l e of p e r c e p t i o n using sketch map i n f o r m a t i o n , an unambiguous i n s t a n c e i s chosen. Bridges and towns are u s u a l l y unique whereas roads, r i v e r s , and mountains g e n e r a l l y are not. Bridges a l s o have the advantage of being p a r t of both road and r i v e r systems and can provide much u s e f u l i n f o r m a t i o n about each. In p a r t i c u l a r , t h e i r a ttached pro-cedures can determine image re'gions that are l i k e l y candidates f o r the a s s o c i a t e d road and r i v e r . A f t e r a s u i t a b l e message i s r e c e i v e d , the top-down procedure a s s o c i a t e d with roads (and r i v e r s ) w i l l search in the region f o r a chain i n the sketch map. That c h a i n probably has s e v e r a l i n t e r p r e t a t i o n s . In t h i s case, however, only the road i n s t a n c e w i l l be used. T h i s reduces search through the space of sketch map i n s t a n c e s . If the sketch map i n f o r m a t i o n source i s not a v a i l a b l e , then model knowledge can be used to r e s t r i c t the regions i n the image that should be searched to f i n d the i n i t i a l o b j e c t . In MISSEE, a simple h e u r i s t i c has been used: c a l c u l a t e a range of s u i t a b l e Chapter 6. R e s u l t s and E v a l u a t i o n 198 areas that a p a r t i c u l a r o b j e c t might have (us i n g the g l o b a l v a l u e , s c a l e ) and then search them i n order of d e c r e a s i n g s i z e . I n s t a n t i a t i o n w i l l again generate messages about l i k e l y i n t e r p r e t a t i o n s f o r other r e g i o n s . Since the sketch map i n f o r -mation i s not a v a i l a b l e , the l e s s a c c u r a t e (and l e s s e f f i c i e n t ) image-based r o u t i n e s w i l l be d i r e c t e d to the a p p r o p r i a t e regions or p o s i t i o n s . Table 6.8 shows the number of regions that were searched to i n s t a n t i a t e the f i v e bottom l e v e l schemata. The f i g u r e s f o r A s h c r o f t Houston Spences Br idge Spences Bridge West Spences Bridge East Cranbrook T o t a l Number 75 1 39 161 1 25 1 1 9 285 of Regions With • Sketchmap Intended #Relevant 7 9 23 31 42 24 #Extra 6 1 2 4 1 1 20 Unintended 10 1 8 0 2 0 T o t a l 23 22 35 32 45 44 Without Sketchmap C o r r e c t #Relevant 1 5 36 31 36 44 1 0 #Extra 17 39 1 5 27 1 3 9 I n c o r r e c t 7 7 51 9 1 3 T o t a l 39 82 97 72 58 22 Table 6.8 The Number of Regions Searched Chapter 6. R e s u l t s and E v a l u a t i o n 199 search with and without the use of a sketch map correspond to the t e s t s that were used to compose Tables 6.1 to 6.7. Relevant regions are d i s t i n g u i s h e d from extraneous ones i n that they are the ones that are e v e n t u a l l y used to f i l l i n instance s l o t s . As would be expected, the number of regions searched when the sketch map was employed was c o n s i d e r a b l y s m a l l e r than without the sketch map. On the average, 22% of the regions were exam-ined when the sketch map was used and 41% when i t was not. The advice given i n messages i s i n v a r i a b l y u s e f u l i n reduc-ing search, because, even i f a l l of the f e a t u r e s that would enable i n s t a n t i a t i o n are not present, the region i s one that g e n e r a l l y would have been searched anyway. For example, the procedures a s s o c i a t e d with roads i n d i c a t e that neighbouring regions with the p o s s i b l e i n t e r p r e t a t i o n URBAN can be towns. The only other requirement f o r town i s that i t s shape be roughly r e c t a n g u l a r . I f the region was a s u i t a b l e s i z e , i t would have been searched anyway due to i t s i n t e n s i t y . P r e s e r v i n g negative (and p o s i t i v e ) knowledge about r e g i o n s and t h e i r attempted i n s t a n t i a t i o n as o b j e c t s prevents needless subsequent p r o c e s s -ing . Search i n the image i s a u t o m a t i c a l l y reduced i n three ways. 1) P o s i t i o n a l i n f o r m a t i o n from the sketch map guides i n s t a n t i a -t i o n i n the r e g i s t e r e d i n t e n s i t y image. 2) Model knowledge f i l t e r s r e g i o n s f o r f u r t h e r p r o c e s s i n g based on the s i z e and average i n t e n s i t y of the r e g i o n . 3) Model knowledge p l u s con-text (the i n f o r m a t i o n gained from the i n t e r p r e t a t i o n process) Chapter 6. R e s u l t s and E v a l u a t i o n 200 suggest l i k e l y i n t e r p r e t a t i o n s f o r regions near the c u r r e n t focus of a t t e n t i o n . Furthermore, p o s s i b i l i t i e s f o r search are made e x p l i c i t , g i v i n g the user c o n t r o l over t h e i r eventual use. 6.5. How User I n t e r a c t i o n Can I n f l u e n c e I n t e r p r e t a t i o n In MISSEE, the user i s able to do as much or as l i t t l e as he/she d e s i r e s to i n f l u e n c e the progress of i n t e r p r e t a t i o n . By t a k i n g an a c t i v e r o l e , the user can guarantee that the r e s u l t s w i l l be to h i s / h e r l i k i n g . Depending on how w e l l the automatic system i s performing, t h i s may take more or l e s s e f f o r t on h i s / h e r p a r t . At one extreme, the user may choose to l e t the system run a u t o m a t i c a l l y . I t i s not r e q u i r e d that he/she draw a sketch map. In the c u r r e n t implementation, he/she must help t r a i n the i n t e n s i t y c a t e g o r i z e r ( S e c t i o n 5.3.1.2.1) to a s s i g n p o s s i b l e c l a s s e s to regions i n the image, but there are ways to make t h i s automatic, too. In t e r s e mode, MISSEE w i l l not i n t e r r u p t i t s e x e cution c y c l e but w i l l continue u n t i l no more ca n d i d a t e s remain[2]. MISSEE w i l l have a u t o m a t i c a l l y produced a p r o t o c o l and p o s s i b l y images ( i f a f l a g was set i n d i c a t i n g an a p p r o p r i a t e output device was a v a i l a b l e ) . I f that output i s i n s u f f i c i e n t , then a f t e r the execution c y c l e i s f i n i s h e d , the user may choose to examine the semantic network in g r e a t e r d e t a i l . [2]In the c u r r e n t system, that would be a f t e r every image r e -gion of s u i t a b l e s i z e and c a t e g o r i z a t i o n has been t r i e d with each of the f i v e bottom l e v e l schemata--bridge, mountain, r i v e r , road, and town. Chapter 6. R e s u l t s and E v a l u a t i o n 201 As i n d i c a t e d i n S e c t i o n 6.2, the r e s u l t s from t h i s type of p r o c e s s i n g w i l l be somewhat i n a c c u r a t e , although they may w e l l be adequate f o r some t a s k s . The user can h e l p the s i t u a t i o n s i g n i f i c a n t l y by drawing a s k e t c h map of the scene over the image. Even b e t t e r r e s u l t s w i l l f o l l o w i f the user takes a more a c t i v e r o l e i n the i n t e r p r e t a t i o n process. The user can provide three types of input to the system as was d e s c r i b e d i n S e c t i o n 4.4.1. In terms of improving the accu-racy of the r e s u l t s , i t may be necessary only to make minor changes i n the parameters of messages on the p o s s i b i l i t i e s l i s t . More d r a s t i c measures would i n c l u d e adding or d e l e t i n g messages. In extreme cases where scene o b j e c t s have not been i n s t a n t i a t e d or n o n - e x i s t i n g e n t i t i e s have, i t i s p o s s i b l e (from L i s p ) to modify the semantic network d i r e c t l y by c r e a t i n g and d e s t r o y i n g i n s t a n c e s . Thus, the user can ensure that the end r e s u l t s w i l l be p e r f e c t . However, s i n c e there i s no " n a t u r a l " user i n t e r -f a c e , more extreme a c t i o n s r e q u i r e the user to have more knowledge about the nature of messages and the schemata manipu-l a t i o n f u n c t i o n s . With regard to task p r i o r i t i e s , the user can narrow the system's focus. If the user i s only i n t e r e s t e d i n o b j e c t s of a c e r t a i n type, such as roads or r i v e r systems, then he/she can d i r e c t MISSEE to look f o r o n l y those o b j e c t s (and t h e i r com-ponents and s p e c i a l i z a t i o n s ) and to ignore i r r e l e v a n t messages. In a d d i t i o n , the user can be very s p e c i f i c about where to f i n d the items of i n t e r e s t . Then, he/she can examine s p e c i f i c p a r t s Chapter 6. R e s u l t s and E v a l u a t i o n 202 of the r e s u l t i n g schemata to o b t a i n the d e s i r e d i n f o r m a t i o n about the o b j e c t s . The user a l s o c o n t r o l s h i s / h e r environment so that the i n t e r a c t i o n f u l f i l l s h i s / h e r needs. G l o b a l input determines how much i s i n c l u d e d i n the p r o t o c o l s , i f and where graphic output i s to be sent, and whether the user wants to have a chance to i n t e r a c t i n each loop of the execution c y c l e . Such environment t a i l o r i n g would become even more convenient i f user models were u t i l i z e d . Appendix E c o n t a i n s a p r o t o c o l f o r a sample s e s s i o n d e a l i n g with the A s h c r o f t image. The "user" has no s p e c i f i c p r i o r i t i e s but i s paying a t t e n t i o n to the i n t e r p r e t a t i o n as i t proceeds so that he can make s l i g h t m o d i f i c a t i o n s i f necessary to ensure that the r e s u l t s w i l l be a c c u r a t e . Sketch map guidance i s a v a i l a b l e and both top-down and bottom-up p r o c e s s i n g are i l l u s -t r a t e d . A l s o , messages of a l l s o r t s are generated: top-down, bottom-up, l a t e r a l , and user generated. While only the p r o t o c o l i s shown i n Appendix E, some of the images produced would be s i m i l a r to those found i n F i g u r e s 5.15, 5.16, 5.21, 6.10, 6.11, 6.16, and 6.18. The r e s u l t i n g i n s t a n c e s are found i n F i g u r e 6.21 . In summary, the user can i n f l u e n c e the what, when, where, and how of i n t e r p r e t a t i o n to get the d e s i r e d r e s u l t s . The amount of i n t e r a c t i o n can range from almost none to a step by step d i a l o g u e . The user would g e n e r a l l y vary the amount of Chapter 6. R e s u l t s and E v a l u a t i o n 203 i n t e r a c t i o n depending on the accuracy of p r o c e s s i n g , h i s / h e r p r i o r i t i e s , or both. T h i s p r o v i d e s a very f l e x i b l e a d j u n c t . t o automatic p r o c e s s i n g . 6.6. D e s c r i p t i v e Adequacy The d e s c r i p t i v e adequacy of a v i s i o n system i s determined by how we l l the d e s c r i p t i o n s of the domain serve to advance the s o l u t i o n of the problem at hand. In MISSEE, the task i s not only the i n t e r p r e t a t i o n of i n t e n s i t y images but a l s o the recovery of p e r t i n e n t scene i n f o r m a t i o n . The schema-based sys-tem employed i s d e f i n i t e l y an asse t i n the p u r s u i t of both of these g o a l s . Questions r e l a t i n g to d e s c r i p t i v e adequacy were r a i s e d i n S e c t i o n 3.3. In response, the VALUEd and LINK types of the schemata make c r u c i a l f a c t s e x p l i c i t . These are the d e f i n i n g a t t r i b u t e s of o b j e c t s that are f i l l e d i n v i a the attached pro-cedures. A l s o , the s l o t values p r o v i d e the source of knowledge a f t e r i n t e r p r e t a t i o n f o r g a t h e r i n g s p e c i f i c i n f o r m a t i o n about the elements of the scene. The atta c h e d procedures do not make e x p l i c i t the methods by which i n s t a n t i a t i o n takes p l a c e . T h i s i s an area where extensions might be made. Facts are g e n e r a l l y encoded i n a uniform manner. The LINKS are maintained by the schemata man i p u l a t i o n system which en f o r c e s an e x p l i c i t , uniform access to them. VALUEd types are s p e c i f i c to schemata but an attempt has been made to use s t a n -dard names f o r commonly o c c u r r i n g a t t r i b u t e s . These i n c l u d e Chapter 6. R e s u l t s and E v a l u a t i o n 204 "sketchmapitem", "regions", and "edges" to refer to the corresponding features from the information sources. Since bot-tom l e v e l schemata (bridges, curbs, mountains, ridges, r i v e r s , roads, and towns) a l l interface to the intensity image by means of regions and/or edges, the method of description i s consistent and uniform. The clear d i s t i n c t i o n between general and pa r t i c u l a r knowledge enables the system to change as new information becomes av a i l a b l e . Instances are hypothetical and can be created and destroyed based on the evidence. In contrast, stereotypic objects are s t a t i c . New facts cause slo t s to be f i l l e d and an image-specific instance network to be b u i l t . This c r u c i a l d i s t i n c t i o n between the general and the s p e c i f i c i s made at a l l levels in the hierarchies, not just at the leaf nodes. Individual schemata are modular but not t o t a l l y indepen-dent. Instantiation of bottom le v e l schemata can take place independently from other schemata because they are s e l f -contained, having both a procedural and a declarative part. However, higher l e v e l schemata that can not be instantiated d i r e c t l y from the IS's must have knowledge about their com-ponents or spe c i a l i z a t i o n s so that control can be passed to an appropriate schema further down in the hierarchies. During i n s t a n t i a t i o n , advisory messages can be sent (for e f f i c i e n c y purposes) l a t e r a l l y to neighbouring schemata which require knowledge of their a t t r i b u t e s . In building the semantic network messages are sent up and down the hierarchies. Chapter 6. Results and Evaluation 205 Even though schemata must "know" the names of the schemata they are sending messages t o , m o d u l a r i t y i s maintained to a l a r g e extent because the messages a l l have a uniform format. The sender s p e c i f i e s only the type of procedure i t wants to address, e i t h e r top-down or bottom-up, and the system f i n d s the a p p r o p r i a t e procedure. Furthermore, the context parameters i n c l u d e i n f o r m a t i o n about what type of i n f o r m a t i o n they encode, such as " r e g i o n 1629", " p o s i t i o n (52 . 70)", "sketchmapitem *road-5", and r e l y on the r e c e i v e r to e x t r a c t the i n f o r m a t i o n i t can u t i l i z e . P r evious s e c t i o n s have shown r e s u l t s that i n d i c a t e that the d e s c r i p t i o n s i n MISSEE are adequate f o r i n t e r p r e t i n g images. Moreover, they are adequate fo r combining f a c t s from d i s p a r a t e IS's i n a way that permits t h e i r ' e f f e c t i v e use i n the i n t e r p r e -t a t i o n p r o c e s s . T h i s i s accomplished through the e x p l i c i t n e s s , u n i f o r m i t y , and modularity of the schemata and the c l e a r d i s -t i n c t i o n made between s t e r e o t y p i c and h y p o t h e t i c a l knowledge. _ . ] _ • P r o c e d u r a l Adequacy By p r o v i d i n g s e v e r a l types of a t t a c h e d procedures f o r each schema, MISSEE ensures that the a p p r o p r i a t e method w i l l be used fo r i n t e r p r e t a t i o n . Since the procedures are a l g o r i t h m s , they are f l e x i b l e , easy to program (compared to a uniform processor o p e r a t i n g i n c o n j u n c t i o n with a d e c l a r a t i v e data s t r u c t u r e ) , and e f f e c t i v e at image i n t e r p r e t a t i o n . A l s o , because the procedures are t i g h t l y coupled with the d e c l a r a t i v e p o r t i o n of the schema Chapter 6. R e s u l t s and E v a l u a t i o n 206 (as opposed to a program and a separate data s t r u c t u r e ) , g r e a t e r m o d u l a r i t y i s maintained. Schemata cooperate by means of the messages that are passed between them. I t has been shown t h a t t h i s i s an e f f e c t i v e means of s h i f t i n g the focus of a t t e n t i o n to b u i l d the semantic net-work, i n t e r p r e t bottom l e v e l schemata, and p r o v i d e context f o r subsequent p r o c e s s i n g . Messages are uniform and i n c l u d e much of the i n f o r m a t i o n r e q u i r e d to decode them. Schemata maintain m o d u l a r i t y by r e l y -ing on the r e c e i v e r s ' of the message to determine the a p p r o p r i -ate procedure to u t i l i z e . Thus a schema does not need to s p e c i f y a s p e c i f i c bottom-up procedure f o r i n s t a n t i a t i o n d i r e c t l y from an I S - - i . e . whether to use s k e t c h map guidance i n s t e a d of the i n t e n s i t y image a l o n e . Rather, the message i s sent to the top-down procedure which takes the c u r r e n t context i n t o account in determining which bottom-up procedure to use. More procedures might be added to a schema (perhaps d e a l i n g with new IS's) and only the r e c e i v e r of the message would have to be m o d i f i e d to i n c o r p o r a t e the new knowledge. The c o n t r o l regime (the c y c l e of p e r c e p t i o n and the execu-t i o n c y c l e ) a l s o maintains p r o c e d u r a l adequacy. I t allows the user to give and r e c e i v e i n f o r m a t i o n at t i m e l y i n t e r v a l s so that he/she can i n f l u e n c e p r o c e s s i n g . Furthermore, i t ranks the i n t e r p r e t a t i o n p o s s i b i l i t i e s f o r e f f i c i e n c y reasons, even though a complete search of a l l the r e g i o n s , sketch map i n s t a n c e s , and Chapter 6. R e s u l t s and E v a l u a t i o n 207 the l i k e would be s u f f i c i e n t . 6.7.J_. S o l u t i o n s to Problems i n Matching Data to Models S e c t i o n 3.2 d e s c r i b e d four problems in matching data to models i n model-based v i s i o n systems. The adequacy of a v i s i o n system i s p a r t l y determined by how w e l l i t s o l v e s those prob-lems. In i n c o n s i s t e n t s i t u a t i o n s (more than one datum f o r a model), h y p o t h e t i c a l models would be c r e a t e d to account f o r each data element, one-to-one. A d d i t i o n a l i n f o r m a t i o n , e s p e c i a l l y from a d i f f e r e n t IS, may show that one instance does not f i t the model w e l l enough and cause i t to be pruned from the network. If pruning does not reduce the number of i n s t a n c e s to one, those that remain can be ranked by t h e i r confidence values which s p e c i f y how c l o s e l y the f e a t u r e s from the IS's f i t the model. As an example, c o n s i d e r the case where one s i d e of a b r i d g e has been f i x e d (to edge-1) but there are two c a n d i d a t e s f o r the other s i d e (with edge-2 and edge-3 as data elements). Since the s i d e s of a bridge should be p a r a l l e l , the o r i e n t a t i o n s of the edges can be used to choose between a l t e r n a t i v e s . If the o r i e n -t a t i o n of e i t h e r edge-2 or edge-3 was too d i s s i m i l a r to that of edge-1, then the in s t a n c e of which i t was a p a r t can be des-t r o y e d . Otherwise, the s i m i l a r i t i e s can be used i n the c a l c u l a -t i o n of the confidence values so that the i n s t a n c e s can be ordered i n a meaningful way. Chapter 6. R e s u l t s and E v a l u a t i o n 208 The ranking i n terms of c o n f i d e n c e values has another consequence with r e s p e c t to the execution c y c l e . Any a d v i c e ( i n the form of messages) that i s i s s u e d from the procedures a t t a c h e d to the two i n s t a n c e s w i l l a l s o be ranked by the c o n f i -dence v a l u e s . T h i s means,, sub j e c t to user i n f l u e n c e , t h a t the most l i k e l y p o s s i b i l i t i e s are examined f i r s t . In the case where a model i s u n s a t i s f i e d (no corresponding data element) context can be a f a c t o r . I f there i s evidence that i n d i c a t e s the s t r o n g l i k e l i h o o d of the e x i s t e n c e of a model, then the t h r e s h o l d s f o r a c c e p t i n g f e a t u r e s can be lowered so that i t i s " e a s i e r " to i n s t a n t i a t e the model. If c r u c i a l data elements ( f o r example, the curbs of a road) can s t i l l not be found, then the h y p o t h e t i c a l i n s t a n c e w i l l be pruned. If the f e a t u r e i s not d e f i n i t i v e , then the confidence value of the r e s u l t i n g i n s t a n c e would be lowered a p p r o p r i a t e l y . Ambiguous s i t u a t i o n s (more than one model f o r a data e l e -ment) are handled i n a manner s i m i l a r to i n c o n s i s t e n t ones. Each data element w i l l cause the c r e a t i o n of a h y p o t h e t i c a l i n s t a n c e . Other i n f o r m a t i o n w i l l e i t h e r r e s u l t in some of the i n s t a n c e s being pruned or w i l l r a i s e (or lower) t h e i r c o n f i d e n c e v a l u e s . For example, an ambiguous curb might be part of e i t h e r a road or a b r i d g e . The i n t e n s i t i e s of the neighbouring regions can be used to disambiguate the s i t u a t i o n . Bridges are next to water which has a lower average i n t e n s i t y than urban regions which are g e n e r a l l y found next to roads. Chapter 6. R e s u l t s and E v a l u a t i o n 209 .The f i n a l problem, incompleteness (no models to account f o r the d a t a ) , has not been s o l v e d . Models and t h e i r a t t a c h e d pro-cedures must be c r e a t e d by the user. Generic knowledge i s s t a t i c d u r i n g the i n t e r p r e t a t i o n p r o c e s s [ 3 ] . To c r e a t e new models as the r e s u l t of i n t e r p r e t a t i o n would r e q u i r e the system to be capable of a b s t r a c t i o n and l e a r n i n g . While t h i s system might provide a convenient framework f o r such r e s e a r c h , i t was not attempted. 6 .8 . Robustness The r e s u l t s d e s c r i b e d p r e v i o u s l y show that MISSEE degrades g r a c e f u l l y when i t i s unable to i n t e r p r e t an image completely ( c f . S e c t i o n 3 . 5 ) . Moreover, the degradation of the r e s u l t s corresponds to the amount of i n f o r m a t i o n a v a i l a b l e from the IS's. I f the user maintains c o n t r o l over the i n t e r p r e t a t i o n , then MISSEE can produce p e r f e c t r e s u l t s . The robustness of the sys-tem ensures that as the user chooses to impart l e s s i n f o r m a t i o n , the accuracy of the r e s u l t s d i m i n i s h e s g r a d u a l l y . If guidance from the sketch map i s not a v a i l a b l e , the value of the r e s u l t s decreases fu r t h e r . . However, even i f the only source of informa-t i o n a v a i l a b l e i s the d i g i t i z e d image, p a r t i a l r e s u l t s w i l l be produced. [3]A user with knowledge of the schemata manipulation func-t i o n s can c r e a t e or modify schemata d u r i n g i n t e r p r e t a t i o n by es-c a p i n g i n t o L i s p . Chapter 6. R e s u l t s and E v a l u a t i o n 210 T h i s range in the r e s u l t s and the robustness of the system has been i l l u s t r a t e d i n p r e v i o u s s e c t i o n s . Appendix E i l l u s -t r a t e s the case where the user ensures that the r e s u l t s are per-f e c t (even though he only provided i n f o r m a t i o n i n 6 out of the 36 c y c l e s of e x e c u t i o n ) . S e c t i o n 6.1 i n d i c a t e d how good r e s u l t s were ob t a i n e d using the guidance of a sketch map to h e l p i n t e r -p r et an image ( e r r o r r a t e , 9%). The r e s u l t s when using the d i g i t i z e d image alone were the s u b j e c t of s e c t i o n 6.2 (where the e r r o r r a t e was found to be 24%). The robustness of MISSEE i s one of i t s main accomplish-ments. I t supports the b a s i c c o n j e c t u r e s about m u l t i p l e i n f o r -mation sources. The more i n f o r m a t i o n and the more d i f f e r e n t the i n f o r m a t i o n that i s u t i l i z e d by the system, the b e t t e r w i l l be the r e s u l t s of the i n t e r p r e t a t i o n . Chapter 6. R e s u l t s and E v a l u a t i o n CHAPTER 7 Summary and C o n c l u s i o n s 7.j_. A Summary o f What i s New and I n t e r e s t i n g 1 ) M u l t i p l e I n f o r m a t i o n S o u r c e s : The u n i f y i n g c o n j e c t u r e of t h i s work has been t h e p o s s i b i l i t y and u s e f u l n e s s of c o m b i n i n g s e v e r a l , p o s s i b l y d i s p a r a t e , t y p e s of i n p u t i n f o r m a t i o n s o u r c e s t o a i d i n t h e i n t e r p r e t a t i o n o f v i s u a l images. O t h e r r e s e a r c h -e r s have u s e d more t h a n one i n f o r m a t i o n s o u r c e , but t h i s work r a i s e s t h e i s s u e of how t o e f f e c t i v e l y combine I S ' s t o one of c e n t r a l i m p o r t a n c e . One way t o combine d a t a t h a t v a r y i n k i n d and i n t h e i r sym-b o l i c n a t u r e i s t o use an o b j e c t - o r i e n t e d d a t a s t r u c t u r e . MIS-SEE i s a schema-based v i s i o n s y s t e m t h a t has been i m p l e m e n t e d t o t e s t t h e u s e f u l n e s s of t h e m u l t i p l e i n f o r m a t i o n s o u r c e c o n j e c -t u r e s . By u s i n g t h e i n f o r m a t i o n t h a t i s a v a i l a b l e t o i n t e r p r e t images, i t h a s p r o v e d e f f e c t i v e i n terms o f adequacy and r o b u s t -n e s s . The d e s i g n of MISSEE l e d t o t h e d e v e l o p m e n t o f s e v e r a l t o o l s w h i c h a r e u s e f u l i n r e c o n c i l i n g d i f f e r e n t t y p e s of i n f o r -m a t i o n t o s o l v e many of t h e p r o b l e m s i n m o d e l - b a s e d v i s i o n . T h e r e a r e two c o n j e c t u r e s m o t i v a t i n g t h i s work. W h i l e t h e r e i s g e n e r a l a r g r e e m e n t a b o u t t h e u s e f u l n e s s of t h e f i r s t h y p o t h e s i s (more i s b e t t e r ) , t h e s e c o n d ( t h e more d i f f e r e n t , t h e b e t t e r ) i s h a r d e r t o s u b s t a n t i a t e . S i n c e t h e r e i s no s a t i s f a c -21 1 212 t o r y d e f i n i t i o n of i n f o r m a t i o n i n images, i t i s d i f f i c u l t t o d e t e r m i n e how d i s p a r a t e two i n f o r m a t i o n s o u r c e s a r e . N o n e t h e -l e s s , i n t u i t i v e l y a s k e t c h map and an image a r e more d i f f e r e n t t h a n two images r e c o r d e d f r o m d i f f e r e n t v i e w p o i n t s . I n f o r m a t i o n p r o v i d e d by a u s e r i s even more d i f f e r e n t t h a n t h a t f o u n d i n i m a g e s. W h i l e i t i s p o s s i b l e t o e x t r a c t f r o m images a l o n e a l l o f t h e s c e n e i n f o r m a t i o n t h a t MISSEE i s a b l e t o d e r i v e , i t i s m a i n t a i n e d t h a t t h e r e s u l t s i n d i c a t e t h e a d v a n t a g e of c o m b i n i n g t h e d i s p a r a t e s o r t s of i n f o r m a t i o n a v a i l a b l e f r o m t h e v a r i o u s s o u r c e s . .2)Schemata: A new v a r i e t y o f schemata has been implemented a l o n g w i t h a s y s t e m f o r m a n i p u l a t i n g them (MAIDS). T h e s e o b j e c t s a d a p t many of t h e s t r u c t u r i n g n o t i o n s of schemata d e v e l o p e d f o r e x p e r t s y s t e m s and n a t u r a l l a n g u a g e u n d e r s t a n d i n g p r o g r a m s t o t h e s p e c i f i c n eeds o f v i s i o n s y s t e m s . In p a r t i c u -l a r , v i s i o n s y s t e m s r e q u i r e i n s t a n c e s t o be h y p o t h e t i c a l a t a l l l e v e l s o f t h e h i e r a r c h i e s and d i s t i n c t f r o m s t a t i c , g e n e r i c o b j e c t s . A l s o , m u l t i p l e r e l a t i o n s h i p s between o b j e c t s a r e n e c e s s a r y f o r t h e i n t e r p r e t a t i o n t a s k s t h a t have been examined. MAIDS schemata p r o v i d e a c o n s i s t e n t , u n i f o r m r e p o s i t o r y f o r i n f o r m a t i o n coming from d i f f e r e n t i n p u t I S ' s . The a t t a c h e d p r o -c e d u r e s a r e a f l e x i b l e means of d r i v i n g t h e i n t e r p r e t a t i o n p r o -c e s s . A l s o , t h e c o n s t r u c t i o n o f a s e m a n t i c n e t w o r k l i n k i n g a l l o f t h e i n s t a n c e s v i a t h e r e l a t i o n s s p e c i a l i z e s t o , decomposes t o , and n e i g h b o u r s , a l l o w s f o r a method of c o n t r o l ( t h e e x e c u -t i o n c y c l e ) t h a t r e d u c e s s e a r c h , a l l o w s f o r u s e r i n t e r a c t i o n , C h a p t e r 7. Summary and C o n c l u s i o n s 213 and promotes c o o p e r a t i o n among the schemata. 3) Using Sketch Maps: Sketch map a n a l y s i s has been used as a v i s i o n problem for some time. Sketch maps are quick and easy to draw even i f they must be superimposed over an image, yet they c o n t a i n much u s e f u l i n f o r m a t i o n about the u n d e r l y i n g scene. T h i s work i s the f i r s t to use sketch map i n f o r m a t i o n to help i n t e r p r e t r e a l imagery i n a geographic domain. 4) Combining Edge and Region Data: Image a n a l y s i s has usu-a l l y c o n c e n t r a t e d on the d e r i v a t i o n of image d i s c o n t i n u i t i e s (edges) or homogeneities ( r e g i o n s ) . Because i n p r a c t i c e they are d e r i v e d from d i f f e r e n t aspects of the image, these two r e p r e s e n t a t i o n s provide d i f f e r e n t types of i n f o r m a t i o n . Follow-ing the m u l t i p l e i n f o r m a t i o n sources c o n j e c t u r e , MISSEE i s one of the f i r s t v i s i o n systems to u t i l i z e both edge and region data. 5) Mixed C o n t r o l S t r a t e g i e s : S t r i c t l y l i n e a r models of con-t r o l i n e f f e c t i v e l y u t i l i z e a l l of the knowledge that i s a v a i l -a b l e i n a model-based v i s i o n system. A t o t a l l y bottom-up system does not g e n e r a l l y take advantage of model knowledge. A top-down system w i l l not u s u a l l y be able to use the context r e s u l t -ing from i n f o r m a t i o n gained from previous p r o c e s s i n g . MISSEE uses a mixed c o n t r o l s t r a t e g y (both top-down and bottom-up) that combines with the s t r u c t u r e of the semantic network so that a t t e n t i o n i s focussed where i t can be most u s e f u l l y a p p l i e d among the c o o p e r a t i n g schemata. The embodiment of t h i s s t r a t e g y Chapter 7. Summary and C o n c l u s i o n s 214 i s the e x e c u t i o n c y c l e which a l s o a l l o w s the user an o p p o r t u n i t y to give and r e c e i v e i n f o r m a t i o n at a p p r o p r i a t e p o i n t s although i t i s not necessary. If d e s i r e d , the user can r e l a t e informa-t i o n of v a r i o u s types to the system and modify the l i s t of pos-s i b i l i t i e s f o r i n t e r p r e t a t i o n to r e f l e c t h i s / h e r own p r i o r i t i e s . The e x e c u t i o n c y c l e d i s t r i b u t e s c o n t r o l w i t h i n the semantic network. Messages d i r e c t e d at "bottom l e v e l " schemata can con-t a i n l o c a t i o n s i n e i t h e r the sketch map or image. Advice about l o c a l i t y reduces the amount of search that must take p l a c e to d i s c o v e r the geographic e n t i t i e s of i n t e r e s t . T h i s r e s u l t s i n i n c r e a s e d e f f i c i e n c y i n c o n t r a s t to the advances mentioned above which a l l concern the s u f f i c i e n c y or i n t e r p r e t i v e power of the system. Open I ssues There are s e v e r a l aspects of the m u l t i p l e i n f o r m a t i o n sources paradigm that have not been addressed. T h i s r e s e a r c h has d e a l t with one way of combining I S ' s - - u s i n g schemata as both a data s t r u c t u r e and o b j e c t - o r i e n t e d i n t e r p r e t a t i o n . There are s e v e r a l other candidates that have been used in Al that might a l s o prove e f f e c t i v e . For example, medical d i a g n o s i s systems that use p r o d u c t i o n r u l e s combine d i f f e r e n t types of t e s t i n f o r -mation in a non-perceptual domain. When comparing d i f f e r e n t approaches to u t i l i z i n g m u l t i p l e i n f o r m a t i o n sources, a method of e v a l u a t i o n , would be of c o n s i d -e r a b l e a i d . If one c o u l d f o r m a l i z e the n o t i o n s of the amount of Chapter 7. Summary and C o n c l u s i o n s 215 i n f o r m a t i o n a v a i l a b l e in an IS, then i t would be p o s s i b l e to determine the e f f e c t i v e n e s s of a method for e x t r a c t i n g and using the i n f o r m a t i o n . Such a formalism would have to account f o r the f a c t that i n f o r m a t i o n from d i f f e r e n t IS's i s not always cumula-t i v e and might be redundant or i n c o n s i s t e n t . With regard to the system that was developed, an open ques-t i o n e x i s t s concerning i t s e x p a n d a b i l i t y . Claims of modularity notwithstanding, i t i s unclear how easy and u s e f u l i t would be i n g e n e r a l to add e i t h e r new o b j e c t s or i n f o r m a t i o n sources. Ongoing experience with adding elements to the developing system was q u i t e p o s i t i v e . However, only continued experimentation w i l l i n d i c a t e whether there are o b j e c t s and IS's that are d i f f i -c u l t to i n c o r p o r a t e (perhaps c l o u d s ? ) and whether i n c r e a s e d s i z e w i l l l e a d to an overwhelming i n c r e a s e i n complexity. 2-3_. Future D i r e c t i o n s There are s e v e r a l extensions to the present system that would make i t more e f f e c t i v e , more e f f i c i e n t , or more modular. l ) R e l a x i n g the R e g i s t r a t i o n C o n s t r a i n t : In the c u r r e n t system, the sketch map i s assumed to be approximately r e g i s t e r e d to the image. T h i s g i v e s s p a t i a l i n f o r m a t i o n regarding the l o c a t i o n of o b j e c t s . I t should be p o s s i b l e to r e l a x t h i s con-s t r a i n t so that i f the system d i s c o v e r s or i s informed that the two IS's are not a l i g n e d , i t w i l l use r e l a t i v e i n s t e a d of abso-l u t e s p a t i a l i n f o r m a t i o n (e.g.. "There i s a town to the l e f t of a road that runs over a b r i d g e " ) . Chapter 7. Summary and C o n c l u s i o n s 216 2) Feedback to Mapsee2: Mapsee2 i s run to completion on the sketch map and sends a l l of i t s r e s u l t s to MISSEE. As has been e x p l a i n e d ( S e c t i o n 5.3.1), MISSEE can p r o v i d e much in f o r m a t i o n that i s u s e f u l to h e l p disambiguate the sketch map. Since Maya schemata, used in Mapsee2, are s i m i l a r to MAIDS schemata, used in MISSEE, i t should be p o s s i b l e f o r them to cooperate so that i n t e r p r e t a t i o n of the image and the sketch map can proceed simultaneously to t h e i r mutual b e n e f i t . However, the methods of c o n t r o l i n Mapsee2 and MISSEE are s u b s t a n t i a l l y d i f f e r e n t , so c o n s i d e r a b l e work would be r e q u i r e d to c o o r d i n a t e them. 3) A B e t t e r User I n t e r f a c e : MISSEE's u s e f u l n e s s as a demonstration system would be improved with the a d d i t i o n of a n a t u r a l language i n t e r f a c e and a mechanism f o r g r a p h i c a l input, such as a l l o w i n g the user to p o i n t to p a r t s of the image or sketch map. 4) A More D e c l a r a t i v e Schemata: I t i s p o s s i b l e to move some of the model knowledge from the attached procedures i n t o the d e c l a r a t i v e p a r t of the schemata. Moving the knowledge would make i t more a c c e s s i b l e , m o d i f i a b l e , and uniform, but would cause the i n s t a n t i a t i o n method to be l e s s f l e x i b l e . There i s a t r e n d i n other areas of Al towards more e x p l i c i t , formal d e s c r i p t i o n s of o b j e c t s . The framework of MISSEE would f a c i l i -t a t e such a move. 5) E f f i c i e n t l y F i n d i n g Edges and Regions: In S e c t i o n 5.3.1.2, i t was p o i n t e d out that edge d e t e c t i o n and region merg-Chapter 7. Summary and C o n c l u s i o n s 217 ing were both l o c a l o p e r a t i o n s that can be a p p l i e d to s e l e c t e d p o r t i o n s of an image. In a d d i t i o n , to reduce search, MISSEE's c o n t r o l mechanism s h i f t s the focus of a t t e n t i o n i n the image based on the c u r r e n t context ( c f . S e c t i o n 6.4). Thus one can c o n f i n e the expensive o p e r a t i o n s of image a n a l y s i s to s p e c i f i c p a r t s of the image and only perform them when needed. T h i s would r e q u i r e two changes to the c u r r e n t system: 1) one would have to r e f e r to p o r t i o n s of the image i n a new way (MISSEE gen-e r a l l y uses the a l r e a d y c a l c u l a t e d region numbers), and 2) boun-dary e f f e c t s between neighbouring subimages would have to be co n s i d e r e d . 1_. 4. Conclusions Schema-based systems are capable of s u f f i c i e n t l y d e s c r i b i n g the a t t r i b u t e s of geographic e n t i t i e s f o r t h e i r i n t e r p r e t a t i o n , as w e l l as d i r e c t i n g the i n t e r p r e t a t i o n process i t s e l f . E x p l o i t i n g i n f o r m a t i o n sources i n a c o o p e r a t i v e f a s h i o n g i v e s r e s u l t s q u a l i t a t i v e l y b e t t e r than those o b t a i n a b l e from i n t e r -p r e t i n g the i n f o r m a t i o n sources s e p a r a t e l y . Chapter 7. Summary and Conclusi o n s 218 References [1] Agin, G.J. and B i n f o r d , T.O. (1973), Computer D e s c r i p t i o n of Curved Objects, Proc. T h i r d I n t e r n a t i o n a l J o i n t Confer-ence on A r t i f i c i a l I n t e l l i g e n c e , S t a n f o r d , 629-640. [2] Agin, G.J. and Duda, R.O. (1975), SRI V i s i o n Research f o r Advanced I n d u s t r i a l Automation, Proc. Second USA-Japan  Computer Conference, Tokyo, 113-117. [3] A n s t i s , S.M. (1970), Phi Movement as a S u b t r a c t i o n Process, V i s i o n Research 10(12), 1411-1430. [4] Bajcsy, R. and Lieberman, L . I . (1974), Computer D e s c r i p t i o n of Real Outdoor Scenes, Proc. Second I n t e r n a t i o n a l  J o i n t Conference on P a t t e r n R e c o g n i t i o n , Copenhagen, 174-179. [5] Bajcsy, R. and J o s h i , A.K. (1978), A P a r t i a l l y Ordered World Model and N a t u r a l Outdoor Scenes, Computer V i s i o n Sys-tems , A.R. Hanson & E.M. Riseman ( e d s . ) , Academic Press, New York, 263-270. [6] Bajcsy, R. and T a v a k o l i , M. (1973), A Computer Re c o g n i t i o n of B r idges, I s l a n d s , R i v e r s and Lakes from S a t e l l i t e Pictures," Proc. Conf. on Machine P r o c e s s i n g of Remotely  Sensed Data, Lab. f o r A p p l i c a t i o n s of Remote Sensing, Purdue, Indiana, 2A-54 to 2A-68. [7] Bajcsy, R. and T a v a k o l i , M. (1976), Computer R e c o g n i t i o n of Roads From S a t e l l i t e P i c t u r e s , Proc. Second I n t e r n a -t i o n a l J o i n t Conference on P a t t e r n R e c o g n i t i o n , Copenhagen, 190-194. [8] Baker, H.H. and B i n f o r d , T.O. (1981), Depth From Edge and I n t e n s i t y Based Stereo, Proc. Seventh I n t e r n a t i o n a l  J o i n t Conference on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 631-636. [9] B a l l a r d , D.H. (1981), Parameter Networks: Towards a Theory of Low-Level V i s i o n , Proc. Seventh I n t e r n a t i o n a l J o i n t  Conference on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 1068-1078. [10] B a l l a r d , D.H. and Brown, CM. (1982), Computer V i s i o n , P r e n t i c e H a l l , Englewood C l i f f s , New J e r s e y . References 219 [11] B a l l a r d , D.H., Brown, CM., and Feldman, J.A. (1978), An Approach to Knowledge-Directed Image A n a l y s i s , Computer  V i s i o n Systems, A.R. Hanson & E.M. Riseman (eds . ) , Academic Press, New York,. 271-282. [12] B a l l a r d , D.H. and Sabbah, D. (1981), On Shapes, Proc. Seventh I n t e r n a t i o n a l J o i n t Conference on A r t i f i c i a l  I n t e l l i g e n c e , Vancouver, 607-612. [13] Barr, A. and Feigenbaum, E.A. (eds.) (1981), The Handbook  of A r t i f i c i a l I n t e l l igence , Volume J_, W i l l i a m Kaufman, Los A l t o s , C a l i f o r n i a . [14] Barrow, H.G. and Tenenbaum, J.M. (1975), R e p r e s e n t a t i o n and Use of Knowledge i n V i s i o n , S i g a r t Newsletter 52, 2-8. [15] Barrow, H.G. and Tenenbaum, J.M. (1976), Msys: A System f o r Reasoning About Scenes, TN 121, AI Center, SRI. [16] Barrow, H.G. and Tenenbaum, J.M. (1978), Recovering I n t r i n -s i c Scene C h a r a c t e r i s t i c s From Images, Computer V i s i o n  Systems, A.R. Hanson & E.M. Riseman ( e d s . ) , Academic Press, New York, 3-26. [17] B a r t l e t t , F.C. (1932), Remembering, A Study i n Experimental  and Soc i a l Psychology, Cambridge U n i v e r s i t y Press, Cam-bri d g e . [18] Bobrow, D.G. and Winograd, T. (1977), An Overview of KRL, A Knowledge Representation Language, C o g n i t i v e Science 1, 3-46. [19] B o l l e s , R.C. (1975), V e r i f i c a t i o n V i s i o n W i thin a Programm-abl e Assembly System: An I n t r o d u c t o r y D i s c u s s i o n , AIM 275, AI Lab, S t a n f o r d . [20] B o l l e s , R.C. (1979), Robust Feature Matching Through Maxi-mal C l i q u e s , Proc. S o c i e t y of P h o t o - O p t i c a l Instrumen-t a t i o n Engineers 182, Imaging A p p l i c a t i o n s f o r Automat-ed I n s p e c t i o n and Assembly, 140-149. [21] B o l l e s , R .C, Quam, L.H., F i s c h l e r , M.A., and Wolf, H.C. (1979), Automatic Determination of Image-to-Database Correspondences, Proc. S i x t h , I n t e r n a t i o n a l J o i n t  Conference on A r t i f i c i a l I n t e l l i g e n c e , Tokyo, 73-78. [22] Brachman, R. (1982), What "ISA" Is and I s n ' t , Proc. Fourth  Conf. of the Canadian Soc i e t y f o r Computat i o n a l S t u d i e s  of I n t e l l i g e n c e , Saskatoon, Canada, 212-221. [23] B r i c e , C R . and Fennema, C.L. (1970), Scene A n a l y s i s Using Regions, A r t i f i c i a l I n t e l l i g e n c e 1(3), 205-226. References 220 [24] Brooks, R.A. (1981a), Model-Based Three Dimensional In-t e r p r e t a t i o n s of Two Dimensional Images, Proc. Seventh  I n t e r n a t i o n a l J o i n t Conference on A r t i f i c i a l I n t e l l i -gence , Vancouver, 619-624. [25] Brooks, R.A. (-198lb). Symbolic Reasoning Among 3-D Models and 2-D Images, Art i f i c i a l I n t e l l i g e n c e 17(1-3), 285-348. [26] Brooks, R.A., G r e i n e r , R., and B i n f o r d , T.O. (1979), The ACRONYM Model-Based V i s i o n System, Proc. S i x t h I n t e r n a -t i o n a l J o i n t Conference on Art i f i c i a l I n t e l l i g e n c e , Tokyo, 105-113. [27] Brown, J.S. and Burton, R.R. (1975), M u l t i p l e Representa-t i o n s of Knowledge for T u t o r i a l Reasoning, Representa-t ion and Understanding, D.G. Bobrow & A. C o l l i n s Teds.), Academic Press, New York, 311-349. [28] Browse, R.A. (1982), I n t e r p r e t a t i o n - B a s e d I n t e r a c t i o n Between L e v e l s of D e t a i l , Proc. F o u r t h Conf. of the Canadian Soc i e t y f o r Computational S t u d i e s of I n t e l l i -qence, Saskatoon, Canada, 27-32. [29] Clowes, M.B. (1971), On Seeing T h i n g s , ' A r t i f i c i a l I n t e l l i -gence 2(1), 79-112. [30] C o l l i n s , A.M. and L o f t u s , E.F.A. (1975), A Spreading-A c t i v a t i o n Theory of Semantic P r o c e s s i n g , P s y c h o l o g i c a l  Review 82(6), 407-428. [31] Davis, L.S. (1975), A Survey of Edge D e t e c t i o n Techniques, Computer Graphics and Image P r o c e s s i n g 4(3), 248-270. [32] Davis, L.S. and Rosenfeld, A. (1981), Cooperating Processes for Low-level V i s i o n : A Survey, A r t i f i c i a l I n t e l l i g e n c e 17(1-3), 245-263. [33] Davis, R. (1980a), Meta-Rules: Reasoning about C o n t r o l , Ar-t i f i c i a l I n t e l l i g e n c e 15, 179-222. [34] Davis, R. (1980b), Report on the Workshop on D i s t r i b u t e d A l , SIGART Newsletter 73, 42-52. [35] Davis, R. (1982), Expert Systems: Where Are We? And Where Do We Go From Here?, AI_ Magazine 3(2), 3-22. [36] Duda, R.O. and Hart, P.E. (1972), Use of the Hough Transformation to Detect L i n e s and Curves i n P i c t u r e s , Communications of the ACM 15(1), 11-15. References 221 [37] Dudani, S.A. and Luk, A.L. (1977), L o c a t i n g S t r a i g h t - L i n e Edge Segments on Outdoor Scenes, Proc. P a t t e r n Recogni-t i o n and Image P r o c e s s i n g Conference, 367-377. [38] Eigen, D.J., Fromm, F.R., and Northouse, R.A. (1974), C l u s -t e r A n a l y s i s Based on Dimensional Information with Ap-p l i c a t i o n s to Feature S e l e c t i o n and C l a s s i f i c a t i o n , Systems, Man, and C y b e r n e t i c s SMC-4(3), 284-294. [39] Erman, L.D. and L e s s e r , V.R. (1975), A M u l t i - L e v e l Organi-z a t i o n f o r Problem S o l v i n g Using Many, D i v e r s e , Cooperating Sources of Knowledge, Proc. Fourth I n t e r n a -t i o n a l J o i n t Conference on A r t i f i c i a l I n t e l l i g e n c e , T b i l i s s i , 483-490. [40] Erman, L.D., Hayes-Roth, F., L e s s e r , V.R., and Reddy, D.R. (1980), The HEARSAY-11 Speech Understanding System: In-t e g r a t i n g Knowledge To Resolve U n c e r t a i n t y , Computing  Surveys 12(2), 213-253. [41] F a l k , G. (1972), I n t e r p r e t a t i o n of Line Data as a Three-Dimensional Scene, A r t i f i c i a l I n t e l l i g e n c e 3(2), 101-1 44. [42] Feigenbaum, E. A. (1977), Themes and Case S t u d i e s of Knowledge E n g i n e e r i n g , Proc. F i f t h I n t e r n a t i o n a l J o i n t  Conference on A r t i f i c i a l I n t e l l i g e n c e , Cambridge, Mass., 1014-1029. [43] F i s c h l e r , M.A. and E l s c h l a g e r , R.A. (1973), The Representa-t i o n and Matching of P i c t o r i a l S t r u c t u r e s , IEEE Trans. On Computers C-22(1), 67-92. [44] F i s c h l e r , M.A., Tenenbaum, J.M.,- and Wolf, H.C. (1981), D e t e c t i o n of Roads and L i n e a r S t r u c t u r e s i n Low-R e s o l u t i o n A e r i a l Imagery Using a M u l t i s o u r c e Knowledge I n t e g r a t i o n Technique, Computer Graphics and Image Pro-c e s s i n g 15(3), 201-223. [45] Flinchbaugh, B.E. and Chandrasekaran, B. (1981), A Theory of Spatio-Temporal Aggregation f o r V i s i o n , A r t i f i c i a l  I n t e l l i g e n c e 17(1-3), 387-407. [46] Fowler, R.J. and L i t t l e , J . J . (1979), Automatic E x t r a c t i o n of I r r e g u l a r Network D i g i t a l T e r r a i n Models, Proc. SI-GRAPH, Chicago, 199-207. [47] Freuder, E.C. (1976), A Computer System f o r V i s u a l Recogni-t i o n Using A c t i v e Knowledge, AI-TR-345, AI Lab, MIT. References 222 [48] Garvey, T.D., Lowrance, J.D., and F i s c h l e r , M.A. (1981), An Inference Technique f o r I n t e g r a t i n g Knowledge From D i s p a r a t e Sources, Proc. Seventh I n t e r n a t i o n a l J o i n t  Conference on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 319-325. [49] G e l e r t n e r , H. (1963), R e a l i z a t i o n of a Geometry-Theorem Proving Machine, Computers and Thought, E. Feigenbaum & J . Feldman (eds . ) , Mcgraw-Hill, New York, 134-152. [50] Glicksman, J . (1982), A Schemata-Based System f o r U t i l i z i n g C ooperating Knowledge Sources i n Computer V i s i o n , Proc. Fourth Conf. of the Canadian Soc i e t y f o r Computational  S t u d i e s of I n t e l l i g e n c e , Saskatoon, Canada, 33-40. [51] Goebel, R. (1977), O r g a n i z i n g F a c t u a l Knowledge i n a Seman-t i c Network, TR77-8, Dept. of Computer Science, U. of A l b e r t a . [52] Grimson, W.E.L. (1981), From Images to Su r f a c e s : A Computa-t i o n a l Study of the Human E a r l y V i s u a l System, MIT Press, Cambridge, Mass. [53] H a l l , E.L. (1979), Computer Image P r o c e s s i n g and Recogni-t i o n , Academic Press, New York. [54] Hanson, A.R. and Riseman, E.M. (1978a), Segmentation of Na-t u r a l Scenes, Computer V i s i o n Systems, A-.R. Hanson & E.M. Riseman (eds . ) , Academic Press, New York, 129-164. [55] Hanson, A.R. and Riseman, E.M. (1978b), VISIONS: A Computer System f o r I n t e r p r e t i n g Scenes, Computer V i s i o n Sys-tems , A.R. Hanson & E.M. Riseman (eds.), Academic Press, New York, 303-334. [56] Havens, W.S. (1978), A Pro c e d u r a l Model of Re c o g n i t i o n f o r Machine P e r c e p t i o n , TR-78-3, Dept. of Computer Scie n c e , U.B.C. [57] Havens, W.S. and Mackworth, A.K. (1980), Schemata-Based Understanding of Hand-Drawn Sketch Maps, Proc. T h i r d  Conf. of the Canadian Soc i e t y f o r Computat i o n a l S t u d i e s  of I n t e l l i g e n c e , V i c t o r i a , Canada, 172-178. [58] Hayes, P., B a l l , E., and Reddy, R. (1981), Breaking the Man-Machine Communication B a r r i e r , Computer 14(3), 19-30. [59] Hayes, P. and Reddy, R. (1979), G r a c e f u l I n t e r a c t i o n i n Man-Machine Communication, Proc. S i x t h I n t e r n a t i o n a l  J o i n t Conference on A r i t f i c i a l I n t e l l i g e n c e , Tokyo, 372-374. References 223 [60] H i l d r e t h , E.C. (1980), Implementation of a Theory of Edge D e t e c t i o n , AI-TR-579, AI Lab, MIT. [61] Hewitt, C , Bishop, P., and S t e i g e r , R. (1973), A U n i v e r s a l Modular ACTOR Formalism f o r A r t i f i c i a l I n t e l l i g e n c e , Proc. T h i r d I n t e r n a t i o n a l J o i n t Conference on A r t i f i -c i a l I n t e l l i g e n c e , S t a n f o r d , 235-245. [62] Horn, B.K.P. (1975), O b t a i n i n g Shape From Shading Informa-t i o n , The Psychology of Computer V i s i o n , P.H. Winston (ed.), McGraw-Hill, New York, 115-155. [63] Horn, B.K.P. and Bachman, B.L. (1978), Using S y n t h e t i c Im-ages to R e g i s t e r Real Images with Surface Models, Com-munications of the ACM 21(11), 914-924. [64] Huffman, D.A. (1971), Impossible Objects as Nonsense Sen-tences, Machine I n t e l l i g e n c e 6, B. Me l t z e r & D. Michie •(eds.), American E l s e v i e r , New York, 295-323. [65] J a i n , R. and Haynes, S. (1982), Imprecision i n Computer V i -s i o n , Computer 15(8), 39-48. [66] Kanade, T. (1980), Region Segmentation: S i g n a l vs. Seman-t i c s , Computer Graphics and Image P r o c e s s i n g 13, 279-297. [67] Kanade, T. (1981), Recovery of the Three-Dimensional Shape of an Object from a S i n g l e View, A r t i f i c i a l I n t e l l i -gence 17(1-3), 409-460. [68] K e l l y , M.D. (1971), Edge D e t e c t i o n i n P i c t u r e s by Computer Using P l a n n i n g , Machine I n t e l l i g e n c e 6, B. Me l t z e r & D. Michi e (e d s . ) , American E l s e v i e r , New York, 397-409. [69] Render, J.R. (1980), Shape from Texture, CMU-CS-81 -102, Dept. of Computer Science, CMU. [70] Koomen, J.A.G.M. (1980), The I n t e r l i s p V i r t u a l Machine: A Study of i t s Design and i t s Implementation as M u l t i l -i s p , M.Sc. d i s s e r t a t i o n , Dept. of Computer Science, U.B.C. [71] Kuhn, T.S. (1970), The S t r u c t u r e of S c i e n t i f i c R e v o l u t i o n s , e d i t i o n 2, U n i v e r s i t y of Chicago Press, Chicago. [72] K u i p e r s , B.J. (1977), Representing Knowledge of Large-Scale Space, AI-TR-418, AI Lab, MIT . [73] Leboucher, G. and Lowitz, G.E. (1978), What a Histogram Can R e a l l y T e l l the C l a s s i f i e r , P a t t e r n R e c o g n i t i o n 10(4), 351-357. References 224 [74] Levesque, H. and Mylopoulos, J . (1979), A P r o c e d u r a l Seman-t i c s f o r Semantic Networks, A s s o c i a t i v e Networks: The  Representation and Use of Knowledge by Computer, N.V. F i n d l e r (ed.), Academic Press, New York, 93-120. [75] L e v i n e , M.D. and Shaheen, S.I. (1981), A Modular Computer V i s i o n System f o r P i c t u r e Segmentation and I n t e r p r e t a -t i o n , P a t t e r n A n a l y s i s and Machine I n t e l l i g e n c e PAMI-3(5), 540-556. [76] L i l l e s a n d , T.M. and K i e f e r , R.W. (1979), Remote Sensing and  Image I n t e r p r e t a t i o n , John Wiley and Sons, New York. [77] L i t t l e , J . J . (1982), Automatic R e g i s t r a t i o n of Landsat MSS Images to D i g i t a l E l e v a t i o n Models, Proc. Workshop on  Computer V i s i o n : R e p r e s e n t a t i o n and C o n t r o l , Rindge, New Hampshire, 178-184. [78] Mackworth, A.K. (1976), Model-Driven I n t e r p r e t a t i o n i n In-t e l l i g e n t V i s i o n Systems, Percept ion 5, 349-370. [79] Mackworth, A.K. (1977a), Consistency i n Networks of Rela-t i o n s , A r t i f i c i a l I n t e l l i g e n c e 8(1), 99-118. [80] Mackworth, A.K. (1977b), How to See a Simple World: an Exe-g e s i s of Some Computer Programs for Scene A n a l y s i s , Machine I n t e l l i g e n c e 8, E.W. Elco c k & D. Mi c h i e (eds.), John Wiley, New York, 510-540. [81] Mackworth, A.K. (1977c), On Reading Sketch Maps, Proc. F i f t h I n t e r n a t i o n a l J o i n t Conference on Art i f i c i a l In-t e l l i g e n c e , Cambridge, Mass., 598-606. [82] Mackworth, A.K. (1978), V i s i o n Research S t r a t e g y : Black Magic, Metaphors, Mechanisms, Miniworlds and Maps, Com-puter V i s i o n Systems, A.R. Hanson & E.M. Riseman (e d s . ) , Academic Press, New York, 53-61. [83] Mackworth, A.K. and Havens, W.S. (1981), S t r u c t u r i n g Domain Knowledge f o r V i s u a l P e r c e p t i o n , Proc. Seventh I n t e r n a -t i o n a l J o i n t Conference on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 625-627. [84] Marr, D. (1976), E a r l y P r o c e s s i n g of V i s u a l Information, P h i l . Trans. Royal S o c i e t y of London 275B(942), 483-524. [85] Marr, D. (1982), V i s i o n : A Computational I n v e s t i g a t i o n i n t o  the Human Repres e n t a t i o n and P r o c e s s i n g of V i s u a l In-formation, W.H. Freeman, San F r a n c i s c o . [86] Marr, D. and H i l d r e t h , E.C. (1980), Theory of Edge Detec-t i o n , Proc. Royal S o c i e t y of London B(207), 187-217. References 225 [87] Marr, D. and N i s h i h a r a , H.K. (1978), Representation and R e c o g n i t i o n of the S p a t i a l O r g a n i z a t i o n of Three Dimen-s i o n a l S t r u c t u r e , Proc. Royal Soc i e t y of London B(200), 269-294. [88] Minsky, M. (1975), A Framework f o r Representing Knowledge, The Psychology of Computer V i s i o n , P.H. Winston (ed. ) , McGraw-Hill, New York, 211-277. [89] Mulder, J.A. (1979), R e p r e s e n t a t i o n and C o n t r o l i n a Pro-gram that Understands L i n e Sketches of a House, M.Sc. d i s s e r t a t i o n , Dept. of Computer Science, U.B.C. [90] Nagao, M., Matsuyama, T., and Ikeda, Y. ( 1 9 7 8 ) , Region Ex-t r a c t i o n and Shape A n a l y s i s of A e r i a l Photographs, Proc. Fourth I n t e r n a t i o n a l J o i n t Conference on P a t t e r n  R e c o g n i t i o n , Kyoto, 620-628. [91] Nagao, M., Matsuyama, T., and Mori, H. (1979), S t r u c t u r a l A n a l y s i s of Complex A e r i a l Photographs, Proc. S i x t h  I n t e r n a t i o n a l J o i n t Conference on Ar i t f i c i a l I n t e l l i -gence , Tokyo, 610-616. [92] N a z i f , A.M. and Levine, M.D. (1982), S e t t i n g S t r a t e g y i n a Rule Based V i s i o n System, Proc. Fourth Conf. of the  Canadian Soc i e t y f o r Computational S t u d i e s of I n t e l l i -gence , Saskatoon, Canada, 52-59. [93] N e i s s e r , U. (1967), C o g n i t i v e Psychology, Meredith, New York. [94] N e i s s e r , U. (1976), C o g n i t i o n and R e a l i t y : P r i n c i p l e s and  I m p l i c a t i o n s of C o g n i t i v e Psychology, W.H. Freeman, San F r a n c i s c o . [95] Nevatia, R. and P r i c e , K.E. (1978), L o c a t i n g S t r u c t u r e s i n A e r i a l Images, Proc. Fourth I n t e r n a t i o n a l J o i n t Confer-ence on Pattern R e c o g n i t i o n , Kyoto, 686-690. [96] Nevatia, R. and P r i c e , K.E. (1982), L o c a t i n g S t r u c t u r e s i n A e r i a l Images, P a t t e r n A n a l y s i s and Machine I n t e l l i -gence PAMI-4(5), 476-484. [97] N i i , H.P., Feigenbaum, E.A., Anton, J . J . , and Rockmore, A.J. (1982), Signal-to-Symbol T r a n s f o r m a t i o n : HASP/SIAP Case Study, AI_ Magazine 3(2), 23-35. [98] N i l s s o n , N. J . (1971), Problem-Solving Methods _n A r t i f i -c i a l I n t e l l i g e n c e , McGraw-Hill, New York. [99] N i l s s o n , N. J . (1980), P r i n c i p l e s of A r t i f i c i a l I n t e l l i -gence , Tioga, Palo A l t o , C a l i f o r n i a . References 226 [100] O'Gorman, F. and Clowes, M.B. (1973), F i n d i n g P i c t u r e Edges Through C o l l i n e a r i t y of Feature P o i n t s , Proc. T h i r d I n t e r n a t i o n a l J o i n t Conference on A r t i f i c i a l In-t e l l i g e n c e , S t a n f o r d , 543-555. [101] Ohlander, R.B. (1975), A n a l y s i s of N a t u r a l Scenes, Ph.D. d i s s e r t a t i o n , Dept. of Computer Sci e n c e , CMU. [102] Ohta, Y., Kanade, T., and Sakai, T. (1978), An A n a l y s i s System f o r Scenes C o n t a i n i n g O b j e c t s with Substruc-t u r e s , Proc. Fourth I n t e r n a t i o n a l J o i n t Conference on P a t t e r n R e c o g n i t i o n , Kyoto, 752-754. [103] Palmer, S. E. (1975), V i s u a l P e r c e p t i o n and World Knowledge: Notes on a Model of Sensory-Cognitive In-t e r a c t i o n , E x p l o r a t ions i n Cogn i t i o n , D.A. Norman & D.E. Rumelhart (eds.), W.H. Freeman, San F r a n c i s c o , 279-307. [104] P o s t a i r e , J.-G. and Vasseur, CP.A. (1981), An Approximate S o l u t i o n to Normal Mixture I d e n t i f i c a t i o n with A p p l i c a -t i o n to Unsupervised P a t t e r n C l a s s i f i c a t i o n , P a t t e r n  A n a l y s i s and Machine I n t e l l i g e n c e PAMI-3(2), 163-179. [105] Ramer, U. (1972), An I t e r a t i v e Procedure f o r the Polygonal Approximation of Planar Curves, Computer Graphics and  Image P r o c e s s i n g 1(3), 244-256. [106] R i c h , E. (1979a), B u i l d i n g and E x p l o i t i n g User Models, Proc• S i x t h I n t e r n a t i o n a l J o i n t Conference on A r t i f i -c i a l I n t e l l i g e n c e , Tokyo, 720-722. [107] Rich, E. (1979b), B u i l d i n g and E x p l o i t i n g User Models, CMU-CS-79-119, Dept. of Computer Sci e n c e , CMU. [108] Roberts, L.G. (1965), Machine P e r c e p t i o n of Three Dimen-s i o n a l S o l i d s , O p t i c a l and E l e c t r o - o p t i c a l Information  P r o c e s s i n g , J.T. T i p p e t t , D. Berkowitz, L. Clapp, C. Koester, & A. Vanderburgh ( e d s . ) , MIT Press, Cambridge, Mass., 159-197. [109] Roberts, R.B. and G o l d s t e i n , I.P. (1977a), The FRL Manual, AIM 409, AI Lab, MIT. [110] Roberts, R.B. and G o l d s t e i n , I.P. (1977b), The FRL Primer, AIM 408, AI Lab, MIT. [111] Robson, D. and Goldberg, A. (1981), The S m a l l t a l k - 8 0 Sys-tem, Byte 6(8), 36-47. [112] Rosenberg, S.T. (1977), Frame-Based Text P r o c e s s i n g , AIM 431, AI Lab, MIT. References 227 [113] Rosenthal, D. and Bajcsy, R. (1978), Conceptual and V i s u a l F o c u s s i n g i n the R e c o g n i t i o n Process as Induced by Q u e r i e s , Proc. Fourth I n t e r n a t i o n a l J o i n t Conference on P a t t e r n R e c o g n i t i o n , Kyoto, 417-420. [114] R u s s e l l , D.M. (1979), Where Do I Look Now?: M o d e l l i n g and I n f e r r i n g Object L o c a t i o n s by C o n s t r a i n t s , Proc. IEEE  Comp. Soc i e t y Conf. on P a t t e r n R e c o g n i t i o n and Image  P r o c e s s i n g , Chicago, 175-181. [115] S a k a i , T. , Kanade, T., and Ohta, Y. (1976), Model-Based I n t e r p r e t a t i o n of Outdoor Scenes, Proc. T h i r d I nterna-t i o n a l J o i n t Conference on P a t t e r n R e c o g n i t i o n , Corona-da, C a l i f o r n i a , 581-585. [116] Schacter, B.J., Davis, L.S., and Rosenfeld, A. (1976), Scene Segmentation by C l u s t e r D e t e c t i o n i n Color Spaces, SIGART Newsletter 58, 16-17. [117] S c o t t , D.W. (1979), On Optimal and Data-based Histograms, Biometrika 66(3), 605-610. [118] S e l f r i d g e , P.G. and Sloan, K.R. (1981), Reasoning About Images: Using Meta-Knowledge i n A e r i a l Image Under-st a n d i n g , Proc. Seventh I n t e r n a t i o n a l J o i n t Conference  on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 755-757. [119] S h i r a i , Y. (1975), A n a l y z i n g I n t e n s i t y A r r a y s Using Knowledge About Scenes, The Psychology of Computer V i -s i o n , P.H. Winston (ed.), McGraw-Hill, New York, 93-113. [120] S h o r t 1 i f f e , E.H. and Buchanan, B.G. (1975), A Model of Inexact Reasoning i n Medicine, Mathematical B i o s c i e n c e s 23, 351-379. [121] Sloan, K.R. (1982), A n a l y s i s of "Dot Product Space" Shape D e s c r i p t i o n , P a t t e r n A n a l y s i s and Machine I n t e l l i g e n c e PAMI-4(1), 87-90. [122] Sloan, K.R. and Bajcsy, R. (1977), World Model Driven R e c o g n i t i o n of N a t u r a l Scenes, Dept. of Computer and Information S c i e n c e , Moore School of E.E., U. of Pennsylvania, P h i l a d e l p h i a . [123] Smith, R.G. and F r i e d l a n d , P. (1980), UNIT Package User's Guide, T e c h n i c a l Memorandum 80/L, DREA. [124] S t e f i k , M.J. (1979), An Examination of a Frame-Structured R e p r e s e n t a t i o n System, Proc. S i x t h I n t e r n a t i o n a l J o i n t  Conference on Art i f i c i a l I n t e l l i g e n c e , Tokyo, 845-852. References 228 [125] S t e f i k , M.J. (1980), Planning with C o n s t r a i n t s , STAN-CS-80-784, Department of Computer Science, S t a n f o r d . [126] Stoch, L. (1981), I n t e r a c t i v e Polygon F i l l i n g on a Raster Graphic D i s p l a y , M.Sc. d i s s e r t a t i o n , Dept. of Computer Science, U.B.C. [127] Stockman, G. (1978), Toward Automatic E x t r a c t i o n of C a r t o -graphic F e a t u r e s , ETL-0153, L.N.K. C o r p o r a t i o n , S i l v e r S p r i n g , Maryland. [128] T a v a k o l i , M. and Rosenfeld, A. (1982), B u i l d i n g and Road E x t r a c t i o n from A e r i a l Photographs, Systems, Man,. and  C y b e r n e t i c s SMC-120 ), 84-91. [129] Tenenbaum, J.M. (1973), On L o c a t i n g O b j e c t s by T h e i r D i s -t i n g u i s h i n g Features i n M u l t i s e n s o r y Images, TN 84, AI Center, SRI. [130] Tenenbaum, J.M. and Barrow, H.G. (1977), Experiments i n I n t e r p r e t a t i o n - G u i d e d Segmentation, A r t i f i c i a l I n t e l l i -gence 8(3), 241-274. [131] Tenenbaum, J.M., F i s c h l e r , M.A., and Wolf, H.C. (1978), A Scene-Analysis Approach to Remote Sensing, TN 173, AI Center, SRI. [132] Thorndyke, P.W. (1979), H e u r i s t i c s f o r Knowledge A c q u i s i -t i o n from Maps, Proc. S i x t h I n t e r n a t i o n a l J o i n t Confer-ence on A r t i f i c i a l I n t e l l i g e n c e , Tokyo, 880-883. [133] Tomita, F. (1981), H i e r a r c h i c a l D e s c r i p t i o n of Textures, Proc. Seventh I n t e r n a t i o n a l J o i n t Conference on A r t i f i -c i a l I n t e l l i g e n c e , Vancouver, 728-733. [134] Ullman, S. (1978), The I n t e r p r e t a t i o n of V i s u a l Motion, MIT Press, Cambridge, Mass. [135] Wesley, L.P. and Hanson, A.R. (1982), The Use of an E v i d e n t i a l - B a s e d Model f o r Representing Knowledge and Reasoning About Images i n the VISIONS System, Proc. Workshop on Computer V i s i o n : Representat ion and Con-t r o l , Rindge, New Hampshire, 14-25. ['136] Weymouth, T.E. (1981), Experiments i n Knowledge-Driven In-t e r p r e t a t i o n of N a t u r a l Scenes, Proc. Seventh I n t e r n a -t i o n a l J o i n t Conference on A r t i f i c i a l I n t e l l i g e n c e , Vancouver, 628-630. [137] Winston, P.H. (1972), The MIT Robot, Machine I n t e l l i g e n c e 7, B. M e l t z e r & D. M i c h i e (eds.TI American E l s e v i e r , New York, 431-463. References 229 [ 138] Winston, P.H. (1977), A r t i f i c i a l I n t e l l i g e n c e , Addison-Wesley, Reading, Mass.. [139] W i t k i n , A.P. (1981), Recovering Surface Shape and O r i e n t a -t i o n from Texture, A r t i f i c i a l I n t e l l i g e n c e 17(1-3), 17-45 . [140] Woodham, R.J. (1980a), Photometric Method f o r Determining Surface O r i e n t a t i o n from M u l t i p l e Images, O p t i c a l En-g i n e e r i n g 19, 139-144. [141] Woodham, R.J. (1980b), Using D i g i t a l T e r r a i n Data to Model Image Formation i n Remote Sensing, Proc. Soc i e t y of  Pho t o - O p t i c a l Instrumentat ion Engineers 238, 361-369. [142] Woodham, R.J. (1981), A n a l y z i n g Images of Curved S u r f a c e s , A r t i f i c i a l I n t e l l i g e n c e 17(1-3), 117-140. [143] Yachida, M., Ikeda, M., and T s u j i , S. (1979), A Knowledge D i r e c t e d L i n e F i n d e r f o r A n a l y s i s of Complex Scenes, Proc. S i x t h I n t e r n a t i o n a l J o i n t Conference on A r t i f i -c i a l I n t e l l i g e n c e , Tokyo, 984-991. [144] Yakimovsky, Y. and Feldman, J.A. (1973), A Semantics-Based D e c i s i o n Theory Region A n a l y z e r , Proc. T h i r d I n t e r n a -t i o n a l J o i n t Conference on A r t i f i c i a l I n t e l l i g e n c e , S t a n f o r d , 580-588. [145] Zucker, S.W., Rosen f e l d , A., and Davis, L.S. (1975), Gen-e r a l Purpose Models: E x p e c t a t i o n s about the Unexpected, Proc. Fourth I n t e r n a t i o n a l J o i n t Conference on A r t i f i -c i a l I n t e l l i g e n c e , T b i l i s i , 716-721. References APPENDIX A The MAIDS System Schemata Schemata are s t o r e d on the p r o p e r t y l i s t of the atom whose pname i s the name of the o b j e c t or i n s t a n c e . There are two types of schemata: OBJECTS — s t e r e o t y p i c or general knowledge: They have % o b j e c t % as the f i r s t p r operty on t h e i r p roperty l i s t . I N S T A N C E S — p a r t i c u l a r knowledge c r e a t e d as p a r t of i n t e r p r e t a t i o n : They are of the form objectname-number where the numbers are incremented from 1 as i n s t a n c e s are c r e a t e d . They have % i n s t a n c e % as the f i r s t p r o p e r t y on the p r o p e r t y l i s t . The Four P a r t s of a Schemata VALUEd type = attribute-name (%VALUE S-expresseion ((%CONFIDENCE . number) (%DEFAULT . S-expression) (%IF-ADDED . form) (%IF-MODIFIED . form) (%IF-REMOVED . form) (%REQUIRED . p r e d i c a t e ) ) ) LINK type = link-name {schema | (schemal schema2 ...)} CONFIDENCE = CONFIDENCE number 230 231 CONF-ALG form PROCEDURES -> procedures to be e v a l u a t e d when a p p r o p r i a t e messages are r e c e i v e d G l o b a l V a r i a b l e s The g l o b a l v a r i a b l e % l i n k _ t y p e s records a l l the known r e l a t i o n -s h i p s and t h e i r i n v e r s e s . I t i s of the form % l i n k _ t y p e s = ((ako . s p e c i a l i z e s - t o ) ( s p e c i a l i z e s - t o . ako) (apo . decomposes-to) (decomposes-to . apo) (a i o . i n s t a n c e s ) ( i n s t a n c e s . a i o ) . . .) %schemata: a l i s t of a l l the g e n e r i c ( s t e r e o t y p i c ) schemata a v a i l a b l e i n the loaded system % i n s t a n c e s : a l i s t of a l l the p a r t i c u l a r i n s t a n c e s a v a i l a b l e i n the loaded system L o c a l V a r i a b l e s In the procedures att a c h e d to CONFIDENCE value s (CONF-ALG) and VALUEd types (IF-ADDED, IF-MODIFIED, IF-REMOVED, IF-NEEDED, REQUIRED), the name of the c u r r e n t schema can be found in the v a r i a b l e %name. Furthermore, in VALUEd type a t t a c h e d procedures, the value that has j u s t been added, m o d i f i e d , e t c . i s l o c a t e d i n the v a r i a b l e % v a l . Convent ions In the f o l l o w i n g d e s c r i p t i o n of f u n c t i o n s , a l l arguments are Appendix A 232 ev a l u a t e d u n l e s s e x p l i c i t l y noted, o p t i o n a l arguments are encl o s e d i n " [ ] " ' s , and the f o l l o w i n g s u f f i x e s are g e n e r a l l y used: A any A t t r i b u t e V VALUEd type a t t r i b u t e s L LINK a t t r i b u t e s I Instances 0 O b j e c t s Schemata M a n i p u l a t i o n Funct ions 1 . ( a d d - l i n k - t y p e l i n k ) Returns: l i n k Side e f f e c t : Adds a l i n k and i t s inv e r s e to the g l o b a l v a r i a b l e % l i n k _ t y p e s : a l i s t of a l l the known l i n k t y pes. The user w i l l be q u e r i e d f o r the name of the i n v e r s e but i t i s not necessary f o r one to be s u p p l i e d . 2 . ( a d d - l i n k - i n v e r s e l i n k i n v e r s e ) Returns: n i l Side e f f e c t : Makes i n v e r s e the i n v e r s e of l i n k i n the g l o -b a l v a r i a b l e % l i n k _ t y p e s . I f l i n k and i n v e r s e do not e x i s t i n % l i n k _ t y p e s , they are both added. 3 . ( i f n e e d schema a t t r i b u t e ) Returns: the r e s u l t of e v a l i n g the % i f - n e e d e d c l a u s e of a t t r i b u t e (a VALUEd type) i n schema. 4.(saddl schemal l i n k schema2) Returns: schema2 Side e f f e c t s : Adds a LINK from schemal to schema2. In a d d i t i o n , the i n v e r s e of the LINK i s added from schema2 to schemal. Link must be known to % l i n k _ t y p e s . Appendix A 233 5 . ( s a n y l i n k ? schemal schema2) Returns: If there i s a path between schemal and schema2 along any type of LINK i t r e t u r n s a l i s t of a l l the l i n k types used i n the path (of the form ( l i n k l l i n k 2 l i n k 3 . . . ) ) . I f there i s no path, i t ret u r n s n i l . 6 . ( s a t t r schema type) Where: Type i s e i t h e r n i l ( r e t u r n a l l a t t r i b u t e s ) or a l i s t of one or more or the f o l l o w i n g types to be retur n e d : valued, l i n k , procedure, or ot h e r . Returns: A l i s t of the a t t r i b u t e s of schema depending on type. 7 . ( s a t t r t y p e schema a t t r i b u t e ) Returns: the type of a t t r i b u t e i n schema. Type w i l l be one of valued, l i n k , procedure, or o t h e r . 8 . ( s c o n s i s t ) Returns: n i l Side e f f e c t : Makes a data base c o n s i s t e n t by adding and removing a p p r o p r i a t e LINKS and o b j e c t s . A f t e r i t i s f i n i s h e d , a l l LINK types w i l l p o i n t to e x i s t i n g o b j e c t s and t h e i r i n v e r s e s w i l l a l s o be i n p l a c e . 9 . ( s c r e a t e [name]) Where: name i s the unevaluated name of the schema that w i l l be c r e a t e d . If name i s not provided the user w i l l be q u e r i e d f o r i t . In a d d i t i o n , the user w i l l be q u e r i e d f o r the v a r i o u s p a r t s of the schema and can input any a t t r i b u t e s t h a t are d e s i r e d . Returns: The newly c r e a t e d o b j e c t . Side e f f e c t : The property l i s t of the named schemata w i l l be m o d i f i e d to i n c l u d e a l l the a t t r i b u t e s that i t i s i n i t i a l l y endowed with. ! 0 . ( s e r a s e i i n s t a n c e [if-rem? [ s p l i c e ? ] ] ) Appendix A 234 Where: if-rem? and s p l i c e ? are e i t h e r n o n - n i l or n i l . Returns: the name of the des t r o y e d i n s t a n c e . Side e f f e c t : Destroys i n s t a n c e as a schema (removes the a t t r i b u t e s from i t s property l i s t ) and removes a l l the LINKs to i t . If if-rem? i s n o n - n i l , then a l l the e x i s t i n g if-removed c l a u s e s on VALUEd types are e v a l e d . If s p l i c e ? i s non-n i l , then the LINKs from adjacent schemata are s p l i c e d around i n s t a n c e (see s p l i c e ) . 11.(seraseo o b j e c t s p l i c e ? ) Where: s p l i c e ? i s n o n - n i l or n i l . Returns:- the name of the des t r o y e d o b j e c t . Side e f f e c t : Destroys the st e r e o t y p e o b j e c t . Since t h i s i s not l i k e l y to be done very o f t e n , the user i s prompted to make sure he/she r e a l l y wants to do i t . If s p l i c e ? i s n o n - n i l , then the sche-mata adjacent to obj e c t have t h e i r l i n k s s p l i c e d around i t (see s p l i c e ) . 12.(serrn form c u l p r i t ) Where: n i s a number from 1 to 12. Side e f f e c t : E r r o r message n i s p r i n t e d using the f u n c t i o n form and value c u l p r i t . Debug i s then c a l l e d on form. 13.(sgeta schema a t t r i b u t e [ i n h e r i t [ a l l o r o n e [ l i n k s ] ] ] ) Where: i n h e r i t i s one of "y" or "n"; a l l o r o n e i s one of "a" or "o"; and l i n k s i s a l i s t of LINKs. Returns: The value of a t t r i b u t e i n schema. I f the a t t r i -bute i n schema has a value, then i t i s retu r n e d . If there i s no a t t r i b u t e i n schema, then p r o p e r t y i n h e r i t a n c e w i l l be used ( i f i n h e r i t = y as opposed to n). I n h e r i t a n c e w i l l be sought up the LINKs found i n the l i s t " l i n k s " . Depending on the value of a l l o r o n e , e i t h e r the f i r s t value (from a b r e a d t h - f i r s t search) w i l l be re t u r n e d ( a l l o r o n e = o) and the g l o b a l v a r i a b l e %wherefrom w i l l be set to the schema, where the a t t r i b u t e was found, or a l l the value s and t h e i r schemata w i l l be returned in a l i s t e,g. ( ( v a i l . s c h l ) ( v a l 2 . sch2)) ( i f Appendix A 235 a l l o r o n e = a ) . Note: The d e f a u l t v a l u e s are i n h e r i t = n , allorone=o, l i n k s = ( a k o ) . 14.(sgetc schema [ a t t r i b u t e ] ) Returns: the con f i d e n c e value of a schema ( i f a t t r i b u t e n i l or "confidence") or of a VALUEd type a t t r i -bute . I 5 . ( s g e t l schema l i n k ) Returns: the schema (or schemata) " l i n k e d " to schema v i a l i n k . 16.(sgetv schema a t t r i b u t e i n h e r i t type a l l o r o n e l i n k s l l i n k s 2 ) Where: i n h e r i t i s one of "y" or "n"; type i s one of "v", "vd", or "d"; a l l o r o n e i s one of "a" or "o"; and l i n k s l and l i n k s 2 are l i s t s of LINKS. Returns: The value (or d e f a u l t ) of a t t r i b u t e (which must be a VALUEd type) i n schema. I f there i s no a t t r i b u t e i n schema, then p r o p e r t y i n h e r i t a n c e w i l l be used ( i f i n h e r i t = y as opposed to n). I n h e r i t a n c e w i l l be sought f o r e i t h e r a value only (type = v, using l i n k s l ) , a value or a d e f a u l t at every step (type = vd, using l i n k s l ) , or the whole s t r u c t u r e w i l l be searched f o r a value (using l i n k s l ) and i f none i s found the whole s t r u c t u r e w i l l again be searched f o r a d e f a u l t ( u s i n g l i n k s 2 ) (type = d ) . Depending on the value of a l l o r o n e , e i t h e r the f i r s t value (from a b r e a d t h - f i r s t search) w i l l be returned ( a l l o r o n e = o) and the g l o b a l v a r i a b l e %wherefrom w i l l be set to the schema where the a t t r i b u t e was found, or a l l the values and t h e i r schemata w i l l be returned i n a l i s t e.g. ( ( v a i l . s c h l ) ( v a l 2 sch2)) ( i f a l l o r o n e = a ) . If a d e f a u l t as opposed to a value i s returned, then the above d e s c r i p t i o n holds f o r d e f a u l t s p l u s the g l o b a l v a r i a b l e % d e f a u l t i s set to t . Note: d e f a u l t s are i n h e r i t = y, type = vd, a l l o r o n e = o, l i n k s l = (ako), and l i n k s 2 = (ako). 1 7 . ( s l i n k ? schemal schema2 l i n k ) Appendix A 236 Returns: If there i s a path from schemal to schema2 along l i n k , then the minimum l e n g t h of the path, e l s e n i l . If l i n k i s n i l , then s a n y l i n k ? i s c a l l e d which looks f o r a path using any type of l i n k . I8.(smerge schemal schema2 doconfalgorithm?) Where: doconfalgorithm? i s n o n - n i l or n i l . Returns: the p r i o r i t y queue. Side e f f e c t : Combines the s l o t s of schema2 i n t o those of schemal, i . e . , f o r VALUEd types, i f the a t t r i b u t e i n schema2 has a n o n - n i l value and the one in schemal does not, put the value i n schemal's s l o t . For LINK types, take the union of the p o i n t e r s . If doconfalgorithm? i s t r u e , evaluate the CONFIDENCE a l g o r i t h m on the r e s u l t i n g schema. A l s o , a l l r e f e r e n c e s to schema2 i n p r e v i o u s messages in the p r i o r -i t y queue are changed to schemal. 19.(snewi o b j e c t ) Returns: the new i n s t a n c e . Side e f f e c t : Creates a new i n s t a n c e of the s t e r e o t y p e o b j e c t . I t w i l l have the name o b j e c t - n , where n i s the next i n t e g e r i n sequence ( s t a r t i n g from 1). 2 0 . ( s p l i c e schema l i n k [ i n v e r s e ] ) Returns: n i l Side e f f e c t : Removes the LINKs from schema to the other nodes att a c h e d by l i n k and s p l i c e s the LINKs around i t . Thus: up i n v e r s e schema l i n k down up goes-> to i n v e r s e l i n k down Appendix A 237 2 1 . ( s p r i n t schemaname) Returns: n i l Side e f f e c t : P r e t t y p r i n t s a schema. 22. ( s p r i n t n schema l i n k ) Returns: n i l Side e f f e c t : P r e t t y p r i n t s the t r e e s t r u c t u r e under schema using l i n k . 23. (sputa schema a t t r i b u t e [ v a l u e ] ) Returns: the value of " v a l u e " . Side e f f e c t : Puts a new d e f i n i t i o n of a t t r i b u t e i n schema and p l a c e s "value" as i t s i n i t i a l v a l u e . 24.(sputc schema [ a t t r i b u t e [ c v a l [spread? [ s k i p ? ] ] ] ] ) Where: c v a l i s a number, and spread? and skip? are e i t h e r n o n - n i l or n i l . Returns: the o l d confidence v a l u e . Side e f f e c t : Changes the CONFIDENCE value of a t t r i b u t e i n schema to c v a l . If a t t r i b u t e = n i l or "con-f i d e n c e " the value of CONFIDENCE at the schema l e v e l i s changed; otherwise the VALUEd type a t t r i b u t e has i t s CONFIDENCE changed. If spread? i s non-NIL then the e f f e c t s of the change ( i f there are any) are spread by r e -e v a l i n g a l l the c o n f - a l g o r i t h m s up the apo and ako h i e r a r c h i e s . If spread? i s non-NIL and skip? i s , then the CONF-ALG i s eva l u a t e d f o r schema ( i . e . the value of c v a l i s i g n o r e d ) . 2 5 . ( s p u t c l schema a t t r i b u t e value o l d c o n f i d e n c e newconfidence spread?) Where: spread? i s e i t h e r n o n - n i l or n i l . Returns: the o l d confidence v a l u e . Side e f f e c t : Replaces o l d c o n f i d e n c e i n the l i s t of c o n f i -dence values of a t t r i b u t e i n schema with Appendix A 238 newconfidence. If spread? i s non-NIL then the e f f e c t s of the change ( i f there are any) are spread by r e - e v a l i n g a l l the conf-a l g o r i t h m s up the apo and ako h i e r a r c h i e s . 26.(sputv schema a t t r i b u t e value) Returns: v a l u e . Side e f f e c t : Puts a new value i n t o the %value s l o t of a t t r i b u t e i n schema. Depending on the p r e v i -ous and new values, c e r t a i n demons may respond. previous new value value ~ n i l % r e q u i r e d i s checked n i l n i l n i l ~ n i l % i f - a d d e d i s e v a l e d " n i l ~ n i l % i f - m o d i f i e d i s e v a l e d ~ n i l n i l %if-removed i s e v a l e d 27.(sremovea schema a t t r i b u t e [ i f - r e m c l a u s e ? ] ) Where: i f - r e m c l a u s e ? i s e i t h e r n o n - n i l or n i l . Returns: the value of the removed a t t r i b u t e . Side e f f e c t : Removes the d e f i n i t i o n of a t t r i b u t e from schema. I f the i f - r e m c l a u s e ? i s n o n - n i l and there i s a n o n - n i l value, then the if-removed form ( i f i t e x i s t s ) i s e v a l e d . 28.(sremoveal schema l i n k ) Returns: the value of the schema(ta) p o i n t e d to from "schema" by l i n k . Side e f f e c t : Removes a l l the LINKs a t t a c h i n g schema to other schemata v i a l i n k . 29.(sremovel schemal l i n k schema2) Returns: schema2 Side e f f e c t : Removes the l i n k from schemal to schema2 and the i n v e r s e l i n k from schema2 to schemal. If schema2 i s n i l , then sremoveal i s c a l l e d which removes a l l the nodes a t t a c h e d to Appendix A 239 schemal by l i n k . 30. (sremovev schema a t t r i b u t e ) Returns: n i l . Side e f f e c t : Removes the %value from a t t r i b u t e (a VALUEd type) i n schema (by s e t t i n g i t to n i l ) . 31. ( s r e s t o r e [ f i l e ] ) Where: f i l e i s ,not e v a l l e d . Returns: n i l . Side e f f e c t : Restores the stereotype schemata from f i l e . i f f i l e i s n i l , then the d e f a u l t f i l e "saved.d" i s used. 32. (ssave [ f i l e ] ) Where: f i l e i s not e v a l l e d . Returns: f i l e . Side e f f e c t : Saves a l l the known s t e r e o t y p e schemata on f i l e . I f f i l e i s n i l , then the d e f a u l t f i l e "saved.d" i s used. 33. ( s s p r i n t n schema) Returns: n i l Side e f f e c t : P r e t t y p r i n t s the t r e e s t r u c t u r e under schema f o l l o w i n g a l l l i n k s . 34. (vcheck schema a t t r i b u t e value) Returns: the r e s u l t of e v a l l i n g the % r e q u i r e d c l a u s e on value ( i f i t e x i s t s ) of a t t r i b u t e (a VALUEd type) i n schema. Appendix A APPENDIX B Edge D e t e c t i o n An edge d e t e c t o r has been implemented based on the Marr-H i l d r e t h theory of edge d e t e c t i o n (Marr and H i l d r e t h , 1980; H i l -d r e t h , 1980). Edges are l o c a t e d at the zero c r o s s i n g s of the L a p l a c i a n of the Gaussian f u n c t i o n convolved with the image. T h i s appendix d e s c r i b e s s p e c i f i c implementation d e t a i l s of how edges and edge i n f o r m a t i o n are c a l c u l a t e d . A mask 31 p i x e l s square, c e n t r e d on the o r i g i n , i s gen-e r a t e d from the f u n c t i o n m(x,y) = ( x 2 + y 2 - 2a 2) * e x p ( - ( x 2 + y 2 ) / 2 a 2 ) and then s c a l e d so that the value of m(0,i) = 0.5 where i i s the l a r g e s t i n t e g e r < 15 such that m(0,i) i s non-zero. In c o n t r a s t to the M a r r - H i l d r e t h theory, only one a value i s used, and i t i s chosen to produce high r e s o l u t i o n edges. A r i t h m e t i c i s c a r r i e d out using f l o a t i n g p o i n t a r i t h m e t i c . The mask i s convolved with the d i g i t i z e d images (128 by 128 by 256 grey s c a l e v a l u e s ) . Zero c r o s s i n g s are found by t r a c i n g out curves i n the convolved image where the values change s i g n s . T h i s produces continuous curves that are e i t h e r c l o s e d or run o f f the edge of the image. 240 241 Since these long curves g e n e r a l l y i n c l u d e s e v e r a l semanti-c a l l y d i s t i n c t o b j e c t s , they are broken i n t o s h o r t e r s t r a i g h t l i n e segments by a method c a l l e d g e n e r a l i z a t i o n (Ramer, 1972; L i t t l e , 1982). G e n e r a l i z a t i o n i s a r e c u r s i v e process wherein a l i n e i s drawn between the endpoints of a curve and the p o i n t on the curve f u r t h e s t from the l i n e i s found. I f the d i s t a n c e from the l i n e to the point i s g r e a t e r than a t h r e s h o l d (a value of 1 p i x e l was used) then the l i n e i s s u b d i v i d e d at the p o i n t and the two new segments are g e n e r a l i z e d i n t u r n . Recursion stops when the d i s t a n c e of a l l the p o i n t s from the l i n e i s l e s s than the t h r e s h o l d , guaranteeing that the d e v i a t i o n i s l e s s than that v a l u e . T h i s produces a number of connected, " s t r a i g h t " seg-ments. From t h i s process a l i n e image i s returned that corresponds to the o r i g i n a l image. Each p i x e l c o n t a i n s e i t h e r 0, meaning that there i s no edge segment at that point,, or a number i d e n t i -f y i n g the edge segment i t i s a part o f . A separate data s t r u c -t u r e c o n t a i n s i n f o r m a t i o n f o r each edge segment. T h i s i n c l u d e s l o c a t i o n , l e n g t h , o r i e n t a t i o n (from 0 to 360 degrees s i n c e the value i s a l s o used to i n d i c a t e that the region to the r i g h t of the segment i s b r i g h t e r than the one on the l e f t ) , and c o n t r a s t . Contrast i s the f i r s t d i r e c t i o n a l d e r i v a t i v e across the edge segment (not the slope of the t h i r d as was suggested by Marr and H i l d r e t h ) . I t i s c a l c u l a t e d by moving along the p o i n t s in the i n t e n s i t y image that correspond to those of the l i n e image. For each p o i n t , the d i f f e r e n c e i s c a l c u l a t e d between the Appendix B 242 p a i r of p i x e l v a l u e s one p i x e l away from the p o i n t and perpen-d i c u l a r to the o r i e n t a t i o n of the segment. The average of the a b s o l u t e v a l u e s of a l l the d i f f e r e n c e s i s the c o n t r a s t (or strength) of the segment. T h i s i n f o r m a t i o n forms a type of raw p r i m a l sketch. I t i s (one of) the intermediate r e p r e s e n t a t i o n s accessed by higher l e v e l f u n c t i o n s . Appendix B APPENDIX C Region Merging Regions i n the i n t e n s i t y image are merged using an imple-mentation of Freuder's a f f i n i t y method (Freuder, 1976). The ge n e r a l n o t i o n i s to merge regions that have the g r e a t e s t a f f i n -i t y towards each other, based mainly on t h e i r i n t e n s i t i e s . Freuder s t a r t e d by forming i n i t i a l r egions 25 u n i t s on a s i d e (out of a 300 by 350 p i x e l image). Since some s i g n i f i c a n t f e a t u r e s i n the images that have been used can be only a few p i x e l s i n extent, a d i f f e r e n t i n i t i a l i z a t i o n was performed. Each p i x e l i n the image that i s not part of a region i s chosen as a seed. A l l neighbours (using 4-connectedness) whose i n t e n -s i t y value i s w i t h i n n of the seed's i n t e n s i t y (a value of 10 was used f o r n) are j o i n e d to the seed's r e g i o n . T h i s continues i t e r a t i v e l y outward while the i n t e n s i t y values remain s i m i l a r . T h i s produces a set of i n i t i a l regions (ranging i n number from 3328 f o r A s h c r o f t to 5959 f o r Cranbrook). The next phase c o n t i n u a l l y merges regions that have an a f f i n i t y towards each other. For each region ( R i ) , a f u n c t i o n i s a p p l i e d to i t and each of i t s neighbours (Rj) to i n d i c a t e t h e i r a f f i n i t y f o r each other. A f f i n i t y i s i n v e r s e l y propor-t i o n a l to the value returned by the f u n c t i o n . The f u n c t i o n used i s 243 244 | I (Ri ) - KRj) | * (A(Ri) + A(Rj) ) where I i s the average i n t e n s i t y of a region and A i s i t s a r e a . The second term favours the merging of small regions but the s i m i l a r i t y of i n t e n s i t i e s i s the major c o n s i d e r a t i o n . Each r e g i o n (Ra) forms a l i n k with the neighbour that i t has the most a f f i n i t y with (Rb). Now, when Rb c a l c u l a t e s a f f i n -i t y v a l u e s f o r i t s neighbours, i t may or may not p o i n t to Ra. Only those p a i r s of neighbours that p o i n t to each other are can-d i d a t e s f o r merging. In another departure from Freuder's method, not a l l doubly l i n k e d r e g i o ns are merged, only the ones with the lowest a f f i n i t y v alues ( u s u a l l y the best 30 were merged). T h i s was done because i t was found that some of the r e s u l t i n g r e g i o ns from "high a f f i n i t y " merges were b e t t e r c a n d i -dates f o r subsequent merges and would have been bypassed by merging some of the "lower a f f i n i t y " " p a i r s . T h i s process i s repeated f o r the new r e g i o n s . Freuder con-t i n u e s i t e r a t i n g u n t i l there i s only one region l e f t and uses the merge h i s t o r y t r e e to s e l e c t regions f o r subsequent a n a l y s i s . P r o c e s s i n g may stop on i t s own, however, i f there are no doubly l i n k e d neighbours to merge ( t h i s happened with the Houston image). By d i s p l a y i n g the r e s u l t i n g regions at each stage superimposed over the o r i g i n a l image, an " i t e r a t e u n t i l s a t i s f i e d " t e r m i n a t i o n c o n d i t i o n was used i n t h i s implementa-t i o n . Appendix C 245 The output from region merging i s s i m i l a r to that from edge d e t e c t i o n . A region image i s produced where each p i x e l c o n t a i n s the number i d e n t i f y i n g the region i t belongs t o . A l s o , f o r each r e g i o n , a t a b l e c o n t a i n s i t s l o c a t i o n , area, perimeter, average i n t e n s i t y , and neighbouring r e g i o n s . Appendix C APPENDIX D Using Mapsee2 The sketch i s drawn over an image of the scene using a t r a c k b a l l . The graphic o v e r l a y i s then t r a n s l a t e d i n t o a s e r i e s of drawing commands ( p l o t s and g o t o s ) , the input form r e q u i r e d by Mapsee2 (the a l g o r i t h m to do t h i s was mo d i f i e d from Stoch, 1981). Mapsee2 (Havens and Mackworth, 1980; Mackworth and Havens, 1981) reencodes the curves as chains and segments the image. I t then f i n d s i n t e r p r e t a t i o n s f o r the chains and b u i l d s a decompo-s i t i o n h i e r a r c h y of the i n s t a n c e s . The i n s t a n c e s are Maya schemata which are made up of a t t r i b u t e / v a l u e p a i r s . Since MAIDS i s upwards compatible with Maya, the a t t r i b u t e values can be accessed i n MISSEE using sgeta, the ge n e r i c "get a t t r i b u t e " f u n c t i o n . The r e l e v a n t sche-mata are then w r i t t e n to a f i l e so that they can be t r a n s f e r r e d from M u l t i l i s p to Franz L i s p . The schemata of i n t e r e s t are p o i n t s , l i n k s , l i n e s , c h a i n s , b r i d g e s , mountains, roads, r i v e r s , towns, road-systems, r i v e r - s y s t e m s , mtn-ranges, and geosystems. 246 I APPENDIX E A Sample P r o t o c o l The f o l l o w i n g s c r i p t i ntermixes user input with the output from MISSEE that c o n s t i t u t e s the p r o t o c o l . A l l three informa-t i o n sources are u s e d — t h e i n t e n s i t y image, the sketch map, and the user's i n p u t . The r e s u l t i n g i n s t a n c e h i e r a r c h y can be found i n F i g u r e 6.21. % missys —> c y c l e * * i * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * SUPERVISOR; Input the s t a r t i n g value f o r the Queue or n i l e.g. ((100 t d %road (smitem *road-3))) n i l ; use the d e f a u l t OBJECT: geosystem-top-down. type: n i l v a l : n i l SCHEDULER: adding to Queue (100 r i v e r - s y s t e m - t d n i l ) ** 2. *************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((100 r i v e r - s y s t e m - t d n i l ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: river-system-top-down, type: n i l v a l : n i l SCHEDULER: adding to Queue (100 bridge-bu-sm ( * b r i d g e - l ) ) ** 3. *************************************** SCHEDULER: Here i s the p r i o r i t y queue:. %QUEUE = ((100 bridge-bu-sm (*bridge-1))) 247 Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: bridge-bu-with-sketch-map f o r * b r i d g e - l bridge-bu-sm: c r e a t e a new bridge i n s t a n c e %bridge-1 bridge-bu-sm: *** model c o n s i s t e n c y f o r bridge from sm bridge-bu-sm: regions l i s t = (130 1186 1629 9 1750 2018) bridge-bu-sm: order l i s t = (0 S B B 0) % b r i d g e : value added to smitem = *bridge-1 bridge-bu-sm: *** model c o n s i s t e n c y f o r bridge bridge-bu-sm: edges l i s t = (88 11 205 206 241 242) bridge-bu-sm: edges i n proper o r i e n t a t i o n (88 11 242) bridge-bu-sm: new edge l i s t = ((88 89) (.10 11 12 13 14 15) (242 243)) bridge-bu-sm: toomany = 0 bridge-bu-sm: c r e a t e new (s ) c u r b i n s t a n c e %curb-1 bridge-bu-sm: edge segments are (88 89) bridge-bu-sm: angles are (248 214) bridge-bu-sm: l e n g t h = 20 maxstrength = 5 avgstr e n g t h = 3.5 bridge-bu-sm: c r e a t e new ( s ) c u r b i n s t a n c e %curb-2 bridge-bu-sm: edge segments are (10 11 12 13 14 15) bridge-bu-sm: angles are (90 57 79 45 90 27) bridge-bu-sm: l e n g t h = 54 maxstrength = 134 av g s t r e n g t h = 96 bridge-bu-sm: c r e a t e new (s ) c u r b i n s t a n c e %curb-3 bridge-bu-sm: edge segments are (242 243) bridge-bu-sm: angles are (228 270) bridge-bu-sm: l e n g t h = 20 maxstrength = 22 avgstr e n g t h = 18. SCHEDULER: adding to Queue (68 r i v e r - t d (2018 r e g i o n ) ) SCHEDULER: adding to Queue (68 r i v e r - t d (1750 r e g i o n ) ) SCHEDULER: adding to Queue (68 r i v e r - t d (9 r e g i o n ) ) SCHEDULER: adding to Queue (68 r i v e r - t d (130 r e g i o n ) ) bridge-bu-sm: neighbouring r i v e r r e g i o ns are n i l SCHEDULER: adding to Queue (68 road-td (1629 r e g i o n ) ) bridge-bu-sm: neighbouring road regions are n i l bridge-bu-sm: %bridge-1 i s model-consistent bridge-bu-sm: %bridge-1 must-be-part of some road-system and r i v e r - s y s t e m SCHEDULER: adding to Queue (73 road-system-bu ( % b r i d g e - l ) ) SCHEDULER: adding to Queue (73 river-system-bu ( % b r i d g e - l ) ) * * A * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 road-system-bu (%bridge-1)) (73 river-system-bu ( % b r i d g e - l ) ) (68 r i v e r - t d (2018 re g i o n ) ) (68 r i v e r - t d (1750 re g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 reg i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n Appendix E 249 OBJECT: %road-system bottom-up f o r %bridge-1 %road-system-bu: e x i s t i n g systems n i l %road-system-bu: componentlist = (%curb-1 %curb-2 %curb-3 % b r i d g e - l ) new schema c r e a t e d %road-system-1 component added %bridge-1 to Queue (73 landmass-bu (%road-system-1)) %road-system-bu: %road-system-bu: SCHEDULER: adding ** *************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 (73 (68 (68 (68 (68 (68 river-system-bu (%bridge-1)) landmass-bu (%road-system-1)) r i v e r - t d r i v e r - t d r i v e r - t d r i v e r - t d road-td (2018 r e g i o n ) ) (1750 r e g i o n ) ) (9 region) ) (130 r e g i o n ) ) (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: % r i v e r - s y s t e m bottom-up f o r %bridge-1 % r i v e r - s y s t e m - b u : % r i v e r - s y s t e m - b u : % r iver-system-bu: % r i v e r - s y s t e m - b u : SCHEDULER: adding e x i s t i n g systems n i l componentlist = (%curb-1 %curb-2 %curb-3 %bridge-1) new schema c r e a t e d %river-system-1 component added %bridge-1 to Queue (73 waterbody-bu (%river-system- 1 )) * * 6. * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 landmass-bu (%road-system-1)) (73 waterbody-bu (%river-system-1)) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (1750 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: %landmass bottom-up f o r %road-system-1 %landmass-bu: %landmass-bu: %landmass-bu: %landmass-bu: e x i s t i n g systems n i l componentlist = (%curb-1 %curb-2 %curb-3 %bridge~1 %road-system-1) new schema c r e a t e d %landmass-1 component added %road-system-1 SCHEDULER: adding to Queue (73 geosystem-bu (%landmass-1)) ** 7. *************************************** Appendix E 250 SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 waterbody-bu (%river-system-1)) (73 geosystem-bu (%landmass-1)) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (1750 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: waterbody-bottom-up f o r %river-system-1 waterbody-bu: new schema c r e a t e d %waterbody-1 waterbody-bu: adding component %waterbody-1 SCHEDULER: adding to Queue (73 geosystem-bu (%waterbody-1)) ** 3^ *************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 geosystem-bu (%landmass-1)) (73 geosystem-bu (%waterbody-1)) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (1750 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: geosystem-bottom-up f o r %landmass-1 geosystem-bu: e x i s t i n g systems n i l geosystem-bu: new schema c r e a t e d %geosystem-1 geosystem-bu: componentlist = (%curb-1 %curb-2 %curb-3 %bridge-1 %road-system-1 %landmass-1 %geosystem-1) ** g < *************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 geosystem-bu (%waterbody-1)) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (1750 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: geosystem-bottom-up f o r %waterbody-1 geosystem-bu: e x i s t i n g systems %geosystem-1 Appendix E geosystem-bu: new schema c r e a t e d %geosystem-2 geosystem-bu: componentlist = (%curb-1 %curb-2 %curb-3 %bridge-1 %river-system-1 %waterbody-1 %geosystem-2) geosystem-bu: new s u p e r i o r system c r e a t e d : %geosystem-3 c o n t a i n i n g %geosystem-2 and %geosystem-1 ; at t h i s p o i n t , the e n t i r e s t r u c t u r e above ; %bridge-1 has been c o n s t r u c t e d ** 10. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (1750 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num --Delete element num a s-exp --Add s-exp to the Queue us i n g i t s p r i o r i m n1 n2 —Move element n1 to a f t e r n2 q --Quit p r o c e s s i n g e --End modifying Queue, continue 1. -> (68 r i v e r - t d (2018 r e g i o n ) ) 2. -> (68 r i v e r - t d (1750 r e g i o n ) ) 3. -> (68 r i v e r - t d (9 r e g i o n ) ) 4. -> (68 r i v e r - t d (130 r e g i o n ) ) 5. -> (68 road-td (1629 r e g i o n ) ) Command:m 2 0 ; t r y region 1750 f i r s t SCHEDULER: What i s " t h e new p r i o r i t y f o r 2?72 1. -> (72 r i v e r - t d (1750 r e g i o n ) ) 2. -> (68 r i v e r - t d (2018 r e g i o n ) ) 3. -> (68 r i v e r - t d (9 r e g i o n ) ) 4. -> (68 r i v e r - t d (130 r e g i o n ) ) 5. -> (68 road-td (1629 re g i o n ) ) Command:e OBJECT: Top Down on % r i v e r . type: region v a l : 1750 SCHEDULER: adding to Queue (72 river-bu-sm ( * r i v e r - 5 ) ) SCHEDULER: adding to Queue (72 river-bu-sm ( * r i v e r - 2 ) ) ** 11. ************************************** Appendix E SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((72 river-bu-sm ( * r i v e r - 5 ) ) (72 river-bu-sm ( * r i v e r - 2 ) ) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) SUPERVISOR: Do your l i s p t h i n g — t y p e d or q to stop ? % p r i n t l e v e l 50 ? ( s e t q % p r i n t l e v e l 11) 1 1 ?d ** 12. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((72 river-bu-sm ( * r i v e r - 5 ) ) (72 river-bu-sm ( * r i v e r - 2 ) ) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (9 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (72 river-bu-sm ( * r i v e r - 5 ) ) 2. -> (72 river-bu-sm ( * r i v e r - 2 ) ) 3. -> (68 r i v e r - t d (2018 r e g i o n ) ) 4. -> (68 r i v e r - t d (9 r e g i o n ) ) 5. -> (68 r i v e r - t d (130 r e g i o n ) ) 6. -> (68 road-td (1629 r e g i o n ) ) Command:d 1 ; * r i v e r - 5 i s an unintended i n t e r p r e t a t i o n 1. —> n i l 2. -> (72 river-bu-sm ( * r i v e r - 2 ) ) 3. -> (68 r i v e r - t d (2018 r e g i o n ) ) 4. -> (68 r i v e r - t d (9 r e g i o n ) ) 5. -> (68 r i v e r - t d (130 r e g i o n ) ) 6. -> (68 road-td (1629 r e g i o n ) ) Command:d 4 1. —> n i l Appendix E 253 2. —> (72 river-bu-sm ( * r i v e r - 2 ) ) 3. -> (68 r i v e r - t d (2018 r e g i o n ) ) 4. -> n i l 5. —> (68 r i v e r - t d (130 r e g i o n ) ) 6. -> (68 road-td (1629 r e g i o n ) ) Command:e OBJECT: river-bu-with-sketch-map f o r * r i v e r - 2 river-bu-sm: c r e a t e a new r i v e r i n s t a n c e % r i v e r - 1 river-bu-sm: *** model c o n s i s t e n c y f o r r i v e r from sm river-bu-sm: regions l i s t = (1750) river-bu-sm: i n t e r p r e t a t i o n s are (WATER) river-bu-sm: one i n t e r p r e t a t i o n must be c o n s i s t e n t with water or f a i l river-bu-sm: added neighbouring regions are (2295 2095 2018) % r i v e r : value added to regions = (1750 2295 2095 2018) river-bu-sm: % r i v e r - 1 i s model-consistent % r i v e r : . value added to smitem = * r i v e r - 2 river-bu-sm: neighbouring bridge regions are (%bridge-1) river-bu-sm: % r i v e r - 1 must-be-part of some r i v e r - s y s t e m SCHEDULER: adding to Queue (73 river-system-bu ( % r i v e r - 1 ) ) ** 13. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 river-system-bu ( % r i v e r - l ) ) (68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: % r i v e r - s y s t e m bottom-up f o r % r i v e r - 1 % r i v e r - s y s t e m - b u : e x i s t i n g systems %river-system-1 % r i v e r - s y s t e m - b u : % r i v e r - 1 added to %river-system-1 ; % r i v e r - 1 j o i n s the same r i v e r - s y s t e m that ; c o n t a i n s %bridge-1 ** 14. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 r i v e r - t d (2018 r e g i o n ) ) (68 r i v e r - t d (130 r e g i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: Top Down on % r i v e r . type: r e g i o n v a l : 2018 Appendix E SCHEDULER: adding to Queue (68 r i v e r - b u (2018)) ** 15. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 r i v e r - t d (130 reg i o n ) ) (68 road-td (1629 reg i o n ) ) (68 r i v e r - b u (2018)))) Do you want to modify i t ( y ) , be in l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (68 r i v e r - t d (130 r e g i o n ) ) 2. -> (68 road-td (1629 r e g i o n ) ) 3. -> (68 r i v e r - b u (2018)) Command:m 3 0 SCHEDULER: What i s the new p r i o r i t y f o r 3?72 1. -> (72 r i v e r - b u (2018)) 2. -> (68 r i v e r - t d (130 r e g i o n ) ) 3. -> (68 road-td (1629 r e g i o n ) ) Command:e OBJECT: river-bu-image-alone r i v e r - b u - i m : region 2018 has alre a d y been t r i e d with r e s u l t % r i v e r - 1 ** 16. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 r i v e r - t d (130 reg i o n ) ) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be in l i s p ( l ) , or q u i t ( q ) ? n OBJECT: Top Down on % r i v e r . type: region v a l : 130 SCHEDULER: adding to Queue (68 river-bu-sm ( * r i v e r - 5 ) ) SCHEDULER: adding to Queue (68 river-bu-sm ( * r i v e r - 1 ) ) ** 17. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 road-td (1629 re g i o n ) ) (68 river-bu-sm ( * r i v e r - 5 ) ) (68 river-bu-sm ( * r i v e r - 1 ) ) ) Appendix E 255 Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m nl n2, q, or e 1. -> (68 road - t d (1629 r e g i o n ) ) 2. —> (68 river-bu-sm ( * r i v e r - 5 ) ) 3. —> (68 river-bu-sm ( * r i v e r - l ) ) Command:d 2 1. -> (68 road-td (1629 r e g i o n ) ) 2. -> n i l 3. -> (68 river-bu-sm ( * r i v e r - ! ) ) Command:m 3 0 SCHEDULER: What i s the new p r i o r i t y f o r 3?75 1. -> (75 river-bu-sm ( * r i v e r - l ) ) 2. -> (68 road-td (1629 r e g i o n ) ) Command:e OBJECT: river-bu-with-sketch-map f o r * r i v e r - 1 river-bu-sm: c r e a t e a new r i v e r i n s t a n c e % r i v e r - 2 river-bu-sm: *** model c o n s i s t e n c y f o r r i v e r from sm river-bu-sm: regions l i s t = (130 943) river-bu-sm: i n t e r p r e t a t i o n s are (WATER WATER) river-bu-sm: one i n t e r p r e t a t i o n must be c o n s i s t e n t with water or f a i l river-bu-sm: added neighbouring regions are (2065 1872 1779 1058 874 725 524 208 873) % r i v e r : value added to regions = (943 130 2065 1872 1779 1058 874 725 524 208 873) river-bu-sm: % r i v e r - 2 i s model-consistent % r i v e r : value added to smitem = * r i v e r - 1 river-bu-sm: neighbouring bridge regions are (%bridge-1) SCHEDULER: adding to Queue (70 b r i d g e - t d (8)) river-bu-sm: % r i v e r - 2 must-be-part of some r i v e r - s y s t e m SCHEDULER: adding to Queue (73 river-system-bu ( % r i v e r - 2 ) ) ** 18. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((73 river-system-bu ( % r i v e r - 2 ) ) (70 b r i d g e - t d (8)) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: % r i v e r - s y s t e m bottom-up f o r % r i v e r - 2 % r i v e r - s y s t e m - b u : e x i s t i n g systems %river-system-1 Appendix E 256 %r i v e r - s y s t e m - b u : % r i v e r - 2 added to %river-system-1 ** 19. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((70 b r i d g e - t d (8)) (68 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. -> (70 b r i d g e - t d (8)) 2. -> (68 road-td (1629 r e g i o n ) ) Command:d 1 1. —> n i l 2. -> (68 road-td (1629 r e g i o n ) ) Command:e OBJECT: Top Down on %road. type: region v a l : 1629 SCHEDULER: adding to Queue (68 road-bu-sm (*road-9)) SCHEDULER: adding to Queue (68 road-bu-sm (*road-5)) ** 20. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 road-bu-sm (*road-9)) (68 road-bu-sm (*road-5))) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (68 road-bu-sm (*road-9)) 2. —> (68 road-bu-sm (*road-5)) Command:d 1 1. —> n i l 2. —> (68 road-bu-sm (*road-5)) Command:e ; *road-5 i s the road that runs over *bridge-1 OBJECT: road-bu-with-sketch-map f o r *road-5 road-bu-sm: c r e a t e a new road i n s t a n c e %road-1 road-bu-sm: *** model c o n s i s t e n c y f o r new-road from sm road-bu-sm: deviance = 53.4132 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-4 Appendix E 257 road-bu-sm: edge segments are (253 255) angles (153 140) road-bu-sm: le n g t h = 16 maxstrength = 57 avgstrength = 32.5 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-5 road-bu-sm: edge segments are (263) angles (136) road-bu-sm: le n g t h = 3 maxstrength = 34 avgstrength = 34.0 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-6 road-bu-sm: edge segments are (260 261) angles (136 172) road-bu-sm: le n g t h = 15 maxstrength = 3 avgstrength = 2.5 road-bu-sm: road-bu-sm: road-bu-sm: road-bu-sm: c r e a t e new curb i n s t a n c e %curb-36 road-bu-sm: edge segments are (351) angles (120) road-bu-sm: l e n g t h = 15 maxstrength = 42 avgstrength = 42.0 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-37 road-bu-sm: edge segments are (50) angles (142) leng t h = 10 maxstrength = 47 avgstrength = 47.0 regions l i s t = (2575 2432 1629 9 738 19 18) i n t e r p r e t a t i o n s are ((URBAN HILLS) (URBAN HILLS) (ROAD MOUNTAIN) (URBAN HILLS) (URBAN HILLS) (URBAN HILLS) (ROAD MOUNTAIN)) road-bu-sm: one i n t e r p r e t a t i o n must be c o n s i s t e n t with road or f a i l %road: value added to smitem = *road-5 SCHEDULER: adding to Queue (50 town-td (2575 r e g i o n ) ) SCHEDULER: adding to Queue (50 town-td (2432 r e g i o n ) ) SCHEDULER: adding to Queue (50 town-td (9 r e g i o n ) ) SCHEDULER: adding to Queue (50 town-td (738 r e g i o n ) ) SCHEDULER: adding to Queue (50 town-td (19 r e g i o n ) ) %road: value added to re g i o n s = (18 1629) road-bu-sm: %road-1 i s model-consistent road-bu-sm: neighbouring bridge regions are (%bridge-l) road-bu-sm: neighbouring road regions are n i l SCHEDULER: adding to Queue (50 road-td (18 r e g i o n ) ) SCHEDULER: adding to Queue (50 road-td (1629 r e g i o n ) ) road-bu-sm: neighbouring town regions are n i l road-bu-sm: %road-1 must-be-part of some road-system SCHEDULER: adding to Queue (53 road-system-bu (%road-l)) ** 2 1 ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((53 road-system-bu (%road-1)) (50 town-td (2575 r e g i o n ) ) (50 town-td (2432 r e g i o n ) ) (50 town-td (9 r e g i o n ) ) (50 town-td (738 r e g i o n ) ) (50 town-td (19 r e g i o n ) ) (50 road-td (18 r e g i o n ) ) (50 road-td (1629 r e g i o n ) ) ) Appendix E Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: %road-system bottom-up f o r %road-1 %road-system-bu: e x i s t i n g systems %road-system-1 %road-system-bu: %road-1 added to %road-system-1 ** 22. ************************************* SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((50 town-td (2575 r e g i o n ) ) (50 town-td (2432 r e g i o n ) ) (50 town-td (9 r e g i o n ) ) (50 town-td (738 r e g i o n ) ) (50 town-td (19 r e g i o n ) ) (50 road-td (18 r e g i o n ) ) (50 road-td (1629 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num i , a s-exp, m n1 n2 1 . —> (50 town- td (2575 r e g i o n ) ) 2. —> (50 town- td (2432 r e g i o n ) ) 3. —> (50 town- td (9 r e g i o n ) ) 4. —> (50 town- td (738 r e g i o n ) ) 5. —> (50 town- td (19 r e g i o n ) ) 6. —> (50 road- td (18 r e g i o n ) ) 7. —> (50 road- td (1629 r e g i o n ) ) Command:d 1 1 . —> n i l 2. —> (50 town- td (2432 r e g i o n ) ) 3. —> (50 town- td (9 r e g i o n ) ) 4. —> (50 town- td (738 r e g i o n ) ) 5. —> (50 town- td (19 r e g i o n ) ) 6. —> (50 road- td (18 r e g i o n ) ) 7. —> (50 road- td (1629 r e g i o n ) ) Command:d 2 1 . —> n i l 2. —> n i l 3. —> (50 town- td (9 r e g i o n ) ) 4. —> (50 town- td (738 r e g i o n ) ) 5. —> (50 town- td (19 r e g i o n ) ) 6. —> (50 road- td (18 r e g i o n ) ) 7. —> (50 road- td (1629 r e g i o n ) ) Command:e or e OBJECT: Top Down on %town. type: r e g i o n v a l : 9 Appendix E 259 SCHEDULER: adding to Queue (50 town-bu-sm (*town-l)) ** 23. ************************************* SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((50 town-td (738 r e g i o n ) ) (50 town-td (19 r e g i o n ) ) (50 road-td (18 r e g i o n ) ) (50 road-td (1629 r e g i o n ) ) (50 town-bu-sm (*town-1))) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. -> (50 town-td (738 r e g i o n ) ) 2. -> (50 town-td (19 r e g i o n ) ) 3. -> (50 road-td (18 r e g i o n ) ) 4. -> (50 road-td (1629 r e g i o n ) ) 5. —> (50 town-bu-sm (*town-l)) Command:m 5 0 SCHEDULER: What i s the new p r i o r i t y f o r 5?70 1. —> (70 town-bu-sm (*town-1)) 2. -> (50 town-td (738 r e g i o n ) ) 3. —> (50 town-td (19 r e g i o n ) ) 4. -> (50 road-td (18 re g i o n ) ) 5. -> (50 road-td (1629 r e g i o n ) ) Command:e OBJECT: town-bu-with-sketch-map f o r *town-1 town-bu-sm: c r e a t e a new town ins t a n c e %town-1 town-bu-sm: *** model c o n s i s t e n c y f o r town from sm town-bu-sm: regions l i s t = (9) town-bu-sm: i n t e r p r e t a t i o n s are ((URBAN HILLS)) town-bu-sm: one i n t e r p r e t a t i o n must be c o n s i s t e n t with urban or f a i l town-bu-sm: %town-1 i s model-consistent %town: value added to regions = (9) town-bu-sm: ce n t r e of town at (51 . 81) %town: value added to smitem = *town-1 town-bu-sm: neighbouring road regions are n i l SCHEDULER: adding to Queue (50 road-td (8)) SCHEDULER: adding to Queue (50 road-td (1006)) town-bu-sm: %town-1 must-be-part of some landmass SCHEDULER: adding to Queue (53 landmass-bu (%town-1)) DEMON: adding neighbours l i n k from %road-1 to %town-1 ** 24 ************************************** Appendix E 260 SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((53 landmass-bu (%town-l)) (50 town-td (738 r e g i o n ) ) (50 town-td (19 r e g i o n ) ) (50 road-td (18 r e g i o n ) ) (50 road-td (1629 re g i o n ) ) (50 road-td (8)) (50 road-td (1006))) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? l SUPERVISOR: Do your l i s p t h i n g - - t y p e d or q to stop ? ( s p r i n t '%town-1) * * * * i n s t a n c e : %town-1 * * * G f s t e r e o t y p e : %town c e n t r e : regions: smitem: v a l u e : (51 val u e : (9) % i f - a d d e d : v a l u e : *town-1 % i f - a d d e d : 81) % c o n f i d e n c e : 100 %co n f i d e n c e : 100 (prog n i l ( p r i n t l b 8 "%town: value added t o " "regions =" % v a l ) ) % c o n f i d e n c e : 100 (prog n i l ( p r i n t l b 8 "%town: value added t o " "smitem = " % v a l ) ) apo-> n i l decomposes-to—> n i l neighbours—> %road-1 c o n f - a l g : (prog n i l ( r e t u r n (cond ( ( s g e t v %name 'smitem 'n) 50) (t 25)))) c o n f i d e n c e : 50 n i l ?d ** 25. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((53 landmass-bu (%town-l)) (50 town-td (738 r e g i o n ) ) (50 town-td (19 r e g i o n ) ) (50 road-td (18 r e g i o n ) ) (50 road-td (1629 r e g i o n ) ) Appendix E (50 road-td (8)) (50 road-td (1006))) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (53 landmass-bu (%town-l)) 2. -> (50 town-td (738 re g i o n ) ) 3. -> (50 town-td (19 r e g i o n ) ) 4. -> (50 road-td (18 reg i o n ) ) 5. -> (50 road-td (1629 r e g i o n ) ) 6. -> (50 road-td (8)) 7. -> (50 road-td (1006)) Command:d 7 1. —> (53 landmass-bu (%town-l)) 2. -> (50 town-td (738 re g i o n ) ) 3. —> (50 town-td (19 re g i o n ) ) 4. -> (50 road-td (18 re g i o n ) ) 5. -> (50 road-td (1629 r e g i o n ) ) 6. -> (50 road-td (8)) 7. —> n i l Command:d 6 1. —> (53 landmass-bu (%town-l)) 2. -> (50 town-td (738 r e g i o n ) ) 3. -> (50 town-td (19 re g i o n ) ) 4. -> (50 road-td (18 re g i o n ) ) 5. -> (5.0 road-td (1629 r e g i o n ) ) 6. —> n i l 7. —> n i l Command:d 5 1. -> (53 landmass-bu (%town-1)) 2. -> (50 town-td (738 re g i o n ) ) 3. -> (50 town-td (19 re g i o n ) ) 4. -> (50 road-td (18 re g i o n ) ) 5. —> n i l 6. —> n i l 7. —> n i l Command:m 4 0 SCHEDULER: What i s the new p r i o r i t y f o r 4?60 1. -> (60 road-td (18 reg i o n ) ) 2. —> (53 landmass-bu (%town-l)) 3. -> (50 town-td (738 re g i o n ) ) 4. -> (50 town-td (19 re g i o n ) ) Appendix E Commandite ; region 18 c o n t a i n s s e v e r a l roads OBJECT: Top Down on %road. type: r e g i o n v a l : 18 SCHEDULER: adding to Queue (60 road-bu-sm (*road-3)) SCHEDULER: adding to Queue (60 road-bu-sm (*road-4)) SCHEDULER: adding to Queue (60 road-bu-sm (*road-8)) SCHEDULER: adding to Queue (60 road-bu-sm (*road-7)) SCHEDULER: adding to Queue (60 road-bu-sm (*road-6)) SCHEDULER: adding to Queue (60 road-bu-sm (*road-5)) ** 26. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: ^QUEUE = ( (60 (60 (60 (60 (60 (60 (53 (50 (50 road-bu-road-bu-road-bu-road-bu-road-bu-road-bu-landmass town-td town-td sm (*road-3)) sm (*road-4)) sm (*road-8)) sm (*road-7)) sm (*road-6)) sm (*road-5)) -bu (%town-l)) (738 region) ) (1 9 region) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) Commands: d num, a s-exp, m n1 n2, q, or e 1 . 2. 3. 4. 5. 6. 7. 8. 9. —> —> —> —> —> —> —> —> —> (60 (60 (60 (60 (60 (60 (53 (50 (50 road-bu-sm road-bu-sm road-bu-sm road-bu-sm road-bu-sm road-bu-sm *road-3)) *road-4)) *road-8)) *road-7)) *road-6)) *road-5)) landmass-bu (%town-1)) town-td (738 r e g i o n ) ) town-td (19 re g i o n ) ) Command:m 8 0 SCHEDULER: What i s the new p r i o r i t y f o r 8?70 1. —> (70 town- td (738 r e g i o n ) ) 2. —> (60 road- bu- sm ( *road-3)) 3. —> (60 road- bu- sm ( *road-4)) 4. —> (60 road- bu- sm ( *road-8)) 5. —> (60 road- bu- sm ( *.road-7) ) 6. —> (60 road- bu- sm ( *road-6)) 7. —> (60 road- bu- sm ( *road-5)) 8. —> (53 landmass -bu (%town-1)) 9. —> (50 town- t d (19 r e g i o n ) ) Command:e Appendix E OBJECT: Top Down on %town. type: r e g i o n v a l : 738 SCHEDULER: adding to Queue (70 town-bu-im (738)) ** 27 **************************** *.* ******** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((70 town-bu-im (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (53 landmass-bu (50 town-td (19 738)) *road-3)) *road-4)) *road-8)) *road-7)) *road-6) ) *road-5) ) (%town-1 )) r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: town-bu-image-alone town-bu-im: region = 738 area 2500 town-bu-im: c r e a t e a new town i n s t a n c e %town-2 town-bu-im: *** model c o n s i s t e n c y f o r town from sm %town: value added to regions = 738 town-bu-im: centr e of town at (33 . 66) town-bu-im: %town-2 i s model-consistent town-bu-im: %town-2 must-be-part of some landmass SCHEDULER: adding to Queue (68 landmass-bu (%town-2)) town-bu-im: neighbouring road r e g i o n s are n i l DEMON: adding neighbours l i n k from %road~1 to %town-2 ** 28. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 landmass-bu (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (53 landmass-bu (50 town-td (19 (%town-2)) (*road-3)) (*road-4)) (*road-8)) (*road-7)) (*road-6)) (*road-5)) (%town-1)) r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (68 landmass-bu (%town-2)) 2. -> (60 road-bu-sm (*road-3)) 3. —> (60 road-bu-sm (*road~4)) 4. -> (60 road-bu-sm (*road-8)) Appendix E 264 5. -> (60 road-bu-sm (*road-7)) 6. —> (60 road-bu-sm (*road-6)) 7. —> (60 road-bu-sm (*road-5)) 8. -> (53 landmass-bu (%town-l)) 9. -> (50 town-td (19 r e g i o n ) ) Command:m 2 0 SCHEDULER: What i s the new p r i o r i t y f o r 2?75 1. —> (75 road-bu-sm (*road-3)) 2. -> (68 landmass-bu (%town-2)) 3. —> (60 road-bu-sm (*road-4)) 4. —> (60 road-bu-sm (*road-8)) 5. —> (60 road-bu-sm (*road-7)) 6. —> (60 road-bu-sm (*road~6)) 7. —> (60 road-bu-sm (*road-5)) 8. —> (53 landmass-bu (%town-l)) 9. -> (50 town-td (19 r e g i o n ) ) Command:e ; *road-3 runs "no r t h " from *town-1 OBJECT: road-bu-with-sketch-map f o r *road-3 reate a new road i n s t a n c e %road-2 c r e a t e new curb i n s t a n c e %curb~38 edge segments are (412) angles (104) l e n g t h = 6 maxstrength = 73 avgs t r e n g t h = 73.0 one o r i e n t a t i o n must be c o n s i s t e n t with other curbs or f a i l c r e a t e new curb i n s t a n c e %curb~39 edge segments are (408) angles (90) l e n g t h = 3 maxstrength = 43 avgstrength = 43.0 max con f i d e n c e i s 50.0 regions l i s t = (18) i n t e r p r e t a t i o n s are ((ROAD MOUNTAIN)) one i n t e r p r e t a t i o n must be c o n s i s t e n t with road or f a i l %road: value added to smitem = *road-3 SYSTEM: Warning, more than one i n t e r p r e t a t i o n f o r region 18 SYSTEM: Adding %road %road~2 to (%road %road-1) %road: value added to regions = (18) road-bu-sm: %road-2 i s model-consistent road-bu-sm: neighbouring bridge r e g i o n s are n i l road-bu-sm: neighbouring road r e g i o n s are n i l SCHEDULER: adding to Queue (67 ro a d - t d (18 r e g i o n ) ) road-bu-sm: neighbouring town, r e g i o n s are (%town-1) road-bu-sm: %road-2 must-be-part of some road-system SCHEDULER: adding to Queue (70 road-system-bu (%road-2)) ** 29. ************************************** road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: road- bu- sm: Appendix E 265 SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((70 road-system (68 landmass-bu (67 road-td (18 (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (60 road-bu-sm (53 landmass-bu (50 town-td (19 -bu (%road-2)) (%town-2)) r e g i o n ) ) (*road-4)) (*road-8) ) (*road-7)) (*road-6)) (*road-5)) (%town-1 ) ) r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (70 road-system 2. —> (68 landmass-bu 3. -> (67 road-td (18 4. —> (60 road-bu-sm 5. —> (60 road-bu-sm 6. —> (60 road-bu-sm 7. —> (60 road-bu-sm 8. —> (60 road-bu-sm 9. —> (53 landmass-bu 10. -> (50 town-td (1 -bu (%road-2)) (%town-2)) r e g i o n ) ) (*road-4)) (*road-8)) (*road-7)) (*road-6)) (*road-5) ) (%town-1)) 9 r e g i o n ) ) Command:a (77 road-bu-sm (*road-4)) 1. —> (77 road-bu-sm ( 2. —> (70 road-system-3. —> (68 landmass-bu 4. -> (67 road-td (18 5. —> (60 road-bu-sm ( 6. —> (60 road-bu-sm ( 7. —> (60 road-bu-sm ( 8. —> (60 road-bu-sm ( 9. —> (60 road-bu-sm ( 10. —> (53 landmass-bu 11. -> (50 town-td (19 *road-4)) bu (%road-2)) (%town-2) ) re g i o n ) ) *road-4) ) *road-8) ) *road-7) ) *road-6)) *road-5) ) (%town-1 )) region) ) Command:e ; *road-4 s t a r t s at *road-5 and runs by ; the end of *road-2 OBJECT: road-bu-with-sketch-map f o r *road-4 road-bu-sm: c r e a t e a new road i n s t a n c e %road-3 road-bu-sm: *** model c o n s i s t e n c y f o r road from sm road-bu-sm: deviance = 0.921442 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-40 road-bu-sm: edge segments are (422) angle (24) road-bu-sm: le n g t h = 15 maxstrength = 74 avgstrength = 74.0 road-bu-sm: one o r i e n t a t i o n must be c o n s i s t e n t with other Appendix E 266 curbs or f a i l road-bu-sm: c r e a t e new curb i n s t a n c e %curb-41 road-bu-sm: edge segments are (411) angles (162) road-bu-sm: le n g t h = 5 maxstrength = 41 avgst r e n g t h = 41.0 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-50 road-bu-sm: edge segments are (146) angles (39) road-bu-sm: length = 18 maxstrength = 14 avgst r e n g t h = 14.0 road-bu-sm: c r e a t e new curb i n s t a n c e %curb-5l road-bu-sm: edge segments are (160 159 161) road-bu-sm: curbangles are (45 180 180) road-bu-sm: le n g t h = 12 maxstrength = 86 avgst r e n g t h = 74.667 road-bu-sm: max con f i d e n c e i s 53.986 road-bu-sm: regions l i s t = (438 738 18) road-bu-sm: i n t e r p r e t a t i o n s are ((URBAN HILLS) (URBAN HILLS) (ROAD MOUNTAIN)) road-bu-sm: one i n t e r p r e t a t i o n must be c o n s i s t e n t with road or f a i l , %road: value added to smitem = *road-4 SCHEDULER: adding to Queue (64 town-td (438 r e g i o n ) ) SYSTEM: Warning, more than one i n t e r p r e t a t i o n f o r region 18 SYSTEM: Adding %road %road-3 to (%road %road-2 %road %road-1) %road: value added to regions = (18) road-bu-sm: %road~3 i s model-consistent road-bu-sm: neighbouring bridge regions are n i l road-bu-sm: neighbouring road regions are (%road-1 %road-2) SCHEDULER: adding to Queue (64 road-td (18 r e g i o n ) ) road-bu-sm: neighbouring town regions are n i l road-bu-sm: %road-3 must-be-part of some road-system SCHEDULER: adding to Queue (67 road-system-bu (%road-3)) ** 30. ************************************* SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((70 road-system-bu (%road-2)) (68 landmass-bu (%town-2)) (67 road-system-bu (%road-3)) (67 road-td (18 r e g i o n ) ) (64 town-td (438 r e g i o n ) ) (64 road-td (18 r e g i o n ) ) (60 road-bu-sm (*road-4)) (60 road-bu-sm (*road-8)) (60 road-bu-sm (*road-7)) (60 road-bu-sm (*road-6)) (60 road-bu-sm (*road-5)) (53 landmass-bu (%town-l)) (50 town-td (19 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? l Appendix E 267 SUPERVISOR: Do your l i s p t h i n g — t y p e d or q to stop ? ( s p r i n t '%road-3) * * * * i n s t a n c e : %road-3 * * * 0 f s t e r e o t y p e : %road smitem: ends: s e p a r a t i o n : deviance: r e g i o n s : apo—> decomposes-to—> neighbours—> c o n f - a l g : conf idence: v a l u e : *road-4 % i f-added: % c o n f i d e n c e : 100 (prog ... v a l u e : ((41 . 62) (29 . 126)) %conf idence: v a l u e : n i l va l u e : 0.921442 v a l u e : (18) % i f - a d d e d : n i l (%curb-40 %curb-41 %curb-45 %curb-46 %curb-49 %curb-50 (%road-1 %road-2) (prog ( v a l 1st) 64 100 %c o n f i d e n c e : n i l %c o n f i d e n c e : n i l %c o n f i d e n c e : 100 (prog ... %curb-42 %curb-47 %curb-51) %curb-43 %curb-48 %curb-44 %curb-48 n i l ?d ** 3 1 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((70 road-system-(68 landmass-bu (67 road-system-(67 road-td (18 (64 town-td (438 (64 road-td (18 (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (53 landmass-bu (50 town-td (19 bu (%road-2)) (%town-2) ) bu (%road-3)) r e g i o n ) ) r e g i o n ) ) r e g i o n ) ) *road-4)) *road-8)) *road-7)) *road-6)) *road-5)) (%town-1 )) region) ) ) Do. you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: %road-system bottom-up f o r %road-system-bu: e x i s t i n g systems %road-2 %road-system-1 Appendix E %road-system-bu: new schema c r e a t e d %road-system-2 %road-system-bu: component added %road-2 SCHEDULER: adding to Queue (68 landmass-bu (%road-system-2)) ** 32. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((68 landmass-bu (%road-system-2)) (68 landmass-bu (%town-2)) (67 road-system-bu (%road-3)) (67 road-td (18 r e g i o n ) ) (64 town-td (438 r e g i o n ) ) (64 road-td (18 r e g i o n ) ) (60 road-bu-sm (*road-4)) (60 road-bu-sm (*road-8)) (60 road-bu-sm (*road-7)) (60 road-bu-sm (*road-6)) (60 road-bu-sm (*road-5)) (53 landmass-bu (%town-l)) (50 town-td (19 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e 1. —> (68 landmass-bu (%road-system 2. —> (68 landmass-bu (%town-2)) 3. —> (67 road-system- bu (%road-3)) 4. —> (67 road-td (18 r e g i o n ) ) 5. —> (64 town-td (438 r e g i o n ) ) 6. —> (64 road-td (18 r e g i o n ) ) 7. —> (60 road-bu-sm ( *road-8)) 8. —> (60 road-bu-sm ( *road-7)) 9. —> (60 road-bu-sm ( *road-6)) 10 . —> (60 road-bu-sm (*road-5)) 1 1 . -> (53 landmass-bu (%town-1)) 12 . -> (50 town-td (19 r e g i o n ) ) ) ) Command:d 1 1 . —> n i l 2. —> (68 landmass-bu (%town-2)) 3. —> (67 road-system- bu (%road-3)) 4. —> (67 road-td (18 r e g i o n ) ) 5. —> (64 town-td (438 r e g i o n ) ) 6. —> (64 road-td (18 r e g i o n ) ) 7. —> (60 road-bu-sm ( *road-8)) 8. —> (60 road-bu-sm ( *road-7)) 9. —> (60 road-bu-sm ( *road-6)) 10. —> (60 road-bu-sm (*road-5)) 11. —> (53 landmass-bu (%town-l)) 12. -> (50 town-td (19 r e g i o n ) ) ) ) Appendix E 269 Command:e OBJECT: %landmass bottom-up f o r %town-2 %landmass-bu: e x i s t i n g systems (%landmass-1) %landmass-bu: %town-2 added to %landmass-1 ** 33. ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((67 road-system-(67 road-td (18 (64 town-td (438 (64 road-td (18 (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (60 road-bu-sm ( (53 landmass-bu (50 town-td (19 bu ^ r o a d -region) ) r e g i o n ) ) r e g i o n ) ) *road-4)) *road-8)) *road-7)) *road-6)) *road-5)) (%town-1)) r e g i o n ) ) ) 3)) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? y Commands: d num, a s-exp, m n1 n2, q, or e —> —> —> (67 road-system-bu (%road-3)) (67 road-td (18 r e g i o n ) ) (64 town-td (438 r e g i o n ) ) -> (64 road-td (18 r e g i o n ) ) —> (60 road-bu-sm (*road-4)) -> (60 road-bu-sm (*road-8)) (60 road-bu-sm (*road-7)) (60 road-bu-sm (*road-6)) 9. -> (60 road-bu-sm (*road-5)) —> (53 landmass-bu (%town-1)) —> (50 town-td (19 r e g i o n ) ) —> —> 10 1 1 Command:m 10 1 SCHEDULER: What i s the new p r i o r i t y f o r 10767 1. —> (67 road-system- bu (%road-3)) 2. —> (67 landmass-bu (%town-1 ) ) 3. —> (67 road-td (18 r e g i o n ) ) 4. —> (64 town-td (438 r e g i o n ) ) 5. —> (64 road-td (18 r e g i o n ) ) 6. —> (60 road-bu-sm ( *road-4)) 7. —> (60 road-bu-sm ( *road-8)) 8. —> (60 road-bu-sm ( *road-7)) 9. —> (60 road-bu-sm ( *road-6)) • 10 . -> (60 road-bu-sm (*road-5)) 1 1 . -> (50 town-td (19 r e g i o n ) ) Appendix E Command:e OBJECT: %road-system bottom-up f o r %road-system-bu: e x i s t i n g systems %road-system-bu: %road-system-bu: %road-3 (%road-system-2 road-system-1 ) %road-3 added to %road-system-2 merging %road-system-1 i n t o %road-system-2 ** 34 ************************************** SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ( (67 (67 (64 (64 (60 (60 (60 (60 (60 (50 landmass road-td town-td road-td road-bu-road-bu-road-bu-road-bu-road-bu-town-td -bu (%town-l)) (18 re g i o n ) ) (438 r e g i o n ) ) (18 re g i o n ) ) sm (*road-4)) sm (*road-8)) sm (*road-7)) sm (*road-6)) sm (*road-5)) (19 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? n OBJECT: %landmass bottom-up for %town-1 %landmass-bu: e x i s t i n g systems (%landmass-1) %landmass-bu: %town-1 added to %landmass-1 ** 35, * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * SCHEDULER: Here i s the p r i o r i t y queue: %QUEUE = ((67 (64 (64 (60 (60 (60 (60 (60 (50 road-td town-td road-td road-bu-road-bu-road-bu-road-bu-road-bu-town-td (18 r e g i o n ) ) (438 r e g i o n ) ) (18 r e g i o n ) ) sm (*road-4)) sm (*road-8)) sm (*road-7)) sm (*road-6)) sm (*road-5)) (19 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) ? l ***of s t e r e o t y p e : %bridge ? ( s p r i n t ' % b r i d g e - l ) * * * * i n s t a n c e : %bridge-1 smitem: v a l u e : *bridge-1 % i f-added: % c o n f i d e n c e : 100 (prog . . . o r d e r l i s t v a l u e : (0 S B B 0) %c o n f i d e n c e : 100 Appendix E neighbour-regions: v a l u e : (2018 1750 9 130) % c o n f i d e n c e : 100 shadowregions: ro a d r e g i o n s : v a l u e : (1186) v a l u e : (1629) % c o n f i d e n c e : 100 % c o n f i d e n c e : 100 decomposes-to—> (%curb-3 %curb-2 % c u r b - l ) neighbours-> (%road-1 % r i v e r - 2 % r i v e r - 1 ) apo—> (%road-system-2 %river-system-1) c o n f - a l g : (prog ... c o n f i d e n c e : 68 n i l ? ( s p r i n t ' % c u r b - l ) * * * * i n s t a n c e : %curb-1 ***of s t e r e o t y p e : %curb a v g s t r e n g t h : v a l u e : 3.5 %conf idence: 75 maxstrength: v a l u e : 5 %conf idence: 75 l e n g t h : v a l u e : 20 %conf idence: 100 a n g l e s : v a l u e : (248 214) %conf idence: (80) edgesegs: v a l u e : (88 89) %conf idence: 100 type: v a l u e : shadow %conf idence: 100 apo—> c o n f - a l g : conf idence: %bridge-1 (prog . 85 n i l ? ( s p r i n t '%curb-2) * * * * i n s t a n c e : %curb-2 *** :of s t e r e o t y p e : %curb a v g s t r e n g t h : v a l u e : 96.33 %conf idence: 100 maxstrength: v a l u e : 1 34 %conf idence: 100 l e n g t h : v a l u e : 54 %conf idence: 100 a n g l e s : v a l u e : (90 57 79 %conf idence: 45 90 27) (80) edgesegs: v a l u e : (10 11 12 %conf idence: 13 14 15) 100 type: v a l u e : road %conf idence: 100 Appendix E 272 apo-> c o n f - a l g : conf idence: %bridge-1 (prog 93 n i l ? ( s p r i n t '%curb-3) * * * * i n s t a n c e : %curb-3 ***of s t e r e o t y p e : %curb av g s t r e n g t h : v a l u e : 18.0 %conf idence: 100 maxstrength: val u e : 22 %conf idence: 100 l e n g t h : v a l u e : 20 %conf idence: 100 a n g l e s : v a l u e : (228 270) %conf idence: (80) edgesegs: val u e : (242 243) %conf idence: 100 type: v a l u e : road %conf idence: 100 apo—> %bridge-1 c o n f - a l g : conf idence: (prog 93 n i l ? % i n s t a n c e s (%landmass-1 %road-system-2 %curb-9 %curb-8 %road-3 %curb-7 %curb-6 %road-2 %town-2 %town-1 %curb-5 %curb-4 %road -1 % r i v e r - 2 % r i v e r - 1 %geosystem-3 %geosystem-2 %geosystem -1 %waterbody -1 %river-system- 1 %curb-3 %curb-2 %curb - 1 % b r i d g e - l ) ? ( s p r i n t '%road-system-2) * * * * i n s t a n c e : %road-system-2 * * * Q f s t e r e o t y p e : %road-system smitem: val u e : n i l % i f-added: ^confidence: n i l (prog ... neighbours—> n i l decomposes-to-> (%bridge-1 %road-1 %road-3 %road-2) apo—> %landmass-1 c o n f - a l g : (prog ... c o n f i d e n c e : 63 ? ( s p r i n t ' % r i v e r - l ) * * * * i n s t a n c e : % r i v e r - 1 * * * 0 f s t e r e o t y p e : % r i v e r r e g i o n s : v a l u e : (1750 2295 2095 2018) %c o n f i d e n c e : 100 % i f - a d d e d : (prog ... Appendix E smitem: v a l u e : * r i v e r - 2 % c o n f i d e n c e : 100 % i f - a d d e d : (prog ... % r iver-system-1 n i l %bridge-1 (prog ... 70 apo—> decomposes-to—> neighbours—> c o n f - a l g : conf idence: ? ( s p r i n t '%landmass-1) * * * * i n s t a n c e : %landmass-1 ***of s t e r e o t y p e : %landmass smitem: v a l u e : n i l % c o n f i d e n c e : n i l % i f - a d d e d : (prog ... neighbours—> n i l decomposes-to—> (%town-1 %road-system-2 )-ako—> %geosystem-1 c o n f - a l g : (prog ... c o n f i d e n c e : 63 ? ( s s p r i n t n '%geosystem-3) %geosystem-3 a i o %geosystem in s t a n c e s %geosystem-3 %geosystem-2 apo %geosystem-3 spec i a l i z e s - t o %waterbody-1 ako %geosystem-2 a i o %waterbody i n s t a n c e s %waterbody-1 s p e c i a l i z e s - t o %river-system-1 a i o % r i v e r - s y s t e m i n s t a n c e s % r iver-system-1 ako %waterbody-1 spec i a l i z e s - t o % r i v e r - 2 apo % r iver-system-1 a i o % r i v e r Appendix E i n s t a n c e s % r i v e r - 2 % r i v e r - 1 apo %river-system-1 a i o % r i v e r neighbours %bridge-1 a i o % b r i d ge i n s t a n c e s %bridge-1 decomposes-to %curb-3 apo %bridge-1 a i o %curb i n s t a n c e s %curb-51 apo %road-3 apo %road-system-2 apo %landmass-1 ako %geosystem-1 apo %geosystem-3 s p e c i a l i z e s %landmass-1 a i o %geosystem a i o %landmass i n s t a n c e s - -%landmass-1 decomposes-to %road-system-2 %town-2 apo %landmass-1 a i o %town ins t a n c e s %town-2 %town-1 apo %landmass a i o %town Appendix E neighbours %road-2 apo %road-system-2 a i o %road i n s t a n c e s %road-3 %road-2 %road-1 a i o %road decomposes-to-%curb-37 apo %road-1 a i o %curb • %curb-4 apo %road-1 a i o %curb neighbours %road-3 %town-2 %town-1 %bridge-1 apo %road-system-2 decomposes-to %curb-39 apo %road-2 a i o %curb %curb-38 apo %road-2 a i o %curb neighbours %road-3 %town-1 %road-1 neighbours %road-1 %town-1 a i o %road-system Appendix E i n s t a n c e s — % r oad - s y s t em-2 decomposes - to % b r i d g e - 1 %road-1 % road -3 % road -2 a i o % road d e c o m p o s e s - t o ~ -% c u r b - 5 l %cu rb -50 apo % road -3 a i o %cu rb • %cu rb -40 apo % road -3 a i o %cu rb n e i g h b o u r s % road -2 %road-1 a i o %cu rb %cu rb -50 %cu rb -2 apo %b r i d ge - 1 a i o %curb %curb-1 a p o — -%b r i d ge - 1 a i o %curb %cu rb - 2 %curb-1 n e i g h b o u r s %road-1 % r i v e r - 2 % r i v e r - 1 apo % r oad - s y s t em-2 %r i v e r - s y s t e m - 1 n e i g h b o u r s Append ix E %bridge-1 % r i v e r - 1 %bridge-1 a i o %geosystem %geosystem-1 decomposes-to %geosystem-1 %geosystem-2 n i l ? ( s p r i n t n '%river-system-1 'decomposes-to) %river-system-1 % r iver-1 % r i v e r - 2 %br idge-1 %curb-1 %curb-2 %curb-3 n i l ? ( s p r i n t n '%landmass-1 'decomposes-to) %landmass-1 %town-1 %road-system-2 %bridge-1 %curb-1 %curb-2 %curb-3 % r o a d - l %curb-4 %curb-37 %road-2 %curb-38 %curb-39 %road-3 %curb-40 %curb-51 n i l ? ( s p r i n t n '%geosystem-3 'decomposes-to) %geosystem-3 %geosystem^1 %geosystem-2 n i l ?d ** 3g ************************************** SCHEDULER: Here i s the p r i o r i t y queue: Appendix E %QUEUE = ((67 road-td (18 r e g i o n ) ) (64 town-td (438 r e g i o n ) ) (64 road-td (18 r e g i o n ) ) (60 road-bu-sm (*road-4)) (60 road-bu-sm (*road-8)) (60 road-bu-sm (*road-7)) (60 road-bu-sm (*road-6)) (60 road-bu-sm (*road-5)) (50 town-td (19 r e g i o n ) ) ) Do you want to modify i t ( y ) , be i n l i s p ( l ) , or q u i t ( q ) SUPERVISOR: Cycl e broken by user--bye n i l -> % Appendix E 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0051869/manifest

Comment

Related Items