T H E RAPID R E C O V E R Y OF THREE-DIMENSIONAL STRUCTURE F R O M LINE DRAWINGS by RONALD A N D Y RENSINK B . S c . ( P h y s i c s ) , T h e University of W a t e r l o o , 1979 M . S c . ( P h y s i c s ) , T h e University of B r i t i s h C o l u m b i a , 1982 M . S c . ( C o m p u t e r Science), T h e University of B r i t i s h C o l u m b i a , A THESIS S U B M I T T E D IN P A R T I A L F U L F I L L M E N T THE REQUIREMENTS FOR THE D E G R E E OF D O C T O R OF PHILOSOPHY in T H E F A C U L T Y OF G R A D U A T E STUDIES D E P A R T M E N T OF C O M P U T E R SCIENCE W e accept this thesis as conforming to the required s t a n d a r d T H E U N i v ^ R S I T Y OF BRITISH September COLUMBIA 1992 Â© R o n a l d A n d y Rensink, 1992 1986 OF In presenting this degree at the thesis in University of partial fulfilment of of department this or thesis for by his or scholarly purposes may be her representatives. permission. of <^<^-^p<yi/v7ii The University of British Columbia Vancouver, Canada Date DE-6 (2/88) gyV. 2 ' ^z, sc/py^ce for an advanced Library shall make it agree that permission for extensive It publication of this thesis for financial gain shall not Department requirements British Columbia, I agree that the freely available for reference and study. 1 further copying the is granted by the understood that head of copying my or be allowed without my written Abstract A c o m p u t a t i o n a l theory is developed that explains how line drawings of p o l y h e d r a l o b j e c t s can be interpreted r a p i d l y a n d i n p a r a l l e l at early levels of h u m a n vision. T h e key i d e a is that a t i m e - l i m i t e d process can correctly recover much of the three-dimensional s t r u c t u r e of these objects when split into concurrent streams, each concerned w i t h a single aspect of scene structure. T h e work proceeds i n five stages. T h e first extends the framework of M a r r to allow a process to be analyzed i n terms of resource l i m i t a t i o n s . T w o m a i n concerns are identified: (i) reducing the amount of nonlocal information needed, and (ii) m a k i n g effective use of whatever i n f o r m a t i o n is obtained. T h e second stage traces the difficulty of line i n t e r p r e t a t i o n to a s m a l l set of constraints. W h e n these are removed, the remaining constraints c a n be grouped into several relatively independent sets. It is shown that each set can be r a p i d l y solved by a separate processing stream, a n d that co-ordinating these streams can yield a lowcomplexity " a p p r o x i m a t i o n " that captures much of the structure of the o r i g i n a l constraints. In p a r t i c u l a r , complete recovery is possible i n l o g a r i t h m i c time when objects have rectangular corners a n d the scene-to-image projection is orthographic. T h e t h i r d stage is concerned w i t h m a k i n g g o o d use of the available i n f o r m a t i o n when a fixed time l i m i t exists. T h i s l i m i t is m o t i v a t e d by the need to o b t a i n results w i t h i n a time independent of image content, a n d by the need to l i m i t the propagation of inconsistencies. A m i n i m a l architecture is assumed, v i z . , a spatiotopic mesh of simple processors. Constraints are developed to guide the course of the process itself, so t h a t candidate interpretations are considered i n order of their l i k e l i h o o d . T h e f o u r t h stage provides a specific a l g o r i t h m for the recovery process, showing how it can be i m p l e m e n t e d on a cellular a u t o m a t o n . F i n a l l y , the theory itself is tested on various line drawings. It is shown t h a t m u c h of the three-dimensional structure of a p o l y h e d r a l scene can indeed be recovered i n very little t i m e . It also is shown that the theory can explain the r a p i d i n t e r p r e t a t i o n of line drawings at early levels of h u m a n vision. Contents Abstract ii List o f F i g u r e s vi List of Tables ix Acknowledgements x 1 Introduction 1 1.1 The Problem 2 1.2 The Approach 4 1.3 L i m i t a t i o n s and K e y A s s u m p t i o n s 6 2 Background 8 2.1 R a p i d P a r a l l e l Processing 8 2.1.1 C o m p u t a t i o n a l Studies 9 2.1.2 P s y c h o p h y s i c a l Studies 19 2.1.3 C o m p u t a t i o n a l versus Psychophysical Studies 26 2.2 2.3 2.4 T h e Interpretation of L i n e Drawings 28 2.2.1 C o m p u t a t i o n a l Studies 28 2.2.2 P s y c h o p h y s i c a l Studies 34 2.2.3 C o m p u t a t i o n a l versus Psychophysical Studies 38 H i g h - l e v e l versus Low-level V i s i o n 39 2.3.1 T h e Structure of Low-level V i s i o n 39 2.3.2 T h e Role of R a p i d ParaUel Recovery 42 T h e A n a l y s i s of R e s o u r c e - L i m i t e d Processes 43 2.4.1 44 M a r r ' s Framework 2.5 3 Extensions 45 2.4.3 A Revised Framework 47 R a p i d Line Interpretation 50 2.5.1 B a s i c Terms 50 2.5.2 F o r m u l a t i o n of the P r o b l e m 51 Low-Complexity Recovery 54 3.1 G e n e r a l Issues 55 3.1.1 Concurrent Streams 55 3.1.2 R e d u c t i o n to C a n o n i c a l Forms 56 3.1.3 A p p r o x i m a t i o n Strategies 60 3.2 3.3 4 2.4.2 I n d i v i d u a l Dimensions 61 3.2.1 Contiguity Labelling 62 3.2.2 Convexity Labelling 67 3.2.3 Slant Sign L a b e l l i n g 70 3.2.4 Slant M a g n i t u d e L a b e l l i n g 75 Integration of Dimensions 78 3.3.1 C o n v e x Objects 79 3.3.2 C o m p o u n d C o n v e x Objects 81 3.3.3 Rectangular Objects 85 Computational Analysis 100 4.1 E x t e r n a l Constraints 101 4.1.1 Image-to-Scene m a p p i n g 101 4.1.2 G e n e r a l Principles 103 4.1.3 Structural Assumptions 105 4.1.4 System of E x t e r n a l C o n s t r a i n t s 106 4.2 4.3 Internal C o n s t r a i n t s 108 4.2.1 Processing A r c h i t e c t u r e 109 4.2.2 G e n e r a l Principles 110 4.2.3 Selection of I n i t i a l Candidates 118 T h e R a p i d Recovery Process 124 4.3.1 A r c h i t e c t u r a l Specifications 124 4.3.2 Robustness 126 4.3.3 5 129 5.1 T h e C e l l u l a r Processor 129 5.1.1 B a s i c aspects 130 5.1.2 C e l l u l a r Processors as Cellular A u t o m a t a 131 5.1.3 Programming 133 5.3 A l g o r i t h m for R a p i d Recovery 136 5.2.1 D e t e r m i n a t i o n of Basic Image Properties 137 5.2.2 D e t e r m i n a t i o n of J u n c t i o n Properties 137 5.2.3 I n i t i a l Assignment of Interpretations 142 5.2.4 P r o p a g a t i o n of Interpretations 142 5.2.5 F i n a l Assignment of Results 144 N e u r a l Implementation 144 Tests of the T h e o r y 146 6.1 Performance on Line Drawings 146 6.1.1 Rectangular Objects 147 6.1.2 N o n c o n f o r m i n g Objects 151 6.1.3 Impossible Objects 157 6.2 7 126 A l g o r i t h m and Implementation 5.2 6 Basic Operation P r e a t t e n t i v e Recovery of Scene Structure 162 6.2.1 Basic Assumptions 162 6.2.2 E x p l a n a t i o n of Psychophysical Results 165 S u m m a r y a n d Conclusions 173 List of Figures 1.1 E a r l y recovery of three-dimensional structure 3 2.1 Linkage between zone and surrounding locations 10 2.2 Types of junctions 31 2.3 Huffman-Clowes labelling set 32 2.4 Penrose triangle 36 2.5 E x t e n d e d c o m p u t a t i o n a l framework 48 3.1 L i n k i n g of l o c a l constraints 59 3.2 Separation into i n d i v i d u a l dimensions 62 3.3 C o n t i g u i t y labelling 63 3.4 Set of contiguity constraints 64 3.5 Inconsistent drawing w i t h consistent contiguity labelling 65 3.6 R e f o r m u l a t i o n of contiguity constraints 66 3.7 Set of convexity constraints 68 3.8 Inconsistent d r a w i n g w i t h consistent convexity labelling 68 3.9 R e f o r m u l a t i o n of convexity constraints 69 3.10 Slant sign labelling 71 3.11 C o n s t r a i n t s on isolated L-junctions 72 3.12 Slant sign constraints for arrow- a n d Y - j u n c t i o n s 73 3.13 Slant sign labeUings for rectangular corners 74 3.14 Huffman-Clowes labellings for convex objects 80 3.15 E x a m p l e s of c o m p o u n d convex objects 82 3.16 Huffman-Clowes labellings for c o m p o u n d convex objects 82 3.17 Free chain complexes 84 3.18 E x a m p l e s of rectangular objects 85 3.19 C o n s t r a i n t s on L-junctions 86 3.20 Huffman-Clowes labeUings for rectangular objects 86 3.21 P l a n a r i t y constraint 87 3.22 Interior angle constraint 87 3.23 C o n t i g u i t y constraints on obtuse L - j u n c t i o n combinations 89 3.24 T e r m i n a t i o n configurations 90 3.25 C o n t i g u i t y constraints on Y - j u n c t i o n combinations 92 3.26 Slant sign constraints applied to impossible figures 95 3.27 C o n d i t i o n s of shallow slant 97 3.28 C o m b i n a t i o n s of angles into corners 97 3.29 Rescaling of image angles 98 4.1 Isolation of inconsistency i n contiguity labelling 103 4.2 S y s t e m of external constraints 107 4.3 E x a m p l e of complementary labelling 112 4.4 E x a m p l e of reformulation of bijective constraint 115 4.5 E x a m p l e of reformulation of nonbijective constraint 115 4.6 I n i t i a l contiguity interpretations 120 4.7 I n i t i a l convexity interpretations 122 4.8 I n i t i a l slant-sign interpretations 123 4.9 E x a m p l e of the r a p i d recovery process 128 5.1 C e l l u l a r processor architecture 130 5.2 C a l c u l a t i o n of orientation differences 139 5.3 D e t e r m i n a t i o n of contiguity relations 141 6.1 Interpretation of convex rectangular object 149 6.2 Interpretation of nonconvex rectangular object 150 6.3 Interpretation of occluded rectangular objects 152 6.4 Interpretation of nonrectangular object 154 6.5 Interpretation of origami object 155 6.6 Interpretation of nonplanar object 156 6.7 Interpretation of object of inconsistent contiguity and convexity 158 6.8 Interpretation of object of inconsistent slant 160 6.9 I n t e r p r e t a t i o n of object of inconsistent depth 161 6.10 Results explained by theory 163 6.11 Slant estimates for C o n d i t i o n A 166 6.12 Slant estimates for C o n d i t i o n B 167 6.13 Slant estimates for C o n d i t i o n C 169 6.14 Slant estimates for C o n d i t i o n E 170 6.15 Slant estimates for C o n d i t i o n G 172 List of Tables 2.1 C o m p l e x i t i e s of coherence classes Acknowledgments T h i s work has benefited greatly from the contributions of m a n y people. In p a r t i c u l a r , I w o u l d like to t h a n k the members of m y committee for their help during m y stay here at U B C . M a n y t h a n k s to J i m E n n s , w h o has worked closely w i t h me on m a n y of the psychophysical experiments described here, a n d who has taught me much about the w o r l d of psychophysical e x p e r i m e n t a t i o n . T h a n k s also to J i m L i t t l e , who has been a source of interesting discussions on the n a t u r e of p a r a l l e l c o m p u t a t i o n a n d its role i n early vision, a n d to D a v i d Lowe for his support a n d help i n b r i d g i n g the worlds of c o m p u t a t i o n a l and biological vision. I a m also indebted to A l a n M a c k w o r t h a n d W h i t m a n Richards for their m a n y helpful comments a n d suggestions. I have been extremely fortunate i n h a v i n g several friends a n d associates able to critique various aspects of this work a n d who were k i n d enough to actually do so. T h i s work has been m u c h i m p r o v e d by their efforts. 1 would p a r t i c u l a r l y like to t h a n k Esfandiar B a n d a r i , James E l d e r , M a r c i a G r a b o w e c k y , R i c h a r d M a n n , R i c h a r d P o l l o c k , G r e g P r o v a n , M a r i o n Rodrigues, a n d M a r c R o m a n y c i a for their help i n this regard. Space restrictions do not allow me t o m e n t i o n a l l the others who also have contributed, but I w o u l d like to assure t h e m that their help is not forgotten. People can of course contribute to one's life i n m a n y other ways, a n d here again I have been rather fortunate i n meeting m a n y people who have helped make m y stay here an enjoyable experience. In addition to the people mentioned above, I would also like to t h a n k Debbie A k s , FranÃ§oise G u y a u x , C h r i s Healey, Julie Johnson, Valerie M c R a e , D a n M c R e y n o l d s , Jane M u l l i g a n , D a n RazzeH, L a n a T r i c k , Lindsey Wey, a n d C a r o l W h i t e h e a d . I wish you aU well. F i n a l l y , 1 would like to express m y deepest gratitude to m y supervisor. B o b W o o d h a m , for a l l the guidance a n d support I have received over the years. I have learned from h i m a great deal about the art of f o r m u l a t i n g and solving a scientific p r o b l e m , a n d this work is i n m a n y ways an attempt to meet his high standards. W i t h o u t this guidance, and without the tolerance he has shown for m y interests i n sometimes rather esoteric fields, this work would not have been possible. Chapter 1 Introduction Those aspects of h u m a n vision most directly involved w i t h the i n c o m i n g image have a characteristic mode of operation: they are r a p i d (usually completed w i t h i n several hundred m i l liseconds), s p a t i a l l y p a r a l l e l (operating simultaneously across the visual field), a n d a u t o m a t i c (unaffected by changes i n goals during the course of processing). T h i s has led to an assumpt i o n t h a t these " e a r l y " processes determine only simple geometric and radiometric properties of the image, e.g., line o r i e n t a t i o n , color, a n d contrast. There is considerable support for this a s s u m p t i o n on c o m p u t a t i o n a l grounds â€” these are the only kinds of properties can be reliably determined by s p a t i a l l y - l i m i t e d processors operating w i t h i n a fixed amount of t i m e . To reliably determine properties of the corresponding scene, therefore, a later stage of more t i m e - c o n s u m i n g operations is needed. T h i s division i n t o early a n d later processes has formed the basis for m a n y c o m p u t a t i o n a l and psychophysical studies of the h u m a n v i s u a l system. However, the underlying assumption is false â€” for some Images, recovery of scene properties can be done at early stages of processing, r a p i d l y a n d i n p a r a l l e l [Ram88, E R 9 0 a , E R 9 1 ] . In figure 1.1(a), for example, the d r a w i n g of the block w i t h a unique three-dimensional orientation can be detected almost Immediately. However, this is not possible when these drawings are altered slightly (figure 1.1(b)), showing that this phenomenon is not due to simple image properties alone, but t o some aspect of the recovered scene structure. T h e goal of this thesis is t o explain how properties of the scene can be recovered r a p i d l y and i n p a r a l l e l at early levels of visual processing. In p a r t i c u l a r , it develops a c o m p u t a t i o n a l theory of how the h u m a n v i s u a l system can r a p i d l y interpret line drawings to o b t a i n the threedimensional structure of the corresponding p o l y h e d r a l objects. Since the general problem of line i n t e r p r e t a t i o n is N P - c o m p l e t e [ K P 8 8 ] , a great deal of time may sometimes be r e q u i r e d for its solution, even when p a r a l l e l processing is used. If recovery is to be r a p i d , therefore, it cannot be based on this m a p p i n g , but rather must be based on an a p p r o x i m a t i o n i n w h i c h the reliabihty a n d completeness of the output have been lowered to some degree. T h e central idea developed here is that a good a p p r o x i m a t i o n can be obtained by s p l i t t i n g the recovery process into several quasi-independent streams, each based on a set of constraints that can be quickly solved. It is shown that relatively few constraints need to be altered i n order to achieve this decomposition, and that the resulting "quick ajid d i r t y " process can recover a s u b s t a n t i a l amount of scene structure i n very little t i m e . It is also shown that this m o d e l can explain the recovery of three-dimensional structure at early levels of h u m a n v i s i o n . In common w i t h other areas of c o m p u t a t i o n a l analysis, this study is first a n d foremost concerned w i t h how Information can be used by a visual system. For r a p i d recovery, however, the structure of the p r o b l e m is no longer dictated entirely by the optics of the s i t u a t i o n â€” Instead, Umits on processing t i m e must also be taken Into account. T h i s work shows how this perspective can be Incorporated Into a c o m p u t a t i o n a l framework, and how It can lead t o a new source of constraints on the representations and processes used In early vision. 1.1 The Problem In what follows, the scene d o m a i n Is taken to be the set of opaque p o l y h e d r a l objects w i t h t r i h e d r a l corners. T h e t e r m ' t r i h e d r a l ' Is used here In a narrow sense, referring to corners formed f r o m the Intersection of three planar surfaces i n such a way that only three edges can radiate f r o m any vertex, a n d that the vertex cannot contact any other edge. T h e Image d o m a i n Is the corresponding set of drawings formed by the projection of these objects o n t o the image plane. T h e r a p i d recovery process must recover f r o m these drawings as m u c h of the scene structure as possible w i t h i n some fixed amount of time. T h e goal of this work Is to develop a c o m p u t a t i o n a l theory of this process, one which accounts for those aspects of three-dimensional structure recovered In h u m a n early vision. T h e r e are several reasons for this choice of problem. F i r s t , there Is evidence that h u m a n vision actually does recover three-dimensional structure r a p i d l y a n d In parallel at early levels [ E R 9 0 b , EI191, E R 9 2 ] . T h e phenomenon is a striking and robust one, w i t h a strong sensitivity to the arrangement of the lines. A s such, there Is considerable p o t e n t i a l for m a k i n g predictions about the kinds of line arrangements for w h i c h recovery wiU a n d wiU not be successful. (a) (b) Figure 1.1: E a r l y recovery of three-dimensional structure. A line drawing that corresponds to a distinct three-dimensional block can be detected almost immediately when the block slants upwards (a). R o t a t i n g the page so that this block slants downwards causes detection to become more difficult, showing that slant has an asymmetry t y p i c a l of many properties of early vision (see [ T G 8 8 , E R 9 0 b ] ) . W h e n line relations are slightly altered (b), detection is equally difficult under a l l conditions (also see [ER91]), indicating that slant is not recovered at a l l . Second, a great deal is k n o w n about the l i m i t s to which three-dimensional structure can be recovered f r o m line drawings,^ this problem having been the focus of several decades of work i n the area of c o m p u t a t i o n a l vision (see section 2.2.1). Moreover, the general p r o b l e m of line i n t e r p r e t a t i o n has been shown to be N P - c o m p l e t e [KP85]. Since the time required to solve an N P - c o m p l e t e problem can (in the worst case) increase exponentially w i t h its size,^ this rules out the possibiUty that the process can always be sped up by parallel processing alone. F i n a l l y , of a l l the r a p i d recovery processes, line interpretation is perhaps that w h i c h most severely taxes the abilities of early v i s i o n . Relations between image and scene are m o r e tenuous here t h a n for most other recovery processes; indeed, m a n y aspects of line i n t e r p r e t a t i o n are often considered to be learned conventions (see, e.g., [Sug86]). T h u s , i f a mechanism can be found for the r a p i d interpretation of line drawings, it becomes plausible that similar mechanisms might also exist for recovery processes based on more realistic associations between image a n d scene. 1.2 T h e Approach For a t i m e - l i m i t e d process, the goal is no longer to extract a l l available i n f o r m a t i o n f r o m an image, but r a t h e r to make g o o d use of the available c o m p u t a t i o n a l resources. T w o factors are therefore of p r i m a r y concern: (i) m i n i m i z i n g the sheer amount of d a t a t r a n s f o r m a t i o n a n d transmission t h a t needs to be carried out i n parallel, and (ii) m a x i m i z i n g the effectiveness of these transformations i n e x t r a c t i n g three-dimensional structure. T h i s work examines how these two factors influence the structure of the recovery process at the levels of c o m p u t a t i o n , algorithm, and implementation. C h a p t e r 2 provides the background m a t e r i a l for this analysis. It begins w i t h a survey of the m a j o r e m p i r i c a l a n d theoretical results o n the limits of r a p i d parallel processing. T h i s is followed by an overview of the i m p o r t a n t results concerning the recovery of three-dimensional structure f r o m line drawings. A discussion is then presented of the ways i n w h i c h these two 'Theories of line interpretation, however, have rarely taken into account noise and other distortions of the image. Complications also arise from shadow edges and texture boundaries. In the interests of simplicity, these will not be discussed here. ^Although there remains a possibility that NP-complete problems are in class P (i.e., can be carried out in polynomial time in the worst case), this situation appears highly unlikely [GJ79, Joh90]. the possibility would stiU exist that such problems are P-complete, substantially increased by the use of parallel processing [GR88]. Even if P = N P , meaning that their speed could not be threads can be d r a w n together. N e x t , M a r r ' s framework of c o m p u t a t i o n a l analysis is extended to cover the case of resource-limited processes. T w o sorts of c o m p u t a t i o n a l constraints are distinguished: " e x t e r n a l " constraints on the static form of the m a p p i n g between image a n d scene, a n d " i n t e r n a l " constraints that guide the course of the process that generates i t . T h e analysis of r a p i d recovery itself begins i n chapter 3, which examines the ways i n w h i c h the image-to-scene m a p p i n g used i n the general problem of Une interpretation can be replaced by an a p p r o x i m a t i o n of lower complexity. In p a r t i c u l a r , it shows that low-complexity recovery can be carried out by weakening the constraints to allow their separation into independent subsets, each concerned w i t h a single aspect of the scene. Four such aspects are considered: the contiguity of edges, the positive convexity of edges, the sign of edge slants, a n d the magnitude of edge slants. It is shown that each of these subsets can be solved i n subUnear t i m e by a processing stream containing a sufficiently large set of parallel processors, and that this complexity is not increased when interaction between the streams involves o n l y a one-way transmission of i n f o r m a t i o n . A l t h o u g h the interpretative power of the resultant m a p p i n g is somewhat reduced, a considerable amount remains; indeed, it is shown t h a t contiguity, convexity, a n d slant can be recovered completely i n l o g a r i t h m i c time when a l l corners are rectangular, i.e., composed of m u t u a l l y orthogonal surfaces. T h e next step is to develop constraints that m a x i m i z e the UkeUhood of successful Interp r e t a t i o n when a Umlt Is placed on processing time. T h i s Is done In chapter 4. A fixed amount of t i m e Is assumed to be available. T h i s choice Is consistent w i t h the limits t y p i c a l for an early v i s u a l process, a n d also has the advantage that the propagation of inconsistencies Is localized. In keeping w i t h this mlnlmaUst vein, c o m p u t a t i o n a l resources are l i m i t e d to a mesh of simple processors. A set of external constraints Is developed to h m l t the space of possible Interpretations. Four principles are used for the choice of constraints: separation of dimensions, l o c a l i t y of constraints, l o c a l coordination of dimensions, and the s t r u c t u r a l a s s u m p t i o n t h a t the corners of the p o l y h e d r a are rectangular. Internal constraints are then developed that guide the course of the recovery process through this space of possible solutions. These are based on four principles: maintenance of interpretative power, locally irreversible c o m p u t a t i o n , m i n i m i z a t i o n of Inconsistency, and an ordering of search to select preferred interpretations of m a x i m u m contiguity and convexity. T a k e n together, the extern a l a n d i n t e r n a l constraints define a process capable of recovering a considerable amount of three-dimensional structure In very h t t l e time. A l t h o u g h the e x t e r n a l a n d internal constraints limit the way i n which the process uses i n f o r m a t i o n , they do not completely specify an a l g o r i t h m . C h a p t e r 5 provides this specificat i o n , a n d implements the resulting a l g o r i t h m on a mesh architecture. T h i s is done v i a the device of a cellular processor. T h i s mechanism is formed by p a r t i t i o n i n g the image into a set of disjoint " c e l l s " , each governed by a Unite-state processing element that can be p r o g r a m m e d to execute a few simple operations on the contents of its ceU, and that can communicate only w i t h its i m m e d i a t e neighbor. A s such, it obeys the general architectural Umitations assumed for the c o m p u t a t i o n a l analysis while simultaneously being easy to control and analyze. T h e resulting a l g o r i t h m provides an existence proof that r a p i d line interpretation can be done on an architecture of the assumed type. T h e final step of the work is to test the theory on a c t u a l line drawings. In chapter 6, the process is tested on domains that range f r o m those i n which a l l underlying assumptions are obeyed to impossible figures w h i c h cannot correspond to any k i n d of polyhedron at a l l . It is shown that a considerable amount of three-dimensional structure can indeed be recovered i n very little t i m e , a n d that this process degrades gracefully as the underlying s t r u c t u r a l assumptions about the scene d o m a i n are violated. These results are then used as the basis of predictions about the kinds of line drawings that can and cannot be r a p i d l y detected by the h u m a n visual s y s t e m . T h e theory is shown to be capable of explaining the ability of early h u m a n vision to recover three-dimensional structure r a p i d l y a n d i n parallel. 1.3 Limitations and K e y Assumptions Before e m b a r k i n g on the development of the theory, it is i m p o r t a n t t o acknowledge a n u m b e r of Umitations a n d assumptions that could potentially limit its relevance. F i r s t of a l l , the treatment here is concerned exclusively w i t h the r a p i d recovery of three-dimensional structure f r o m line drawings. T h e advantage of this approach is that the p r o b l e m d o m a i n is small a n d has a simple m a t h e m a t i c a l description, m a k i n g the analysis of the recovery process as simple as possible. B u t this domain is a highly artificial one â€” the figures contain no gaps or any other k i n d of noise, nor do they describe cracks and markings which are found on just about any real surface. Indeed, these stimuli are so artificial that the analysis runs the risk of saying n o t h i n g at a l l about processes that interpret more reahstic images. T h e scope of the theory is also l i m i t e d by the architectural constraints required for t h e c o m p u t a t i o n a l analysis. T h e analysis here assumes only a two-dimensional array of relatively simple processors, each connected only to its immediate neighbors (chapter 4). Since this l i m i t a t i o n is relatively severe, the resulting process provides a lower b o u n d to what might be reasonably expected f r o m a spatiotopic array of processors. B u t the assumption of a m i n i m a l processing architecture also means that the predictions are applicable only to the extent t h a t such an architecture actually is representative of that used i n h u m a n early vision. T h e r e is also considerable l a t i t u d e i n the choice of the finer details of the theory. Several of the choices made here are somewhat tentative, intended only to show that such a t h e o r y can be developed. Consequently, they are unUkely to w i t h s t a n d the test of time. Insofar as the theory can explain the r a p i d recovery of three-dimensional orientation by the early h u m a n v i s u a l system, it assumes that this system actually does carry out this process. Results t o date [ E R 9 0 b , E R 9 1 , E R 9 2 ] show that the recovery of three-dimensional o r i e n t a t i o n at early levels is sufficient to explain most k n o w n results concerning the sensitivity of early vision to line drawings of opaque p o l y h e d r a . B u t while this sensitivity cannot be explained i n terms of simple operations on the image (e.g., spatial filtering), there still remains the possibility of some other "image-based" explanation, e.g., a sensitivity to p a r t i c u l a r s p a t i a l relations between the lines, or the " l o a d i n g - i n " of a complete object m o d e l v i a lookup t h a t based on image features (e.g. [PE90]). F i n a l l y , even i f three-dimensional orientation actually is computed at these early levels, there is still no guarantee that the process is i n any way a t t e m p t i n g to make g o o d use of available c o m p u t a t i o n a l resources. E v o l u t i o n often produces biological systems that are adequate r a t h e r t h a n o p t i m a l (see, e.g., [Ram85, Gou89]), and it may well be that r a p i d recovery falls Into this category. those based on effectiveness, If so, its operation Is governed by constraints other t h a n a n d the c o m p u t a t i o n a l model developed here w i l l be largely Irrelevant for explaining h u m a n performance. A scientific theory, however, u l t i m a t e l y succeeds or falls to the degree that It explains phenomena In a succinct way, a n d suggests new avenues of research to explore (e.g., [Lak78]). A s the following chapters show, the theory developed here Is able to account for the recovery of three-dimensional structure at early levels of h u m a n vision, and can make predictions as to w h a t other kinds of line drawings can a n d cannot be recovered i n this way. F u r t h e r m o r e , It does so by developing principles that are potentially applicable to other areas of perception and c o g n i t i o n , and t o b o t h artificial and biological systems. Chapter 2 Background T h e p r o b l e m of r a p i d line i n t e r p r e t a t i o n is a fusion of two concerns that historically have been quite separate: (i) determining the extent to which properties can be extracted r a p i d l y and i n p a r a l l e l f r o m an image, a n d (ii) determining the extent to w h i c h line drawings can be interpreted as opaque p o l y h e d r a l objects. T h u s , a g o o d way to begin is to survey the p r i n c i p a l developments i n each of these areas. It is then shown how aspects of b o t h can be usefuUy combined into a r a p i d recovery process, and how this process can be analyzed by an extension of the c o m p u t a t i o n a l framework of M a r r [Mar82]. 2.1 Rapid Parallel Processing T h e earliest stages of v i s u a l processing are characterized by the u n i f o r m application of relatively simple operations at each location i n the visual field (see, e.g., [Zuc87b, L B C 8 9 ] ) . It is evident that the problems solved at these levels make great use of parallelism, w i t h one or more processors assigned to each p a t c h of the image. It is less evident, however, what the l i m i t s of this k i n d of processing might be. T h i s section surveys some of the m a i n results p e r t a i n i n g to r a p i d parallel processing. T h e o r e t i c a l results are presented first, w i t h discussion focusing on the way i n which a problem's structure determines its complexity on a p a r a l l e l processor. T h i s is followed by an overview of what is known about the extent of r a p i d parallel processing i n h u m a n early v i s i o n . 2.1.1 Computational Studies T w o different routes can be taken when studying the limits of parallel processing. The first starts w i t h a given architecture and then determines its suitability for various classes of problems. Such "processor-dependent" analysis is widely used, p a r t i c u l a r l y to ascertain the capabilities of an existing machine (e.g., [ P D 8 4 , L B C 8 9 ] ) . on problems rather t h a n architectures per se. B u t the emphasis here is Consequently, a " d u a l " approach is t a k e n : a class of problems is specified a n d the suitability of various architectures for this class t h e n e x a m i n e d . T h i s approach can be based on the amount of coherence i n the m a p p i n g between input a n d o u t p u t image. It is shown t h a t this coherence has a large influence on the l i m i t s to which an operation can be sped up by a parallel architecture. A . Basics To estabUsh what is meant by the " s u i t a b i l i t y " of an architecture for a p a r t i c u l a r p r o b l e m , consider first a network of T u r i n g machines joined together by h i g h - b a n d w i d t h connections. Such a " m a x i m a l " architecture obviously allows the greatest use to be made of p a r a l l e l i s m , regardless of p r o b l e m structure. Its generality, however, means that c o m p u t a t i o n a l resources are often wasted. A natural architecture is therefore defined as one which best matches t h e given p r o b l e m , i.e., w h i c h uses a m i n i m a l set of resources to carry out the task. Such a n architecture can be obtained (conceptually, at least) by starting w i t h a m a x i m a l architecture a n d then weakening the power of the i n d i v i d u a l processors and the communication patterns between t h e m u n t i l a change occurs i n the time or space required. T h e m i n i m a l configurations at each of these transitions are exactly the n a t u r a l architectures. Since different choices of t i m e a n d space bounds are possible, there is usually more t h a n one n a t u r a l architecture for a given p r o b l e m . In general, finding the n a t u r a l architecture for a problem is difficult â€” even the m a p p i n g of neighbors i n the problem space to neighbors i n the architecture is N P - h a r d ^ [ N K P 8 7 ] . B u t when problems are based on the m a p p i n g of images to images^ the coherence i n the m a p p i n g simplifies m a t t e r s considerably (see, e.g., [Sto87, Sto88]). F i n d i n g a n a t u r a l architecture for a p r o b l e m then reduces to determining its m a p p i n g coherence a n d r e l a t i n g it to the time a n d ' T h i s means that the problem is at least as hard as any NP-complete task. ^Without loss of generality, the calculation of a lower-dimensional result can be expressed as a mapping in which the output image contains repeated instances of the result. For example, calculation of the average value in an image can be expressed in terms of an output image containing the average at each location. <â€” - s â€¢ â€¢ â€”Â» Forward Linkaae + From zone in Input T s â€¢ To image in output - Contributes to result ' â€¢ Backward Linkage From image in input To zone In output - Selects local operator F i g u r e 2.1: Linkage between zone a n d surrounding locations, space required on various kinds of architectures. i) Mapping Coherence To begin w i t h , let a zone be some sxs that each mxm p a t c h of pixels i n the image. Zones m a y overlap, so image contains ( m - s ) ^ zones. M a p p i n g coherence is described here i n t e r m s of how the input inside a zone influences the output image i n the surrounding locations (figure 2.1). A zone is said to interact with the rest of the image if there is at least one direction i n which the range of this influence is u n l i m i t e d . T h e linkage of the m a p p i n g is defined as t h e number of degrees of freedom i n this interaction."' For strictly l o c a l operations, such as those t y p i c a l of early vision, no interaction exists between a zone a n d locations sufficiently far away i n the o u t p u t . These are consequently zeroUnkage problems - any changes w i t h i n a zone are propagated only a finite distance away. A t the opposite extreme, consider the sorting of image intensities. Here, changing any of t h e p i x e l values i n the zone potentially changes the value of the output at a location a r b i t r a r i l y far away. T h e linkage is therefore p r o p o r t i o n a l to zone area. ^Many of the ideas presented here regarding mapping coherence have their origin in the work of Stout [Sto87, Sto88]. However, the definition of linkage used here is considerably different, being based on degrees of freedom in a more uniform fashion. Distinctions such as unidirectional and bidirectional linkage, as well as the resulting set of coherence classes, are also novel. Linkages can r u n b o t h ways, however, so that two kinds of problem can be distinguished. In unidirectional problems, the information t r a n s m i t t e d from the zone can be computed purely locally â€” no p i x e l values are needed from the rest of the image beyond some finite surrounding region. F o r example, determining the average pixel intensity requires only one parameter (the s u m of p i x e l intensities) to be t r a n s m i t t e d from any zone. Since the pixels outside the zone do not affect this value, this interaction is clearly unidirectional, w i t h n o outside i n f o r m a t i o n needed. In bidirectional problems, on the other h a n d , the interaction between a zone a n d its surroundings runs b o t h ways: not only do changes i n the zone influence the rest of the image, but the rest of the image influences what is required of the zone. M o r e precisely, t h e i n f o r m a t i o n to be t r a n s m i t t e d from a zone cannot be determined i n isolation for b i d i r e c t i o n a l problems, since values f r o m the rest of the image are needed to select the appropriate q u a n t i t y to be calculated locally. A n example of this is the calculation of the median intensity of a n image i n w h i c h the range of p i x e l values is unHmited (e.g., r a n d o m variables w i t h a gaussian d i s t r i b u t i o n ) . A single degree of freedom can be assigned to the l o c a l o u t p u t : the n u m b e r of pixels above (or below) the global median. B u t the value of this median must first be t r a n s m i t t e d to the zone, and this value may be affected by changes i n p i x e l intensities at locations a r b i t r a r i l y far away.^ In essence, the k i n d of linkage back f r o m the image t o the zone reflects the amount of contextual i n f o r m a t i o n needed to select the appropriate l o c a l o p e r a t i o n (figure 2.1). G i v e n this characterization of m a p p i n g coherence, problems can be grouped according to the strength a n d directionality of their Unkage.^ Four classes are considered here: zero linkage, constant linkage, linkage p r o p o r t i o n a l to zone perimeter, and linkage p r o p o r t i o n a l t o zone area. C o n s t a n t - a n d perimeter-hnkage problems are further divided into u n i d i r e c t i o n a l and b i d i r e c t i o n a l subclasses. A n y operation involved i n visual processing can be placed i n t o one of these classes,^ a n d it is shown below that placement into a class puts bounds on its c o m p l e x i t y on various kinds of architectures. *In a sense, the difference between unidirectional and bidirectional problems corresponds to that between deterministic and nondeterministic problems: in the unidirectional case, the outputs of isolated zones are sufficient to produce the solution, whereas in the bidirectional case, they are sufficient only to verify it. (For a discussion of the relation between deterministic and nondeterministic problems see, e.g., [GJ79].) ^ T h e strength of the linkages can be different in the two directions. But the simple classification into unidirectional and bidirectional problems is sufficient for present purposes. ^Operations having a structure that does not match well with the examples discussed here (e.g., those involving fractal quantities) wUl require a finer division of coherence classes. But as a first approximation, lower bounds for such problems can be obtained by "rounding down" to the nearest coherence class. ii) Complexity Measures T h e complexity of a p r o b l e m can be analyzed i n a relatively processor-independent w a y v i a the methods of complexity theory (see, e.g., [Baa78, G J 7 9 , JohQO]). Here, t h e basic unit is taken t o be the time required to merge t w o independent quantities into one â€” for example, the a d d i t i o n or m u l t i p l i c a t i o n of two numbers, or the testing of their equality. T h e complexity of a given a l g o r i t h m is then measured by the number of such operations required for t h e most difficult case i n t h e p r o b l e m set.^ T h e complexity of a given problem is t h a t of the least-complex a l g o r i t h m capable of solving it on a given architecture. Differences i n the speed of basic operations â€” such as arise i n different mechanical or biological systems â€” are eliminated by the use of 0 - n o t a t i o n , which describes the t i m e o n l y to w i t h i n a constant factor. A n a l g o r i t h m is said t o require 0 ( / ( n ) ) time i f there exist positive constants c, d a n d N such that for any input of size n > N, the time cf(n) < T{n) < df(n). If the a l g o r i t h m is the least complex k n o w n t o solve the problem, the complexity of the p r o b l e m is said t o have a n upper b o u n d of 0(f{n)). Similarly, the problem has a lower complexity b o u n d Q,{g{n)) when any a l g o r i t h m t o solve it must have a complexity of at least If a p r o b l e m is bounded above by 0(f{n)) a n d below by Q(f(n)), 0{g{n)). it has (exact) complexity 0 ( / ( n ) ) (see, e.g., [Baa78, Har87]). Defined i n this way, the time needed t o solve a problem is - t o w i t h i n a p o l y n o m i a l factor - independent of the p a r t i c u l a r set of instructions of the machine (see, e.g., [Har87]). T h i s q u a n t i t y is therefore an invariant of the problem, (see, e.g., [Baa78, T W W 8 8 ] ) , a n d so can be used for abstract, machine-indifferent analysis [RP91]. A s for the case of m a p p i n g coherence, processes can be grouped into various complexity classes. (For a g o o d survey of these classes, see [Joh90].) One of these is the set P of processes that c a n be carried out i n p o l y n o m i a l t i m e ; such processes m a y have a different complexity on different (serial) architectures, but this complexity will always be p o l y n o m i a l (see, e.g., [ G J 7 9 , Har87]). T h i s class can be further subdivided according to the degree that complexity is lowered by the i n t r o d u c t i o n of parallel processing. T h e class NC is defined as the set of problems h a v i n g subUnear complexity when a sufficient number of processors are provided.^ '^Complexity measures can also be based on average-case analysis, as well as on a probabilistic analysis that ignores exceptional cases of small measure (see [TWW88]). However, worst-case measures are those most often used, in part because of the relative ease of analysis. These measures are also preferred here since they avoid the need to develop extra procedures to handle cases in which the computational hmitations are exceeded. *More precisely, these are problems of complexity 0{\og,^ n) when 0{n'') processors are available (where exponents k,p G Z). In contrast, a class of " P - c o m p l e t e " problems has been found that is apparently i n c a p a b l e of being sped up this way; i n essence, these problems remain "serial" no m a t t e r how m a n y processors are allowed (see, e.g. [GR88]). N o t e that this view of complexity is based on the number of operations needed to d a t a a n d so the time needed for d a t a transmission combine across space is often ignored. W h i l e this is suitable for m a n y situations, it is less so for others, especially for operations on images, where d a t a is often moved a r o u n d a considerable distance during the course of the c o m p u t a t i o n . Transmission delays are severe i n biological systems (where speeds are typically on the order of I m / s [She83]), a n d are also a factor i n the operation of machine systems [Uhr87, p. 261]. W h e n a p p l y i n g complexity measures to image-processing problems,^ therefore, the effects of transmission delay must be kept i n m i n d . Hi) Architectural Parameters G i v e n t h a t the complexity of an image-processing problem depends on the u n d e r l y i n g architecture, it is i m p o r t a n t to estabUsh what the relevant parameters might be. In w h a t foUows, architectures are described by graphs where each node represents a separate processi n g element ( P E ) a n d each edge a direct connection between the corresponding P E s . T h u s , a m a x i m a l architecture corresponds to a complete graph i n w h i c h each P E (equivalent to a T u r i n g machine) is directly connected to aU the others. T h i s m o d e l is superficiaUy different f r o m the parallel r a n d o m access machine ( P R A M ) often used i n theoretical studies of p a r a l l e l processing (e.g., [GR88]), since the P R A M is defined as an abstract machine w i t h a shared m e m o r y i m m e d i a t e l y accessible to any of the processors.â€¢'^ B u t this shared m e m o r y allows direct c o m m u n i c a t i o n between P E s , a n d so the P R A M and m a x i m a l architectures are essentially equivalent. In this f o r m u l a t i o n , the complexity of a problem can be analyzed by t r a c i n g the flow of i n f o r m a t i o n t h r o u g h the network. T h e p a t h taken by each piece of i n f o r m a t i o n can be represented by a p a t h t h r o u g h the g r a p h that begins at the point where it is picked up i n the image, a n d terminates at its final position(s) i n the o u t p u t . T h e nodes of the g r a p h are Â®As used here, the terms 'image-processing problem' and 'image operation' are largely synonymous. The only difference is that the specification of a problem does not necessarily contain an explicit rule to obtain the output from the input, whereas this is generally true of the term 'operation'. ^"Strictly speaking, this characterizes only the most powerful variant: the P R I O R I T Y concurrent read concurrent write ( C R C W ) P R A M . Since no other P R A M variants will be considered here, this qualification wiU not be explicitly mentioned. assigned weights representing the time required for local computation.^^ A weight can be assigned to each d a t a p a t h by accumulating the weights of aU nodes encountered along the way. T h e processing t i m e for a c o m p u t a t i o n is then the m a x i m u m weight of all the d a t a p a t h s i n the c o m p u t a t i o n . T h u s , complexity is largely governed by two sets of parameters: (i) those concerned w i t h the processing resources available to each P E , a n d (ii) those concerned w i t h the p a t t e r n of d a t a transmission between P E s . A wide variety of processing elements are possible for a p a r a l l e l architecture. A t one extreme, each P E has the power of a T u r i n g machine and can operate completely independently of the others. T h i s is basically the multiple i n s t r u c t i o n , m u l t i p l e data-stream ( M I M D ) architecture [Fly72], i n w h i c h each P E may carry out a different set of instructions. Weakeni n g the power of the P E s decreases their abiUty to respond to different signals so that they become less able to respond to the structure of the image and less flexible i n c o m m u n i c a t i n g w i t h their neighbors. In the extreme case this becomes a single i n s t r u c t i o n , multiple d a t a s t r e a m ( S I M D ) architecture [Fly72], where a l l P E s operate i n lockstep, carrying out the same o p e r a t i o n everywhere i n the network. A similar spectrum of possibilities exists for d a t a transmission. T h e simplest network is a t w o - d i m e n s i o n a l ^/n X ^/n array of n processing elements. Here, each of the processors is assigned to some p a r t i c u l a r zone or set of zones i n the image, a n d operates i n complete isolation f r o m the others. T h i s is the k i n d of architecture generally thought to exist at the very earliest stages of visual processing, i.e., the r e t i n a and the striate cortex (e.g., [RobSO]). T h e simplest form of processor-processor interaction occurs i n the mesh, a network, where each P E i n the array is connected w i t h its nearest neighbors (e.g., [Ros83]). Here, d a t a can be sent f r o m any P E to any other P E w i t h time p r o p o r t i o n a l to the distance i n the mesh. Transfer can be greatly sped up (at least i n terms of the number of switches involved) by way of a pyramid network, i n which a hierarchical communication structure is used. T h e basic -v/n X ^/n mesh forms the lowest level of this hierarchy. T h i s mesh is then p a r t i t i o n e d into a set of A; X A; nonoverlapping sections, w i t h the P E s i n each section then connected to a single P E i n a higher-level y/nik x y/n/k mesh. T h i s higher-level mesh is i n t u r n sectioned and connected to the P E s i n a still higher-level mesh, this process continuing u n t i l only one P E exists at the highest level (see, e.g. [Ros86]). T h e resulting structure is hierarchical, a l l o w i n g any two P E s separated by distance m to communicate t h r o u g h O ( l o g m ) switches. '^In this view, switches and memories are regarded as nodes corresponding to simple computations. Linkage Zero Array Mesh Pyramid Hypercube 0(1) 0(1) 0(1) 0(1) 0(n) 0(logn) 0(logra) ? 0{\og^n) ? 0(n) 0(7zi/2) 0(logn) 0{n'') 0{n) ? Constant (unidirectional) Constant (bidirectional) Perimeter (unidirectional) Perimeter (bidirectional) ? Area f2(n) O(logn) Table 2.1: Complexities of coherence classes. P y r a m i d networks have been proposed for several of the more " g l o b a l " processes of v i s i o n , such as line t r a c i n g [Ede87] a n d selective visual attention [ K U 8 4 , Tso90] A more highly-connected architecture is that of the hypercube (e.g., [Hil84]). A d- dimensional hypercube has 2'^ corners; i f the positions of neighbors i n the hypercube differ by some constant distance a > 0 along any dimension, corners w i l l be separated by at most a distance of ad. T h u s , i f n = 2'^ P E s are connected such that each corresponds t o a different corner of the hypercube, then t w o P E s can communicate w i t h i n O ( l o g n ) time. A l t h o u g h this is the same as for t h e p y r a m i d , t h e greater number of possible paths yields a greater effective b a n d w i d t h , w h i c h allows the hypercube t o avoid the bottlenecks that can arise at t h e higher levels of the p y r a m i d [StoST]. â€¢ B . Classes of I m a g e - P r o c e s s i n g Problems T h i s section presents the m a j o r results k n o w n about the limits to which various kinds of image-processing problems can be sped up by parallel processing. Problems are grouped according the coherence of the corresponding m a p p i n g between input a n d o u t p u t . A r r a n g e d i n this way, an interesting p a t t e r n emerges f r o m these results - the lower-bound complexities due to d a t a transmission are the same for a l l problems i n any coherence class (table 2.1.1). A n d these lower bounds prove t o be the dominant factors i n the complexity of m a n y imageprocessing problems. i) Zero-linkage Problems B y definition, zero-linkage problems have no interaction between a zone and a l o c a t i o n that is sufficiently far away. These are exactly the problems best handled by local operations. T h e simplest of these are local measurements, l i m i t e d template across the image. i.e., the uniform apphcation of a s p a t i a l l y - These include pointwise remappings of intensity (e.g., g a m m a correction) a n d convolutions by functions of l i m i t e d spatial extent. M o r e generally, zero-Unkage problems include those that can be solved using properties of fixed support, i.e., where the p r o p e r t y can be extracted f r o m a fixed set of points i n each zone [UU84]. T h e l i m i t o n transmission distance means that each P E can complete its operation w i t h i n a fixed t i m e independent of image size, a n d so these problems have 0 ( 1 ) complexity. T h e l i m i t e d t i m e and spatial extent also mean that each P E need only be a finite-state a u t o m a t o n . A n a t u r a l architecture for a zero-linkage problem is therefore a simple array i n which each finite-state P E takes its i n p u t f r o m the corresponding zone i n the image. A l t h o u g h c o m m u n i c a t i o n time is m i n i m a l i n an array, an extensive amount of w i r i n g is usually required t o connect pixels to their P E s , especially if the zones are large. F u r t h e r m o r e , such a network w o u l d be impossible to reconfigure when a different zone size is required. These drawbacks are largely eliminated by using a mesh. Here, the input is p a r t i t i o n e d i n t o nonoverlapping sections, w i t h each P E t a k i n g its i n p u t from a single section. Since P E s do not generally have direct access to a l l information i n a zone, information must be t r a n s m i t t e d t h r o u g h the mesh. In essence, a mesh trades off time for space. For zero-linkage problems, the n a t u r a l mesh architecture is the cellular automaton [TM87, C H Y 9 0 , T M 9 0 ] , for which the processing elements are simple finite-state a u t o m a t a . By U m i t i n g the n u m b e r of iterations allowed for each P E , a cellular automaton can c a r r y out zero-linkage problems such as spatial filtering [PD84]. A s the P E s are given more power, they are able to combine simple measurements i n interesting ways â€” for example, to determine the color or o r i e n t a t i o n of hne segments by comparing the magnitudes among a basic set of l o c a l measurements (see, e.g., [Gra85]). ii) Constant-linkage Problems Constant-Unkage problems are characterized by a positive Unkage whose strength does not depend o n the size of the zone. T w o variants can be distinguished: u n i d i r e c t i o n a l a n d bidirectional. Unidirectional problems One of the simplest unidirectional problems is to determine the average value of t h e pixels i n an image. A s discussed i n section 2.1.1, determining this quantity requires o n l y one parameter (the s u m of p i x e l intensities) from any zone. Similarly, the c a l c u l a t i o n of the standard deviation is also undirectional, requiring two parameters (the sum of p i x e l intensities, together w i t h the s u m of their squares) to be obtained from each zone. Other problems which can be formulated this way include finding the m i n i m u m distance between black (or white) pixels i n the image [Sto87], determining the center of mass [Tan84], a n d detecting h o r i z o n t a l or v e r t i c a l concavities [Sto87]. A H these tasks can be carried out i n O(logn) time on a p y r a m i d architecture [Tan84, Ros86, Sto87]. In general, l o g a r i t h m i c complexity can be achieved for any process i n w h i c h each P E reduces the d a t a f r o m the level below it to a constant amount, and passes this d a t a upwards [Sto87]. T h e p y r a m i d is consequently a n a t u r a l architecture for the entire class of u n i d i r e c t i o n a l constant-Unkage problems. Hypercubes allow no additional reductions of complexity. Bidirectional problems Relatively little work has been done on this class of problems. D e t e r m i n i n g the m e d i a n can be done i n O(log'^ra) t i m e on a p y r a m i d ; it is not k n o w n whether this quantity can be lowered [Sto87]. A n o t h e r b i d i r e c t i o n a l constant-linkage problem is the determination of extreme points, i.e., those points located at the corners of the smallest convex p o l y g o n containing a l l points i n the image. T h e complexity of this problem also is 0(log'^ n) on a p y r a m i d [Sto87]. It m a y be that this (provisional) h m i t applies to a l l such problems. Hi) Perimeter-linkage Problems A large set of problems can be characterized by linkage p r o p o r t i o n a l to the perimeter of the zone. A g a i n , b o t h u n i d i r e c t i o n a l a n d bidirectional variants can be distinguished. Unidirectional problems T h i s class is exemplified by connected component labelling ( C C L ) , where each distinct component i n the image is to be assigned a unique l a b e l . N o t e that a special case of this p r o b l e m is the determination of whether aU lines i n the image are connected. P r o v i d e d that the components passing t h r o u g h the perimeter of a zone are correctly labelled (as far as the zone is concerned), no other aspect of the zone's contents are needed to solve this p r o b l e m . T h e number of degrees of freedom is therefore equal to the number of perimeter crossings. A s s u m i n g a u n i f o r m d i s t r i b u t i o n of components in the image, this is directly p r o p o r t i o n a l to perimeter length. A n o t h e r perimeter-linkage problem is the determination of the lengths of a l l Unes i n the image. Here, two parameters (label and t o t a l length inside the zone) are required at each perimeter crossing. C C L can be done i n Q{n) time on a mesh [Sto88], 0(ra^/^) time on a p y r a m i d [ M S 8 7 ] , and 0 ( l o g ( n ) ) time o n a hypercube [ L A N 8 9 ] . T h i s latter Umit is the same when a P R A M is used [SV82]. It m a y be that these Umits also apply to the other problems i n this class. Bidirectional problems T h i s class includes problems of constraint r e l a x a t i o n , w h i c h are characterized by l o c a l measurements that require context for their complete interpretation [HZ83, K I 8 5 ] . Since these problems depend only o n l o c a l constraints, the effect of a zone on the result is determined by a b a n d of pixels along the border, the exact w i d t h depending on the range of the l o c a l process. Linkage is consequently p r o p o r t i o n a l to the length of this border. B u t values i n t h e l o c a l zone must also be consistent w i t h their surroundings, m a k i n g the problem b i d i r e c t i o n a l . T w o types of relaxation p r o b l e m exist: continuous a n d discrete. For continuous relajca t i o n , l o c a l values (as weU as interaction terms) are represented as real numbers. Among other things, this allows non-zero probabiUties to be assigned to different interpretations of any l o c a l feature. T h e problems themselves are generally formulated i n terms of m a x i m i z i n g or m i n i m i z i n g some global quantity, w h i c h then allows t h e m to be recast as finite difference equations [HZ83]. One p a r t i c u l a r l y interesting set of problems involves reconstructing surfaces by finding the e x t r e m u m of some global measure such as the smoothness or error of the reconstructed surface. T h i s approach is the basis of general frameworks of visual processing such as regularization theory [ P T K 8 5 ] a n d M a r k o v r a n d o m fields [ G G 8 4 ] . C o n t i n u o u s r e l a x a t i o n can also be formulated i n terms of Unear p r o g r a m m i n g [ B B 8 2 , p. 420-430]. Since Unear p r o g r a m m i n g can be done i n p o l y n o m i a l time [Kar84], it is Ukely that continuous r e l a x a t i o n is of this complexity. For problems that can be cast as the solution of eUiptical equations (either Unear or nonUnear), the number of iterations required to solve the p r o b l e m to w i t h i n a given accuracy is 0{n^), where L is the order of the equation [Bra77, p. 281]. O n a p y r a m i d architecture, where multiresolution techniques [Bri87] can be used. These are often referred to as relaxation processes. T h e y are described here as problems, however, since they are abstract specifications of input-output mappings that are quite independent of the particular processes used to carry them out. this is reduced to 0{n) iterations [Gla84]. In contrast to continuous r e l a x a t i o n , discrete relaxation requires that values assigned to pixels be integers, and that only one interpretation be allowed for each object. M a n y of these problems are N P - c o m p l e t e , including the interpretation of line drawings [ K P 8 5 ] . It is strongly suspected (although not proven) that N P - c o m p l e t e problems take an exponentially large amount of time i n the worst case [GJ79]. A s s u m i n g this to be true, the c o m p l e x i t y of discrete r e l a x a t i o n results more f r o m the cost of search t h a n from bottlenecks o n d a t a transmission. iv) Area-linkage Problems F i n a l l y , problems exist for which linkage is p r o p o r t i o n a l to the area of the zone. area-linkage These problems have m i n i m a l coherence between values i n the input and the o u t p u t at any l o c a t i o n . A n example is the r o t a t i o n of a discrete image by 180Â°. Here, a change i n one part of the i n p u t can change the output at a position a r b i t r a r i l y far away. A n o t h e r example is the sorting of p i x e l intensities. Here again, a change i n the value of a single p i x e l can lead to changes at locations far removed f r o m the original zone. Area-linkage problems involve such large amounts of d a t a transmission that p y r a m i d architectures (and variants thereof) cannot efficiently handle the transmission of d a t a . B o t tlenecks exist at the higher-level P E s of the p y r a m i d , and so considerable time is therefore required to move d a t a over large distances. Image r o t a t i o n , for example, is of complexity Q,{n) on a p y r a m i d . A similar limit exists for sorting [Sto87]. T h i s latter value may a general l i m i t for area-Unkage problems on this architecture. W h e n the c o m m u n i c a t i o n bottlenecks are bypassed by the use of more connected architectures, the complexity of area-linkage problems is reduced. completelyFor example, sorting on a hypercube requires O ( l o g n ) time [Sto87], the lowest complexity possible on any architecture [GR88]. 2.1.2 Psychophysical Studies E a r l y vision consists of those operations i n the h u m a n v i s u a l system that are r a p i d , spatially p a r a l l e l , a n d require little a t t e n t i o n . Since these operations are directly involved w i t h the i n c o m i n g image, they are relatively easy to study empirically. Consequently, they have l o n g been the subject of psychophysical investigation (see, e.g., [Zuc87b]). Since the focus here is on the limits to this k i n d of processing, this section surveys only the results of psychophysical studies on the descriptions used at the highest stages of early vision. A s this survey shows, there is a remarkable degree of convergence to these results. A . Basics P s y c h o p h y s i c a l studies of r a p i d visual processing have largely been concerned w i t h those activities t h a t occur almost instantaneously and without conscious effort. For example, w h e n a h o r i z o n t a l Une is placed among a group of v e r t i c a l Unes, it invariably "pops o u t " of the image, no m a t t e r how m a n y vertical Unes there may be. O n the other h a n d , detecting a T - s h a p e d figure a m o n g L - s h a p e d figures requires a much slower and more effortful serial scan of the display [Tre88]. T h i s is generally taken as evidence that fast search is based on " v i s u a l p r i m i t i v e s " formed r a p i d l y a n d i n paraUel across the visual field, while slow search is based on constructs formed serially at higher levels. T h i s difference i n performance (in b o t h accuracy and response time) can be used to determine the set of properties determined rapidly a n d i n parallel i n early v i s i o n . F o r t h e most p a r t , experiments have been based on one of three types of task: visual search, t e x t u r e segmentation, or grouping. i) Visual Search In v i s u a l seaxch, the task is to determine whether a displayed image contains a subset of some given collection of target patterns. Performance is generaUy measured by the accuracy or speed of the response. P s y c h o p h y s i c a l studies explore how this performance varies as a function of the number a n d type of target patterns In the coUectlon, the number and t y p e of Items i n the display, a n d the d u r a t i o n of the display Itself (see, e.g. [Rab78, Rab84]). V i s u a l search experiments date back to the work of Green a n d A n d e r s o n [ G A 5 6 ] , w h o demonstrated that search speed for a target was unaffected by variations In the shapes of the other Items, except for those of the same color. T h i s suggested that color is available at early levels t o aUow the selective processing of visual information. Further work by Neisser [Nel63] showed this to h o l d for simple geometrical properties as weU: target letters embedded i n a group of nontargets are detected more quickly when they have a distinctive shape or orientation. M o r e recent studies (e.g., [Tre82, TreSS, Dun89]) measure response time for a single target as a function of the number of items i n the display, w i t h response accuracy b e i n g h e l d constant. These experiments show that if the target is sufficiently distinct from the other items, response t i m e is effectively independent of the number of items present â€” subjectively, the target "pops o u t " of the display. Otherwise, detection time is roughly p r o p o r t i o n a l t o the number of items i n the display, w i t h the constant of proportionality being t w i c e as large for target-absent displays as for target-present ones. T h i s latter p a t t e r n is t a k e n as evidence for a serial scanning process that terminates when the target p a t t e r n has been found [Rab78, Tre82, Tre88]. ii) Texture Segmentation A n alternative way to investigate r a p i d parallel processing is based on the perception of visual texture. Texture perception has several different aspects. These include o b t a i n i n g surface shape f r o m texture gradient, determining the intrinsic structure of a surface, a n d finding the boundaries between regions of different texture (see, e.g., [Wil90]). M u c h of w h a t is k n o w n about texture perception is based mostly on studies of this latter aspect, called texture segmentation. In p a r t i c u l a r , studies have concentrated on finding the determinants of "effortless" segmentation, i.e., segmentation occurring w i t h i n several hundred milliseconds of i n i t i a l viewing a n d w i t h no conscious scrutiny (e.g., [Jul81]). Segmentation itself has several different aspects, including the detection of regions of different texture, a n d the determination of the shape of the possible boundaries [Wil90]. F o r the most p a r t , experiments proceed either by measuring the time required to perform these tasks to w i t h i n a given accuracy or by measuring performance accuracy as a function of display t i m e . In b o t h cases, a p a t t e r n of results is found that is m u c h the same as that for visual search. T e x t u r e d regions can be separated effortlessly from each other when they differ sufficiently i n the density of their elements, or if these elements are sufficiently distinct from each other (e.g., a region of h o r i z o n t a l lines against a region of vertical lines) [Jul86]. It must be kept i n m i n d , however, that t e x t u r e segmentation is a process w i t h goals that are i n m a n y ways different from those of v i s u a l search, and so may not necessarily involve the same set of elements. ni) Visual Grouping T h e representations used i n early vision can also be studied by finding the determinants of visual grouping [Bec66, Bec82]. T h i s approach has its origins i n the Gestalt laws of g r o u p i n g . Disconnected elements can be grouped together into larger units (such as lines and regions) on the basis of similarity a n d s p a t i a l organization [Zuc87a]. T u r n i n g this around provides a way to define these properties operationally â€” similarity and spatial organization are exactly those properties t h a t lead to visual grouping. Since studies of visual grouping often are based on the conscious perception of g r o u p i n g strength (e.g., [Bec82, S B G 8 9 ] ) a n d not on processing speed or accuracy, their results do not necessarily p e r t a i n to r a p i d parallel processing. B u t it has been found that "spontaneous" grouping is not based on the overall shapes of objects, but rather on the similarity of their "elementary p a r t s " [Bec82]. To the extent that these parts are consistent w i t h the elements of v i s u a l search or texture perception, they can provide a check on the descriptions formed at early levels of v i s i o n . T h r e e types of grouping are commonly studied: (i) segregation into regions, (ii) segregation into p o p u l a t i o n s , a n d (iii) creation of intrinsic surface structures. T h e first of these is similar to t e x t u r e segmentation. B u t , whereas segmentation is generally concerned w i t h the boundaries of t e x t u r e d regions, segregation focuses on the h n k i n g of items into distinct regions. P o p u l a t i o n segregation is similar i n most ways except that l i n k i n g based on p r o x i m i t y i n the image is replaced by l i n k i n g based on p r o x i m i t y i n a more abstract space of intrinsic properties (e.g., color or orientation). T h u s , for example, a group of yellow dots i n t e r m i x e d w i t h blue dots can be separated into two distinction populations, even though no geometrical boundaries exist. E x p e r i m e n t s are generaUy based on judgements of whether two or more kinds of features are scattered throughout the image. A l t h o u g h it is also sometimes termed " t e x t u r e segmentation" [Bec82], this task is conceptually quite different, i n v o l v i n g the pooUng of v i s u a l elements based on their intrinsic properties rather t h a n their locations. T h e t h i r d type of grouping is the f o r m a t i o n of " i n t r i n s i c " structures, such as onedimensional contours or two-dimensional flow patterns, which can arise even i n images formed only of dots [Ste78, ZSS83]. Like the segregation of elements into regions or populations, this process is generally thought to be based on simple properties computed over l o c a l zones i n the image [Bec82]. B . M o d e l s of R a p i d V i s u a l P r o c e s s i n g T h e general p a t t e r n of results f r o m experiments on visual search, texture segmentation, a n d grouping is much the same: performance is governed by a small set of simple image properties such as line o r i e n t a t i o n , curvature, contrast, and color (see [TG88]). T h e e x p l a n a t i o n of these results, however, is far f r o m straightforward. F i r s t , the h m i t e d range of conditions over w h i c h d a t a i s collected can make it difficult to determine whether a process requires c o n s t a n t , l o g a r i t h m i c , or linear t i m e . A n d there is no necessary connection between those properties that are computed quickly a n d those that are computed i n parallel â€” a description m a y be the result of an extremely fast-acting serial mechanism, or conversely, a parallel m e c h a n i s m may still require time that increases w i t h the size of the input [Tow72]. In spite of these reservations, several theories have been proposed to explain m a n y aspects of the results. A l t h o u g h differing i n details, these theories agree that simple properties are c o m p u t e d r a p i d l y a n d i n parallel at an early "preattentive" stage, and that complex properties require the application of more sophisticated serial operations at a subsequent " a t t e n t i v e " stage of processing [Bec82, J u l 8 6 , Tre88]. i) Feature Integration Theory Feature-integration theory [Tre82, T G 8 8 ] was originally developed to explain w h y " p o p o u t " i n v i s u a l search occurs when the properties of targets differed sufficiently f r o m those of nontargets, but not when they differed only i n the spatial arrangement of their p a r t s . A c c o r d i n g to this theory, the preattentive system is composed of a set of parallel spatiotopic m a p s , each describing the d i s t r i b u t i o n of a p a r t i c u l a r property (or "feature") across the visual field. These features are simple properties of the two-dimensional image, i n c l u d i n g o r i e n t a t i o n , curvature, binocular disparity, color, and contrast [TG88]. Once these maps have been c o m p u t e d , a target containing a unique feature can be detected simply by checking for a c t i v i t y i n the relevant m a p [Tre88]. T h e separation of the m a p s , however, means that spatial relations between features cannot be represented explicitly. Instead, the coherence of items is represented i n d i r e c t l y v i a a "master m a p " Unking together the appropriate locations i n the feature maps. T h e comput a t i o n of complex structures therefore requires a spotlight of attention to access the master m a p a n d Unk up aU the relevant features into a coherent whole. Since this spotUght must seriaUy inspect each coUection of features present i n the image, the detection of complex features requires time p r o p o r t i o n a l to the number of features present. T h i s explains why, for example, targets distinguished only by inside/outside relations do not pop out [TG88]. In its o r i g i n a l f o r m , feature-integration theory did not account for several phenomena. A m o n g these were the finding that conjunctions of simple features at the same l o c a t i o n can be r a p i d l y detected when their constituents are strongly discriminable, and the finding t h a t search rate increases smoothly w i t h the d i s c r i m i n a b i l i t y of the stimuli [Tre88]. T h e first of these has since been explained by postulating an i n h i b i t i o n (or excitation) of the master m a p at locations where elements are strongly activated. T h i s allows all items containing nontarget features t o be effectively ignored, leaving a small remainder among which the target can be quickly detected [Tre88, C W 9 0 , T S 9 0 ] . T h e second effect is accounted for by p o s t u l a t i n g t h a t the spotlight of attention operates not on i n d i v i d u a l items but on groups of items, the size of the group v a r y i n g w i t h the d i s c r i m i n a b i l i t y of its members [TG88]. B o t h these refinements, however, m a i n t a i n the assumption that only simple l o c a l operations are carried out i n p a r a l l e l at preattentive levels. ii) Resemblance Theory Resemblance theory [DII89, Dun89] is an alternative account of visual search that differs f r o m feature-integration theory i n several ways. It shares the basic premise that simple features are c o m p u t e d at the preattentive level but postulates that the speed of search depends entirely on the resemblance between the target a n d nontarget patterns i n the image. It explains the relatively slow search for conjunctions as due to the similarity of target a n d nontarget items arising f r o m their c o m m o n features. One of the more interesting aspects of this theory is that resemblance is based on the degree of t r a n s f o r m a t i o n needed to m a p the features of one figure into those of another [DH89]. It therefore is a first step away from the idea that preattentive processes are necessarily based on simple l o c a l properties. A l t h o u g h some of the difficulty of conjunction search is a p p a r e n t l y due to conjunction itself [Tre91], the possibihty remains that some aspects of preattentive operation are best explained i n terms of features resulting from procedures applied to simple line elements. iii) Texton Theory In contrast to b o t h feature-integration and resemblance theory, t e x t o n theory was developed to account for effortless texture segmentation. Here, perceived texture is thought to depend entirely on the first-order densities of spatial patterns called textons. These are localized geometric shapes w i t h simple properties, i n c l u d i n g endpoints, elongated blobs. Une crossings, a n d Une segments of various lengths, w i d t h s , a n d orientations [Jul84a]. T e x t o n theory explains texture segmentation by a model similar to those used for v i s u a l search, w i t h processing being separated into distinct preattentive and attentive systems. T h e preattentive system is composed of a set of spatiotopic maps, each describing the d i s t r i b u t i o n of a p a r t i c u l a r t e x t o n across the visual field. Effortless segmentation occurs when the regions differ sufficiently i n the first-order densities of their constituent textons. Because only t e x t o n densities are involved, textures cannot be effortlessly segmented when they differ only i n t h e relative arrangements of their textons (e.g., a region of L-shaped figures against a region of T - s h a p e d figures). To separate such regions therefore requires conscious " s c r u t i n y " by higher-level processes [Jul84a]. Textons have m u c h i n c o m m o n w i t h the set of features postulated for visual search. T h e y include not only l e n g t h , w i d t h , and o r i e n t a t i o n , but also color, m o t i o n , binocular disparity, a n d flicker [Jul84a]. Indeed, given that Une-crossings are no longer considered t o be t r u e textons [Not91], the two sets appear to be almost identical. Textons have even been used to explain visual search itself, using a mechanism analogous to the spotUght of a t t e n t i o n being postulated to account for the detection of particular t e x t o n combinations [JB83]. L i k e feature-integration theory, t e x t o n theory has also been revised to allow groups of items t o be searched i n parallel w i t h i n U m i t e d regions, the size of these regions v a r y i n g w i t h the strength of the density gradient [Jul87]. B u t i m p o r t a n t differences also exist. Whereas feature-integration a n d resemblance theories are based o n the absolute presence or absence of features, t e x t o n theory posits boundaries based on the l o c a l differences between t e x t o n densities [Jul86]. F u r t h e r m o r e , while most (if not aU) textons have properties similar to those of preattentive features, they are quite different ontologically: textons are geometric elements containing specific properties, a n d are not the properties themselves. In essence, each t e x t o n contains a conjunction of simple properties. T h u s , although effortless texture segmentation cannot be based on spatial relations, it can be based on the conjunction of simple features [Jul84b]. iv) Spatial Filtering Recent attempts t o provide an algorithmic framework for texture segmentation have shown that m u c h of it can be explained i n terms of the spatial filters postulated for edge detection, v i z . , localized Unear filters of differing widths a n d orientations (e.g.. [Cae84, B A 8 8 , V P 8 8 ] ) . It also has been suggested that the texture boundaries themselves are determined v i a operations analogous to those used for edge-detection, w i t h the a r r a y of filter outputs being smoothed and the lines of m a x i m u m change then used to m a r k the texture boundaries [ V P 8 8 , G B 8 9 , B C G 9 0 ] . Direct psychophysical evidence has been o b t a i n e d In favor of this view [Not91]. A s a consequence, there is now some doubt about the need t o m a i n t a i n textons as a separate set of texture primitives (see, e.g., [Not91]). B u t consensus remains that texture p r i m i t i v e s â€” whatever these u l t i m a t e l y may be â€” are based o n l y o n simple l o c a l properties c o m p u t e d r a p i d l y and reliably from the image. T h e spatial-filter m o d e l also helps to explain the grouping of image elements. Such filters respond not only to a c t u a l hues of a given orientation a n d length but also to simple s t r u c t u r a l groups h a v i n g the same general outUnes, such as the " v i r t u a l Hues" formed by a row of dots of similar contrasts [Zuc86]. However, although filters are thought to be necessary for the grouping process, they are not usually beHeved to be sufficient. A p a r t from exceptions such as the r a p i d detection of "locally p a r a l l e l " structure In Glass patterns [Ste78]), g r o u p i n g processes are generally thought to require nonlocal Integration of Information across t h e image (e.g., [Zuc87b]). It also appears unlikely that the properties determined at the preattentive level c a n be explained entirely In terms of a single set of filter-based elements. For example, p o p u l a t i o n grouping Is based on the lightness differences of the elements, rather t h a n by the contrast ratios t h a t govern region segregation. F u r t h e r m o r e , conjunctions of these properties do not support p o p u l a t i o n segregation, whereas they do support region segregation [BGS91]. T h e two sets of processes cannot therefore involve the same set of basic elements. Further support for this view comes f r o m studies that show texture Identification and texture segmentation t o be based on different sets of p r i m i t i v e elements [Not91]. T h u s , given the possible existence of several different sets of preattentive elements. It Is likely that at least some of t h e m are not directly related to s p a t i a l 2.1.3 filters. C o m p u t a t i o n a l versus Psychophysical Studies F r o m a c o m p u t a t i o n a l v i e w p o i n t , there are good theoretical grounds for the assumption of a distinct stage of early v i s u a l processing. T o begin w i t h , almost aU. the properties at this level have two i m p o r t a n t characteristics: (i) they are zero-Unkage (section 2.1.1),^"^ a n d A s a convenient way of speaking, the linkage of a property is identified with the linkage of the corresponding image-processing problem. (ii) they have a fixed support, i.e., the relevant property can be extracted from a fixed set of points i n the zone. A spatially-bounded template can therefore determine the relevant property at each point and the corresponding m a p can be computed rapidly a n d i n p a r a l l e l . A l t h o u g h recent experiments have shown that conjunctions of preattentive features can p o p out when sufficiently distinct [TS90], this has little effect on the general argument since, as several models have shown (e.g., [ C W 9 0 , TS90]), this can be accounted for entirely by a m o r e sophisticated search mechanism that selectively suppresses (or excites) the outputs of t h e simple feature m a p s . In contrast to these " t e m p l a t e " properties, others are neither zero-linkage nor have a fixed support. F o r example, determining whether a given object is inside or outside a neighboring object cannot be done w i t h i n some fixed zone, since there are no Umits to the extent of the neighbor's boundaries. E v e n i f limits were imposed, there w o u l d stiff be no fixed points w h i c h could always be used. T h u s , a different template is required for each of the exponentiallyincreasing number of possible s h a p e s . I t has therefore been suggested that " n o n l o c a l " properties, i n c l u d i n g v i r t u a l l y a l l types of grouping and s p a t i a l relations, are determined procedurally v i a specialized m'iiuaZ roufmes applied to earlier "base" descriptions [U1184]. M a n y of these routines are serial, spatially inhomogeneous, and are thought to be controlled by higher-level processes. A s such, their application is sometimes Unked (to greater or lesser extent) w i t h the spothght of attention required at attentive levels [UI184, T G 8 8 ] . T h i s point of view receives some confirmation from the finding that spatial relations such as p a r a l l e l i s m and inside-outside cannot be detected preattentively [TG88]. However, this grouping of visual processes into distinct early a n d later stages is not w i t h o u t its difficulties. Consider first the property of length. T h i s is generally regarded as a p r i m i t i v e quantity, b o t h i n e m p i r i c a l studies on visual search (e.g., [TG88]) a n d i n c o m p u t a t i o n a l models of early vision (e.g., [Mar82]). B u t length is not a zero-linkage property â€” a gap a r b i t r a r i l y far away can change the value assigned to a Une. It also is not easily determined by a template, or even a set of templates along the Une, since the value f r o m any i n d i v i d u a l template depends on the overlap between it and the Une being measured. A t best, length might be determined f r o m competition among the set of templates along the given Une (cf. [Zuc87a]) but this begins to introduce a nonlocal element into the c o m p u t a t i o n . It also has been found that binocular disparity (and possibly depth) can be determined ^*For example, consider a surface patch divided into n intervals. exist for each interval, then fc" different combinations are possible. If k possible values (e.g., color, height) preattentively [NS86]. It is possible to call disparity a zero-linkage property, i n the sense t h a t the value at any point depends only on some finite surrounding zone i n the image. B u t there is no way i n which it can be given a compact fixed support â€” to ascertain disparity requires the m a t c h i n g of patches i n the left a n d right images, and the contents of these patches can be quite a r b i t r a r y . M a t c h i n g must therefore be done procedurally. Recent results have also shown that the preattentive system can determine properties such as direction of Ughting and three-dimensional orientation â€” properties not of the i m a g e , b u t of the scene to which It corresponds [RamSS, E R 9 0 a , ERQOb, E R 9 1 ] . Such recovery appears to have a nonlocal procedural component, since its success does not depend completely o n the presence or absence of any p a r t i c u l a r local property, but instead depends on the entire system of line relations present i n the i t e m [ E R 9 0 b , E R 9 1 , E R 9 2 ] . These findings c a l l Into question the basic assumptions b e h i n d the conventional assignment of visual processes t o early a n d later levels. In p a r t i c u l a r , they call into question the reasons for believing t h a t three-dimensional structure cannot be rapidly determined by a spatiotopic array of p a r a l l e l processors. 2.2 The Interpretation of Line Drawings T h e p r o b l e m of Une i n t e r p r e t a t i o n is one that is simple enough to allow easy f o r m u l a t i o n a n d e x p e r i m e n t a l m a n i p u l a t i o n , yet complex enough that Its solution requires at least some degree of InteUlgence. A s such, it provides an interesting arena i n which to study processes of a type generally thought to be restricted to higher levels of cognition. T h i s section surveys some of the m a i n results of the c o m p u t a t i o n a l and psychophysical studies that have been carried out i n this area. In p a r t i c u l a r , It examines the case where the drawings correspond to two-dimensional projections of opaque p o l y h e d r a . Some of the more i m p o r t a n t theoretical results are first surveyed. T h i s is foUowed by an overview of w h a t is k n o w n about the abiUty of humans to Interpret such drawings. 2.2.1 Computational Studies T h e p r o b l e m of determining the three-dimensional structure of an object from its corresponding Une drawing has been the focus of a considerable amount of work i n the field of c o m p u t a t i o n a l vision (see, e.g., [CF82]). T h i s section reviews several i m p o r t a n t results that have been obtained. These results have for the most part been developed w i t h i n a single framework - the blocks world. W h e n the basic assumptions of this framework are met, a great deal can be said about what can and cannot be recovered from a hne d r a w i n g . A . Basics E a r l y work on the machine i n t e r p r e t a t i o n of Une drawings (e.g., [Rob65]) a t t e m p t e d to analyze scenes composed of a s m a l l set of k n o w n p o l y h e d r a . T h e goal was to identify Unes i n the image w i t h edges of p a r t i c u l a r instances of these objects. Recognition proceeded by using a priori knowledge of the p o l y h e d r a l shapes to determine which image regions corresponded to w h i c h surfaces. A l t h o u g h research i n "model-based" vision (e.g., [Low85]) continues to use such global constraints, attention also t u r n e d to the use of " l o c a l models", i.e., constraints on the l o c a l structure of the objects i n the scene. G u z m a n [GA68] showed that the s t r u c t u r a l relations a m o n g the Unes of the j u n c t i o n s were often sufficient for the extraction of three-dimensional structure. Subsequent work (e.g., [Clo71, H u f Z l , W a l 7 2 , Mac76]) gave this approach a more soUd theoretical framework i n w h i c h to formulate a n d discuss issues of Une i n t e r p r e t a t i o n . T h i s theoretical framework was based on the so-caUed "blocks w o r l d " , a scene domain comprised of p o l y h e d r a l objects w i t h trihedral corners, i.e., corners formed f r o m the intersect i o n of three p o l y g o n a l faces (see, e.g., [CF82]). T h e corresponding image d o m a i n is formed by the orthographic projection of these objects onto the image plane. G i v e n that corners are t r i h e d r a l i n the narrow sense (section 1.1), this projection consists of straight-Une segments connected by junctions of either two or three Unes. B y using Une drawings alone, aU effects of surface coloration (e.g., reflectance and texture) and shading are discounted. V i e w i n g d i rection a n d direction of Ughting are held constant, w i t h the two directions often being made coincident i n order to avoid shadows. T h e result is a " m i n i w o r l d " i n which attention can be focused entirely on the recovery of surface geometry. T h e most comprehensive, a n d difficult, problem concerning Une i n t e r p r e t a t i o n i n this m i n i w o r l d is that of realizability: G i v e n a Une d r a w i n g , does it correspond to an a c t u a l arrangement of p o l y h e d r a l objects i n some three-dimensional scene? If so, what are the threedimensional shapes a n d positions of these objects? V i r t u a l l y aU the work done on the blocks w o r l d has proceeded by spUtting this problem into two parts: a quaUtative aspect concerned w i t h the s t r u c t u r a l relations between edges and surfaces i n the scene, a n d a quantitative aspect concerned w i t h the slants of the Unes a n d the depths of the vertices [Sug86, K P 8 8 ] . A l t h o u g h b o t h aspects can be approached Independently, the results of quahtatlve analysis have usually been used as the s t a r t i n g point for quantitative analysis (see, e.g., [Sug86]). B . Qualitative Interpretation Since the faces of a p o l y h e d r a l object are planar, its structure is completely determined b y the locations of its edges, i.e., locations where t h e orientation of adjoining faces suddenly changes. Q u a l i t a t i v e analysis is based on the s t r u c t u r a l relations between these edges (see, e.g., [Mal87]). Edges can be subdivided into convex and concave forms according to w h e t h e r or not the edge folds o u t w a r d , i.e., whether or not an external plane can be placed i n t o contact w i t h the edge. A further subdivision results f r o m the relation between object a n d viewer. Edges can be grouped according to whether b o t h or just one face is visible.^^ T h i s l a t t e r k i n d of edge is referred t o as a boundary edge. These correspond exactly t o places i n the viewer-centered description where depth changes discontinuously. T w o types of b o u n d a r y edge are often distinguished, according to which side of the line corresponds to the visible face. T h e r e m a i n i n g edges are referred to as interior edges. These correspond to locations where the depth gradient changes discontinuously. These are divided into two types, according t o whether the corresponding edge is convex or concave. Between t h e m , boundary a n d interior edges describe not only an object's shape but also the segmentation of the image, i.e., w h i c h regions i n the image do or do not correspond t o connected surfaces i n the scene (see, e.g. [Sug86, K a n 9 0 ] ) . To interpret a Une d r a w i n g , each Une must be labeUed as a p a r t i c u l a r k i n d of edge (convex, concave, or b o u n d a r y ) . T h e interpretation is guided by a set of expUcit constraints on t h e various edge labeUings. These constraints can be provided largely by restrictions o n t h e labeUing of junctions [Huf71, Clo71]. Four types of j u n c t i o n can be distinguished (figure 2.2). T h e first three are triUnear, formed f r o m the j o i n i n g of three Unes: (i) arrow-junctions, for w h i c h t h e greatest angle between two Unes is greater t h a n 180Â°, (ii) Y - j u n c t i o n s , for w h i c h i t is less t h a n 180Â°, a n d (ni) T - j u n c t i o n s , for which i t is exactly 180Â° (see figure 2.2). There also exist L - j u n c t i o n s , formed from the j o i n i n g of t w o noncoUinear Unes. E a c h type of j u n c t i o n leads t o a p a r t i c u l a r set of constraints. These constraints, first given by H u f f m a n [Huf71] and Clowes [Clo71], are shown i n figure 2.3. These correspondences fail to h o l d when there is an accidental aUgnment of viewing direction w i t h p a r t i c u l a r arrangements of surface edges. Since accidental alignments are exceedingly rare, the assumption usuaUy is made that they ^*It is evident that this distinction applies only to convex edges. T-junction Y-junction Arrow-junction L-junction Figure 2.2: Types of junctions. do not occur; under this general viewpoint constraint, the correspondences between j u n c t i o n a n d edge types w i l l always h o l d . T h e p r o b l e m of finding a consistent set of labels for a given drawing is k n o w n as the line labelling ^Tohlem. E v e r y p o l y h e d r a l scene gives rise to a unique set of labels [Ric88], a n d i f a d r a w i n g is realizable, it can be consistently labelled [Huf71, Sug86]. not t r u e . T h e separation into independent B u t the converse is quahtative and quantitative components means that the metric structure of the scene is not available to the qualitative labelling process. Consequently, Hne drawings can be consistently labeHed, but have no correspondence w i t h any p o l y h e d r a l object [Huf71, K a n 9 0 ] . T h e labeHing of a given drawing can be carried out by a relatively straightforward p r o cedure. A s figure 2.3 shows, each type of junction can be labeHed i n several different ways. To reduce the number of l o c a l candidates, interpretation often begins w i t h the appHcation of "Waltz filtering" to eHminate labels that are locaHy inconsistent [Wal72, Mac77]. T h i s k i n d of consistency check is a relatively simple procedure that can be carried out i n p o l y n o m i a l time [MF85]. W a l t z fHtering finds a correct interpretation if the locally consistent labels are globaUy consistent as weH. B u t this does not always occur. Consequently, it is sometime necessary to < I < < I < t Boundary edges: Interior edges: Solid + space Convex Concave F i g u r e 2.3: Huffman-Clowes labelling set. explore a l l possible combinations of the remaining labels, each combination then tested for g l o b a l consistency. Since the labelling problem is N P - c o m p l e t e [ K P 8 5 ] , it is highly unlikely that a globally consistent solution can always be found i n p o l y n o m i a l time. Instead, the worst-case t i m e is likely to be an exponential function of the number of junctions (and lines) i n the image. A l t h o u g h the Huffman-Clowes constraints never lead to inconsistency i n a drawing t h a t corresponds to a physically reahzable object, they sometimes consistently label an object that cannot be realized. T w o types of error occur: inconsistencies i n the global topological struct u r e , a n d inconsistencies i n the depths of the surfaces (see, e.g., [ D r a S l , Kan90]). Topological inconsistencies can be e l i m i n a t e d when a l l corners are rectangular [Kan90]. Inconsistencies i n d e p t h , however, must be h a n d l e d v i a more powerful constraints based on the m e t r i c structure of the scene. T h e use of metric constraints was pioneered by M a c k w o r t h [Mac73], who developed an approach based on the observation that regions i n the image must correspond to flat planes i n the scene. E a c h plane is represented by its gradient, a two-dimensional measure of its orientation i n space. Since aU faces of a p o l y h e d r a l object are p l a n a r , its coherence can be captured by constraints i n the gradient space, which eUminate m a n y inconsistent interpretations. A l t h o u g h constraints on gradient space are useful, they do not eliminate a l l inconsistent interpretations [ D r a S l ] . T h i s is because only p a r t i a l use is made of three-dimensional inform a t i o n â€” gradients ignore the fact that planes are also specified by their depth along the hne of sight. T h i s l a t t e r q u a n t i t y forms the basis of sidedness reasoning [ D r a S l ] , i n w h i c h constraints are based on the condition that one plane must always be i n front of the other on a given side of their intersection line. T h e resulting set of constraints then ensures that a l l consistent interpretations correspond to physically realizable objects [ D r a S l ] . C . Quantitative Interpretation A n alternative to qualitative line interpretation is to work directly w i t h the quantitative structure t o o b t a i n the depths and the three-dimensional orientations of the objects i n the scene. T h i s technique, first suggested i n [Fal72], is based on the observation that the junctions a r o u n d a c o m m o n region correspond to points and edges around a common planar face. T h i s plane can be described by a linear equation, w i t h the unknowns being the depths of the corners i n contact w i t h the face. C o l l e c t i n g the equations for each region i n the drawing yields a system of Unear equations, which can be solved by straightforward means [Sug86, K a n 9 0 ] . In general, these systems of equations are underdetermined. T h u s , even when an absolute depth is a t t a c h e d to one p o i n t , several degrees of freedom still r e m a i n [Sug86]. A d d i t i o n a l constraint o n the solutions is therefore required. One such constraint is an a priori specifica- t i o n of the three-dimensional orientation of particular faces. Since each of these specifications is independent, the number of degrees of freedom is reduced by the number of o r i e n t a t i o n specifications t h a t can be given. Other kinds of l o c a l constraint also are possible. If a j u n c t i o n corresponds to a rectangular corner, the slant and tilt of the corresponding faces a n d edges are completely determined by the angles of the Unes about its vertex, the values depending only one whether the j u n c t i o n is concave or convex [Per68, M a c 7 6 , KanQO]. F u r t h e r m o r e , there exists a set of necessary (but not sufficient) conditions on a j u n c t i o n that corresponds to a rectangular corner [Per68]: a n arrow-junction must have one angle greater t h a n 90Â°and the other two less t h a n 90Â°, w h i l e a Y - j u n c t i o n must have a l l its angles greater t h a n 90Â°[Per68, K a n 9 0 ] . Three-dimensional orientations can also be recovered when only two of the angles are 90Â°[Kan90]. In t h i s case, the hnes must be correctly identified w i t h the corresponding edges i n the scene. R e c o v e r y of slant is possible for b o t h o r t h o g o n a l a n d perspective projection of the object onto the i m a g e plane [Kan90, ch. 8]. G l o b a l constraint also is used. One approach is to specify the recovered surface as the smoothest of aU possible candidates[Kan90, ch. 10]; this loosely corresponds to the regularization technique suggested for several aspects of early vision [ P T K 8 5 ] . M o r e generally, there is an interplay between l o c a l and global constraints. For example, M u l d e r & D a w s o n [MD90] have shown that for some objects, a complete quantitative interpretation requires that only a subset of corners be rectangular. T h i s essentially is a special case of m a x i m i z i n g the rectangularity i n the recovered figure. 2.2.2 Psychophysical Studies In contrast w i t h the work on c o m p u t a t i o n a l aspects of hne interpretation, work i n psychophysics has been rather heterogeneous. methodologies It encompasses a wide variety of e x p e r i m e n t a l a n d s t i m u l i , as w e l l as different theoretical frameworks. However, there is wide agreement i n the general p a t t e r n of experimental results, a n d these patterns also are consistent w i t h m a n y of the results from c o m p u t a t i o n a l studies. A . Basics Investigations into the perception of Hne drawings extend back to the very beginnings of experimental psychology. T h e first comprehensive explanation of how drawings could be perceived as three-dimensional objects was given at the t u r n of the century by M a c h , who proposed that the v i s u a l system operates on a "principle of economy" (see [Att82]). This gave way to the Gestalt principle of figurai "goodness", which selected those interpretations that required m i n i m a l "energy" for their representation (see e.g., [Hoc78, p p . 131-155]). A c c o r d i n g to this p r i n c i p l e , a Hne drawing of a cube is perceived as a three-dimensional object rather t h a n as a coHection of two-dimensional Unes because this requires less energy for its representation. S i m i l a r reasons also explained the tendency to perceive its sides a n d angles as equal whenever possible. T h e vagueness of Gestalt laws eventually led to their abandonment by m a n y workers i n the field. However, the central insight remained that interpretation must involve constraints on the interpreted object. T h i s provided the s t a r t i n g point for later investigations (e.g., [ H M 5 3 , Att54]) w h i c h a t t e m p t e d to provide a more rigorous study of these constraints. A s i n the case of machine v i s i o n , these later studies can be categorized into two groups: those concerned w i t h the qualitative aspects of hne interpretation, and those concerned w i t h its q u a n t i t a t i v e aspects. Studies i n the first group focus on the factors that determine whether a hne d r a w i n g is perceived as a set of Unes or as a three-dimensional structure. Studies i n the second group are concerned w i t h the perception of the metric properties of the structure itself. Reflecting a bias t o w a r d viewing Une interpretation as a "high-level" activity, b o t h kinds of studies t e n d t o rely on verbal reports of consciously perceived structure. B . Qualitative Interpretation In contrast w i t h c o m p u t a t i o n a l studies of quaUtative structure, psychophysical studies have tended to focus on global aspects of the interpretation rather t h a n local properties such as the convexity or concavity of i n d i v i d u a l edges. A t least some of this emphasis Ukely is due to the legacy of the Gestalt school, w i t h its emphasis on the m i n i m u m energy of the entire interpretation. One of the first attempts to put this approach on a more rigorous footing was the work of A t t n e a v e [Att54], who recast the principle of m i n i m u m energy into one of " m a x i m a l s i m p U c i t y " , where simpUcity was based upon the " i n f o r m a t i o n " contained i n the percept. B y identifying this information w i t h that used i n information theory, it was hoped to have a m o r e objective basis for the rules of the interpretation process. Since the absolute amount of i n f o r m a t i o n depends upon the coding scheme, such rules cannot be entirely objective. Nevertheless, a few general principles can be derived. For example, m a x i m a l l y simple structures have Unes of equal length, s y m m e t r y about the o r i g i n , corners of equal angle, etc. In the case of a cube, this approach correctly predicts that its Une d r a w i n g is interpreted as a symmetric structure w i t h edges of equal length rather t h a n as an a s y m m e t r i c a l set of Unes of unequal length. A l t h o u g h this approach could explain the perception of simple Une drawings, it could not do so for more complex ones without i m p o s i n g ad hoc rules on how various regularities could be t r a d e d off against each other [ H M 5 3 , A t t 5 4 ] . In t u r n , this could not be done without the specification of a p a r t i c u l a r coding scheme. F i g u r e 2.4: Penrose triangle. Such schemes have been proposed (e.g. [Lee71]). If a large enough set of rules is i m p o s e d , it can indeed e x p l a i n the perception of m a n y kinds of line drawings [Res82, B L 8 6 ] . B u t such " m i n i m a l description" approaches suffer from serious drawbacks. First of a l l , the emphasis on global measures means that a drawing that cannot be consistently interpreted must be represented as a two-dimensional structure. T h i s is at odds w i t h the finding that globally inconsistent figures such as the Penrose triangle (figure 2.4) are perceived as three-dimensional objects. ( A l s o see, e.g. [Hoc78, pp 152-155].) mechanism for finding m i n i m a l encodings. It also is difficult to provide a plausible E v e n if this could be done, the search for t h e m i n i m a l description w o u l d still take considerable time [Att82, Res82]. Most importantly, perhaps, it is difficult to justify why the size of the description itself should be the m a i n determinant of the process, rather t h a n some property of the structure being described. A rather different approach was taken by Weisstein [ W M 7 8 ] , who investigated how various line drawings influenced the a d a p t a t i o n of the visual system to sinusoidal gratings. A simple blank hexagon placed o n a g r a t i n g extending over the entire visual field resulted i n a complete lack of a d a p t a t i o n at the locations it covered. A t these locations, relatively low-contrast gratings could be easily detected, although this was not possible i n the rest of the v i s u a l field. B u t when a Y - j u n c t i o n was added t o the hexagon, adaptation suddenly appeared i n the blank field, as i f that area h a d been "fiHed i n " by the surrounding gratings. These results were explained by a tendency for the early visual system to perceive this figure as a cube, which was then separated f r o m the flat background. M o r e generally, it was found that Une segments can be more accurately identified when they are part of a d r a w i n g of a coherent three-dimensional object t h a n when they are among a set of u n s t r u c t u r e d Unes [ W H 7 4 ] . T h i s was found to hold for junctions as weU [ B W H 7 5 ] , i n d i c a t i n g that l o c a l properties govern this process. A similar set of results was obtained by W a l t e r s , who found hues t o be perceptually brightened when interprÃ©table as edges of a coherent three-dimensional object. T h i s brightening was found to be unaffected by g l o b a l properties, depending only on j u n c t i o n type a n d hne length (over distances of less t h a n 1Â°) [Wal87]. C . Quantitative Interpretation A s c o m p u t a t i o n a l studies show, hne interpretation can be achieved v i a constraints o n t h e quantitative structure of the recovered object. Interestingly, psychophysical studies suggest t h a t several quantitative constraints are indeed involved i n the h u m a n perception of line drawings. One of these constraints is that of rectangularity, i.e., the requirement that p o l y h e d r a l objects have sides at right angles to each other. T h e projection of rectangular corners yields junctions h a v i n g a p a r t i c u l a r set of constraints on the angles between their hues (section 2.2.1). E m p i r i c a l tests [Per72, SheSl] have shown that subjects are highly sensitive t o these constraints, being able to determine accurately whether a hne drawing does or does not correspond to a rectangular cube. T h i s can be done even when some of the hues are removed f r o m the figure, p r o v i d e d that at least one Une is kept from each of the three orientations [Per82]. In contrast, subjects are far worse at recognizing which structures contain corners w i t h a n gles of 60Â° or 120Â° [She81]. Consequently, it is hkely that the critical factor is r e c t a n g u l a r i t y rather t h a n simple equahty among the angles. R e c t a n g u l a r i t y also makes it possible to determine the orientation of an object i n threedimensional space (see section 2.2.1). A very high correlation has been shown between a c t u a l slant a n d judgements obtained f r o m Une drawings of rectangular figures [AF69]. A l t h o u g h the perceived slants are less steep t h a n the a c t u a l slants, this " f l a t t e n i n g " can be lessened when contributions f r o m other cues are reduced [Att72]. A n o t h e r useful s t r u c t u r a l property is that of b i l a t e r a l symmetry, i.e., s y m m e t r y about a plane t h r o u g h the center of an object. T h i s property is generally perceived i n a hne d r a w i n g whenever it is consistent w i t h the laws of projective geometry (e.g., [Per76, P C 8 0 , Per82]) E v e n when the task itself makes no use of i t , subjects spontaneously perceive symmetry about half the t i m e . Indeed, it is even possible to alternate between interpretations based o n s y m m e t r y a n d rectangularity [Per76]. Not a l l kinds of s y m m e t r y can be detected, since e q u a l angles of 60Â° or 120Â° are not generally perceived as s y m m e t r i c a l [SheSl]. T h e preference for b i l a t e r a l s y m m e t r y m a y have ecological origins â€” most animals are bilaterally s y m m e t r i c [Per 76]. Subjects can also detect the coplanarity of two planes i n a Une drawing, and although the accuracy for this is somewhat lower t h a n for the detection of rect angularity, it is stiU quite g o o d [Per82]. T h i s shows that the h u m a n visual system can recover at least some q u a n t i t a t i v e spatial relations f r o m Une drawings. L i t t l e is k n o w n about the mechanisms that carry out Une interpretation i n h u m a n v i s i o n . T h e two extremes have been proposed: "convergence" a n d "direct c o m p u t a t i o n " mechanisms [ P C 8 0 , Per82]. T h e former is essentially a general-purpose relaxation process (section 2.1.1) that can incorporate various constraints into its operation. Recovered properties are o b t a i n e d as the c o m p u t a t i o n settles i n t o ah equiUbrium state. T h e latter is a special-purpose device that computes properties directly (i.e., i n a non-iterative w a y ) , o b t a i n i n g its speed at the price of decreased flexibiUty. T h e r e is insufficient evidence to determine which of these two processes (if either) is responsible for Une i n t e r p r e t a t i o n , but the sensitivity of the v i s u a l system to several k i n d s of geometrical properties has been t a t e n to support the existence of the general-purpose " i n d i r e c t " process [Per82]. 2.2.3 C o m p u t a t i o n a l versus Psychophysical Studies A s is evident f r o m sections 2.2.1 a n d 2.2.2, there is considerable agreement between the results of c o m p u t a t i o n a l a n d psychophysical studies i n the areas where they overlap. A c c o r d i n g to c o m p u t a t i o n a l models, there is enough i n f o r m a t i o n i n the junctions to allow the recovery of almost a l l quaUtative structure f r o m a Une drawing. T h i s result is echoed i n the finding t h a t the perceived three-dimensionaUty of a Une d r a w i n g depends on the types of the j u n c t i o n involved. T h e similarities extend to the quantitative aspect of interpretation as weU, where the i m p o r t a n c e of s t r u c t u r a l constraints such as rect angularity has been estabUshed i n b o t h areas of study. A g r e e m e n t , however, is not the same as completeness â€” many aspects of Une interpretat i o n have not yet been investigated by either k i n d of study. For example, most c o m p u t a t i o n a l a n d psychophysical studies have been based on perfect or near-perfect Une drawings, so that i n t e r p r e t a t i o n i n the presence of noise is a relatively unexplored d o m a i n . A n o t h e r largely unexplored area is the complexity involved w i t h interpreting various kinds of Une drawings. In its most general f o r m , the reaUzabiUty problem (section 2.2.1) is N P - c o m p l e t e [ K P 8 5 ] , and so the i n t e r p r e t a t i o n of some drawings must sometimes be difficult a n d time-consuming, regardless of whether the system is artificial or biological. N o studies, however, have explored the way i n w h i c h complexity issues are handled by the h u m a n visual system. T h e fact t h a t psychophysical studies are usually based on reports of relatively high-level percepts shows a tacit agreement t h a t hne i n t e r p r e t a t i o n requires sophisticated processing. B u t how t h e n t o account for r a p i d hne i n t e r p r e t a t i o n i n early vision? E v i d e n t l y , r a p i d i n t e r p r e t a t i o n must be possible for only a subset of the scene d o m a i n , one for w h i c h the t i m e complexity is very low. A few sub domains of this k i n d are k n o w n t o exist. One of these is the orthohedral w o r l d , where aU objects are constrained to have surfaces parallel to the x,y, a n d z planes. Here, the labehing of n hnes can be done i n 0(n) a serial processor, a n d i n O(log^ n) time when time on processors are available [ K P 8 8 ] . B u t this result does not necessarily p e r t a i n to r a p i d recovery i n h u m a n vision, since 'nP processors m a y not always be available, a n d O(log^ n) time may not always be allowed. M o r e generaUy, it is not clear w h i c h aspects of scene structure can be r a p i d l y determined, or even w h i c h aspects should be. These questions can only be examined i n the context of a c o m p u t a t i o n a l theory. 2.3 High-level versus Low-level Vision In order to develop a c o m p u t a t i o n a l theory of r a p i d parallel recovery, it is necessary to k n o w w h a t role this process could play. T h i s section re-examines the reasons for separating v i s i o n into high a n d low levels, a n d for the p a r t i c u l a r assignment of various processes to these levels. It then examines what can be expected of a r a p i d recovery process, a n d shows how it can help bridge the gap between the two levels. 2.3.1 T h e S t r u c t u r e of L o w - l e v e l V i s i o n Information-processing tasks generaUy have aspects common to a l l inputs a n d aspects appUcable only to special cases. In vision, these two aspects take the form of distinct levels: a " l o w " level based on the general constraints of geometry, physics, and information theory, and a " h i g h " level based on the more specific relations between i n d i v i d u a l objects in the scene (e.g.,[Mar82, Fel85]. T h e b o u n d a r y between low- and high-level vision therefore reflects t h e Umits of a " c o m m o n core" beUeved to be derivable (usually i n a b o t t o m - u p fashion) f r o m general considerations alone. O w i n g to the inverse relation between the generaUty of a constraint a n d the s t r u c t u r a l c o m p l e x i t y of the objects it appHes to (see, e.g., [Sal85, p. 49]), this common core must involve structures of a relatively simple structure. In p a r t i c u l a r , the common core is usually taken to be a viewer-centered m a p (or set of maps) of various scalar properties of the image or scene, w i t h possibly some explicit representation of structural grouping as well [ B T 7 8 , M a r 8 2 ] . T h i s characterization of low-level vision differs from that of early vision, i n that emphasis is placed on properties of the i n p u t - o u t p u t m a p p i n g itself rather t h a n on properties of the process that generates it.-^^ B u t processing speed is used (often i m p h c i t l y ) to decompose lowlevel v i s i o n i n t o a sequence of " h o r i z o n t a l " modules. E a c h of these is concerned w i t h a distinct stage of the c o m p u t a t i o n , a n d is apphed to the entire image (see, e.g., [Mar82, UU84, U h r 8 7 ] ) . W h i l e there is no consensus on the exact structure of these stages, there is considerable agreement on their existence a n d general operation. A . E a r l y Stage T h e first stage of low-level vision is generally identified w i t h early vision, i.e., based on operations carried out r a p i d l y a n d i n parallel across the visual field. T h i s is sometimes described as the "image processing" stage, since the representations for b o t h input a n d o u t p u t are generally arrays of pixels, usually w i t h the same spatial dimensions [Ree84]. E a r l y v i s i o n is beUeved to provide a quick i n i t i a l analysis of the image, m a k i n g expUcit those properties useful for subsequent stages of processing (e.g., the locations a n d orientations of Unes i n the image) [ M a r 7 9 , Mar82].-^'' Its p r i m i t i v e elements therefore describe properties that can be reUably determined i n this way. T y p i c a l l y , this is done by the concurrent appUcation of fixed templates to each point i n the image (e.g., spatial filtering [Gra85]). T h i s early stage is c o m m o n to virtuaUy aU c o m p u t a t i o n a l models of low-level vision, t a k i n g o n forms such as the " r a w p r i m a l sketch" of M a r r [Mar82], the " M I R A G E m o d e l " of W a t t a n d M o r g a n [ W M 8 5 , W a t 8 8 ] , and the "cortex t r a n s f o r m " of W a t s o n [Wat87]. A l t h o u g h the p r i m i t i v e s used i n these models differ i n detail, they generally describe simple properties of the image, such as color, orientation, a n d spatial frequency. It has been recognized (e.g. [Mar82]) that primitives should describe properties of the scene whenever possible (e.g., using ^^This distinction between early and low-level vision is not one that is usually drawn. However, it helps to illustrate one of the points being made here, viz., that computational models must incorporate issues of resource use. ^'^Interestingly, in his earlier work, (e.g. [Mar79]), M a r r emphasized that "there seems to be a clear need for being able to do early visual processing roughly and fast as well as more slowly and accurately" p. 31]. T h i s idea became less prominent in later work. [Mar79, contrast to o b t a i n changes i n surface reflectance). B u t scene-based properties t h a t c a n be rehably determined w i t h templates are few a n d far between. For the most p a r t , a complete determination of scene properties requires subsequent stages of processing that employ m o r e sophisticated a n d time-consuming operations. B . L a t e r Stages There are m a n y ways t o associate properties of the scene w i t h p r i m i t i v e image elements (see [ d Y v E 8 8 ] ) . Because of t h i s , and because of the shortage of relevant information f r o m psychophysical a n d neurophysiological studies, there is no general consensus as to how subsequent processing is carried out. One possibihty is that reconstruction is based directly on the image elements, u s i n g constraints derived from the n a t u r e of the scene and the way it is projected to the image plane. T h i s is sometimes assumed t o be done v i a separate streams for each k i n d of visual m e d i u m (e.g., i n f o r m a t i o n obtained v i a luminance, m o t i o n , or texture) or for different scene a n d image properties (see, e.g., [ C A T 9 0 ] ) . B u t although some recovery processes can be carried out a l most i m m e d i a t e l y when p a r a l l e l processing is available (e.g. the recovery of three-dimensional surface o r i e n t a t i o n v i a p h o t o m e t r i c stereo [W008I]), much more time is generally r e q u i r e d . For example, the i n t e r p r e t a t i o n of hne drawings is an N P - c o m p l e t e problem [ K P 8 5 ] , w h i c h effectively rules out the possibihty of always speeding it up sufficiently by p a r a l l e l processing alone (section 2.1.1). Recovery processes described by the frameworks of r e g u l a r i z a t i o n theory [ P T K 8 5 ] or M a r k o v r a n d o m fields [GG84] also are relatively time-intensive, t y p i c a l l y requiring several t h o u s a n d iterations for images of moderate size (e.g., [Bla89]). F u r t h e r m o r e , their close association to relaxation problems makes it Ukely that the time required increases at least Unearly w i t h the size of the i n p u t . A n alternative approach is to b u i l d up the scene descriptions more gradually, v i a an intermediate stage containing " n o n - t e m p l a t e " properties that can be determined quickly. F o r example, the p r i m i t i v e elements of M a r r ' s raw p r i m a l sketch [Mar82] are grouped together o n the basis of l o c a l properties (e.g., common orientation) to form higher-level symboUc structures. T h i s grouping is done recursively, so that highly complex elements can be built up. T h e result is a " f u U " p r i m a l sketch that is available to subsequent processes, such as those i n v o l v e d w i t h t e x t u r e segregation [Mar82]. U U m a n [UU84] has suggested that m a n y spatial relations (including those described i n the f u l l p r i m a l sketch) are obtained v i a the application of visual routines t o the elements of early vision (section 2.1.3). These routines are based on a small set of simple operations such as m a r k i n g a n d p r o p a g a t i o n , which are then concatenated together to form the desired procedure. T h i s allows m a n y " n o n - t e m p l a t e " properties to be extracted from the image i n time p r o p o r t i o n a l to the size of the i n p u t . B u t although the complexity of these strategies is relatively low, the Unear complexity bounds are stiU insufficient for m a n y purposes, especiaUy i f images are large a n d complex. F u r t h e r m o r e , m a n y of these operations are spatially inhomogeneous, suggesting t h a t they may be based on a higher-level serial control [UU84]. A n o t h e r alternative is to choose a more modest common core, e.g., the image itself, w i t h perhaps a few of its properties (such as the orientations of Une fragments) made expUcit. E s sentially, this identifies low-level vision w i t h some variant of early vision, perhaps augmented by high-speed grouping processes. T h i s approach is found i n m a n y model-based recognition schemes (e.g., [ B r o S l , B i e 8 5 , Low85]), where recognition proceeds v i a the m a t c h i n g of image features to projections of a predefined m o d e l onto the image plane. It also is found i n techniques that use image features to index directly into a large set of predefined models (e.g., [PE90]). B u t models are not always available, especiaUy for u n k n o w n environments. E v e n when they are, occlusion often removes m a n y of the relevant features, raising the possibiUty of confusion w i t h other objects that share the same subset of visible features. F u r t h e r m o r e , this approach must be able to handle a l l possible views of aU possible objects at a l l possible orientations i n the scene. T h i s makes the system unwieldy as the number of objects to be represented increases: m e m o r y requirements can become s u b s t a n t i a l if aU possibiUties are to be stored; i f procedures are used to reduce the memory requirements, c o m p u t a t i o n t i m e increases. T h u s , a " m i n i m a l core" based on simple image properties is often too m i n i m a l for low-level v i s i o n . A more complete intermediate description of the scene is therefore required. 2.3.2 T h e R o l e of R a p i d P a r a l l e l R e c o v e r y It w o u l d appear that low-level vision faces a d i l e m m a of sorts, since a common core based on properties of the scene cannot be computed quickly, while simple image-based properties are insufficient for general purposes. B u t there is a way around this d i l e m m a : instead of demanding that interpretations make o p t i m a l use of available i n f o r m a t i o n , demand only t h a t they be "reasonably correct". In p a r t i c u l a r , instead of demanding that interpretations be consistent over the entire image, demand only that they be consistent over spatially U m i t e d zones. Relaxing consistency i n this way allows the recovery of scene properties to have a c o m plexity far below that of " o p t i m a l " recovery: not only is m a x i m a l use made of paraHehsm, but the interaction of each processor w i t h its neighbors can be considerably simphfied (cf. section 2.1.1). Since nonlocal context provides much of the information for i n t e r p r e t a t i o n , the outcome is usually s u b o p t i m a l ; i n fact, interpretations m a y exist only over a sparse set of locations i n the v i s u a l field. T h u s , a r a p i d recovery process cannot be expected to p r o d u c e a description that is complete, or even globally coherent. W h a t can be expected, however, is that some of this description wiU be accurate enough for tasks further along the processing stream. Such " q u a s i - v a l i d " estimates could be useful i n several ways. F o r example, they c o u l d help guide processes that cannot afford to wait for a complete analysis of the scene (e.g., active visual processes such as gaze or focus of attention [Bal91]). T h e y could also act as precursors to serve as the i n i t i a l estimates for slower processes that restore some degree of g l o b a l consistency [ER92]. T h e y might also be used as (invariant) indexes into higherlevel object models, thereby increasing the efficiency of model-based recognition. In any event, this view of early vision suggests that parallel processes may play a greater role t h a n previously suspected â€” i n essence, the " h o r i z o n t a l " stages of the conventional theories m a y be complemented w i t h " v e r t i c a l " islands of locally-consistent interpretations. G i v e n the p l a u s i b l h t y of this viewpoint, the problem now is to develop it into a rigorous theory of early v i s u a l processing. It is essential to find a way to describe a r a p i d recovery process precisely a n d t o j u s t i f y its operation. W h a t is required for this is a framework t h a t allows it to be given a c o m p u t a t i o n a l analysis i n the sense of M a r r [Mar82]. 2.4 The Analysis of Resource-Limited Processes If r a p i d recovery is to be given a rigorous c o m p u t a t i o n a l analysis, a general framework m u s t exist that allows a clear f o r m u l a t i o n of the problem and sets the ground rules for its e x p l a n a t i o n . T h e framework proposed by M a r r [Mar82] goes a long way towards this end. However, it can only be used to analyze processes for which the h m i t e d resource is the i n f o r m a t i o n available i n the image [RP91]. A few studies (e.g., [ F B 8 2 , Ros87, Tso87]) have grappled w i t h the issue of how t i m e a n d space h m i t a t i o n s influence the structure of a visual process, but a general framework for the i n c o r p o r a t i o n of resource h m i t a t i o n s has not yet appeared. Such a framework is therefore developed here, based on a direct extension of M a r r ' s framework. 2.4.1 Marr's Framework A c c o r d i n g t o M a r r [Mar82], the complete analysis of a visual process involves three different levels of e x p l a n a t i o n : 1. C o m p u t a t i o n a l level. T h i s is concerned w i t h the functional aspects of the task. It consists of two p a r t s : (i) a description of the constraints between the i n p u t and o u t p u t of a v i s u a l process, a n d (ii) a justification of why these p a r t i c u l a r constraints were chosen. 2. A l g o r i t h m i c level. A n a l y s i s at this level describes and justifies the representations a n d algorithms used. It is essentially a constructive demonstration that an a l g o r i t h m exists capable of generating the required m a p p i n g . 3. I m p l e m e n t a t i o n a l level. T h i s level is concerned w i t h the description a n d justifica- t i o n of the p h y s i c a l substrate on which the algorithms are implemented. A n " i m p l e m e n t a t i o n a l " explanation provides a constructive demonstration that there exists a p h y s i c a l system that can carry out the required computations. One of the strengths of this framework is its recognition of a separate " c o m p u t a t i o n a l " level of e x p l a n a t i o n focusing on b o t h the what a n d the why of the i n p u t - o u t p u t m a p p i n g . T h e what is concerned w i t h the expUcit description of the constraints on the f o r m of the m a p p i n g . T h i s aspect of analysis is complete when the constraints are shown to determine a unique m a p p i n g . T h e why is concerned w i t h the justification of these constraints, showing that t h e resulting set of associations between input a n d output is suitable for the purposes at h a n d . To use an example taken f r o m M a r r [Mar82, p p . 22-24], the what of a cash register's function is explained by describing its output as the s u m of its i n p u t s . T h e why of this function is explained by the need for a p r i c i n g mechanism that has a zero value, is commutative a n d associative, a n d that allows inverse operations. In this approach, constraints on the i n p u t - o u t p u t m a p p i n g of a visual process are assumed to be machine-indifferent, originating f r o m the laws of optics or f r o m the structure of the objects under consideration. A s such, it i m p U c i t l y assumes that the mappings are shaped only by the i n f o r m a t i o n available i n the image, a n d not by Umits on the c o m p u t a t i o n a l resources. T h i s aUows analysis to be completely general, w i t h no dependence on the structure of the processor c a r r y i n g out the computations. W h e n the process is U m i t e d p r i m a r i l y by the ' * M a r r [Mar82] does consider efficiency to be important, but only once the task itself has been laid out. Efficiency itself is therefore addressed at the algorithmic rather than the computational level of analysis. available i n f o r m a t i o n , it can be completely explained by this k i n d of analysis. B u t when it is h m i t e d by other factors, something more is needed. 2.4.2 Extensions A . E x t e r n a l a n d Internal C o n s t r a i n t s If resource h m i t a t i o n s are to be incorporated into a c o m p u t a t i o n a l framework, several i m p o r t a n t distinctions must first be made. T h e first is that between external a n d internal constraints. E x t e r n a l constraints are those on the " s t a t i c " aspect of the m a p p i n g , i.e., those definable w i t h o u t regard to the way the output is generated. These are essentially the constraints that apply when the processor is viewed as a "black b o x " . In the case of the cash register, for example, the requirements of c o m m u t a t i v i t y a n d associativity are external constraints, apphcable only to the final form of the output function. These constraints c a n operate either directly v i a relations between input a n d output elements, or more i n d i r e c t l y v i a relations between the elements of the input or output domains (see, e.g., [RM89]). W h e n a process can be analyzed entirely i n terms of external constraints, the i n t e r n a l details of the processor are irrelevant. B u t when h m i t e d c o m p u t a t i o n a l resources enter i n t o the p i c t u r e , it becomes i m p o r t a n t to consider exactly how the process is carried out. T h i s is specified by the i n t e r n a l constraints, w h i c h apply to the way the output is generated [RP91]. M o r e precisely, these are invariants of the information flow that occurs d u r i n g the course of the c o m p u t a t i o n . These constraints include hmits on the communication b a n d w i d t h a n d a r c h i t e c t u r a l constraints o n the set of basic operations to be used. Internal constraints can therefore influence the complexity of a given operation on various kinds of processors. B . Resources a n d R e s o u r c e L i m i t a t i o n s It is i m p o r t a n t to recognize that when an information-processing task is analyzed, a subset of constraints is usuaUy specified that is fbced and not subject to further discussion. Eor example, when a n a l y z i n g a process to recover shape f r o m shading, the available i n f o r m a t i o n is determined by the viewing conditions and sensor array specified i n the problem f o r m u l a t i o n . E x p l a n a t i o n then centers a r o u n d the constraints used to recover shape from this i n f o r m a t i o n , but the available i n f o r m a t i o n itself remains as a given throughout this analysis, and does not need to be explained. A s such, the available information is effectively a "boundary c o n d i t i o n " for the analysis, Umiting the set of i n p u t - o u t p u t mappings that can be considered. quantities are referred to here as resources, and the corresponding constraints as Such resource Umitations. Resources can involve either the external or the internal aspects of processor o p e r a t i o n . E x t e r n a l resources are quantities that can be defined independently of processor s t r u c t u r e . These include not only the available i n f o r m a t i o n , but also such things as the t o t a l a m o u n t of time or energy used. T h e corresponding constraints are referred to as external limitations. These generally result f r o m higher-level factors i n the surrounding environment. A s s u c h , they can be considered to be constant over the course of processing (cf. [Sal85, ch.4]). Internal resources can be similarly defined as those quantities relevant to the i n t e r n a l structure of the processor. E x a m p l e s of these include communication b a n d w i d t h , the d i s t r i b u t i o n of m e m o r y buffers w i t h i n the architecture, a n d the p r o p o r t i o n of m a t t e r t a k i n g the f o r m of processing elements. T h e corresponding constraints on these quantities are referred to as internal limitations. Considering again the example of the cash register, the e x p l a n a t i o n of a p a r t i c u l a r design m a y involve an i n t e r n a l h m i t a t i o n such as the requirement that m e t a l gears be used for a l l operations. N o t e that a similar complementarity exists for b o t h kinds of constraint - as a given analysis requires fewer resources i n its " b o u n d a r y conditions", it appHes to a wider range of processes. If an analysis requires the existence of five identifiable points i n an image, it also is apphcable to processes based on six identifiable points. If only four points are required i n the analysis, it can be appHed to an even larger set of processes. In the same way, an analysis that explains the operation of a cash register containing twenty gears also applies to a wider range of processes t h a n one based on a l i m i t a t i o n of forty gears. C. Abstractness A q u a n t i t y is said to be abstract to the degree that its physical composition is relevant to the analysis. T h e most abstract quantities are purely formal ones, i.e., those that are independent of the properties of the u n d e r l y i n g substrate. Such f o r m a l quantities include i n f o r m a t i o n a n d c o m p u t a t i o n a l measures of t i m e (section 2.1.1). T h e corresponding constraints a n d l i m i t a tions are as abstract as the least abstract quantity involved. For example, the requirement of c o m m u t a t i v i t y is a purely abstract constraint on the operation of a cash register, being completely independent of its m a t e r i a l composition. Similarly, the requirement that a base 10 representation be used is also independent of physical structure. A s these examples show, b o t h i n t e r n a l and external constraints can be completely abstract. M o r e concrete quantities contain intrinsic constraints due to the physical properties of the u n d e r l y i n g substrate, such as its density or t h e r m a l conductivity. These properties can affect b o t h external and i n t e r n a l aspects of the processor's operation. For example, setting an upper Umit on the weight of a cash register hmits the t o t a l value that can be represented. T h i s results i n an " a p p r o x i m a t i o n " of the addition operation i n which the output is given a definite upper b o u n d . T h i s upper b o u n d may not necessarily be i m p o r t a n t for p r a c t i c a l purposes if the cash register is an electronic device, but it may weU have a serious effect i f a d d i t i o n is required t o be done mechanically. D . Completeness A n analysis is said to be complete to the extent that the constraints determine the uniqueness of the m a p p i n g , a l g o r i t h m , or Implementation being analyzed. For example, requiring a d d i tion to be based on a p o s i t i o n a l numeric representation provides only a p a r t i a l specification of its a l g o r i t h m i c structure, since â€” among other things â€” the particular base has not been specified. T h e choice of base has no Impact on functional properties such as c o m m u t a t i v i t y and associativity. It may, however. Influence the efficiencies possible for various operations. N o t e that the Initial set of h m i t a t i o n s assumed In the formulation of a problem already sets Umits t o the kinds of m a p p i n g s , algorithms, or implementations that are possible. The set of constraints obtained f r o m a c o m p u t a t i o n a l analysis therefore serves to complete this o r i g i n a l set of specifications. 2.4.3 A Revised Framework T h e above considerations can be Incorporated Into a coherent system by a straightforward extension of M a r r ' s framework. T h e resulting system Is summarized i n figure 2.5. A s In the original framework, analysis Is carried out at three different levels of e x p l a n a t i o n . T h e most general of these is the c o m p u t a t i o n a l level, where analysis Is centered around the description and justification of the m a p p i n g between image and reconstructed scene. A n a l y s i s at this level is complete when It Is shown that the m a p p i n g described by the constraints is (i) unique, and (II) Is consistent w i t h the given h m i t a t i o n s . 1. Computational Level Constraints sufficient to determine input-output mapping that is (i) unique, and (ii) exists witliin given limitations Abstract j n Concrete External All Possibly some Internal ^H::;:::;:::;: Possibly some iov:::;: Possibly some :::!:::::.â€¢: 2. Algorithmic Level Constraints sufficient to determine procedural decomposition that is (i) unique, and (ii) exists within given limitations External Abstract Concrete Internal g I:;:;:;!;:;:; Possibly some J All remaining Possi bl y 30 me 3. Implementational Level Constraints sufficient to determine physical instantiation that is (i) unique, and (ii) exists within given limitations F i g u r e 2.5: E x t e n d e d c o m p u t a t i o n a l framework. If the analysis is to be general, these hmitations must be abstract (i.e., involve no p h y s i c a l properties) a n d external (i.e., have no dependence on the internal structure of the processor). T h e constraints derived under these conditions (shown i n the upper left quadrant of figure 2.5) are therefore independent of any assumptions about the processor ItseK. T h i s essentially corresponds to an analysis carried out at the c o m p u t a t i o n a l level i n M a r r ' s framework, except that constraints m a y now be justified by an appeal to abstract resources other t h a n available i n f o r m a t i o n (e.g., t i m e or space). It m a y not be possible to e x p l a i n a m a p p i n g i n such a general way if it has been s h a p e d by the p h y s i c a l properties or i n t e r n a l structure of the processor. A n a l y s i s must t h e n be completed by Invoking h m i t a t i o n s that are less general. These m a y be less abstract or m a y involve the Internal structure of the processor to some degree. T h e corresponding constraints are located i n the r e m a i n i n g quadrants of figure 2.5. N o t e that if h m i t a t i o n s p e r t a i n t o only a few aspects of the process, they can give rise to only a p a r t i a l set of constraints o n Its p h y s i c a l substrate or architecture. If so, this still allows the analysis to be apphcable t o a relatively large set of processes. Similar considerations apply to the algorithmic level of analysis, where the goal Is t o decompose the given process Into a system of more elementary d a t a structures a n d operations. E x p l a n a t i o n at this level describes a n d justifies the constraints that make this decomposition unique. If Internal constraints exist at the c o m p u t a t i o n a l level, the two levels of analysis w i n not be completely Independent â€” the algorithmic analysis must not only o b t a i n a set of abstract Internal constraints, but also ensure that they are consistent w i t h those o b t a i n e d f r o m the c o m p u t a t i o n a l level (upper right quadrants i n figure 2.5). A n algorithmic analysis Is complete to the extent that It specifies a decomposition that is b o t h unique and consistent w i t h a l l other constraints. It is general to the extent that n o t h i n g is assumed about the p h y s i c a l composition of the processor itself. T h e final level of analysis is that of implementation. A s In M a r r ' s framework, the goal Is t o specify a set of constraints that determine a unique physical i n s t a n t i a t i o n of the processor. However, there are now two sources of constraint to contend w i t h : external constraints on the t o t a l a m o u n t of m a t e r i a l , a n d i n t e r n a l constraints on its d i s t r i b u t i o n w i t h i n the processor. N o t e t h a t the i m p l e m e n t a t i o n a l constraints do not determine the physical i m p l e m e n t a t i o n precisely, b u t only to the " g r a n u l a r i t y " of the algorithmic analysis. Once a process is understood at the three levels, analysis can be recursively apphed to each of the components this decomposition. of 2.5 Rapid Line Interpretation C o m p u t a t i o n a l models have been most successful when (i) the parameters of the p r o b l e m (such as i n p u t , o u t p u t , a n d resource use) can be clearly specified, (ii) the problem c a n be solved by a m o d u l a r process, a n d (ui) the constraints obtained by the analysis lead to testable predictions (see, e.g., [ P T K 8 5 ] ) . It is evident from section 2.3.2 that the r a p i d recovery process is highly m o d u l a r , r e q u i r i n g v i r t u a l l y no interaction w i t h other aspects of visual processing. It is also evident that knowledge of the constraints on this process can lead to p r e d i c t i o n s about the kinds of hne drawings that can and cannot be r a p i d l y detected at early levels of h u m a n v i s i o n . T h i s section shows that the problem itself is weh defined, w i t h a l l relevant parameters clearly specified. 2.5.1 Basic Terms A . Time In order t o keep the analysis as general as possible, the basic unit of time is taken to be t h a t required to combine two independent quantities, or to t r a n s m i t across some unit distance. B y describing t i m e i n terms of 0 - n o t a t i o n , this basic unit does not need to be specified i n greater d e t a i l (see section 2.1.1). It is also i m p o r t a n t here to distinguish between serial a n d parahel measures of t i m e . S e r i a l t i m e refers t o t h a t required o n a serial machine; essentiaUy, this describes the t o t a l amount of " w o r k " needed. P a r a l l e l t i m e is the m i n i m u m time required on a given parallel architecture, a n d is often less t h a n serial time.^^ Unless otherwise specified, time is identified here w i t h paraUel t i m e . B . R a p i d processing F o r m a n y v i s u a l processes, it Is assumed (often ImpUcltly) that o p t i m a l or n e a r - o p t i m a l use Is made of the Information available In the image. T h i s effectively places a fixed lower b o u n d o n the i n f o r m a t i o n to be used for a process, the exact b o u n d depending on the input i m a g e . Since every p r o b l e m has an intrinsic complexity, any such " i n f o r m a t l o n - h m l t e d " p r o b l e m must have a lower b o u n d on the time It requires (see section 2.1.1). 'Â®The solution of some problems cannot be sped up by using a paraUel architecture â€” see section 2.1.1. Similar considerations h o l d for other resource Umitations. In p a r t i c u l a r , a " t i m e - U m i t e d " process can be defined by placing upper bounds on the available processing t i m e , these bounds depending on the input image. N o upper b o u n d is expUcitly given for the information used by such a process, but complexity considerations i m p l y that such an b o u n d must exist. G i v e n the complementary n a t u r e of their upper a n d lower Umits, it is seen that â€” at least i n a very b r o a d sense â€” i n f o r m a t i o n - U m i t e d and time-Umited processes are duals of each other. Intuitively, a r a p i d process is a time-Umited process for which the upper bounds on t i m e are relatively low. In the interests of precision, the t e r m ' r a p i d ' refers here to any process for w h i c h the complexity is a subUnear function of the number of Unes i n the i m a g e . T h i s choice is m o t i v a t e d by two considerations. F i r s t , processes that can be carried out i n p o l y n o m i a l time f o r m a n a t u r a l complexity class, w i t h aU p o l y n o m i a l processes retaining p o l y n o m i a l complexity even when carried out on various machines (section 2.1.1). A s such, Unear- t i m e processes cannot be readily isolated. G i v e n that processes of high-degree p o l y n o m i a l c o m p l e x i t y cannot be considered as r a p i d , the subUnear criterion must be imposed if an a w k w a r d theoretical b o u n d a r y is t o be avoided. T h e choice of the subUnear criterion also is m o t i v a t e d by p r a c t i c a l reasons: it is generally impossible to distinguish a Unear-time parallel process from a constant-time process appUed sequentiaUy to each l o c a t i o n i n the visual field [Tow72]. Consequently, only subUnear processes can be readily identijfied as being carried out i n paraUel. 2.5.2 F o r m u l a t i o n of t h e P r o b l e m In w h a t foUows, the expression ' r a p i d recovery' refers to the r a p i d interpretation of Une drawings. T h e scene d o m a i n is a restriction of the blocks w o r l d (section 2.2.1) i n w h i c h only three edges can be i n contact about the vertex of any corner (section 1.1). T h e scene is assumed to be projected onto the image plane v i a a monocular orthographic p r o j e c t i o n . T h e inputs are therefore drawings composed of straight Une segments w i t h no dangUng ends a n d w h i c h meet i n j u n c t i o n s composed of either two or three Unes. T h e outputs are viewercentered dense descriptions (i.e., maps) of the structure of the corresponding p o l y h e d r a i n ^Â° Note that this is completely separate from considerations of efficiency. T h e efficiency of an informationlimited process is a measure obtained by comparing the time it requires against the absolute lower bound imposed by complexity considerations. Similarly, the efficiency of a time-limited process is measured by comparing the amount of information it extracts from the image against the maximum that could be achieved. In both cases, efficiency is described in the same terms. ^^The term 'real-time' has been suggested for processes requiring at most Knear time [Vol82]. the s c e n e . A n estimate of the relevant properties is assumed to exist at every point along these hnes. A s for r a p i d processing generally, the available time is h m i t e d to a subhnear f u n c t i o n of the size of the p r o b l e m , i.e., the number of the hnes and vertices i n the d r a w i n g . T h e p r o b l e m is to recover as m u c h of the scene structure as possible w i t h i n the aUocated time. In what foUows, a rather severe h m i t a t i o n is imposed: the recovery process must use only a constant amount of t i m e , i.e., the amount of time must be independent of the size or content of the i n p u t . T h i s is m o t i v a t e d by several considerations. F i r s t , if the output of the r a p i d recovery process were the basis for more complex operations at higher levels (section 2.3), c o n t r o l of this interface would be greatly simphfied if it could be assured that recovery was always completed w i t h i n a fijced amount of time. Second, constant-time hne i n t e r p r e t a t i o n is an extreme case of r a p i d recovery, a n d therefore a n interesting p r o b l e m i n its own right. A m o n g other things, any structure recovered under these conditions sets a lower b o u n d on what can be expected of any r a p i d recovery process. A n d given that extremely low h m i t a t i o n s are involved, the results o b t a i n e d w o u l d be apphcable to the widest variety of processes (section 2.4.2). F i n a l l y , constant-time interpretation leads i n a very n a t u r a l way to the locally-consistent estimates assumed to be provided by r a p i d recovery (section 2.3). Since transmission speeds are finite, a constant-time h m i t a t i o n translates into a constant-distance h m i t on the t r a n s m i s s i o n of i n f o r m a t i o n i n the o u t p u t . A s such, inconsistencies resulting from violations of the u n d e r l y i n g assumptions are not propagated throughout the image, but are restricted to relatively s m a l l regions, or "patches". T h i s consequently avoids the destruction of Interpretations In areas where these assumptions do h o l d . T o m a k e the analysis relevant for the greatest range of processors, relatively severe h m i t a t i o n s are also placed on the available processing resources (cf. section 2.4.2). Since the r a p i d i n t e r p r e t a t i o n is hkely to be done in-place by a spatiotopic array of processing elements (section 2.3), the number of processors must be p r o p o r t i o n a l to the number of locations u p o n w h i c h the hne drawing falls; accordingly, 0{n) processors are assumed to be available for a n i n p u t of size n. T h e simplest way to coordinate these elements is as a t w o - d i m e n s i o n a l a r r a y of independent processors. B u t although this architecture is i n some sense a m i n i m a l ^^The term 'structure' refers here to properties of the scene. These are chosen to be the (positive) convexities, slant signs and slant magnitudes of the edges, as well as the contiguity relations between edges and surfaces (section 4.1.1). one, it requires a considerable amount of wiring for each processing element (section 2.1.1). Processors are therefore assumed to be arranged i n a mesh, w i t h each element connected only t o its nearest neighbor. It also is assumed that each processor is simple enough t h a t its operation requires only a fixed amount of space and time.^^ A l t h o u g h the p a r t i c u l a r space a n d time Umitations that apply to r a p i d recovery are not k n o w n , most aspects of this process can be analyzed without knowing their exact values. T h i s can be done by assuming that the t i m e required for l o c a l processing is less t h a n that r e q u i r e d for transmission across some smaU fraction of the image. T h i s amounts to an assumption t h a t the complexity of r a p i d recovery is d o m i n a t e d by transmission t i m e , a point of view largely i n accord w i t h k n o w n Umits on biological a n d artificial processors (section 2.1.1). A m o n g other things, this assumption removes the need to distinguish between time as defined by s i g n a l propagation a n d time as defined by the number of switches along the p a t h (section 2.1.1), since these two measures are directly p r o p o r t i o n a l to each other for a mesh architecture. T h e i n t e r p r e t a t i o n process can therefore be described i n terms of the percolation of i n f o r m a t i o n t h r o u g h a mesh network at some constant speed. T h e absolute size of the image, the size, speed a n d spacing of the processors, and the speed of transmission do not need to be k n o w n â€” a l l that is relevant is the r a t i o of transmission speed to the length of the Unes i n the d r a w i n g . E v e n this can be eUminated by a rescaUng of the image (e.g., setting the average Une length to some constant). Consequently, the c o m p u t a t i o n a l analysis is largely independent of the details of any p a r t i c u l a r representation or architecture used. These assumptions, of course, do not rule out the use of a more complex architecture such as a pyramid. Rather, they merely avoid assuming the extra processing power, allowing the analysis to apply to a larger set of processes. Chapter 3 Low-Complexity Recovery T h e success of a r a p i d recovery system rests u p o n its ability to recover a large a m o u n t of scene structure w i t h i n a s m a l l amount of t i m e . Since the interpretation of a Hne d r a w i n g is a n N P - c o m p l e t e p r o b l e m (section 2.2.1), a m a p p i n g that recovers a l l possible three-dimensional structure is not generally suitable for this purpose. Instead, a low-complexity " a p p r o x i m a t i o n " must be used t h a t captures only part of the relevant structure. A n a p p r o x i m a t i o n can differ f r o m a more complete m a p p i n g i n three ways: (i) fewer degrees of freedom i n the i n p u t , (ii) fewer degrees of freedom i n the o u t p u t , a n d (iii) fewer transformations of the given d a t a . T h e first way (see, e.g., [Tso87]) essentially reduces the input resolution, while the second (see, e.g., [Lev86]) reduces the expressiveness of the o u t p u t . B u t r a p i d recovery is assumed to have the same k i n d of m a p p i n g as for o p t i m a l i n t e r p r e t a t i o n , v i z . , an association of scene properties to each (high-resolution) Une i n the image (section 2.5.2). A p p r o x i m a t i o n is therefore based on the t h i r d way â€” fewer transformations of the data. Because fewer transformations are involved, scene properties cannot always be recovered successfuUy at each zone i n the image. If constraints are chosen carefuUy, however, the UkeUhood of this recovery can remain h i g h . G i v e n the Umitations on time a n d transmission distance, this UkeUhood is highest for those aspects that (i) are easy to compute, a n d (U) require m i n i m a l " n o n l o c a l " i n p u t , i.e., m i n i m a l input from areas outside the zone. T h i s chapter examines the extent to w h i c h low-complexity recovery can be carried out along concurrent streams, each concerned w i t h a single dimension of scene analysis. F o u r p a r t i c u l a r dimensions are considered: the contiguity and convexity of edges a n d the sign a n d m a g n i t u d e of edge slants. C o m p l e x i t y bounds are derived that show the extent to w h i c h each of these properties can be computed i n subhnear time and w i t h m i n i m a l nonlocal i n f o r m a t i o n . It also is shown t h a t these streams can be combined to completely recover b o t h q u a l i t a t i v e and q u a n t i t a t i v e structure i n subUnear t i m e for several subdomains of polyhedral o b j e c t s , including convex p o l y h e d r a a n d p o l y h e d r a w i t h rectangular corners. 3.1 General Issues Several general issues are involved i n the specification of a m a p p i n g for a r a p i d recovery process. T h i s section discusses three of the more i m p o r t a n t ones: the degree to w h i c h p r o cessing power can be increased by concurrent processing streams, the complexity of s o l v i n g the constraints w i t h i n each stream, and the tradeoffs that exist when a p p r o x i m a t i n g a given m a p p i n g by one of lower complexity. 3.1.1 Concurrent Streams A decomposition into separate processing streams is found i n m a n y c o m p u t a t i o n a l models of early vision (see, e.g., [ P T K 8 5 ] ) . T h i s decomposition often has its origins i n the processing of different m e d i a (e.g., contours defined by luminance, m o t i o n , or texture [ C A T 9 0 ] ) , or i n the processing of different aspects of the output (e.g., m o t i o n , color, and binocular dispari t y ) . E a c h of these streams essentially contains a bundle of highly-correlated i n f o r m a t i o n ( a " d i m e n s i o n " ) that describes some p a r t i c u l a r aspect of structure i n the image or scene. T h e existence of separate streams is beUeved to faciUtate the development of p e r c e p t u a l processes, since n a t u r a l selection can act independently on each one [ S i m S l , M a r 8 2 ] . B u t there also is another reason for their existence â€” they m a x i m i z e the sheer amount of d a t a t r a n s f o r m a t i o n t h a t can be done w i t h i n a given amount of time. If a set of operations are independent of each other, they can be carried out faster i n paraUel rather t h a n i n sequence. A l s o , the complexity of each operation is often lower when fewer and less complex variables are involved. If the constraints can be reformulated such that each dimension involves only a few variables, then a m a x i m u m amount of d a t a transformation is possible. Such a dimension is readily obtained by coalescing the original variables into a few groups, w h i c h are t h e n treated as coarse-grained variables governed by a smaller set of "coUapsed" constraints [ M M H 8 5 , M a l 8 7 ] . B y grouping the original set of variables i n several different ways, the o r i g i n a l problem can be largely decomposed into several simpler subproblems, each of w h i c h can be solved by a concurrent processing stream. Note that the sets of properties handled by these streams do not need to be independent of each other â€” only the systems of constraints need to be this way. Decomposing a process into concurrent streams can lead to a considerable r e d u c t i o n of processing t i m e , but the price of this reduction is a loss of coherence: the solutions o b t a i n e d i n each stream are not necessarily compatible w i t h those obtained i n the other s t r e a m s . T h u s , cross-dimensional constraints must be incorporated if aU (or even much) of the p o w e r of the original set of constraints is to be retained. A n i m p o r t a n t aspect of developing a successful a p p r o x i m a t i o n is therefore to m a x i m i z e the use of cross-dimensional constraints w i t h o u t increasing the complexity of the problem. O n e such strategy is based on a hierarchical decomposition of variables [ M M H 8 5 ] . H e r e , the original variables are grouped together into a few sets of coarser-grained variables t h a t obey a simple set of collapsed constraints. Once the set of coUapsed constraints is solved, the result is used as the basis for a new problem involving finer-grained variables. T h i s c a n i n t u r n be applied to yet another set of constraints on even finer-grained variables. In essence, the problem has been decomposed into a sequence of simple streams i n which the o u t p u t s of the coarser-grained systems help w i t h the solution of the finer-grained ones. M o r e generally, low-complexity interaction is possible if information from an u n a m b i g u ous set of results i n one stream can be t r a n s m i t t e d to help constrain possible solutions i n another. Since the transmission is based on unambiguous (local) results, the b a c k w a r d flow of i n f o r m a t i o n f r o m the second stream to the first one has no further effect on the o r i g i n a l result. T h i s essentially corresponds to a unidirectional hnkage (section 2.1.1) between streams, w i t h Unkage now generalized to apply not only to interactions across geometrical space, b u t across more abstract dimensions as well. If the amount of information to be t r a n s m i t t e d is s m a l l , the cross-dimensional constraints w i l l not a d d to the complexity of the p r o b l e m . 3.1.2 R e d u c t i o n to C a n o n i c a l F o r m s If a n a p p r o x i m a t i o n is to capture m u c h of the structure of the original m a p p i n g , it must focus on those aspects of the scene that are (i) easy to compute a n d (ii) need a m i n i m a l amount of i n f o r m a t i o n f r o m outside the local zone. One way to help ensure that these conditions are met is to select dimensions such that their determination can be reduced to the solution of some l o w - c o m p l e x i t y problem. T w o problems are of p a r t i c u l a r importance i n this regard: 2-Satisfiabihty ( 2 - S A T ) a n d connected components labeUing ( C C L ) . In order to simplify the reduction to these problems, only the constraints on arrow-, Y , a n d L - j u n c t i o n s are considered exphcitly. T h e constraints on T-junctions are h a n d l e d by a preprocessing step where v i r t u a l gaps are introduced between the stems and the crossbars of each T - j u n c t i o n , the two hnes afterwards treated as unconnected. T h e crossbar of the T - j u n c t i o n then corresponds to an occluding edge, while the stem becomes an unconstrained hne that has at least one " d a n g h n g " end. T h i s reformulation has the advantage that the r e m a i n i n g hnkages automaticahy spht the n hnes i n the image into separate partitions, each of w h i c h can be treated separately.^ A . R e d u c t i o n to 2-SAT One way to ensure that a dimension is easy to compute is to restrict the set of " i n t r a d i m e n s i o n a l " constraints so that they correspond to an instance of the 2-Satisfiabihty ( 2 - S A T ) p r o b l e m . T h i s can be defined i n terms of a set of boolean variables^ V â€” {vi,V2,. â€¢., Vn) a n d a set of clauses C = ( c j , C 2 , . . . , c^), w i t h each Ci containing either one or two variables. T h e p r o b l e m is to assign t r u t h values to the Vi such that a l l clauses i n C have at least one ' t r u e ' h t e r a l (see, e.g. [GJ79]). Since the clause Ck = {vi, Vj} is a disjunction of variables, it has the equivalent f o r m = ~ {vi Avj). Consequently, any problem involving b i n a r y constraints o n two-valued variables can be treated as an instance of 2 - S A T [MacQl]. For the hne labelhng p r o b l e m , the variables are the possible edge labels, w i t h the set of Huffman-Clowes constraints (section 2.2.1) determining their allowable combinations. Since these variables arc four-valued, reduction to 2 - S A T can only be done by decomposing the set of variables into sets of simpler elements. For the most p a r t , such a reduction occurs v i a the direct t r a n s c r i p t i o n of edge labels i n t o two-valued variables a n d the recasting of the remaining j u n c t i o n constraints into b i n a r y f o r m (i.e., into a f o r m i n v o l v i n g only two variables). For example, edge convexity could be expressed i n 2 - S A T by t a k i n g the '-|-' and ' - ' labels as the complementary values to be attached to the (interior) edges, and â€” as far as possible â€” implementing the constraints o n the j u n c t i o n interpretations v i a b i n a r y constraints on these variables. Because of the 'Structures such as holes soraetimes lead to separate sets of labels being used for different parts of the same object. But this occurs even in Huffman-Clowes labelling. ^More precisely, the 2 - S A T problem is defined in terms of a set of literals U = ( Â« i , Â¥ i , M2, Â«2, â€¢ . . , uâ€ž, Â¥â€ž). These literals are constrained such that only one of the pair Ui or Â«i can be used, allowing them to be treated as two-valued boolean variables, the value of variable i being 'true' if ui is selected and 'false' if a; is selected. independence of the p a r t i t i o n s , variables i n the image can take on more t h a n two values, provided t h a t t h i s does not occur w i t h i n any single p a r t i t i o n . B r i n g i n g together the above considerations yields T h e o r e m 3.1 7/ a line drawing of n lines has one or more partitions, 1. All variables take on 2 values, 2. All constraints then the relevant and on more than one variable are labelling problem each such that binary, can be reduced to an instance of 2-SAT. A U 2 - S A T problems of size n can be solved i n 0 ( n ) t i m e on a serial processor [E1S76], a n d i n O(log'^n) t i m e when O(n^) processors are available [ K P 8 8 ] . B . R e d u c t i o n to C C L T h e r e d u c t i o n to 2 - S A T makes Uttle appeal t o the geometrical organization of the constraints per se, using t h e m only i n regards to the ease of c o m p u t a t i o n . B u t spatial coherence can also be used, b o t h to o b t a i n an a p p r o x i m a t i o n of lower complexity, a n d to help m i n i m i z e the amount of n o n l o c a l i n p u t needed for the l o c a l interpretations. In p a r t i c u l a r , note t h a t there sometimes exists a coordination among the sets of edge labels possible for a j u n c t i o n , or more generally, for some connected subset of Unes i n the drawing. For example, if boundary edges are ignored, a l l Unes i n a Y - j u n c t i o n must have the same l a b e l , either '-|-' or T o capture this n o t i o n of c o o r d i n a t i o n , define a bijective constraint on a set of variables Ui as one where the number of possible values for each variable is the same, a n d w i t h a 1:1 Unking between the allowable values (figure 3.1). F o r a bijectiVe constraint, therefore, the value of one variable determines the values of the others."^ The variables are essentially "locked together", a n d can be treated as a single quantity, w i t h a p p r o p r i a t e l y reformulated constraints being appUed to neighboring variables. W h e n two bijective constraints apply to a c o m m o n variable, the resultant set of constraints is also bijective. T h i s can simpUfy analysis considerably, allowing sets of junctions w i t h bijective constraints on m variables to be treated as a single m-valued complex when these j u n c t i o n s are connected to each other by Unes i n the drawing (figure 3.1). A n example ^More precisely, the value of any variable is related to that of any other by a bijective function. bijective / nonbijective nonbijective N / \ / /-\ /-â€¢: bijective S / bijective ^ / S /--.. complex j /-\ complex j+1 F i g u r e 3.1: L i n k i n g of l o c a l constraints. Lines connect values compatible w i t h each other. of such a co-ordinated complex is the Necker cube. Here, two globally-consistent interpre- tations are possible, each of which has no l o c a l interpretations i n c o m m o n w i t h the other. Because the junctions are h n k e d v i a bijective constraints, the interpretation of any one j u n c t i o n i m m e d i a t e l y determines those of aU the others. For a bijective complex, globahy inconsistent labeUings can be removed by sending a signal f r o m locations where legal values are missing and propagating it along the hnes of the complex, the signal causing the w i t h d r a w a l of the relevant value at each location along the way. T h i s p r o p a g a t i o n can be stopped at locations where the value has already been r e m o v e d , a n d so when aU propagated signals have stopped, only the consistent labeUings w i U r e m a i n . T h i s process is essentiaUy a variant of connected component labeUing ( C C L ) , w i t h connections made on the basis of the bijective constraints found at each j u n c t i o n . ' ' If only one p a r t i c u l a r i n t e r p r e t a t i o n is required, a l l but one value can be deleted f r o m one of the l o c a t i o n s , a n d the interpretations associated w i t h the deleted variables removed. T h e i n t e r p r e t a t i o n process for a complex of bijective constraints can therefore be reduced to C C L . Since a Une d r a w i n g may contain several complexes separated or surrounded by "free" variables not i n a complex, this is not necessarily true of the interpretation of the d r a w i n g itself. A lower-complexity a p p r o x i m a t i o n wiU only be possible when m i n i m a l effort is used i n assigning values to the free variables and co-ordinating the interpretations of the various complexes. A t least two such conditions exist: when variables can have only one * M o r e abstractly, this is a unidirectional perimeter-linkage problem (section 2.1.1), since all that is required is knowing which of the m labels to attach to each of the hnes crossing the boundary of the zone. T h e result of joining together two complexes across adjacent zones is always a single complex, since the constraints across the boundary are also bijective. value, or when they a^e not subject to any constraints at a l l . T h i s consequently yields T h e o r e m 3.2 If a drawing 1. All variables of n lines has one or more partitions, take on m values, 2. All constraints each such that and on more than one variable are bijective then the problem can be reduced to an instance of CCL. T h e complexity of C C L is Q{n) time for a serial processor, and O ( l o g n ) time when n p r o cessors are available^ [SV82, L A N 8 9 ] . Bijective constraints can also simpUfy the analysis of cross-dimensional interactions. If t h e i n t e r p r e t a t i o n i n one stream corresponds to a single complex that covers the entire d r a w i n g , the n u m b e r of possibilities is fixed, being at most the number of values possible for any l o c a l variable. A n d if the i n t e r p r e t a t i o n i n a different stream also corresponds to such a complex, it too w i n have a fixed number of possible interpretations. Since only a fixed number of possible combinations needs to be examined, the interaction between the two streams w i l l increase complexity by at most a constant factor. T h i s remains true even if the complexes do not cover the entire d r a w i n g , or if complexes of different streams are not aligned w i t h each other â€” the i m p o r t a n t factor is only that at each location i n the image only a fixed n u m b e r of combinations is possible. 3.1.3 Approximation Strategies It is often the case t h a t a set of constraints must be altered if a problem is t o be reduced to a l o w - c o m p l e x i t y f o r m . A l t h o u g h there are a large number of ways that this can be done, two general strategies â€” each diametricaUy opposed to the other â€” can be discerned. T h e first of these is a conservative strategy, which increases the number of constraints u n t i l aU can be re-expressed i n the appropriate (e.g., b i n a r y or bijective) f o r m . T h i s approach effectively rejects a subset of legal labellings, avoiding those that require greater time (cf., " u n s o u n d reasoning" [Lev86]). Loosely speaking, speed is gained by increasing the number of " T y p e I " errors, i.e., increasing the number of realizable drawings (i.e., those that correspond * T h e number of processors is actually linear in the number of edges and the number of vertices. But since all vertices in the Hne drawings considered here have at least two and at most three edges, only the number of edges is used here. to a p o l y h e d r a l scene) that are not detected as such. T h e result is a "quick and d i r t y " e s t i m a t e as to w h a t can exist i n the scene. T h e opposite of this is a liberal strategy, which removes constraints u n t i l the r e m a i n d e r can be put into the appropriate f o r m . Here, low complexity becomes possible by increasing the " T y p e IV error rate, i.e., increasing the number of unrealizable drawings deemed to be reahzable. Such a strategy can be used as the basis for a quick "preprocessor" that provides hmits as to what cannot exist i n the scene. In general, elements of b o t h strategies may be used to develop an a p p r o x i m a t i o n , the T y p e I a n d T y p e II error rates being traded off against each other. 3.2 Individual Dimensions G i v e n t h a t hne i n t e r p r e t a t i o n is to be carried out along separate dimensions, w h i c h d i m e n sions should these be? Several sets of considerations must be taken into account. JS the determination of a dimension is to have a low complexity, it must involve as few values as possible; indeed, if the associated problem is to be reduced to 2 - S A T , the variables m u s t have only two possible values. Similarly, i f use of nonlocal i n f o r m a t i o n is to be kept l o w , constraints should be bijective. A n d if interactions between dimensions is to be m i n i m i z e d , each dimension must involve constraints that interact w i t h the others only i n a u n i d i r e c t i o n a l way. It also is assumed that the dimensions involve quantities that are viewer-centered, a condition generaUy assumed for aU of early visual processing [Mar82]. A m o n g other t h i n g s , this ensures that the recovery process obeys the more general viewpoint consistency constraint [Low87], w h i c h assumes that the scene is viewed from a single direction. It also entails t h a t three-dimensional o r i e n t a t i o n must always be defined w i t h respect to the direction of v i e w i n g . E a c h dimension must also obey a second constraint used by v i r t u a l l y aU theories of hne i n t e r p r e t a t i o n : the general viewpoint constraint (section 2.2.1). T h i s requires any i n t e r p r e t a tion to be stable under s m a l l changes i n viewing direction. One of the consequences of this constraint is that no two edges i n the scene can be contained i n a plane at right angles to the image plane. T h i s allows accidental ahgnments to be ruled out â€” arrow- and Y - j u n c t i o n s w i U always correspond to coherent corners i n the scene, a n d since corners are assumed to involve no more t h a n three edges (section 1.1), T-junctions wiU always correspond to occlusions of ///////Ay/// ///////////// ///////////// 1. Contlgultij stream -contiguity relations between line and flanking regions 2. Convexity Stream - convex/nonconvex values to lines 5. Slant Sign Stream - sign of slant values to lines 0. Input Image 4. Slant Magnitude Stream //////////Ã»!y - magnitude of slant values to lines F i g u r e 3.2: Sepaxation into i n d i v i d u a l dimensions. one edge b y a noncontiguous surface. I n general, there are usually only a few viewing directions t h a t give rise t o unstable interpretations, and so only a small penalty i n i n t e r p r e t a t i v e power is given u p i n r e t u r n for a large gain i n performance (see, e.g., [Sug86]). In w h a t follows, a t t e n t i o n is given t o b o t h the quaUtative and the quantitative aspects of " o p t i m a l " Une i n t e r p r e t a t i o n (section 2.2.1). T o further increase t h e amount of concurrent processing (section 3.1.1), each of these is further spUt, yielding four largely independent dimensions: edge contiguity, edge convexity, slant sign, and slant magnitude (figure 3.2). 3.2.1 Contiguity Labelling M u c h of the effectiveness of processing at early levels depends o n k n o w i n g whether neighb o r i n g regions i n the image correspond t o contiguous or noncontiguous surfaces i n the scene [Hor86, p p . 354-355]. Consequently, a reasonable candidate for an independent processing s t r e a m is one concerned w i t h the determination of contiguity. To be as independent as possible, the corresponding dimension must avoid quantities that describe the i n t e r n a l structure of the objects (e.g., convexity a n d concavity). F u r t h e r m o r e , it w o u l d also help reduce complexity i f labels can have only two possible values. Thus, H u f f m a n - C l o w e s ( H C ) labelUng cannot be used. A somewhat different scheme is therefore proposed â€” labels indicate only whether t h e sides flanking a Une correspond t o surfaces that F i g u r e 3.3: C o n t i g u i t y labelling. are contiguous ( C ) or noncontiguous ( N ) w i t h the corresponding edge i n the scene^ (figure 3.3). In contrast w i t h H C labelhng, each hne therefore has two labels, one for each side. A . Constraints on Contiguity Labelling In order to exclude doubly discontiguous edges (i.e., wires), contiguity constraints are r e q u i r e d for the hnes. These are subject to the constraint that b o t h sides cannot be labelled w i t h ' N ' , since the p o l y h e d r a l w o r l d contains no wires; however, aU other combinations of ' C a n d ' N ' are possible (figure 3.4). C o n s t r a i n t s on junctions are taken f r o m the Huffman-Clowes scheme by identifying convex and concave edges w i t h doubly-contiguous hnes, and boundary edges w i t h singly-contiguous hnes (figure 3.4). T h i s coUapses the H C constraints into the set shown i n figure 3.4. It is apparent that any coherent scene wiU give rise to a consistent set of labels, a n d that this can be done by a process similar to that used for H C labelhng. T h e result is a segmentation of the image into sets of regions corresponding to noncontiguous surfaces i n the scene. Because these constraints have been derived from the Huffman-Clowes set, any d r a w i n g w h i c h can be given a consistent H C labeUing can also be given a consistent contiguity l a beUing. T h e converse s i t u a t i o n , however, does not necessarily h o l d : a consistent contiguity labeUing m a y not correspond to a consistent H C labeUing (e.g., the d r a w i n g i n figure 3.5). T h e increased susceptibihty of the contiguity system to false labeUings stems f r o m the loss ^Mackworth [Mac74, M M H 8 5 ] describes a somewhat similar scheme of "connect" and "nonconnect" edges, based on the distinction between interior and boundary edges. However, it differs from the present scheme in using one rather than two labels per hne. < I < -^z c -4 ^ -vN c F i g u r e 3.4: Set of contiguity constraints. / N C C C c C N N F i g u r e 3.5: Inconsistent drawing w i t h consistent contiguity labeUing. of the correlations between contiguity and convexity, which are not taken into account w h e n contiguity alone is considered. T h u s , a consistent H C solution that has been "weakened" by coUapsing the convex a n d concave labels is only one of perhaps several possible solutions t o the contiguity labelUng p r o b l e m . B . Complexity of Contiguity Labelling Since contiguity labelUng involves only two values, its reduction to 2 - S A T depends entirely on the extent to w h i c h it can be described by a set of binary constraints. figure 3.6, almost a l l of these constraints can be converted into binary f o r m . A s shown i n Constraints on Unes are quite simple, since only a p r o h i b i t i o n against double discontiguity is needed. F o r Y - j u n c t i o n s , an a d d i t i o n a l bijective constraint is imposed: the "inside edges" of a region (i.e., Unes sharing a c o m m o n region) must have the same contiguity labeUing. A m o n g other things, this yields the constraint that at most one of the three faces bordering a Y - j u n c t i o n can be noncontiguous. F o r arrow-junctions, a l l Unes except for those on the "outside" of t h e arrowhead must be m a r k e d as contiguous, and the outer sides of these junctions are subject to the bijective constraint that b o t h must have the same value. There are 16 possible combinations of N and C labels on L-junctions, of which 6 are a l lowed. A constraint against doubly-discontiguous Unes leaves 3 x 3 = 9 possibiUties. A constraint against diagonal N labels removes another two. T h i s leaves only one more constraint â€” that against 4-way contiguity (figure 3.6) â€” to be enforced. T h i s constraint, however, cannot be enforced u s i n g binary constraints. L o w complexity can therefore be guaranteed only for approximations i n w h i c h this constraint has somehow been replaced. ~(N,) (N, ^ N3) (N^ N4) (N, ^ N^) ~(C3-N4) F i g u r e 3.6: R e f o r m u l a t i o n of contiguity constraints. Conservative approximation One w a y t o remove the need for an exphcit constraint against 4-way contiguity is to require t h e inside a n d outside edges of an L - j u n c t i o n t o have identical contiguity values; alternatively, one of t h e inside edges can be constrained to be discontiguous. V i a theorem 3.1, this results i n P r o p o s i t i o n 3.1 L-junctions, Liberal When binary contiguity constraints are added that prohibit labelling can be reduced to the 4-way contiguity of 2-SAT. approximation A low-complexity a p p r o x i m a t i o n can also result by o m i t t i n g the need to exclude 4-way contiguity o n L - j u n c t i o n s . T h i s leads t o P r o p o s i t i o n 3.2 contiguity When the constraint labelling can be reduced to against 4-way contiguity on L-junctions is omitted, 2-SAT. N o t e t h a t similar reductions t o C C L are not possible unless extremely severe alterations are made t o t h e constraints. 3.2.2 Convexity Labelling G i v e n that contiguity is concerned w i t h inter-object relations, its n a t u r a l complement is i n t r a object structure, v i z . , edge convexity. A s for the case of contiguity, the standard H C labels are not suitable for present purposes, and must be replaced. A two-valued system is used here, based on t h a t of [Mal87]: ' - h ' for edges of positive convexity (this has the same meaning as i n the H C system), a n d 'o' for aU others. Note that the l a b e l 'o' does not necessary correspond to negative convexity, but r a t h e r , serves as the complement required i n a two-valued s y s t e m . In what follows, the t e r m 'convexity' refers to positive convexity, i n the sense defined here. A . Constraints on Convexity Labelling T h e constraints on convexity labelling can be determined f r o m those of the HuflFman-Clowes set by collapsing the labels i n a manner similar to that done for contiguity. T h e resultant set is shown i n figure 3.7. A n y coherent scene wiU give rise to a consistent set of labels, w h i c h can be found by a " s t a n d a r d " labeUing process (section 2.2.1). Because the convexity constraints are a subset of the H C constraints, any drawing t h a t can be given a consistent H C labelUng can also be given a consistent convexity l a b e l l i n g . A s for the case of contiguity, however, the converse situation does not necessarily h o l d . A n example of this is shown i n figure 3.8, w h i c h can be given a consistent convexity labelUng even t h o u g h a consistent H C labelUng is impossible. T h e results of the contiguity and convexity streams can be combined if the edges m a r k e d as '4-' i n the convexity stream m a t c h a subset of the doubly-contiguous edges i n the contiguity s t r e a m . T h e r e m a i n i n g 'o' edges can then be assigned H C labels on the basis of contiguity alone. It is evident that combining the results i n this way is possible exactly when a solution of the H C constraints can be found. B u t such a co-ordination requires the results i n b o t h streams to be weakened versions of the H C solution a n d , since the streams are separated, this does not generally occur. B . C o m p l e x i t y of C o n v e x i t y L a b e l l i n g Since convexity labelUng involves only two values, its reduction to 2 - S A T depends on the extent to w h i c h the constraints can be put into bijective or binary f o r m . A s shown i n figure + F i g u r e 3.7: Set of convexity constraints. u u u ~(0, - ~(0, ^ O^) +2) -(+2-03) -(+,-03) ~(02^ ~(02-+3) +3) - ( N j ^ O,) ~(+3- - ( 0 3 - N,) ~(03-0,) u u2 +,) (+,) F i g u r e 3.9: R e f o r m u l a t i o n of convexity constraints. 3.9, a l l of these can be put i n t o this f o r m . T h e o r e m 3.1 therefore yields: P r o p o s i t i o n 3.3 Convexity labelling of line drawings can be reduced to 2-SAT. A s is evident f r o m figure 3.9, a l l of these constraints are also bijective, except for t h e p r o h i b i t i o n against the double convexity of L - j u n c t i o n s . T h i s suggests that a p p r o x i m a t i o n s can be derived w i t h o u t a great alteration of the set of constraints. Conservative approximation A low-complexity a p p r o x i m a t i o n t o convexity labelhng can be obtained b y r e q u i r i n g a l l L - j u n c t i o n s t o either have b o t h sides labeUed w i t h ' o ' , or else t o have only one side labeUed w i t h ' o ' . T h e o r e m 3.2 then leads t o : P r o p o s i t i o n 3.4 If all L-junctions are constrained to have both sides labelled'â€¢o\ or to have only one side labelled ' o ' , convexity labelling can be reduced to CCL. Liberal approximation A h b e r a l approach w o u l d be simply t o allow interpretations t o contain doubly-convex L - j u n c t i o n s . T h e o r e m 3.2 then yields: P r o p o s i t i o n 3.5 If both sides of L-junctions can be reduced to CCL. are allowed to be convex, convexity labelling 3.2.3 Slant Sign L a b e l l i n g T h e quantitative aspect of Une interpretation considered here is the three-dimensional o r i e n t a t i o n of the edges of each p o l y h e d r o n . T h i s property has two aspects: tilt, the t w o - d i m e n s i o n a l orientation i n the image plane, a n d slant, the deviation away from this plane. Since tilt is already available i n the image, processing can focus entirely on the recovery of slant. T h e determination of slant can itself be spUt into two components, concerned w i t h its sign a n d m a g n i t u d e respectively. Slant sign (see, e.g., [Kan90]) describes whether the d e p t h of the edge increases or decreases as it travels along some direction. It remains invariant under any positive rescaling of the depth, i.e., it can represent the "z-affine" structure, w h i c h m a y be the most i m p o r t a n t aspect of the recovered scene [TB90]. In this sense it is similar to convexity. B u t slant sign is viewer-centered rather t h a n object-centered, a n d so is more t y p i c a l of the properties thought to be handled by early vision (section 2.1.2). Slant sign is represented here by a double arrow (>- ), the direction of the arrow i n d i - c a t i n g the direction to foUow to increase distance f r o m the viewer (figure 3.10). ^ T h e o n l y consequence of using this representation is that under the general viewpoint constraint (sect i o n 3.2), the slant sign must r e m a i n the same under smaU changes of viewing p o s i t i o n . Zero slant is therefore not aUowed. T h i s can be stated as a constraint that no edges i n the scene can be at right angles to the Une of sight. It is evident that any polyhedral scene o b e y i n g this general constraint w i U give rise to a consistent labelUng of the Une drawing. A . C o n s t r a i n t s on Slant Sign L a b e l l i n g A l t h o u g h m a n y approaches (e.g., [Sug86]) require the quaUtative aspects to be solved before t h e q u a n t i t a t i v e aspects, the demands of r a p i d processing (section 3.1.1) require that t h e t w o types of aspects be determined largely concurrently. B u t if this is to be done, some '^It may be useful to view the arrowheads as parallel lines receding into the distance. *In contrast to the other quantities, slant sign can only be defined with respect to a particular direction of travel. If slant sign is to be treated as a pure scalar, a canonical direction must therefore be defined. A natural choice for a coordinate system is one based on the lines surrounding each vertex, the reference direction being that in which the vertex is approached. Represented in this way, slant sign is subject only to an additional constraint that the labeUing of lines be spMt, with opposite ends of the Mnes having opposite values. A directional component also exists in the labelling of lines by arrows in the H C system, and the splitting required to put it into scalar form has becomes the basis of the contiguity system developed here. B u t because constraints on the slant system are binary and bijective, using a ' " s p l i t " representation will affect neither the power nor the complexity of slant sign labelling. In the interests of clarity, the "directional" form is used. F i g u r e 3.10: Slant sign labelling. a d d i t i o n a l a priori assumptions are needed about the structure of the p o l y h e d r a i n the scene â€” otherwise, any combination of slant signs can be attached to the hnes about a j u n c t i o n . ^ Such s t r u c t u r a l assumptions do indeed seem to be used by the h u m a n visual system to determine three-dimensional structure (section 2.2.2). Convex polyhedra A very general s t r u c t u r a l assumption is that the p o l y h e d r a are convex. T h i s prohibits Y - j u n c t i o n s f r o m h a v i n g a l l hnes slanted towards the viewer,-^^ since this w o u l d correspond to a dent i n the surface. Similarly, an arrow-junction could not have its stem slanted away f r o m the viewer while its two outer edges h a d to opposite slant. C o n s t r a i n t s also come into p l a y v i a the p l a n a r i t y of the faces: If the face is convex, two " c h a i n s " of arrow labels exist, w h i c h diverge from the j u n c t i o n at greatest distance and converge on the j u n c t i o n nearest the viewer. Directangular corners A more specific assumption is that p o l y h e d r a have directangular corners, i.e., corners for w h i c h two edges are at right angles to a t h i r d about which they can " s w i v e l " . C o n s t r a i n t s can be based on the observation that two edges meeting at a right angle i n the scene w i h Â®For example, a junction can always be interpreted as a very shallow corner, and this can be tilted or flipped to achieve any combination of signs. ' Â° M o r e precisely, the distance to the viewer cannot be decreased as the distance from the vertex is increased. " F o r example, a book partway open has directangular angles at points where the spine meets the top and bottom of the covers. One possibility if y = 90Â° Intersection at 90" to line in image = zero slant Two possibilities U y ^ 90Â° 161 < 90Â°: Slant signs A and B have opposite values 151 = 90Â°: Slant sign B has zero value 151 > 90Â°: Slant signs A and B have same val ues F i g u r e 3.11: C o n s t r a i n t s o n isolated L-junctions. always give rise to lines of opposite sign when the angle i n the image plane is less t h a n 90Â°, and to hnes of the same sign when the image angle is greater t h a n 90Â°(figure 3.11). G i v e n the hne corresponchng to the swivel edge, then, the slant signs of the other two hnes can be immediately determined (figure 3.12). It foHows that the constraints on the slant signs of arrow- and Y - j u n c t i o n s are bijective (section 3.1.2), the exact constraints depending o n whether the angles between the hne pairs are greater or less t h a n 90Â°. However, constraints o n L-junctions cannot be determined unless the orientation of the swivel axis i n the image can be identified, since otherwise the angle i n the image m a y not correspond to a 90Â°angle i n the scene. Note that although the orientation must be given, it does not matter on w h i c h side of the j u n c t i o n the hidden swivel hes â€” a change of 180Â°will result i n the same set of bijective constraints. Rectangular corners A powerful constraint apparently used by the h u m a n visual system is that of rectangul a r i t y , the assumption t h a t ah edges i n each corner are at right angles to each other (section 2.2.2). A s i n the more general case of directangular corners, k n o w i n g the label attached to one of the hnes on an arrow- or Y - j u n c t i o n immediately determines those of the others. B u t now it is not necessary to know i n advance which of the hnes corresponds to the swivel edge, since a l l edges are equivalent. T h e constraints themselves take on a simple form â€” for arrow-junctions, the slant signs of the wings must be opposite that of the stem, whereas a l l Opposite sign as (a) Stem is swivel axis (b) Stem is not sv/ivel axis F i g u r e 3.12: Slant sign constraints for arrow- and Y - j u n c t i o n s . lines i n Y - j u n c t i o n s must be given the same slant signs [Kan90]. Requiring a consistent set of slant signs for these junctions leads to P e r k i n s ' laws (section 2.2.1): for Y - j u n c t i o n s , a l l angles must be greater t h a n 90Â°, while for arrow-junctions, the largest angle must be less t h a n 270Â°, a n d the second-greatest less t h a n 90Â°. T h e set of constraints on slant sign labels for rectangular corners are shown i n figure 3.13. N o t e that the slant sign labels on arrow- a n d Y - j u n c t i o n s become closely matched t o t h e convexity labellings: for Y - j u n c t i o n s , edges are convex exactly when they are slanted away f r o m the viewer, a n d are nonconvex when they are slanted towards the viewer. Similarly, a n a r r o w - j u n c t i o n w i l l have its stem slanting towards the viewer when it is nonconvex, a n d away when convex, the other lines t a k i n g on complementary values. T h e homogeneity of angles also means that there is no ambiguity about the angle between the edges of the L-junctions. A n d since this angle is 90Â°, the slant sign of one Une automaticaUy determines that of the other (figure 3.11). T h u s , L-junctions can be described entirely i n terms of bijective constraints, w i t h o u t any need for a priori knowledge about the direction of the hidden swivel axis. 1 1 F i g u r e 3.13: Slant sign labellings for rectangular corners. B , C o m p l e x i t y of Slant Sign L a b e l l i n g Directangular corners W h e n corners are directangular, they give rise to bijective constraints on arrow- a n d Y j u n c t i o n s . A n d when the directions of the hidden swivel axes are k n o w n , L-junctions have a similar set of constraints. T h u s , f r o m theorem 3.2, P r o p o s i t i o n 3.6 all junctions Rectangular When all comers are directangular and the directions are known, slant sign labelling can be reduced to of the swivel axis at CCL. Corners W h e n corners are rectangular, a special swivel axis need not be singled out. A n d since L - j u n c t i o n s always have bijective constraints under this c o n d i t i o n , this yields P r o p o s i t i o n 3.7 When all corners are rectangular, slant sign labelling can be reduced to CCL. N o t e t h a t the differences between directangular and rectangular corners do not lead t o a significant difference i n the complexity of slant sign labelling. R a t h e r , the m a i n difi"erences are i n the amount of a priori i n f o r m a t i o n needed from nonlocal sources. 3.2,4 Slant M a g n i t u d e L a b e l l i n g Slant m a g n i t u d e is an absolute value which represents the "steepness" of an edge w i t h respect to the image plane. T h i s quantity is completely independent of slant sign, being i n variant under inversion about the image plane, but sensitive to the rescahng of depth. It is also a q u a n t i t y t h a t takes on a continuous value. A m o n g other things, this latter p r o p e r t y means t h a t the p a r t i c u l a r representation used (e.g., angle, gradient) is not i m p o r t a n t f r o m a c o m p u t a t i o n a l point of view, since these quantities can be transformed into each other v i a information-preserving operations. A . C o n s t r a i n t s o n Slant M a g n i t u d e L a b e l l i n g A s for slant sign, assumptions must be made about the structure of the p o l y h e d r a i n t h e scene if this dimension is to be determined independently of the others. Known corners If the three-dimensional structure of a corner is k n o w n and its edges have been identified w i t h the corresponding hnes i n the image, a system of equations can be set up between t h e slants of these edges a n d the angles between the Unes of the j u n c t i o n [ K a n 9 0 , p.288] sin sin ^ 2 cos(^i â€” ^ 2 ) + cos sin <jf>2 sin (^3 cos(^2 - sin <?i>3 sin 1^1 cos(^3 - cos ^ 2 = ^3) + cos 4>2 cos <?!>3 6 * 1 ) - f cos <;Ã¨3 cos ij!>i = cos712, â€” cos 7 2 3 , COS731, (3.1) where (pi is the slant of edge i ( w i t h zero being along the hne of sight towards the viewer), 9i the angle of edge i i n the plane, a n d 7 i j the angle between edge i a n d j. A solution can be f o u n d for any value of angles chosen. However, this solution requires an iterative scheme (e.g., N e w t o n - R a p h s o n ) unless a d d i t i o n a l constraints are i n t r o d u c e d [KanQO]. In order t o keep the measure of slant s y m m e t r i c a l about the image plane, the angle a = 7 r / 2 - < ^ i s used for the slant m a g n i t u d e ItseH, w i t h a always i n the interval (â€”7r/2,7r/2). Slant magnitudes cannot be determined for L-junctions i n i s o l a t i o n , even when the angle between their edges is k n o w n . B u t equation 3.1 shows t h a t if the slant magnitude of one of the edges is k n o w n , t h a t of the other can be determined. A n d because this is an equation Unear i n sin<p a n d c o s ^ , <> / (and therefore a) can be solved for analytically. If 7 is not 90Â°, two values are possible, corresponding to edges of greater or lesser slant (figure 3.11). These can take on different slant signs, depending on the particular value of 7; if this occurs, slant magnitudes must become signed i n order to m a i n t a i n the correct b i n d i n g betvi^een the signs and the magnitudes assigned to the edge. Otherwise, the slant sign of the k n o w n edge need not be given, since the solutions are s y m m e t r i c a l about the image plane. Directangular corners If the corner is k n o w n to be directangular and if the Une corresponding to the swivel edge can be identified i n the image, the set of equations 3.1 takes on the f o r m [ K a n 9 0 , p.289]: B = 2 C0S(^1 - ^ 2 ) C 0 S ( ^ 1 - ^ 3 ) V C05(^2-^3) C = COs2(^i-e3) ; COs2(^2-^3), COs'^/3 sin^/3 -B X , 1+ + VB^ - 4AC 2A 7r/2-tan-\/X Â«2 = 7r/2-tan-('Â°'[^^~^\^otai), COS(e/3 - Â«3 = (3.2) 7r/2-tancos(^3 - (3.3) (f2) ~J ^ , , ^i)cotai (3-4) where /? denotes the angle about the swivel axis, taken here to be edge 3. These values are c o o r d i n a t e d sets, a n d so allow magnitude and sign to be completely separated. Since the t w o solutions of these equations are reflections of each other about the image plane [Kan90], arrow- a n d Y - j u n c t i o n s have unique magnitude estimates for each edge. A l t h o u g h slant magnitudes cannot be determined for L - j u n c t i o n s , one constraint still appUes â€” i f the angle between corresponding edges is 90Â°, the m a g n i t u d e of one edge uniquely determines t h a t of the other. If the angle is not 90Â°, two values are possible (figure 3.11). T h u s , if the swivel angle cannot be identified, three values are possible for the slant of the second edge. Rectangular corners For a rectangular corner, aU. edges are orthogonal to each other, a n d the relation b e t w e e n slant and j u n c t i o n angle can be expressed i n the much simpler form [Per68, Kan90] N o t e that the equahty of a l l angles between edges also ehminates the need to know w h i c h hne is the p r o j e c t i o n of the swivel axis. T h e rectangularity of the corner also means that there is no ambiguity i n i d e n t i f y i n g the angle 7 between the edges of any corner corresponding t o an L - j u n c t i o n . A n d since 7 is 90Â°, the slant m a g n i t u d e of one edge is uniquely determined by the magnitude of the other. F o r rectangular corners, therefore, L-junctions are completely described by bijective constraints. B . C o m p l e x i t y o f Slant M a g n i t u d e L a b e l l i n g Directangular Corners W h e n corners are directangular, there are three possible sets of magnitudes for a j u n c t i o n , corresponding to the three possible choices of swivel axis. If the direction of the swivel axis a n d the swivel angle (3 are k n o w n , unique magnitude estimates can be assigned to edges contacting arrow- a n d Y - j u n c t i o n s (equations 3.3 - 3.4). F u r t h e r m o r e , this condition also leads t o b i n a r y constraints on the magnitudes possible for L - j u n c t i o n s . B u t a chain of such L - j u n c t i o n s could cause the number of possible values to increase exponentially w i t h its l e n g t h , these values being impossible to resolve except by sequentially proceeding along the chain. In the worst case, therefore, the determination of slant magnitude for directangular corners could require at least hnear t i m e , even on a parallel architecture. Rectangular Corners JS aU corners are rectangular, the magnitudes for the edges of arrow- and Y - j u n c t i o n s can be o b t a i n e d directly f r o m equation 3.5. Values for L-junctions can be determined f r o m the fact that slant m a g n i t u d e remains invariant under a reflection of one edge by 180Â°; consequently, only the angle of the hidden edge is needed, and not its direction i n respect t o the vertex. B y determining a n d rebroadcasting the values of a l l orientations i n the p a r t i t i o n to a l l j u n c t i o n s , the direction of the hidden edge can be made available to aU L-junctions.^'^ Once the l o c a l estimates of slant magnitude have been obtained, it only remains to check their consistency. A s discussed i n section 3.1.2, such a consistency check can be carried out v i a C C L . Since aU other operations can be done i n constant t i m e , this yields When all corners P r o p o s i t i o n 3.8 has a complexity are rectangular, no greater than that of the determination of slant magnitude CCL. N o t e t h a t although the c o m p u t a t i o n of the magnitude as given by equation 3.5 can be done i n constant t i m e , it does involve several trigonometric functions. B u t this calculation can be done quite simply if the slope of the slant rather t h a n its angle is the relevant quantity. In p a r t i c u l a r if the square of the slope (essentially, a "slope energy") is used, this removes the need for b o t h an inverse tangent a n d a square root function. T h e only remaining quantities then become cosine functions of the angles between j u n c t i o n lines, which can be determined quite simply v i a the dot product (cf. section 5.2.2). Since slope energy and slope angle are related by a monotonie f u n c t i o n , the p a r t i c u l a r quantity chosen is of no great i m p o r t a n c e for most purposes. In the interests of m a i n t a i n i n g a parallel between two-dimensional a n d three-dimensional orientations, slope is represented here by its angle. 3.3 Integration of Dimensions A s shown i n section 3.2.2, completely separated dimensions are often unable to capture large parts of the m a p p i n g structure contained i n the original set of constraints. F o r example, a d r a w i n g m a y have several different contiguity and convexity labeUings, a n d if these are chosen such that the edges w i t h positive convexity correspond to Unes that are doubly contiguous, the two can be combined into a complete H C labelling. T h e separation of streams, however, means that it wiU generally be impossible to pick out the appropriate contiguity a n d convexity interpretations from among the alternatives. Instead of yielding a completely coherent i n t e r p r e t a t i o n , the process w i U be more Ukely to y i e l d two p a r t i a l interpretations t h a t are i n c o m p a t i b l e w i t h each other. ^^If only two directions exist in the drawing, any magnitudes compatible with equation 3.5 can be assigned to the edges. T h i s loss of interpretative power, however, can be lessened by a controUed amount of interaction between streams. A s discussed i n section 3.1.1, this can be done without r a i s i n g the complexity of the process provided that it is based on the unidirectional t r a n s m i s s i o n of unambiguous results. In order to quickly achieve unambiguous results, a conservative strategy must be employed, based on a d d i t i o n a l s t r u c t u r a l constraints which rule out m a n y legal interpretations (section 3.1.3). It is shown here that such a strategy can succeed for several sub domains of p o l y h e d r a l objects. 3.3.1 Convex Objects A p a r t i c u l a r l y simple d o m a i n i n which to begin is that of convex objects. These are p o l y h e d r a l objects i n which aU edges of the object are convex; consequently, " m a t e r i a l " always exists along the shortest p a t h connecting any two points on two contacting edges (i.e., two edges t h a t meet at a corner). A . C o n s t r a i n t s o n L a b e l l i n g of C o n v e x O b j e c t s B y definition, the interior edges of convex objects are convex. A s such, a l l arrow- a n d Y j u n c t i o n s have a unique interpretation i n b o t h the contiguity and the convexity streams. C o n v e x i t y also forces the inner edges of any L - j u n c t i o n to be contiguous; this i n t u r n forces a unique convexity labehing of aU. L - j u n c t i o n s , i.e., aU edges nonconvex. T h e resulting set of constraints, shown i n figure 3.14, leads to a unique set of convexity labels. The only indeterminate quantities are the contiguity labels on the outer edges of arrow-junctions a n d L-junctions. C o n s t r a i n t s on the outer edges of arrow-junctions are binary and bijective, r e q u i r i n g b o t h edges t o have the same value, whereas those on L-junctions are simple b i n a r y constraints that p r o h i b i t more t h a n one side f r o m being contiguous. B . C o m p l e x i t y of L a b e l l i n g C o n v e x O b j e c t s A U convexity labeUings are unique, and so the contiguity labels i n any consistent interpretat i o n must necessarily be compatible w i t h the convexity labels. Consequently, the determination of a complete quahtatlve interpretation reduces to the determination of a consistent set of contiguity labels. F i g u r e 3.14: HufFman-Clowes labellings for convex objects. i) Reduction to 2-SAT Since a l l relevant constraints are binary, and since only two labels apply, p r o p o s i t i o n 3.1 leads directly to The line labelling T h e o r e m 3.3 of convex objects has a complexity no greater than that of 2-SAT. N o t e that if the H C labels are required, they can be recovered s i m p l y by assigning ' - ' t o any doubly-contiguous non-convex hnes, and assigning b o u n d a r y Unes to singly-contiguous lines. S i m i l a r l y , any consistent i n t e r p r e t a t i o n based on separate convexity a n d contiguity labels can be p u t i n t o H C f o r m . ii) Reduction to CCL For convex objects, the contiguity of the outer edges of the L-junctions results o n l y f r o m the contact of adjacent blocks, and not on any intrinsic s t r u c t u r a l property. It is therefore evident that a legal labelling for a drawing exists if a n d only if it can be assigned an i n t e r p r e t a t i o n i n w h i c h a l l blocks have been moved to a position i n which they are "free f l o a t i n g " . T h i s latter condition can obtained by setting a l l outer edges of L-junctions to be discontiguous; the result is a set of unique contiguity labels on aU outer edges. Since a l l constraints are unique, no additional work is required to coordinate the results i n the contiguity a n d the convexity streams. T h e complexity of the interpretation process is therefore exactly that needed to check for the presence of inconsistencies i n each (section 3.1.2). T h i s depends u p o n the conditions assumed for the image and scene domains. If only one set of connected hnes exists i n the image, the preprocessing to remove T-junctions can be o m i t t e d , a n d the need to m a i n t a i n p a r t i t i o n s ehminated. Under these conditions, t h e complexity of the i n t e r p r e t a t i o n process is exactly that of detection. B u t the blocks w o r l d generaUy aUows several blocks to exist simultaneously i n the scene, m a k i n g it necessary t o distinguish between various groups of hnes i n the image. Since the basic H C labeUings are unique, so are those of the contiguity a n d convexity streams. A n d since b o t h these streams involve the same p a r t i t i o n s , the integration of values is s t r a i g h t f o r w a r d , leading t o T h e o r e m 3.4 If objects are assumed to not contact each other, the line labelling of convex objects has a complexity no greater than that of CCL. fn essence, t h e n , a principle of " m i n i m a l exterior contiguity" has been invoked to o b t a i n a p r o b l e m of lower complexity by reducing the set of preferred solutions. A s opposed t o a purely conservative strategy (section 3.1.3), however, this strategy does not affect the labeUing p r o b l e m i n its narrowest sense, v i z . , a determination if at least one solution exists. N o t e also that i f an i n t e r p r e t a t i o n of a p a r t i t i o n exists, it is necessarily the only f l o a t i n g " i n t e r p r e t a t i o n possible. "free T h u s , a complete determination of scene structure (i.e., solving the reaUzabiUty problem) can be reduced t o finding a solution t o a system of Unear equations a n d inequaUties [Sug86]. T h i s can be solved v i a hnear p r o g r a m m i n g , w h i c h can be carried out i n p o l y n o m i a l time [Kha79]. Linear p r o g r a m m i n g , however, is a P-complete p r o b l e m [Joh90, p.80], a n d as such is unhkely to be solvable by a sub-hnear a l g o r i t h m even when p a r a l l e l processing is available (section 2.1.1). 3.3.2 Compound Convex Objects Consider now a sUghtly less restricted domain i n which it is stiU assumed that m a t e r i a l always exists along the shortest p a t h connecting any points along two contacting edges (as for convex objects), but for which the edges themselves are no longer required to be convex. These objects are referred to here as compound convex objects, since they can be readily reahzed by the attachment of convex objects to each other, this attachment being subject t o the general constraint that only three edges can make contact at any vertex. E x a m p l e s of such objects are shown i n figure 3.15. < I < t F i g u r e 3.16: Huffman-Clowes labellings for c o m p o u n d convex objects. A . C o n s t r a i n t s o n L a b e l l i n g of C o m p o u n d C o n v e x O b j e c t s C o m p o u n d convex objects give rise to almost the same set of arrow-, Y - , a n d T - j u n c t i o n labellings as f o u n d i n the " s t a n d a r d " H C set. B u t because of the shortest-path requirement, there must always be a c o m m o n surface on the side formed by the interior angle of any L j u n c t i o n , a n d on the surfaces between edges of an arrow-junction. T h e interpretation process can therefore be based on the set of j u n c t i o n labeUings shown i n figure 3.16. ( T h e conversion into separate contiguity a n d convexity constraints is straightforward.) Note that only four constraints have been removed f r o m the original Huffman-Clowes set.^'^ ' ^ T h e interpretation of an arrow-junction with a concave stem should not be allowed if consideration is focused on compound convex objects per se, drawings not obeying the constraints assumed. since it can reduce the ability of the system to detect line However, this interpretation can easily be removed, with all arguments going through unaffected. It is left in to show that the set of constraints in figure 3.16 potentially applies to a slightly larger domain of polyhedral objects. B . C o m p l e x i t y of L a b e l l i n g C o m p o u n d C o n v e x O b j e c t s To establish bounds on the complexity of hne labelhng, consider first the convexity s y s t e m . F r o m figure 3.16, it is seen that no L-junctions can have a convex edge; consequently, a l l must have a 'o' l a b e l attached to b o t h edges. V i a proposition 3.4, it follows that convexity labelhng for this d o m a i n can be reduced to C C L , the computation proceeding independently for each p a r t i t i o n . C o m p a t i b i h t y between the convexity and contiguity streams can be guaranteed by t r a n s m i t t i n g the identities of any edge m a r k e d as '-f ' and constraining the relevant edges t o be doubly contiguous. U n i q u e contiguity values can also be assigned to the inside edges of Y j u n c t i o n s , a n d to the crossbars of T - j u n c t i o n s . Since the partitions are the same for b o t h streams, contiguity labeUing needs to be done at most two times for each p a r t i t i o n â€” once for each of the two possible convexity interpretations. i) Reduction to 2-SAT A U contiguity constraints on Unes a n d L-junctions i n figure 3.16 can be put into the b i n a r y f o r m described i n section 3.2.1. A p p h c a t i o n of proposition 3.1 then yields For compound convex objects, line labelling has a complexity T h e o r e m 3.5 the maximum ii) Reduction of that of 2-SAT to and no greater than CCL. CCL Since the labelhng of arrow-junctions, Y - j u n c t i o n s , a n d T-junctions can a l l be based o n bijective constraints, the possibihty is raised that the hne labeUing of compound convex objects can be reduced to C C L . T h i s can be done by showing that the bijective constraints o n L - j u n c t i o n s a n d hnes are unnecessary. N o t i c e t h a t each complex of bijective constraints beginning on the outside of an L - j u n c t i o n can be considered a " c h a i n " that travels along the sides of arrow-junctions, t e r m i n a t i n g either when it contacts an edge w i t h a unique value (e.g., the stem of an a r r o w - j u n c t i o n ) , another outer L - j u n c t i o n edge, or a danghng edge (figure 3.17). These chains can be readily d e t e r m i n e d v i a C C L . Because the j u n c t i o n constraints preserve contiguity, a l l values along a chain must have the same value. T h u s , if a chain terminates at a j u n c t i o n which forces it t o have a unique value, or contains an edge which is simUarly constrained, aU of its elements chain A chain B Figure 3.17: Free chain complexes. must be set t o that value, an operation which can be carried out by C C L . Otherwise, the chain is free to take on either contiguity value. A s long as it ensures that the basic contiguity constraints at its ends are obeyed, the chain can be considered t o be essentially decoupled f r o m the rest of the i n t e r p r e t a t i o n , its values then unaffected by subsequent assignments i n the rest of the d r a w i n g . If a free chain has at least one end i n contact w i t h an L - j u n c t i o n , interpret its constituent variables as discontiguous. figure 3.16. T h i s assignment is always compatible w i t h the constraints of A n interpretation constrained i n these ways is therefore possible if and o n l y if it is possible to interpret the d r a w i n g as a set of compound convex objects. T h e use of this r e s t r i c t i o n is essentially a generalized application of the principle of " m i n i m a l exterior c o n t i g u i t y " used i n the analysis of convex objects. Invoking this principle essentially causes these objects t o be dismantled into separate convex components whenever possible. Somewhat similar considerations apply to a chain that has dangUng edges at b o t h its ends, except that here the chain w i l l be interpreted as contiguous. In a direct paraUel w i t h the previous principle, this can be seen as a principle of " m a x i m u m interior c o n t i g u i t y " . N o t e that the t w o contiguity principles have been invoked to obtain a p r o b l e m of lower complexity by reducing the set of preferred solutions. A s opposed to a purely conservative strategy (section 3.1.3), however, this strategy does not affect the labelling problem i n its narrowest sense, since a restricted solution wiU be found if at least one more "general" solution exists. H a v i n g dealt w i t h L - j u n c t i o n s , it must now be shown that an explicit binary constraint is not needed against doubly-discontiguous Unes. If no junctions are present, a line can i m m e d i a t e l y be given any legal labeUing. Otherwise, as figure 3.16 shows, the p r o h i b i t i o n against double discontiguity is a u t o m a t i c a l l y imposed for aU junctions. T h i s proves T h e o r e m 3.6 labelling For compound convex objects that are assumed to not contact each other, line has a complexity no greater than that of CCL. F i g u r e 3.18: Examples of rectangular objects. 3.3.3 Rectangular Objects L o w - c o m p l e x i t y i n t e r p r e t a t i o n is also possible for rectangular objects, i.e., p o l y h e d r a l objects for which a l l corners have edges that meet at right angles. These constitute a large d o m a i n of objects, examples of which are shown i n figure 3.18. A . C o n s t r a i n t s o n L a b e l l i n g of R e c t a n g u l a r Objects R e c t a n g u l a r objects impose no a d d i t i o n a l exphcit constraints on the H C labeUings of a r r o w - , Y - , and T-junctions. C o n s t r a i n t s only apply to L - j u n c t i o n s , the particular choice of con- straints depending on the angle i n the image.^'* There are two cases to consider here. T h e first is w h e n this angle is acute (i.e., less t h a n 90Â°). Because the angles between the corresponding edges i n the scene are 90Â°, the hnes of an acute L - j u n c t i o n must have opposite slant signs (section 3.2.3). A n d since the hidden edge of a rectangular corner is always slanted away f r o m the viewer [Kan90], the consistency of the slant signs leads to a bijective constraint o n the convexity labeUing (figure 3.19(a)). A p a r a l l e l s i t u a t i o n exists for obtuse L-junctions (i.e., those for w h i c h the angle is greater t h a n 90Â°). R e c t a n g u l a r i t y now forces b o t h sides to take on the same slant signs. When b o t h edges are slanted away f r o m the viewer, they must be interpreted as a pair of singlycontiguous hnes; if they are slanted towards the viewer, three interpretations are possible (figure 3.19(b)). T h e resultant set of junctions labeUings, shown i n figure 3.20, is much the same as t h a t of H u f f m a n - C l o w e s , except that the constraints shown i n figure 3.19 have been added. A more q u a n t i t a t i v e constraint that can also be used is that of planarity: if the p l a n a r i t y of the faces is to be m a i n t a i n e d , any chain of three connected hnes h a v i n g three different " T o avoid possible confusion, this angle is taken to be the smaller of the two possibilities. Hidden edge must be in this zone Hidden edge must be in this zone Hidden edge must be in this zone K Hidden edge must be in this zone F i g u r e 3.19: Constraints on L-junctions. 1 F i g u r e 3.20: Huffman-Clowes labellings for rectangular objects Figure 3.21: P l a n a r i t y constraint. Surface normals for a common surface must be the same. ~ ( C 3 - N4) F i g u r e 3.22: Interior angle constraint. directions i n the image cannot aU be labeUed as contiguous (figure 3.21). T h i s constraint apphes to b o t h sides of the chain. A s figure 3.21 shows, this constraint stems from a p r o h i b i t i o n against h a v i n g different surface normals defined from each of the hne pairs. A n o t h e r q u a n t i t a t i v e constraint is the interior angle constraint: if a slant sign is t o have l o c a l consistency, any chain of two connected obtuse L-junctions must alternate i n the consistency labels attached to the edges on their interior angles (figure 3.22). T h i s arises f r o m the close connection between slant sign a n d contiguity for these L-junctions (figure 3.19), together w i t h the requirement that slant signs on these junctions must alternate. B . C o m p l e x i t y of L a b e U i n g R e c t a n g u l a r Objects Since the convexity l a b e l l i n g of edges involves only b i n a r y bijective constraints, p r o p o s i t i o n 3.4 ensures that this can be reduced to C C L . A s for the other domains discussed here, o n l y t w o convexity interpretations are possible for each p a r t i t i o n . A n d as before, c o m p a t i b i l i t y can be ensured by first solving for convexity and then requiring hnes labelled as ' + ' t o be doubly contiguous. In order to reduce contiguity interpretation to a low-complexity problem, constraints must be put into an appropriate form. T h e constraints on arrow- and T-junctions are a l ready b i n a r y a n d bijective, as are the constraints on acute L-junctions. Since the c o n t i g u i t y constraints on Y - j u n c t i o n s can also be described by a set of b i n a r y bijective constraints (sect i o n 3.2.1), the reduction of the contiguity labelling problem centers on the constraints for obtuse L-junctions a n d Unes, i) Reduction to L e m m a 3.1 constraint 2-SAT For rectangular objects, the planarity against the 4-way contiguity reformulation having complexity and interior of obtuse L-junctions no greater than that of angle constraints allow the to take on binary form, this CCL. P r o o f : If an obtuse L - j u n c t i o n is isolated, simply assign it one of the legal labeUings of figure 3.19. V i a an exhaustive enumeration of aU possibiUties, it can be seen that the p l a n a r i t y constraint rules out a j o i n i n g of acute and obtuse junctions. U n i q u e values can also be assigned when the shared edge is an "outer" edge of an a r r o w - j u n c t i o n , w i t h the stem p o i n t i n g away f r o m the interior angle (figure 3.23). W h e n the stem points towards the interior angle, a b i n a r y (although not bijective) constraint can be i m p o s e d on the possible values. A unique set of contiguity labels can also be made possible for Y - j u n c t i o n s by i n v o k i n g the p l a n a r i t y constraint (figure 3.23). O t h e r w i s e , a l l shared edges are w i t h others of the same type, a n d so the j u n c t i o n is part of a chain of obtuse L - j u n c t i o n s . These chains can be detected i n a preprocessing step based on C C L . In such a chain, each interior side of a j u n c t i o n is an exterior side of its neighbor. A n d since the interior angle constraint forces the interior labeUings of neighbors to be different, it is impossible that such a j u n c t i o n can have b o t h of its interior a n d exterior sides labeUed as contiguous. Since only a direct assignment of values a n d constraints to local configurations are involved, the complexity of these c N H N Required by unary, binary constraints Impossible due to planarity constraint Required by b i j e c t i v e , planarity c o n s t r a i n t s Impossible due to i n t e r i o r angle constraint F i g u r e 3.23: C o n t i g u i t y constraints on obtuse L - j u n c t i o n combinations. operations are no greater t h a n that of C C L . T h e proof then fohows from the observation that the interior angle constraint is b o t h b i n a r y a n d bijective. T h e o r e m 3.7 labelling For rectangular to have a complexity objects, the planarity and interior no greater than the maximum â€¢ angle constraints of that of 2-SAT and allow line CCL. P r o o f : Since hnes can be handled by a b i n a r y constraint (section 3.2.1), it is only necessary to express the constraint against the 4-way contiguity of obtuse L-junctions i n b i n a r y f o r m . F r o m l e m m a 3.1, it foUows that these can be cast into the appropriate f o r m v i a a preprocessing step that assigns unique values to m a n y of the obtuse L - j u n c t i o n labels, a n d b i n a r y constraints to the rest. A n d it foUows from the l e m m a that this step has a complexity no greater t h a n C C L . â€¢ These are the same complexity bounds found by K i r o u s i s and P a p a d i m i t r i o u [ K P 8 8 ] for the somewhat more restricted case of the orthohedral world, i n which aU edges are required to be paraUel to one of the three m a i n axes i n the scene. T h e addition of exphcit p l a n a r i t y a n d interior angle constraints, therefore, makes similar low-complexity recovery possible for the more general d o m a i n of rectangular objects. (a) L-termination (A) (b) L-termination (B) (c) Y-termination F i g u r e 3.24: T e r m i n a t i o n configurations. ii) Reduction to CCL A s for the case of convex and connected convex objects, it is also possible to show t h a t Une labelUng for rectangular objects is of complexity no greater t h a n that of C C L . T h i s requires a careful isolation of the r e m a i n i n g binary constraints on obtuse L-junctions and on Unes. L e m m a 3.2 Bijective L-junctions, except for arrow-junction constraints can determine the case of the correct contiguity "L-terminations', with its stem oriented toward the interior in which labelling an L-junction of obtuse contacts an angle (figure 3.24). P r o o f : A s l e m m a 3.1 shows, most values on obtuse L-junctions can either be uniquely assigned or given bijective constraints, except for the case where it contacts an arrowj u n c t i o n w i t h its stem p o i n t i n g t o w a r d from the interior angle. If the L - t e r m i n a t i o n is of type A (i.e., free variables for b o t h exterior edges), the exterior edges require some a d d i t i o n a l constraint to ensure that they cannot b o t h be contiguous. If the L - t e r m i n a t i o n is of type B (i.e., free variables on only one of the exterior edges), the side contacting the L - j u n c t i o n is uniquely determined, and so no constraints are needed for the other side. â€¢ It must now be shown that there is no need for an expUcit binary constraint against doubly-discontiguous Unes. L e m m a 3.3 Bijective constraints alone can enforce the prohibition guity, except for the case of a "Y-termination", contacts two arrow-junctions i.e., a configuration against double disconti- in which a dangling edge with stems oriented away from that edge (figure 3.24(c)), P r o o f : If no junctions axe present, a line can immediately be given any legal l a b e l l i n g . Otherwise, as figures 3.19 and 3.20 show, the prohibition against double discontiguity is a u t o m a t i c a l l y imposed for a l l junctions except obtuse L-junctions and Y - j u n c t i o n s . A s shown i n figure 3.23, the constraints on obtuse L-junctions ensure that each hne has at least one contiguous side. T h i s leaves only Y - j u n c t i o n s to be considered. T h e only constraints on Y - j u n c t i o n s are the set of bijective constraints shown i n figure 3.6, w h i c h require that the same value be assigned to hnes contacting a common region. If a d r a w i n g does correspond to a scene containing rectangular objects, however, two Y - j u n c t i o n s wiU never contact each other, for to do so would immediately violate the p l a n a r i t y constraint. T h e constraint against double contiguity can therefore be i n h e r i t e d directly f r o m the junctions that the Y - j u n c t i o n contacts. C o m p h c a t i o n s arise f r o m the danghng edges arising from occlusion (i.e., the stems of T - j u n c t i o n s ) , but these can be handled by a preprocessing stage: If all three edges are dangling: the j u n c t i o n can simply take on any legal interpretation. If two edges are dangling: the r e m a i n i n g edge necessarily contacts another j u n c t i o n , and so obeys the constraint; the inner edges of the danghng hnes are undeterm i n e d , but w i t h o u t loss of generality the appropriate constraints can be enforced by requiring these to be contiguous. If o n l y one edge is dangling: A n exhaustive e x a m i n a t i o n of a l l cases (figure 3.25) shows that the p l a n a r i t y constraint ensures the appropriate contiguity condition for aU configurations except the Y - t e r m i n a t i o n . â€¢ G i v e n lemmas 3.2 and 3.3, it must now be shown that the remaining contiguity constraints on L - and Y - t e r m i n a t i o n s can be handled appropriately. A n y complex beginning at an L or Y - t e r m i n a t i o n can be seen as a " c h a i n " t h a t travels along the sides of Y - j u n c t i o n s a n d the outer sides of arrow-junctions, and terminates at another L - or Y - t e r m i n a t i o n . These chains are similar to those used to analyze the interpretation of c o m p o u n d convex objects (sec.tion 3.3.2), and are handled in much the same way. Since aU other aspects of the hne i n t e r p r e t a t i o n can be handled by bijective constraints, it is only necessarily to show that the chains can also be labeUed i n a consistent way v i a a process of complexity no greater t h a n that of C C L . c c c Required by unary, binary c o n s t r a i n t s Impossible due to p l a n a r i t y c o n s t r a i n t Required by unary, binary c o n s t r a i n t s Impossible due to p l a n a r i t y c o n s t r a i n t Cannot e x i s t f o r a r e c t a n g u l a r object Impossible due to p l a n a r i t y c o n s t r a i n t F i g u r e 3.25: C o n t i g u i t y constraints on Y - j u n c t i o n combinations. C o n t i g u i t y labelling can be carried out by first assigning labels to variables c o n s t r a i n e d to take on unique values, and to those chains that contain such a variable. N e x t , any c h a i n w i t h more t h a n two orientations to its set of edges must necessarily be discontiguous i f it is to obey the p l a n a r i t y constraint; consequently, each free chain that remains cannot b e n d , but must t r a v e l i n one general direction only. T h e r e m a i n i n g free chains can now be labelled. Note that since these chains are decoupled f r o m the rest of the interpretation (section 3.3.2) it does not m a t t e r whether this is done before or after determining labels for the rest of the drawing. Indeed, it follows from l e m m a 3.2 a n d the bijective nature of the constraints that a l l variables outside the chains w i l l have o n l y one possible value. In order to reformulate the constraints on the remaining free chains, the range of possible interpretations is restricted i n a manner similar to that employed i n the other two d o m a i n s , v i z . , a restriction such that a solution for the restricted variant exists if and only if a solut i o n exists for the more general case. Consider first the chains that are connected together cycUcally i n a group, i.e., connected by common L - or Y-terminations^^ If the n u m b e r i n such a group is even, let a l l t e r m i n a t i o n configurations be contiguous on one side only. If t h e number is o d d , pick a t e r m i n a t i o n configuration and set b o t h of its sides to be contiguous if it is a Y - t e r m i n a t i o n or discontiguous i f it is an L - t e r m i n a t i o n , and then constrain t h e r e m a i n i n g configuration to be contiguous only on one side. T h i s results i n a set of bijective constraints t h a t allow a l l chains i n the cycle to be interpreted while continuing to p r o h i b i t double discontiguity. It is evident that this process can be carried out i n parallel for a l l cychc chains i n the p a r t i t i o n . Those chains t h a t are not cycUc must contact a t e r m i n a t i o n configuration for w h i c h one side of the central L - or Y - j u n c t i o n has already been assigned a definite contiguity value. If there is only a single chain between such junctions, it can be determined i n a fixed amount of t i m e whether or not it can be given a value consistent w i t h those already assigned; this is possible exactly when a legal labeUing exists for the d r a w i n g . A similar situation holds for two connected chains. T h e common L-terminations are necessarily of type A . If three or more chains are connected together, a shghtly more complex procedure c a n be used: 1. Determine the values for the endpoints if they are to have a legal labelhng of their t e r m i n a t i o n configurations. 2. C o n s t r a i n a l l t e r m i n a t i o n s between the chains along this p a t h to have sides of opposite contiguity. 3. If the number of chains is even and the endpoints have different contiguity labels, the alternation of contiguity at the terminations w i h suffice for a legal labelhng. A s i m i l a r situation holds when the number of chains is o d d a n d the endpoints have the same contiguity labels. Otherwise, pick one of the inner t e r m i n a t i o n configurations: If it is a n L - t e r m i n a t i o n : it must be of type A ; constrain b o t h sides of its c e n t r a l j u n c t i o n to be discontiguous. If it is a Y - t e r m i n a t i o n : constrain b o t h sides of its central j u n c t i o n to be contiguous. T h e result of this process is a sequence of chains that must alternate i n value at each term i n a t i o n c o n d i t i o n , except at the configurations described above. A solution for this set of constraints is possible exactly when a legal labelhng can be obtained for the d r a w i n g . T h e detection of the chains a n d the propagation of values along their extent can be carried out entirely by C C L . Since the constraints along these chains are bijective, their solution can also be obtained v i a this procedure. A n d since b o t h convexity a n d contiguity labeUing can be reduced t o C C L , this yields T h e o r e m 3.8 labelling For rectangular to have a complexity objects, the planarity no greater than that of and interior angle constraints allow line CCL. C . Slant Sign C o n s t r a i n t s F o r rectangular objects, constraints exist o n the slant sign of each edge i n the drawing (sections 3.2.3 a n d 3.2.4). T h e resultant set of constraints are shown i n figure 3.13. Since a l l constraints are bijective, the entire drawing is i m p h c i t l y Unked together i n one entire complex w i t h only two possible interpretations. F r o m proposition 3.7, it then foUows that the d e t e r m i n a t i o n of slant signs can be reduced to C C L under these conditions. (a) Drawing delectad as impossible rectangular object - cannot be given consistent set of slant sign labels (b) Drawing not detected as Impossible rectangular obiect - can be given consistent set of slant sign labels F i g u r e 3.26: Slant sign constraints applied to impossible figures. Described i n this way, the determination of slant sign does not rely on any other system, and so can serve as an independent source of constraint on the final interpretation. Indeed, the use of slant sign yields a process of higher accuracy that that obtained from qualitative labeUing alone [Kan90], since it permits a greater rejection of drawings that cannot correspond t o a rectangular object (figure 3.26(a)). However, because only l o c a l orientations about the j u n c t i o n s are involved, these a d d i t i o n a l constraints are not sufficient to eUminate aU impossible figures (figure 3.26(b)). C o n s t r a i n t s on slant sign not only provide more information about the corresponding p o l y h e d r o n , but can also speed up the interpretation process itself. For example, if a Y - j u n c t i o n has a l l three edges slanted away f r o m the viewer, it must correspond to a convex corner. Similarly, if the stem of an arrow-junction slants away f r o m the viewer, it must be concave a n d the other edges convex. A s shown i n figure 3.19, the slant sign determines the labeUing of acute L-junctions and of obtuse L-junctions for which the edges slant away, aUowing aU contiguity constraints to be put immediately into b i n a r y f o r m . Note also that i f the slant sign stream is used (together w i t h the convexity stream) as the basis for contiguity i n t e r p r e t a t i o n , it eUminates the need for expUcit p l a n a r i t y and interior-angle constraints. D . Slant M a g n i t u d e Constraints Slant magnitudes are constrained v i a equation 3.5. It foUows f r o m this equation that t h e m a g n i t u d e of one edge immediately determines that the other (section 3.2.4). W h e n o n l y two Une directions are present i n a p a r t i t i o n , slant magnitude is underconstrained; the slant magnitudes of the edges can then be fixed simply by assigning some a r b i t r a r y value to one of the directions. Since the p a r a h e l hnes i n each p a r t i t i o n represent paraUel edges i n the scene, once the p a r t i c u l a r magnitudes have been chosen, C C L can be used to propagate t h e m to aU Unes i n the p a r t i t i o n . A similar s i t u a t i o n exists when three different hne directions exist i n the image, except now the slant magnitudes are uniquely determined. A s discussed i n section 3.2.4, the determ i n a t i o n of the slant magnitudes for this situation can also be reduced to C C L . E. Robustness i) Image perturbations T h e quaUtative a n d quantitative interpretation of rectangular objects reUes on a special f o r m of the general viewpoint assumption, v i z . , the assumption that slants i n the scene are never zero, a n d consequently, that hnes i n the image are never perpendicular to each other. A l t h o u g h this assumption is sufficient for theoretical purposes, any p r a c t i c a l system must be able t o compensate for errors that arise f r o m the measurement of image properties. A s such, an a d d i t i o n a l set of techniques is required to ensure that the interpretation process remains robust against smaU perturbations of the input image. For the i n t e r p r e t a t i o n of rectangular objects, perturbations have their greatest effect when one or two edges have a slant differing only shghtly from zero. W h e n only one of the edges is very shaUow, the other two are i n a plane closely ahgned w i t h the hne of sight; as a consequence, their projections onto the image are be nearly at right angles to the projection of the shaUow edge (figure 3.27(a)); among other things, this makes it difficult to distinguish arrow- a n d Y - j u n c t i o n s f r o m T - j u n c t i o n s . If two of the edges are shaUow, their projections are nearly at right angles to each other (figure 3.27(b)), m a k i n g it difficult to distinguish between acute a n d obtuse L - j u n c t i o n s . A s such, shallow edges can cause a potential instabiUty i n the labelhng of convexity, contiguity, and slant sign. Furthermore, from equation 3.5 it also foUows that estimates of slant magnitude are also sensitive to smaU errors i n hne orientation angle 9i under these conditions. One way to o b t a i n robustness against such perturbations is to alter shghtly the angles of the hnes i n the j u n c t i o n s , setting t h e m to values that are aU the same. T h i s helps b o t h t o remove the effect of l o c a l p e r t u r b a t i o n s , a n d to reduce the effects of perturbations i n t r o d u c e d at any later stage of processing. T h e exact procedure depends on which of the two situations (a) One edge shallow (b) Two edges shallow F i g u r e 3.27: Conditions of shallow slant. F i g u r e 3.28: C o m b i n a t i o n s of angles into corners. is encountered. In b o t h cases, the procedure begins by o b t a i n i n g the distribution of Une directions i n the p a r t i t i o n . Ideally, only three directions w o u l d exist, corresponding to the three directions of the edges of the corresponding object. If more t h a n three exist, a procedure (e.g., t a k i n g the m e a n of each of the three distributions) can be used to remap the hnes onto a smaller set of angles. Once determined, these new values can be broadcast to a l l junctions. T h i s r e m a p p i n g applies t o a l l possible corners (figure 3.28), since it follows from equation 3.5 that slant m a g n i t u d e is indifferent to the p a r t i c u l a r combination chosen. Since a similar reassignment can also be used for the final set of directions obtained, alteration of Une direction can be based entirely on a pair of canonical j u n c t i o n s , obtained from the appropriate rearrangement of Unes i n the image (figure 3.29). C o n s i d e r first the case where one edge has a shaUow slant. A s figure 3.27(a) shows, this is signaUed by the existence of two nearly-paraUel Une directions i n the image. Using the Normal to projection of nonshallov edge Slopes increased with respect to normal j I Normal to projection of shallow edge (a) One edge shallow slopes decreased with respect to normal (b) Two edges shallow F i g u r e 3.29: Rescaling of image angles. n o r m a l t o t h e shallow edge t o complete a local coordinate frame, t h e slopes of the other t w o hnes c a n be rescaled u n t i l at least one has a value 6min (figure 3.29(a)). Such a rescahng of orientations corresponds t o a r o t a t i o n of each corner of the object about an axis perpendicular t o the shallow edge, w i t h the slant of the shallow edge being increased. N o t e that this is not a true r o t a t i o n of the object as a whole, since this requires a change i n the distances between junctions as well. B u t i f the r o t a t i o n is small, the change i n foreshortening is neghgible, a n d the transformation can be Interpreted as a shift i n v i e w i n g p o s i t i o n t h a t results i n more robust interpretation. A similar technique can be used when t w o shahow edges exist. F r o m figure 3.27(b), It Is seen t h a t this condition Is signahed b y the existence of two hne orientations that are nearly perpendicular t o each other. A s for the case of one shallow edge, slopes can be rescaled, except that n o w the rescahng Is done w i t h respect t o an axis perpendicular to the edge w h i c h Is not shallow (figure 3.29(b)). T h e resulting transformation corresponds to a r o t a t i o n about an axis at right angles t o the nonshaUow edge, the r o t a t i o n serving t o increase t h e slant of the t w o shaUow edges. If i t is necessary to apply b o t h transformations to the hnes of an Image, this can be done simply be a p p l y i n g the required corrections i n some fixed way. T h u s , provided that three different hne directions can be distinguished In a p a r t i t i o n , they can always be r e m a p p e d i n t o a new set of orientations that can disambiguate any local ambiguities caused b y smaU p e r t u r b a t i o n s i n the Input Image, a n d that m i n i m i z e the effects of any other perturbations t h a t might be introduced b y subsequent processing stages. ii) Perspective distortion T l i e results obtained here assume the scene-to-image projection to be orthographic, i.e., rays f r o m the scene contact the image plane at right angles. Since the scene-to-image m a p p i n g is the same at a l l points i n the image, a spatially-uniform set of rules can therefore be used to recover the scene structure. T h i s greatly simphfies the development and analysis of the recovery process. O r t h o g r a p h i c projection is almost always a good a p p r o x i m a t i o n of the perspective project i o n that a c t u a l l y governs the m a p p i n g of objects onto the image plane. However, it breaks down when an object extends over a large fraction of the visual field. In such a case, only one point corresponds to a perpendicular projection from the object to the image plane, and a " r a d i a l " d i s t o r t i o n arises that is centered about this p o i n t . A l t h o u g h this distortion complicates the recovery process, it does not affect its interpretative power â€” a global t r a n s f o r m a t i o n of the image can always be found that maps each j u n c t i o n to its equivalent under o r t h o graphic p r o j e c t i o n (see, e.g., [ K a n 9 0 , ch.8]). Consequently, b o t h qualitative and q u a n t i t a t i v e structure c a n always be recovered. Since the emphasis of this work is p r i m a r i l y on r a p i d recovery a n d not on robustness per se, special corrections for perspective distortion are not developed here. Note that perspective d i s t o r t i o n alters only the angles a n d lengths of lines i n the image, a n d so the basic q u a l i t a t i v e aspects of the recovery process are largely unaffected. F u r t h e r m o r e , if these distortions are s m a l l , the angles can be reahgned by the broadcast mechanism used to handle s m a l l p e r t u r b a t i o n s i n the i n p u t . T h u s , the only situation not encompassed by this approach is t h e relatively rare case where the projection of an object extends over a considerable fraction of the v i s u a l field. Chapter 4 Computational Analysis A c o m p u t a t i o n a l analysis describes and justifies an image-to-scene m a p p i n g that is (i) u n i q u e , and (ii) possible w i t h i n the given external and internal hmitations (section 2.4). F o r the m a p p i n g considered here, the external Hmitations are that the information comes f r o m a single orthographic projection of a blocks w o r l d scene of the type described i n section 1.1, and t h a t a constant amount of time is available for its operation (section 2.5.2). I n t e r n a l h m i t a t i o n s are that a mesh architecture is used, a n d that the processors have a fixed n u m b e r of states. T h i s chapter develops a set of constraints that defines a process capable of recovering a large amount of the scene structure w i t h i n these l i m i t a t i o n s . To ensure that l o c a l c o m p u t a t i o n is relatively simple, a set of external constraints is chosen t h a t Hmit the range of possible mappings to those that can be determined i n subhnear time.^ A set of i n t e r n a l constraints is developed to control the search through the space of possible solutions. These constraints are chosen so that a reasonable chance exists of finding a plausible i n t e r p r e t a t i o n w i t h i n the allotted t i m e . T h e constraints developed here, of course, are not necessarily those used by the h u m a n early v i s i o n system. M a n y factors are potentially involved i n the r a p i d recovery of three- dimensional structure, not a l l of which are k n o w n or fuhy appreciated at the present t i m e . A s such, this analysis is not intended p r i m a r i l y as a definitive treatment of the r a p i d recovery process, but rather as an illustration of how a c o m p u t a t i o n a l analysis of this process can be carried o u t . *More precisely, the complexity of the mapping must be a subUnear function of the number of lines in the image (see section 2.5.1.) 4,1 External Constraints A s discussed i n section 2.5.2, tlie output of a r a p i d recovery process is a dense set of estimates assigned to each spatiaUy-hmited p a t c h (or "zone") i n the image.^ These estimates must be b o t h l o c a l l y consistent a n d computable i n a constant amount of time. If they are to have a g o o d chance of corresponding to the a c t u a l structure of the scene, the corresponding p r o p e r t y must be easy to compute a n d use m i n i m a l information from outside the zone. G i v e n the correlation between the amount of nonlocal i n f o r m a t i o n that must be t r a n s m i t t e d a n d the complexity of an operation (section 2.1.1), it fohows that recovered quantities must be computable by a low-complexity process, ideally one of no more t h a n l o g a r i t h m i c c o m p l e x i t y (cf. chapter 3.1.2). T h i s can be ensured by an appropriate choice of external constraints (section 2.4) on the final form of the m a p p i n g . These constraints serve to ehminate those mappings that cannot be computed i n subhnear t i m e . 4.1.1 Image-to-Scene mapping For a r a p i d recovery process, the goal is to reconstruct as m u c h of the three-dimensional scene as possible, w i t h interpretations required to be consistent only over s p a t i a l l y - h m i t e d zones (section 2.3.2). T h i s goal is somewhat different from that of the "classical" p r o b l e m of hne i n t e r p r e t a t i o n , w h i c h assigns unambiguous interpretations to each hne, a n d completely rejects drawings that cannot be given a globally consistent i n t e r p r e t a t i o n . Consequently, t h e image-to-scene mappings need not be the same for the two types of tasks. Possible differences i n the image-to-scene m a p p i n g include not only differences i n t h e p a r t i c u l a r associations between input a n d output quantities, but also differences i n the q u a n tities themselves. T h e first step to find a m a p p i n g suitable for r a p i d recovery is therefore t o determine an appropriate set of inputs a n d outputs. A . B a s i c Quantities A fixed h m i t on time translates into a fixed hmit on the distance over which i n f o r m a t i o n can be t r a n s m i t t e d . If recovery is to be robust w i t h regards to this h m i t , it cannot be based on g l o b a l properties (e.g., the number of features present i n the image), or on extensive ^ T h e exact size of these zones is not critical, the main constraint being that they are small enough that each contains no more than a few Une segments (section 4.3.1). properties (e.g., the lengths of Hnes a n d edges). Instead, it involves only those properties that can be determined locally, i.e., over arbitrarily small areas of the visual field (cf. section 2.1.1). L o c a l i t y apphes to properties of b o t h the input and the o u t p u t . In w h a t foUows, the basic quantities i n the image domain are taken to be the t w o dimensional orientations of the Unes a n d the locations of their endpoints.-^ T h e quantities i n the scene d o m a i n are taken to be the (positive) convexities, slant signs and slant magnitudes of the edges, as weU as the contiguity relations between edges and surfaces. In p a r t i c u l a r , the f o r m of the output is taken to be exactly that used i n chapter 3.1.2 â€” a set four spatiotopic m a p s , one for each property, i n which the variables describe the absence or presence of the corresponding quantity. A U properties are assumed to be "dense", being attached t o all points along the Unes i n each zone. N o t e that this differs from the "sparse" form used i n the " c l a s s i c a l " p r o b l e m of Une i n t e r p r e t a t i o n (section 2.2.1), where properties are a t t a c h e d to i n d i v i d u a l Hnes, rather t h a n points (see, e.g., [Mal87]). T h i s choice of properties is m o t i v a t e d not only by the requirement that the properties be l o c a l , but also by the results of chapter 3.1.2, which show that these quantities can i n deed be r a p i d l y recovered for several sub-domains of polyhedral objects. Note that these are not " t e m p l a t e " properties, w h i c h can be calculated reUably on the basis of local i n f o r m a t i o n (section 2.1.1), but instead require at least some nonlocal i n f o r m a t i o n for their complete d e t e r m i n a t i o n . However, the low complexity indicates that relatively Uttle nonlocal i n f o r m a t i o n is needed. T h i s is the key to the effectiveness of a r a p i d recovery process â€” even though nonlocal information this structure is generally needed for a complete local interpretation, can be rapidly recovered if the amount of information at least some of needed is small. O f course, other quantities (such as the slants of the surfaces) could also be used, a n d conversely, some of the quantities used here m a y not actuaUy be recovered by the h u m a n early v i s u a l system. B u t the quantities chosen here encompass b o t h the quaUtative a n d q u a n t i t a t i v e aspects of Une i n t e r p r e t a t i o n (section 2.2.1), and are therefore adequate for present purposes, v i z . , iUustrating how r a p i d parallel recovery can be done. B . Isolation of Indeterminate Values Since the i n t e r p r e t a t i o n provided by a rapid-recovery process does not need to be consistent over the entire image (section 2.3.2), the surrounding interpretations do not need to be ^Locations are always relative to a particular zone, so that absolute coordinates are not needed. N H F i g u r e 4.1: Isolation of inconsistency i n contiguity labeUing removed when a l o c a l inconsistency is found. Indeed, this is not even desirable, for the propagation required for this removal can take considerable time (cf. section 3.1.2), a n d also results i n i n f o r m a t i o n being lost from those areas which do obey the assumptions. Similarly, the h m i t e d time available m a y be insufhcient to completely determine a l l interpretations, so that ambiguities also must be exphcitly handled i n some way. In order t o allow these possible outcomes to be exphcitly signaUed, the set of o u t p u t states is expanded to include an ' F l a b e l for inconsistencies, a n d an ' A ' label for ambiguities. These can be apphed to any variable i n any dimension. B y careful apphcation of these labels, indeterminate values can be isolated so as to allow a definite i n t e r p r e t a t i o n to be given to the other parts of the drawing. A n example of this is shown i n figure 4.1. If the contiguity constraints of section 3.3.3 are used on this figure, a globaUy consistent labeUing is not possible. A t t a c h i n g an T l a b e l to one of the hnes on the central n o t c h , however, allows the remaining edges to be given a definite i n t e r p r e t a t i o n . In order that these labels can be used wherever needed, no exphcit constraints are placed on their apphcation; if necessary, they can be assigned to aU hnes in drawing. However, the use of these labels must be m i n i m i z e d i f the greatest amount of i n f o r m a t i o n is to be obtained about the three-dimensional structure of the scene. Since this cannot be done by constraints o n the static assignment of the labels, it must be done v i a constraints on the process that generates these assignments (section 4.2). 4.1.2 General Principles C h a p t e r 3.1.2 shows that low-complexity approximations can be formed i n m a n y different ways, depending on the constraints selected. It is now necessary to select one p a r t i c u l a r set of constraints, and to j u s t i f y this selection. Three general principles are relevant here. A . Separation of D i m e n s i o n s A s shown i n chapter 3.1.2, a low-complexity m a p p i n g can be obtained when e x t e r n a l constraints apply p r i m a r i l y to simple variables w i t h i n separate dimensions, w i t h only a l i m i t e d amount of i n t e r a c t i o n between the corresponding streams. T h i s strategy is taken as a basic principle here. In p a r t i c u l a r , a l l constraints that are nonbijective (i.e., do not have a 1:1 m a p p i n g between allowable values â€” see section 3.1.2) must involve only two variables, each w i t h two possible values.'' T h i s ensures that the problem is easy to solve (section 3.1.2). Since it has been shown to lead to low-complexity mappings for m a n y subdomains (section 3.3), the set of dimensions used here is exactly that of chapter 3.1.2: contiguity, p o s i t i v e convexity, slant sign, a n d slant magnitude. B . Locality of Constraints M o s t constraints used i n hne interpretation (sections 2.2.1 and 3.3) are l o c a l , p e r t a i n i n g t o i n d i v i d u a l hnes a n d j u n c t i o n s . A n example of this is the requirement that a l l edges i n a Y j u n c t i o n must be labelled as '-f-' or 'o' (section 3.2.2). Constraints such as the interior angle constraint (section 3.3.3), however, involve relations Ã´eiiyeen junctions, and so are not of this f o r m . Since n o n l o c a l constraints require image-based as well as scene-based i n f o r m a t i o n t o be t r a n s m i t t e d , they increase the demands on c o m p u t a t i o n a l resources. They are also a w k w a r d to enforce on a mesh processor, where i n f o r m a t i o n travels at a constant speed across the image (section 2.5.2). O n l y l o c a l constraints must therefore be used. A s a special case of this principle, note that it is impossible to ensure that a single l a b e l can be attached to any Une (section 2.2.1), since a Une can extend over a considerable distance i n the image. Consequently, constraints on j u n c t i o n labeUings are stiU allowed, but they now apply t o Une segments of fixed length rather t h a n Unes of arbitrary length. A n auxiUary set of constraints is required to constrain neighboring segments to take on the same values. *Note that more values are possible for these variables, but that only two must enter into the nonbijective constraints. T h i s allows T and ' A ' labels to be used in addition to the two definite values, since they do not enter into any explicit constraint. c. L o c a l C o o r d i n a t i o n of D i m e n s i o n s M u c h of the power of the original m a p p i n g can often be captured by a low-complexity approxi m a t i o n only if there is an interaction between the interpretations i n the separate dimensions (section 3.3). Ideally, this w o u l d be carried out by a globally-coordinated sequencing between the different streams. Since global coordination is not feasible here, however, the interactions between dimensions must be reformulated to occur at a l o c a l level. A s discussed i n section 3.3, the key to the successful integration of dimensions is the u n i d i r e c t i o n a l transmission of i n f o r m a t i o n . B u t global coordination is not needed â€” this can be achieved by t r a n s m i t t i n g i n f o r m a t i o n from a local interpretation after it has been assigned an unambiguous value. For example, when the edges of a Y - j u n c t i o n have a unique labelhng as convex, they are necessarily contiguous, and so can determine the corresponding i n t e r p r e t a t i o n i n the contiguity stream. T h i s k i n d of interaction allows l o c a l constraints to assist the i n t e r p r e t a t i o n process w i t h o u t any danger of increasing the complexity of the process. 4.1.3 Structural Assumptions A s shown i n section 3.3, approximations of subhnear complexity can exist when appropriate restrictions are placed on the contiguity a n d convexity interpretations of L - j u n c t i o n s . These restrictions can be achieved v i a assumptions about the structure of the p o l y h e d r a , a n d as discussed i n section 3.3, there are several sets of assumptions w h i c h can be used towards this end. L i w h a t fohows, i n t e r p r e t a t i o n is based on constraints obtained from the assumption of rectangular corners (section 3.3.3). T h e r e are several reasons for this choice. F i r s t of ah, the visual system is exceptionaUy g o o d at detecting junctions corresponding to rectangular corners, and using the rectangularity assumption t o determine the three-dimensional orientations of the corresponding edges (see section 2.2.2). There is no reason to suppose that this preference is h m i t e d only to the higher stages of v i s u a l processing. A second set of reasons involves issues of s y m m e t r y a n d structure. If an angle between two edges i n a corner is u n k n o w n , 90Â° is a n a t u r a l default, simply because it is midway on the range of a l l possible angles;^ i n some sense, it may be considered to be an expected ^If edges are assumed to be unmarked, a rotation of 180Â° is an identity transform. Attaching an edge to a value. F u r t h e r m o r e , the fact that a l l edges are perpendicular to each other makes it s i m p l e to convert between the slants of the edges and the slants of the faces: for rectangular corners, the n o r m a l to the surface corresponding to a region is parallel to the edge that is opposite it i n the j u n c t i o n . F i n a l l y , there are reasons based on c o m p u t a t i o n a l complexity. T h e i n t e r p r e t a t i o n of rectangular objects requires relatively httle i n the way of processing resources, since it is among the least complex of a l l hne interpretation problems. Rect angularity also allows edge slant to be determined r a p i d l y by l o c a l processes, something not generally possible for t h e other domains considered i n section 3.3. Indeed, the e s t i m a t i o n of edge slant does not even need to wait for the quahtative analysis to be completed (cf. [Sug86]), but can proceed i n t a n d e m w i t h the determination of contiguity and positive convexity. 4.1.4 S y s t e m of E x t e r n a l C o n s t r a i n t s B r i n g i n g together the requirements outhned above, the r a p i d recovery of three-dimensional structure is assumed here to be governed by the external constraints shown i n figure 4.2. These involve four quasi-independent dimensions: contiguity, positive convexity, slant s i g n , and slant m a g n i t u d e . T h e intra-dimensional constraints are given by the permissible l a - beUings of the j u n c t i o n s ; these are essentiaUy the constraints developed i n section 3.3.3. Interactions between dimensions, shown by the arrows i n figure 4.2, are readily derivable f r o m this same set of constraints. T h i s system of constraints is largely a reformulation of those of section 3.3.3. In p a r t i c u l a r , the nonlocal p l a n a r i t y a n d interior-angle constraints have been replaced by l o c a l constraints o n slant sign, w h i c h then influence contiguity v i a the inter-dimensional interactions. T h e r e also exists a more direct l o c a l (nonbijective) constraint against doubly-discontiguous Unes. A l t h o u g h not required to a t t a i n a process of l o g a r i t h m i c complexity when global coo r d i n a t i o n is possible (section 3.3.3), this constraint can improve the speed and power of the i n t e r p r e t a t i o n process when only l o c a l processing is allowed. N o t e that the contiguity constraints on obtuse L-junctions cannot i n general be put i n t o b i n a r y or bijective f o r m . In the absence of a definite interpretation for the two inner or two outer Unes, only a single c o m m o n constraint (that requiring b o t h inner Unes to take on the same value) can be appUed. B u t the remaining constraints can be put into b i n a r y f o r m i f corner does mark it, but two adjacent edges that differ by 180Â° are stiU effectively the same edge. Contiguity Convexity Slant Sign F i g u r e 4.2: S y s t e m of e x t e r n a l constraints. A r r o w s indicate interactions between dimensions. a dependency on the state of particular "trigger" variables is introduced so that constraints are put into effect only after a definite interpretation has been assigned.^ Consequently, aU bijective a n d binary constraints on the junctions have been kept a n d the rest reformulated to take on one of these two forms. A l t h o u g h global co-ordination of the type required t o achieve the results of section 3.3.3 is no longer possible, these constraints do allow the three-dimensional structure of the scene t o be recovered. G i v e n sufficient t i m e , a consistent i n t e r p r e t a t i o n w i t h o u t ambiguities or inconsistencies is possible whenever t h e d r a w i n g corresponds to a set of rectangular objects. Because of the T and ' A ' labels, however, the requirements of section 2.3.2 are also met: a scene that does not contain rectangular objects everywhere m a y stiU give rise to local interpretations i n those regions where the basic s t r u c t u r a l assumptions are obeyed. 4.2 Internal Constraints A l t h o u g h e x t e r n a l constraints h m i t the range of interpretations which can be given to a line d r a w i n g , they are not generally sufficient to determine its f o r m completely. For example, the m a r k i n g of edges as inconsistent or ambiguous must be kept to a low level (section 4.1.1), but this requirement conflicts w i t h the p r o h i b i t i o n against global measures (section 4.1.1). M o r e generally, the i n t e r p r e t a t i o n process must operate w i t h i n a fixed amount of t i m e , a n d while the e x t e r n a l constraints developed i n section 4.1 do select a set of solutions that can be calculated quickly a n d w i t h a m i n i m u m of nonlocal i n f o r m a t i o n , they do not provide any guidance as t o what should be done when such h m i t s are imposed. To complete the specification of this m a p p i n g , therefore, an independent set of i n t e r n a l constraints must be imposed to help ensure that the best use is made of the available p r o cessing time.^ These are constraints on the generation of the i n t e r p r e t a t i o n itself (section 2.4.2). For the process here, these constraints are required to lead to a subset of interpre^These constraints, obtained from figure 4.2, are as follows: 1. If one of the inner edges is contiguous, no more than one outer edge can be contiguous. 2. If one of the outer edges is discontiguous, both inner edges must be contiguous. 3. If both inner edges are discontiguous, both outer edges must be contiguous. 4. If both outer edges are contiguous, both inner edges must be discontiguous. '^The use of such constraints is essentially an elaboration of Marr's principle of graceful degradation [Mar82, p. 106], extended to cover not only reductions of available information, but reductions of other resources as well. t a l i o n s that have a relatively h i g h hkehhood of corresponding to the structure actually i n the scene. A l t h o u g h the probabiUties of various image-to-scene associations depend on the particular scene d o m a i n under consideration, exact knowledge of these probabihties is not generaUy necessary â€” aU that is required is an ordering of the various candidates. A s such, it is possible to provide a set of principles that are potentially appUcable to m a n y domains encountered i n the n a t u r a l w o r l d . 4.2.1 Processing Architecture Internal constraints act on the flow of i n f o r m a t i o n that occurs d u r i n g the course of c o m p u t a t i o n . In order to develop such constraints it is first necessary to specify â€” at least at a general level â€” an "abstract architecture" that describes the way i n which i n f o r m a t i o n processing and i n f o r m a t i o n transmission are carried out. A . P r o c e s s i n g over Zones F r o m the definition of the r a p i d recovery problem (section 2.5.2), the only processing resources assumed to be available are a spatiotopic mesh of processors, w i t h each processor h a v i n g a relatively smaU set of states. If g o o d use is to be made of these resources, each processor (or group of processors) i n the mesh must be assigned to a separate zone i n the image, i.e., to a compact contiguous area of h m i t e d spatial extent (section 2.1.1). T h i s requirement stems p r i m a r i l y from considerations of efficiency. W h e n only a fixed amount of t i m e is aUowed, each processor can only act on a fixed number of inputs.^ Since the number of processors increases w i t h the size of the input (cf. section 2.5.2), effectiveness can be m a i n t a i n e d by assigning each processor to a separate region of the image. A n d if processors are u n i f o r m i n regards to their processing power (as assumed here), it is best if these regions have the same size. Since transmission delays w i t h i n a zone must be kept to a low level, regions must also be contiguous a n d compact.^ ( T h i s is related to the preference for l o c a l quantities described i n section 4.f .1.) T h e d e m a n d for contiguity is forced not only by the need for compactness, but * T h i s is a generalization of an order-limited perceptron [MP69], with the output function being any function that can be calculated in a fixed amount of time. ^This restriction means that the process can be carried out by a generalized version of a diameter-hmited perceptron [MP69]. by the recognition that the external constraints are highly sensitive to breaks i n the l i n e s , and since hne drawings may fall anywhere, no part of the image can afford to be s k i p p e d . C o n t i g u i t y also reduces the c o m p u t a t i o n a l power required of the i n d i v i d u a l processors, since it is easier to handle a small set of unbroken hnes t h a n a large set of disconnected hne fragments. B . C o m m u n i c a t i o n between N e i g h b o r i n g Zones If the m a p p i n g between i n p u t a n d output could be described entirely i n terms of n o n interacting zones, recovery could be carried out on an array of processors, each of w h i c h calculates only simple template properties of its corresponding zone. B u t if a n y t h i n g be- y o n d the most r u d i m e n t a r y hne interpretation is to be carried out, communication between processors is required. In what foUows, it is assumed that the assignments of processors to zones m a i n t a i n s a spatiotopic organization and that neighboring processors i n the mesh are assigned to neighboring zones i n the image. Once a g a i n , this is m o t i v a t e d by considerations of efiiciency. Since a l l constraints between the l o c a l interpretations are themselves l o c a l , there is relatively httle to be gained by h a v ing some other assignment of zones to processors. F u r t h e r m o r e , the operation of the l o c a l processors (as weU as the analysis itself) is simpUfied, since the transmission of i n f o r m a t i o n takes place only v i a the zone-to-zone percolation of i n f o r m a t i o n t h r o u g h the " v i r t u a l m e s h " formed by the lattice of zones over the image. In p a r t i c u l a r , this i n f o r m a t i o n flow originates from zones containing an interprÃ©table j u n c t i o n , a n d propagates at a constant rate along the connecting Unes. Internal constraints therefore act by controUing the i n i t i a l assignment of interpretations w i t h a zone, a n d by controUing the propagation of these values along the hnes of the drawing. 4.2.2 General Principles A t the most general level, i n t e r n a l constraints can take effect i n two ways: (i) constraints on the basic operations used, a n d (ii) constraints on the representations operated u p o n . These effectively provide general constraints on the propagation of i n f o r m a t i o n around the " v i r t u a l m e s h " t h a t occurs d u r i n g the i n t e r p r e t a t i o n process. to provide constraints on this p r o p a g a t i o n . Four general principles are used here A . M a i n t e n a n c e of Interpretive P o w e r If the process is to have a g o o d chance of recovering some part of the scene structure, it must not be too quick to t h r o w away possible interpretations. Consequently, a hberal i n t e r p r e t a t i o n strategy is used: a candidate interpretation is kept unless an inconsistency is detected. Since inconsistencies are determined by the set of external constraints, this becomes the requirement t h a t i n t e r n a l constraints must not exclude any interpretation consistent w i t h the e x t e r n a l constraints. In other words, i n t e r n a l constraints must not have any ehminative power â€” they must be entirely concerned w i t h the ordering of the various possible solutions. In order to allow a l l possible interpretations to be handled i n a systematic way while keeping true to the d e m a n d that only two values exist i n each constraint (section 4.1.2), the o u t p u t s of each of the four streams are spht into two separate subsystems: C o n t i g u i t y : T w o complementary subsystems to represent the possibihty of c o n t i g u i t y a n d noncontiguity. C o n v e x i t y : T w o complementary subsystems to represent the possibihty of convexity a n d nonconvexity. Slant Sign: T w o complementary subsystems to represent the possibihty of the two types of slant sign.^Â° Slant M a g n i t u d e : T w o different subsystems â€” a quantitative subsystem to c a r r y the value of the estimate, a n d a quahtatlve subsystem to represent the possibihty that this value can legitimately be assigned. F o r the complementary subsystems, the existence of a possible interpretation is signalled by a 'possible' state attached to the relevant edge, while its impossibihty is hkewise signaUed by an ' i m p o s s i b l e ' state (figure 4.3).^^ T h e use of these subsystems allows a l l possible interpretations t o be represented quite s i m p l y : D e f i n i t e : assignment of 'possible' to an edge i n one of the subsystems a n d 'impossible' to its complement. A m b i g u o u s : assignment of 'possible' i n both subsystems. Inconsistent: assignment of 'impossible' i n b o t h subsystems. Slant towards or away from the viewer is not a pure scalar like the other two quantities â€” it has a directional component that must be taken into account (cf. section 3.2.3). T h i s can be handled simply by having each subsystem represent an increase in depth as the line segment is traversed from one of the ends. " T h i s is somewhat analogous to the use of relevance logic in reasoning (see, e.g., [Lev86]. C o n t i g u i t y Subsystem '6 '4 6 Noncontigulty Subsystem U 2 3 { possible, impossible } G { possible, impossible } F i g u r e 4.3: E x a m p l e of complementary labelling. B . L o c a l l y Irreversible C o m p u t a t i o n If a process is to make g o o d use of available t i m e , it must avoid doing a n d undoing the same operations w i t h o u t any net effect. T h i s requirement â€” essentially a form M a r r ' s principle of least c o m m i t m e n t [Mar82, p p . 106-107] â€” rules out hypothesize-and-test strategies, f a v o r i n g instead "one-shot" processes that require only a few steps for their completion. In the absence of g l o b a l c o n t r o l , this must be done by a l o c a l mechanism t h a t forces the process to avoid redundant processing while simultaneously ensuring that it w i l l not exclude any consistent interpretation. To combine this principle w i t h that of m a i n t a i n i n g interpretative power, the following scheme is used: 1. A 'possible' state is i n i t i a l l y assigned to a l l values of a l l complementary subsystems as weU as the qualitative subsystem of the slant magnitude stream. 2. W h e n e v e r a l o c a l inconsistency is found, the corresponding value is m a r k e d as 'impossible', a n d this value w i l l never be w i t h d r a w n . T h i s is essentially a simple f o r m of W a l t z filtering (section 2.2.1), w i t h an i n i t i a l m a x i m u m uncertainty steadily reduced to the point where no l o c a l inconsistencies r e m a i n . G i v e n the s t r u c t u r a l assumptions that have been made, httle nonlocal information is required for a l o c a l i n t e r p r e t a t i o n . Convergence to a definite interpretation i n each zone is therefore hkely to be fast. E v e n if only a h m i t e d amount of time is available, the result is hkely to provide at least some i n f o r m a t i o n to higher stages of processing. T h e s i t u a t i o n is similar for the slant magnitude stream, except that the quahtative subsystem serves to indicate the confidence of the corresponding magnitude estimate. When l o c a l inconsistencies are found i n the estimates of slant magnitude, an 'impossible' l a b e l is assigned to the variables involved, a n d then propagated along the Une. N o t e t h a t the state of the interpretation can be described i n terms of the d i s t r i b u t i o n of the 'possible' states over the edges i n the complementary subsystems. M o r e precisely, l o c a l u n c e r t a i n t y exists only when a value and its complement are b o t h possible. T h i s aUows the overaU uncertainty attached to an interpretation to be described by an "entropy" measure that includes aU the i n d i v i d u a l l o c a l uncertainties. A l t h o u g h never used by the a c t u a l process itself, this measure can provide a way to describe the overaU state of uncertainty i n the i n t e r p r e t a t i o n at any given m o m e n t . L o c a l irreversibiUty can be incorporated into complementary labeUing by a r e f o r m u l a t i o n of the constraints found at the l o c a l junctions. In order to preserve the power of the o r i g i n a l set of constraints, this reformulation is subject to the foUowing condition: any set of definite values r u l e d out by the o r i g i n a l constraints of figure 4.2 must also be ruled out by the new set of constraints. F r o m figure 4.2, it is seen that aU constraints except the contiguity constraints on obtuse L - j u n c t i o n s are "context free", i.e., the particular set of constraints depends only o n the geometrical configuration i n the image, and not on the set of labels attached (cf. figure 4.2). R e f o r m u l a t i o n is based o n the idea that once such constraints have been set u p , eUmi n a t i o n of possible interpretations can occur by a simple p r i o r i t y mechanism that aUows an 'impossible' state to replace any 'possible' state. Since 'possible' states are initially assigned to a l l variables, the reformulation involves only the ways i n which 'impossible' states are t o be t r a n s m i t t e d . T h e s i t u a t i o n for state-dependent junctions is similar, except that no constraints are apphed u n t i l definite assignments have been made to the inner or outer edges. Consequently, it is possible to reformulate the constraints i n a way that aUows the process to be locally irreversible while m a i n t a i n i n g the complete set of external constraints: i) Unary constraints Since only two values can exist i n a subsystem, a unary constraint (i.e., a constraint that acts on a single variable) necessarily requires the variable to have a unique value. F o r example, a u n a r y constraint exists on the inside edges of an arrow-junction that force t h e m to be contiguous. G i v e n two complementary subsystems, u n a r y constraints can be easily enforced by m a r k ing the corresponding value i n the complementary subsystem as 'impossible'. T h e value i n the o r i g i n a l subsystem, however, remains unaffected. T h i s leaves a definite i n t e r p r e t a t i o n w h i c h can only be altered by becoming inconsistent (i.e., the value i n b o t h subsystems b e i n g ' i m p o s s i b l e ' ) . For an a r r o w - j u n c t i o n , therefore, the Inside edges In the non-contiguity subsyst e m are m a r k e d as 'impossible' a n d the corresponding edges In the contiguity system r e t a i n their o r i g i n a l state of 'possible'. ii) Bijective constraints Bijective constraints are such that a 1:1 correspondence exists between the values of the variables Involved (section 3.1.2). For example, a bijective constraint exists on the c o n t i g u i t y labels of the inside edges of an acute L - j u n c t i o n , since b o t h of these edges must be either contiguous or discontiguous (section 3.2.1). Since o n l y two values exist for each variable, bijective constraints take on a simple f o r m : either the variables have the same values, or else they have opposite values. If two adjacent hnes are required to have the same values, b o t h must have the same definite values. I.e., the corresponding variables In the complementary subsystem must be 'impossible'. This constraint can be enforced by the requirement that If one of the corresponding variables In a subsystem Is 'Impossible', so must be the other variable i n the same subsystem. If two adjacent hnes are required to have opposite values, their corresponding variables are similarly constrained, except that now the constraint apphes to variables In "opposing" (figure 4.4). subsystems N o t e that this latter type of constraint provides a b i n d i n g between the two subsystems, w h i c h are otherwise largely Independent. iii) Nonbijective constraints T h e Intradlmenslonal constraints that are not bijective Involve a single p r o h i b i t i o n against a p a r t i c u l a r c o m b i n a t i o n of values (figure 4.2). These constraints can be reformulated quite simply. If one of these values i n such a p r o h i b i t e d combination definitely occurs (i.e., its complement is 'impossible'), then the other value must be excluded (i.e., its " d i r e c t " state Is 'Impossible'). O t h e r w i s e , n o t h i n g else is done. Note that i n contrast to bijective constraints, only a one-way transmission of 'impossible' states Is Involved. For e x a m p l e . If one side of a hne has been m a r k e d as definitely discontiguous (i.e.. Its value i n the contiguity subsystem is 'impossible') then the other side must be contiguous .u^ u G ( possible, impossible } if ( V j = impossible) (u^= impossible) if <v^= impossible)-> (u^ = impossible) V, G { possible, impossible ) = impossible) (v,= impossible) if (u, = impossible) -> 1 (v, = impossible) 4 if (u^= impossible) -> (v^ = impossible) F i g u r e 4.4: E x a m p l e of reformulation of bijective constraint. Contiguity Subsystem u^ G {possible, impossible} Noncontiguity Subsystem Vj G { possible. Impossible } if (u^ = impossible) -> ( v ^ Â» impossible) if (U2= impossible) -> (v^ = impossible) F i g u r e 4.5: E x a m p l e of reformulation of nonbijective constraint. (i.e., i t s value i n the discontiguity subsystem is 'impossible'). B u t i f one side is m a r k e d as contiguous, n o t h i n g else necessarily follows, a n d so no transmission results (figure 4.5). iv) State-dependent constraints T h e contiguity constraint t o be apphed t o obtuse L-junctions depends on the i n t e r p r e t a tion attached t o its edges. T h i s k i n d of constraint can be reformulated i n a straightforward way by i m p o s i n g the appropriate set of constraints when the "trigger" variables take o n definite values, i.e., when their values i n the noncontiguity subsystem are 'impossible'. v) Interdimensional constraints Since i n f o r m a t i o n f r o m one dimension is sent t o another only when a definite i n t e r p r e t a tion has been achieved (section 4.1.2), the reformulation of the relevant constraints is fairly straightforwaxd: when a p a r t i c u l a r set of values has been definitely assigned to the edges about a j u n c t i o n (i.e., the corresponding complementary values are 'Impossible'), the associated values i n the other stream can be given definite values (I.e., their complements are set to 'impossible'). N o t e that i f a variable is deemed to be inconsistent, its value i n b o t h subsystems is 'impossible'. Consequently, the transmission of information across dimensions can cause t h e corresponding variable i n some other dimension to also be labehed as inconsistent. Since processing t i m e Is h m i t e d , however, the propagation of these Inconsistencies Is unhkely to affect greatly the quahty of the final i n t e r p r e t a t i o n , an assumption borne out by tests o n a variety of hne drawings (chapter 6). C . M i n i m i z a t i o n of Inconsistency It is i m p o r t a n t t o control the propagation of labels so that m i n i m a l Inconsistency results, I.e., 'impossible' are assigned to no more variables t h a n necessary. T h i s condition is a u t o m a t i - cally obeyed i f the drawing corresponds to a rectangular object, for the constraints are such that appropriate values can always be assigned to the variables. Indeed, when redundant constraints are added, more routes become available for p r o p a g a t i o n , a n d the faster spread of 'Impossible' states then speeds up the interpretation process. However, when the scene contains objects that do not conform to the underlying s t r u c t u r a l assumptions, inconsistencies can arise i n the resulting Interpretation. Consequently, the more routes available for p r o p a g a t i o n , the greater the spread of inconsistent interpretations. In order to h m l t the spread of such Inconsistencies, therefore, some care must be t a k e n when selecting the p a r t i c u l a r set of constraints to be used. In p a r t i c u l a r , if a variable is subject to a u n a r y constraint, no other constraints must be apphed to i t . For example, the Inner edges of an arrow-junction are constrained to be contiguous. If they are also constrained to have the same value, the Interpretative power of the system Is not affected regarding rectangular objects, since no Interpretation exists In w h i c h these Unes can be assigned definite Interpretations as discontiguous. However, i f an 'impossible' l a b e l has been t r a n s m i t t e d to one of these hnes, such a constraint w i h cause it to be propagated to the others a n d assign t h e m values that could never exist i n any p o l y h e d r a l scene. B y disallowing such a constraint, the o p p o r t u n i t y for inconsistencies to spread is m i n i m i z e d while the power of the o r i g i n a l system of constraints Is m a i n t a i n e d . D . Priority Marking T h e preceding principles have brought w i t h t h e m a shift in emphasis from individuals ensembles. to B u t h u m a n perception tends towards i n d i v i d u a l interpretations. W h e n v i e w i n g a Necker cube, for example, perception alternates between single unambiguous cubes, r a t h e r t h a n being a superimposed set of aU possible interpretations. If the recovery process is t o reduce the set of interpretations that are simultaneously possible, and if it is to r e m a i n effective, it must focus on those that have the greatest hkehhood of corresponding t o the structure i n the scene. T h i s can be done by m a r k i n g such preferred interpretations as distinct. If this m a r k i n g does not otherwise affect the interpretation process, it can allow p r i o r i t y to be given t o the most Ukely candidates while stiU keeping available aU other possible interpretations. T h e simplest way to incorporate p r i o r i t y m a r k i n g is to extend the set of values t h a t can be given to a variable ~ i n addition to 'possible' a n d 'impossible', include a 'preferred' state, w h i c h has p r i o r i t y over a 'possible' s t a t e . I n regards to aU constraints developed so far, the 'possible' a n d 'preferred' states can be treated as equivalent. T h e i n t r o d u c t i o n of this d i s t i n c t i o n can be viewed as a way to spUt the recovery process into two concurrent substreams, dealing w i t h ensembles a n d individuals respectively. A s such, the a d d i t i o n a l constraints required for p r i o r i t y m a r k i n g must be Umited entirely to 'possible' a n d 'preferred' values. T h e only exception is a requirement that when the complement of a 'possible' value is m a r k e d as 'impossible', the value itself must be upgraded to 'preferred'. However, this " i n t r a - s t r e a m " transmission does not affect the set of constraints deaUng w i t h ensembles, since these are involved entirely w i t h the propagation of 'impossible' states. Consequently, the i n t r o d u c t i o n of the possible-preferred distinction has no adverse effects on the abiUty of the recovery process t o eUminate inconsistent interpretations. If desired, the final o u t p u t can be represented using " s t a n d a r d " labels that express the two definite interpretations, the inconsistent interpretation, a n d the ambiguous i n t e r p r e t a t i o n : 1. If one subsystem has a 'preferred' state and the other does not, take its value as a definite i n t e r p r e t a t i o n . 2. Otherwise, if b o t h subsystems have 'preferred' or 'possible', set the i n t e r p r e t a t i o n to be ambiguous. ^^Although interpretations can be even better distinguished by the use of several different priority levels, selection is usually from just a few alternatives, so that this system is sufficient for present purposes. 3. Otherwise, b o t h subsystems must have 'impossible' states. Set the interpretation t o be inconsistent. P r i o r i t y m a r k i n g is a " d u a l " to the interpretation of ensembles. Instead of using a h b e r a l strategy to ehminate impossible interpretations, it uses a conservative strategy to generate hkely ones. It is therefore based on the same set of external constraints as the "ensemble" system, except that it involves 'preferred' a n d 'impossible' labels. Initiahy, ah variables are set to 'possible', w i t h the exception of a s m a l l i n i t i a l set that are assigned a 'preferred' s t a t e . C o n s t r a i n t s are enforced i n m u c h the same way as those of the ensemble substream, except that they involve the transmission of 'preferred' states to the " d i r e c t " subsystems i n s t e a d of 'impossible' states to the complementary subsystems. Consequently, the complexity of p r i o r i t y m a r k i n g is the same as that of the ensemble substream. T h e only a s y m m e t r y between the two processes is that interpretations i n the ensemble substream can override those of the p r i o r i t y substream, but not vice versa. In p a r t i c u l a r , w h e n an ' i m p o s s i b l e ' state is assigned to a variable i n some subsystem, it is effectively w i t h d r a w n f r o m p r i o r i t y m a r k i n g . In a d d i t i o n , the value of the corresponding variable i n the complement a r y subsystem is upgraded f r o m 'possible' to 'preferred'. T h i s a s y m m e t r y reflects the basic difference u n d e r l y i n g the assignment of the two kinds of label: possible-impossible distinctions are based on necessary consequences of the set of assumptions, while possible-preferred distinctions are generally based on considerations of hkehhood. Since complementary subsystems are not used for slant magnitude, there is no need t o distinguish between 'possible' a n d 'preferred' values. However, 'preferred' labels can be used to signal when there is some evidence for a definite assignment of magnitude (e.g., magnitudes o b t a i n e d v i a P e r k i n s ' laws). T h i s proves especially useful i n distinguishing slants that are zero by default f r o m those that have been determined to be zero, since the latter can be treated equivalently to any nonzero slant magnitude. 4.2.3 S e l e c t i o n of I n i t i a l C a n d i d a t e s T h e course of processing is controhed not only by constraints on the d y n a m i c flow of i n f o r m a t i o n , but also by constraints on the i n i t i a l interpretations that are considered. In p a r t i c u l a r , the p r i o r i t y m a r k i n g mechanism developed in the previous section provides a way to dist i n g u i s h a subset of selected candidates, but does not itself provide any principles to guide this selection. A l t h o u g h the most appropriate selection of i n i t i a l candidates depends on the paxticular d o m a i n being modelled, a few general principles appear t o be widely applicable. A . Contiguity T h e first of these principles is that of maximum interior contiguity, which assumes that the inner surfaces of corners are contiguous whenever possible. T h i s principle is an extension of the one used to reduce the complexity of labeUing convex and compound convex objects by selecting a preferred set of interpretations (sections 3.3.1 and 3.3.2). It may also be a basis for the h u m a n perception of Une drawings [Mac74][Hor86, p. 355]. T h e principle of m a x i m u m interior contiguity appUes only to the contiguity s t r e a m . T h e p a r t i c u l a r set of j u n c t i o n m a r k i n g s , shown i n figure 4.6, is as foUows: A r r o w - j u n c t i o n s : C o n t i g u i t y is preferred for aU Unes except for the outer pair, w h i c h do not generaUy form an interior angle. Since contiguity is necessary for inner hnes, noncontiguity is impossible. Y-junctions: C o n t i g u i t y is preferred for aU Unes. L - j u n c t i o n s (obtuse): C o n t i g u i t y is preferred for the inside edges, since most of the possible interpretations assign t h e m this value. T h e outer edges of these j u n c t i o n s , however, cannot be given preferred interpretations, for although one of these edges can be contiguous, the other cannot, a n d the s y m m e t r y of the situation makes it impossible t o prefer one over the other. L - j u n c t i o n s (acute): O u t e r edges are necessarily contiguous, while s y m m e t r y makes it impossible to prefer any interpretations of their inner edges (figure 4.2). T-junctions: T h e crossbars have a necessary contiguity relation, so that preferred values foUow immediately. N o preference is given to values on T - j u n c t i o n stems, since these are effectively isolated Unes.^^ B. Convexity Preference i n the convexity stream is determined by the principle of maximum convexity, i n w h i c h assumes that aU t r i h e d r a l corners i n the scene have positive convexity. T h i s p r i n c i p l e '^Strictly speaJdng this is not true for more reahstic surfaces, where the stem represents not only an edge of a different surface, but can also represent a crack or thin surface marking on the same surface. T o address this issue more fully would require the development of a rapid-recovery system based on a more extensive set of labels, and this is beyond the scope of the present work. Contiguity Subsystem â€¢ â€¢â€¢â€¢ Impossible Noncontiguity Subsystem â€¢::-::-:y::-:> Possible Preferred stems f r o m the observation t h a t convex corners are more c o m m o n t h a n concave ones; i n d e e d , concave corners do not necessarily correspond to a c t u a l structures of the object itseff, b u t m a y instead result f r o m contact between adjacent objects [Bie85]. T h e i n i t i a l set of j u n c t i o n markings i n the convexity stream, shown in figure 4.7, is as foUows: A r r o w - j u n c t i o n s : C o n v e x i t y is preferred for the stems, while non-convexity is preferred for the outer wings. Y - j u n c t i o n s : C o n v e x i t y is preferred for aU hnes. L - j u n c t i o n s (obtuse): A U Unes are necessarily non-convex, leading to preferred values i n the nonconvexity subsystem. L - j u n c t i o n s (acute): A l t h o u g h constrained to have one convex a n d one nonconvex side, s y m m e t r y makes it impossible to assign a preference. T - j u n c t i o n s : T h e crossbars of the T-junctions correspond to occluding edges, a n d so are necessarily non-convex. C . Slant S i g n O w i n g t o the close connection that exists between convexity and slant sign when the corners are rectangular (section 3.2.3), the principle of m a x i m u m convexity can also determine preferred states for values i n the slant sign stream. These are shown i n figure 4.8. Since t h e corners are assumed to be rectangular, the convex edges i n arrow- and Y - j u n c t i o n s correspond directly to edges t h a t are slanted away from the viewer, a n d non-convex edges to edges slanted towards the viewer. Consequently: A r r o w - j u n c t i o n s : Slant t o w a r d the viewer is preferred for the stems, while slant away is preferred for the outer wings. Y - j u n c t i o n s : Slant away from the viewer is preferred for aU hnes. O t h e r j u n c t i o n s do not contain enough information to determine slant sign directly, a n d so no preference can be assigned on their account. Convexity Subsystem â€¢ â€¢â€¢â€¢ Impossible Nonconvexity Subsystem Possible Preferred Slant Sign Subsystem - away â€¢ â€¢â€¢â€¢ Impossible Slant Sign Subsystem - toward Possible Preferred D . Slant Magnitude If an arrow- or Y - j u n c t i o n obeys P e r k i n s ' laws (section 3.2.4), the values i n its quantitative subsystem axe assigned the corresponding slant magnitudes, and the values i n the quahtatlve subsystem axe set to 'preferred' to show that a definite interpretation has been made. Oth- erwise, the m a g n i t u d e of the edges is set to a default value of zero, a n d the corresponding quahtatlve l a b e l is set t o 'possible' so that it can be overridden by any definite i n t e r p r e t a t i o n . 4.3 T h e Rapid Recovery Process Taken together, the external a n d i n t e r n a l constraints developed above go a long way towards specifying a m a p p i n g that allows a large amount of scene structure to be recovered i n very httle t i m e . M i n i m a l assumptions have been made about processing resources â€” it is assumed only that a mesh of relatively simple processors is available, a n d that the time required for l o c a l c o m p u t a t i o n is less t h a n that of d a t a transmission to nearby locations. Consequently, these constraints are largely independent of the details of the u n d e r l y i n g mechanism. If the theory is to be complete, however, it must lead to a m a p p i n g that is uniquely specified. Several architectural parameters must therefore be specified. It must also be shown how the e x t e r n a l a n d internal constraints can be smoothly combined into a r a p i d recovery process that is robust to small perturbations i n the i n p u t . 4.3.1 Architectural Specifications T h e constraints developed i n the previous sections have the advantage that they are apphcable t o a range of possible processors. Because they depend on a few aspects of the processor, however, these aspects must be given a definite specification i f the resultant m a p p i n g is t o be unique. T h e choices made here are intended to be as general as possible, and to reflect what is k n o w n of the h u m a n visual system when the specification of p a r t i c u l a r parameters is unavoidable. To begin w i t h , the processing elements are assumed to be finite-state, m a k i n g it necessary to convert continuous quantities such as two-dimensional orientation a n d slant magnitude i n t o discrete f o r m . S p a t i a l location must be represented w i t h a high degree of precision, reflecting the h i g h acuity possible even at early stages i n h u m a n vision (see, e.g., [WB82]). E a c h cell is therefore assumed to be able to represent location to w i t h i n l / 1 6 t h of its own size.^^ O n the other h a n d , the measurement of hne orientation i n the early visual system is based o n channels of a half-amphtude b a n d w i d t h of about 1 0 - 2 0 Â° [TG79], and so is much less precise. Consequently, orientation measurements are quantized to intervals of 10Â°. T h e estimates of slant magnitude must also be quantized. Like two-dimensional orientat i o n , these are given a relatively coarse-grained representation, w i t h magnitude quantized t o intervals of 20Â°, centered around values of 0Â°, 20Â°, 40Â°, 60Â°, and 80Â°, A n o t h e r issue is the way i n which the zones can be arranged over the image. Three m a i n types of regular tesselation are possible: rectangular, triangular, and hexagonal. T h e p a r t i c ular choice does not greatly m a t t e r when processing does not involve coordinate-dependent quantities, but this must be made definite for purposes of analysis. In order to simphfy the i m p l e m e n t a t i o n as m u c h as possible, it is assumed that aU. zones have the same shape a n d size, a n d that they f o r m a rectangular lattice over the image. T h e coordination of communication between zones must also be specified. Processing over each zone is carried out by a separate processor or group of processors, a n d c o m m u n i c a t i o n between these processors may proceed either synchronously (coordinated by a global clock) or asynchronously. Since the process acts v i a an irreversible priority override m e c h a n i s m , and since the available propagation paths are c o n s t a n t , p r e c i s e t e m p o r a l coordination of operation is not i m p o r t a n t . Consequently, the issue of synchronous communication has h t t l e i m p a c t on the performance of the process. T h e m a j o r difference between the two types of c o m m u n i c a t i o n is therefore i n the ease of implementation and analysis. In what foUows, synchronous c o m m u n i c a t i o n is assumed. F i n a l l y , an appropriate size must be chosen for the zone themselves. T h i s depends i n part on the absolute number of available processors, or more precisely, on the r a t i o of processors to the size of the i n p u t . It is assumed here that each zone can be made small enough to contain at most three hnes (i.e., enough for a single j u n c t i o n ) . B e y o n d these requirements, the exact size of the zones is u n i m p o r t a n t for present purposes â€” since processing speed is d o m i n a t e d by transmission time (section 2.5.2), changing the size of the zones only leads to Since each cell is later assumed to correspond to a visual area of roughly 10 min arc (section 5.3), this yields an precision of less than 1 min arc, roughly comparable to the limits of human visual acuity [WB82]. ^^State-dependent constraints are similar, the only difference being that a delay is introduced by the re- quirement that a definite set of labels be assigned to the critical variables. a rescaling of the time course of the process. 4.3.2 Robustness T h e assumption of rectangularity carries w i t h it an obhgation to protect the process f r o m the instabihties that result when hnes i n the image are nearly paraUel or are nearly at right angles to each other. For arrow- a n d Y - j u n c t i o n s , techniques similar to those of section 3.3.3 c a n be apphed i n a s t r a i g h t f o r w a r d fashion, at least locaUy. In p a r t i c u l a r , an arrow- or Y - j u n c t i o n containing a 90Â°angle is treated as if the angle were shghtly larger. Since a g l o b a l broadcast of the reassigned angles is not feasible using a mesh architecture, ambiguous L - j u n c t i o n s cannot be immediately resolved. T h e y are consequently t r e a t e d here as junctions containing constraints c o m m o n to b o t h acute and obtuse L-junctions (see, e.g., [Mal87]). One such constraint is that at least one edge must be nonconvex (see figure 4.2). T h e sensitivity of slant magnitude estimation can be reduced by a few a d d i t i o n a l measures. For junctions i n clear v i o l a t i o n of P e r k i n s ' laws (section 3.2.4), edges are given no i n i t i a l preferred slant magnitude (i.e., the values i n the qualitative system are set to 'possible'). C o n s t r a i n t s are also weakened so that neighboring estimates are acceptable only if they are w i t h i n adjacent ranges. FinaUy, to Umit the accumulation of errors that would result if estimates of slant magnitude were propagated v i a L - j u n c t i o n s , estimates are taken only f r o m direct sources (i.e., at the arrow- and Y - j u n c t i o n s ) , w i t h values propagated only as far as the next j u n c t i o n . 4.3.3 Basic Operation G i v e n the a d d i t i o n a l refinements of the sections 4.3.1 a n d 4.3.2, the process is completely specified. Since m a n y of the constraints apply to the generation of the interpretation, and not simply its final f o r m , the image-to-scene m a p p i n g cannot be given a closed-form description, fnstead, the i n t e r p r e t a t i o n of a given hne drawing can only be obtained by carrying out the process itself. T h e detailed operation of the recovery process is discussed i n chapter 5, where an algo^^Note that the absolute scale is important for any real system, leading to a preference for cell sizes that are as large as possible. T h u s , the absolute size of a cell involves a time-space trade-off (cf. section 5.1.3): a larger number of smaller, simpler cells increases computational simphcity, while a smaller number of larger, more complex cells reduces transmission time (when internal transmission is not a factor). T h e choice of appropriate size is hkely to be based on some compromise between these two sets of conflicting requirements. r i t h m is developed that embodies a l l the relevant external and internal constraints. H o w e v e r , the recovery process itself is shghtly more abstract t h a n this, since it is completely specified w i t h o u t the a d d i t i o n a l details of the a l g o r i t h m . T h e basic elements of its operation, t a k i n g place i n each zone concurrently, are as follows: 1. I n i t i a l measurements are made of the t e r m i n a t i o n locations and the orientations of the hne segments w i t h i n the zone. Terminations include not only true endpoints of the hnes, but also crossings of the zone boundaries. T h e locations of these t e r m i n a t i o n s are represented w i t h high precision ( 1 / 1 6 of the zone size). O r i e n t a t i o n measurements are quantized i n units of 10Â°. 2. T h e t y p e of j u n c t i o n present (if any) is the zone is estabhshed, a n d the angles between its hnes determined. 3. I n i t i a l interpretations are assigned t o aU variables i n ah. substreams. If the zone contains one or more disconnected hnes, a l l values are assigned 'possible'. If it contains a j u n c t i o n , the hnes are labeUed according to the rules described i n section 4.2.2. T h i s is done separately for the values a n d complementary values i n each of the streams. 4. Values are propagated along connecting hnes to neighboring zones v i a the p r i o r i t y m e c h a n i s m described i n section 4.2.2. T h i s is done i n t a n d e m for b o t h subsystems i n a l l streams. Since c o m m u n i c a t i o n is only possible between zones that are i m m e d i a t e neighbors (section 4.2.1), this leads to a percolation of information along the hnes at a constant speed. P r o p a g a t i o n of labels proceeds by assigning 'impossible' states t o eUminate inconsistent interpretations, a n d by assigning 'preferred' labels to select a preferred subset of the r e m a i n i n g possibihties. 5. Simultaneous w i t h this " i n t r a - d i m e n s i o n a l " process, an " i n t e r - d i m e n s i o n a l " p r o p a g a t i o n is also o c c u r r i n g , t r a n s m i t t i n g i n f o r m a t i o n from zones that contain a variable w i t h a definite i n t e r p r e t a t i o n . T h i s transmission appUes only to zones at the same l o c a t i o n i n the image, and foUows the rules given i n figure 4.2. 6. T h e transmission of i n f o r m a t i o n along Unes and between dimensions continues u n t i l the t i m e h m i t is reached. Inconsistent interpretations are identified by the assignment of 'impossible' to an edge i n b o t h subsystems. A m b i g u o u s interpretations are identified by the assignment of 'possible' i n b o t h subsystems. O f the remaining interpretations, those deemed to be most Ukely are distinguished by the 'preferred' state. A n example of this process is shown i n figure 4.9, which iUustrates how the i n i t i a l convexity estimates assigned to a d r a w i n g evolve into a more complete i n t e r p r e t a t i o n . Convexity (+) jQzMagnitude Nonconvexity (o) iiiii Impossible Â«ftssÃ¯Possible ^â€¢â€¢Preferred Narrow gray lines mark cell boundaries Chapter 5 Algorithm and Implementation T h e c o m p u t a t i o n a l analysis of chapter 4 has yielded a set of external constraints o n the " s t a t i c " associations between image and recovered scene, and a set of internal constraints o n the " d y n a m i c " aspects of the recovery process itself. These specify a unique image-to-scene m a p p i n g , a n d provide some general hmitations on the transformations that are to be used. W h a t is now required is to show that these constraints can be incorporated into a complete, well-defined system. In p a r t i c u l a r , the process must be decomposed to the point where it can be carried out v i a the operations available on a device h a v i n g the processing h m i t a t i o n s assumed i n the c o m p u t a t i o n a l analysis (section 2.4). T h e analysis here is based on a device called the cellular processor. T h i s is a t y p e of cehular a u t o m a t o n (section 5.1.2) formed by p a r t i t i o n i n g a dense mesh of processors i n t o a relatively sparse set of disjoint " c e l l s " , each of which is assigned a simple processing element to carry out the l o c a l interpretations. It is shown that the basic operations of this mechanism can be implemented on a mesh of simple finite-state processors. T h e a l g o r i t h m itself is t h e n developed v i a a simple p r o g r a m based on these basic operations. T h e general properties of this mechanism are shown to be compatible w i t h what is k n o w n of p r i m a t e cortical structure, and a tentative suggestion put forward regarding the way i n which it might be implemented i n h u m a n visual cortex. 5.1 The Cellular Processor If it is to be effective, a r a p i d recovery process must be based on estimates made over regions of the image that are contiguous and compact, i.e., over zones (section 4.2.1). T h i s introduces two different s p a t i a l scales into the recovery process: (i) a fine-grained scale that supports the F i g u r e 5.1: Cellular processor architecture. high resolution of the i n p u t a n d output representations, and (ii) a coarser-grained scale based on the size of the zone. A useful mechanism to handle this situation is the cellular T h i s is a device consisting of two spatiotopic meshes: (i) a dense mesh of processor. measurement elements that determine basic image properties (e.g., color, contrast, and orientation), a n d a sparser mesh of more complex control elements that carry out the l o c a l interpretations (figure 5.1). 5.1.1 B a s i c aspects T h e cellular processor allows a l g o r i t h m i c analysis to be carried out i n a straightforward way, w i t h issues of measurement a n d control separated as much as possible. E a c h measurement element ( M E ) can be loosely identified w i t h a mechanism that measures some t e m p l a t e p r o p e r t y , such as the color or orientation of hnes. These M E s are assumed to have a small set of possible o u t p u t values that are determined entirely by the contents of a s p a t i a U y - h m i t e d neighborhood a r o u n d the corresponding point i n the image. A s such, they have no i n t e r n a l states a n d operate independently of each other. T h e spatiotopic order of the set of inputs is assumed to be m a i n t a i n e d i n the set of M E o u t p u t s , so that the array of M E s and the array of their outputs can b o t h be referred to as the "measurement l a y e r " , the d i s t i n c t i o n between the M E s and their outputs being clear f r o m context. A d j a c e n t elements i n this layer may or may not have overlapping i n p u t regions. It is assumed that the density of M E s is sufficiently high that that no i n f o r m a t i o n i n the image is lost.^ ^This layer has some interesting similarities with the dense set of localized filters found in the striate cortex T o carry out more complex operations, the measurement layer is p a r t i t i o n e d into a n u m ber of compact, contiguous sections (or "ceUs"),^ w i t h the outputs i n each ceU assigned t o a control element ( C E ) at the corresponding location i n a higher-level " c o n t r o l l a y e r " . E a c h C E is assumed to be sufficiently complex that it can respond to a l l possible combinations of outputs i n its ceh. Towards this end, each C E is given a small finite memory to hold i n t e r m e diate quantities derived f r o m the M E outputs (e.g., the number of hne segments it contains, their l o c a t i o n , a n d the areas of any region bounded by t h e m ) . Note that these quantities need not be determinate â€” i n situations where space or time is extremely h m i t e d , or where there is some inherent uncertainty i n the measurements themselves, statistical quantities m a y be preferred (see, e.g., [Ros86]). ft also is assumed that each C E can control at least some of its M E s v i a backprojections that override the M E output.'^ In a d d i t i o n , each C E is assumed to have a small set of operations that it can perform on its m e m o r y locations and on a its M E s . These operations f o r m the basis of the l o c a l processing carried out by the processor. In contrast to the isolated elements of the measurement layer, elements i n the c o n t r o l layer are able to interact w i t h their nearest neighbors, h a v i n g access to at least some aspects of their neighbor's current state. T h i s adds a degree of " l a t e r a l " control to the " b o t t o m - u p " and " t o p - d o w n " strategies generally employed i n visual processing. 5.1.2 C e l l u l a r P r o c e s s o r s as C e l l u l a r A u t o m a t a Since each M E is an isolated mechanism performing a single operation, a l l interesting aspects of the recovery process are carried out by the processors i n the control layer. Consequently, the evolution of a cellular processor can be completely described by a rule that maps the current state of each c o n t r o l element onto a new state, the new value being determined by (i) the outputs of the M E s w i t h i n its ceh, (u) the contents of its memories, a n d (in) the states of its i m m e d i a t e neighbors. Since processing must be indifferent t o the absolute spatial coordinates i n the image, this m a p p i n g must be spatially u n i f o r m . F u r t h e r m o r e , the process is assumed t o operate v i a synchronous c o m m u n i c a t i o n between ah zones (section of primates (see section 5.3). ^ T h e meaning of the term 'cell' corresponds to that of 'zone', but at the level of architecture rather than that of image. ^This can be accomplished by special internal memories, each capable of overriding the outputs of one particular M E . In this formulation, the output of the C E can be expressed either as the set of M E outputs or as the set of G E memory states. 4.3.1). Described i n this way, a ceUular processor (or more precisely, its control layer) is seen to be a special type of ceUular automaton. CeUular a u t o m a t a are discrete deterministic systems formed by a cÃ®-dimensional g r i d of i d e n t i c a l processors operating according to a fixed l o c a l law. M o r e precisely, a ceUular a u t o m a t o n ( C A ) can be defined as a quadruple [Kar90] (5.1) A = {d,S,NJ), where is a positive integer describing the dimension of A , 5 is a finite set of states, is a set of n neighborhood vectors (each of the f o r m x â€” {xi, ...Xd)), and / is the local t r a n s i t i o n f u n c t i o n f r o m 5 " to 5 . T h e ceUs of A are arranged along an infinite c?-dimensional g r i d , their positions indexed by elements Z'^, the rf-dimensional space of integers. CeUular a u t o m a t a were developed originaUy by U l a m and von N e u m a n n as tractable a p p r o x i m a t i o n s of highly nonhnear differential equations i n biological systems (see [ T M 9 0 ] ) . B u t they are also interesting i n their own right, since local rules can lead to a variety of complex, spatiaUy-extended structures (see, e.g., [ C H Y 9 0 , Smi90, T M 9 0 ] ) . CeUular a u t o m a t a have been used for simple image operations, including thresholding, pointwise a r i t h m e t i c o n image p a i r s , a n d convolution (see, e.g., [Gol69, P D L + 7 9 , Ros83]). O t h e r operations include the s h r i n k i n g a n d expansion of elements i n the image, and the formation of their convex huU [ P D L + 7 9 ] . Indeed, it is Ukely that C A s can do considerably more t h a n this, since given the a p p r o p r i a t e t r a n s i t i o n functions a n d i n i t i a l configurations, they are capable of universal c o m p u t a t i o n , i.e., c o m p u t i n g any function that can be computed by a T u r i n g machine (see, e.g., [ C H Y 9 0 ] ) . In order t o conform w i t h the general constraints of the recovery process, a two-dimensional g r i d is used, a n d the neighborhood set N is the set of ceUs at most a unit distance away i n the h o r i z o n t a l or v e r t i c a l direction.'' T h u s , the neighborhood is composed of nine ceUs: the ceU itseK, a n d a layer formed by overlapping 3 x 1 arrays of ceUs i m m e d i a t e l y to the t o p , b o t t o m , r i g h t , a n d left. Consequently, the t r a n s i t i o n function / is described by a m a p p i n g ^ S that associates each possible p a t t e r n of neighborhood states to the new state of the center ceU. â€¢*A rectangular tesselation is not necessary for cellular automata that operate on images â€” several applications (e.g., [Gol69]) are based on a hexagonal array. Indeed, any C A vs^ith an arbitrary neighborhood N is equivalent in its computing power to one with a von Neumann neighborhood, i.e., one with neighbors to the top, bottom, left, and right (see [PDL+79]). 5.1,3 Programming A . Basic Considerations V i e w i n g tlie control layer of a cellular processor as a cellular a u t o m a t o n , its p r o g r a m m i n g reduces to the design of an appropriate transition function and selection of an a p p r o p r i a t e i n i t i a l configuration of values. There are, however, three i m p o r t a n t constraints p a r t i c u l a r to its o p e r a t i o n . F i r s t , to rule out the necessity for any k i n d of higher-level global mechanism, the i n i t i a l value of each C E must be determined entirely by the M E s w i t h i n its corresponding ceh. T h i s means that the i n i t i a l configuration of values must be i n spatial register w i t h the i n p u t image, thereby p r o h i b i t i n g the use of the special-purpose i n i t i a l patterns or auxihary elements often used i n general C A design. Similarly, the final configuration also is required t o be i n register w i t h the image, since the output is required to be a spatiotopic m a p . T h i s rules out algorithms that deform the spatial organization found i n the i n p u t , such as the s h r i n k i n g process used to count the number of items i n an image [Gol69]. F i n a l l y , the operation of the processor itself must be in-place, i.e., the memory i n inactive ceUs cannot be used as scratch space for intermediate calculations. T h e use of scratch space is a viable option when the i n i t i a l configurations are such that k n o w n subsets of the g r i d can be guaranteed to r e m a i n inactive (see, e.g., [ A r b 8 7 , ch. 7]). However, this condition cannot i n general be met when a n a r b i t r a r y set of i n p u t images (and therefore Initial configurations) is possible. T h e power of a cehular architecture cannot therefore be harnessed i n the manner used for m a n y classes of general C A problems, v i z . , by designing an appropriate i n i t i a l configuration. Instead, the appropriate i n f o r m a t i o n must be stored locally i n each ceh. T h i s can be done by increasing the n u m b e r of states i n S (i.e., increasing the number of states i n each c o n t r o l element). Increasing power i n this way also ahows the t r a n s i t i o n function to have a more n a t u r a l structure, simphfying the design a n d analysis of the system's behavior [Arb87, ch. 7]. A t the lowest possible level, therefore, the p r o g r a m m i n g of a cehular processor reduces to the selection of a set of states for each C E , together w i t h a t r a n s i t i o n function that operates on these states. B u t to help ensure that the processor respects the constraints described above, it is convenient to p r o g r a m at the shghtly higher level of simple operations on p a r t i c u l a r properties accessible by the C E . Once such a set of operations has been specified, any p a r t i c u l a r recovery process can then be specified by the appropriate concatenation of operations. T h i s is effectively a general mechanism for the "abstract p r o g r a m m i n g " of parallel processes, with the resultant p r o g r a m loaded into each of the C E s , where it acts somewhat hke paraUehzed version of a v i s u a l routine [UU84]. B . E l e m e n t a r y Structures a n d Operations T h e d a t a structures to be used i n p r o g r a m m i n g the control elements are straightforward: the values of the M E s i n the corresponding ceU, the contents of the internal C E memories, a n d the accessible properties of the adjacent C E s . These are a l l simple scalars, w i t h only a s m a l l set of possible values. A s such, they can be handled i n a u n i f o r m way. M o r e l a t i t u d e exists i n the choice of elementary operations. There is i n some sense a " n a t u r a l " set of basic operations â€” if too few exist, it m a y not be possible to carry out a h the intra-ceU operations w i t h i n a single time step; if too m a n y exist, they merely add to the space required by the C E . T h e elementary operations chosen here are simple forms of d a t a input, output, and transformation: 1. Input o f information f r o m M E s to m e m o r y elements. C o n n e c t i v i t y w i t h i n a ceh is assumed to be high enough to aUow a C E to estabhsh direct access from any M E to any of its i n t e r n a l m e m o r y elements. 2. O u t p u t of information f r o m m e m o r y elements to M E s . C o n n e c t i v i t y also is assumed h i g h enough to allow a C E to estabhsh backprojections f r o m any of its i n t e r n a l m e m o r y to at least some of its M E s . T h e interpretation output by the processor takes the f o r m of values of these latter M E s (or equivalently, of the corresponding m e m o r y elements that override t h e m ) . 3. S i m p l e operations o n information in m e m o r y elements. It is assumed that each C E can a d d , s u b t r a c t , m u l t i p l y , a n d perform integer division on the contents of the m e m o r y elements. It also is assumed that a two numbers can be compared t o determine the higher value. Inputs and outputs for these operations are always t a k e n f r o m the m e m o r y elements; transfer of contents between m e m o r y elements is s i m p l y a special case where no operation is performed. In a d d i t i o n , each C E is assumed to have an input from higher levels that provides a simple c o n t r o l o n its operation. Depending on the value of this signal, the C E either resets its memories to some default state, begins/continues its operation, or halts its operation. c. C o m b i n i n g Basic Operations T h e cehular processor is p r o g r a m m e d by creating an appropriate t r a n s i t i o n function a n d set of states f r o m the elementary structures and operations described above. T h i s can be done most simply by concatenating elementary instructions together into a sequence, an o p e r a t i o n which corresponds to the composition of the corresponding t r a n s i t i o n functions. B o t h simple and c o m p o u n d operations can be concatenated into new c o m p o u n d operations. N o t e t h a t the resultant t r a n s i t i o n need not be carried out i n a sequence of separate transitions â€” it can be "fused" into a more complex function that can be carried out i n one step. T h e replacement of a sequence of simple operations by a single t r a n s i t i o n corresponds to the use of a lookup table (cf. section 7). In this sense the process is consistent w i t h the loading-in of a complete object m o d e l based on its features i n the image (e.g. [PE90]). However, the approach here involves items of a smaller " l o c a l models" composed entirely of locally-definable properties. Note that the issue here here centers a r o u n d the advantages of a larger sequence of simple transitions as opposed to a smaller sequence of more complex transitions â€” an instance of the basic time-space tradeoff found i n more general models of c o m p u t i n g (see, e.g., [Har87]). Operations can similarly be combined v i a the " i f - t h e n " conditional construct, the result simply formed f r o m the two alternative functions. T h e loop construct of conventional p r o g r a m m i n g languages also is aUowed, but only if the b o d y of the loop is carried out a h m i t e d number of times. A s used here, the loop is a simple p r o g r a m m i n g convenience, w h i c h is " u n r o h e d " i n the a c t u a l i m p l e m e n t a t i o n of the t r a n s i t i o n function. A loop controUed by a variable can be t r a n s l a t e d i n t o several separate unroUed loops, w h i c h are then selected v i a c o n d i t i o n a l constructs. In a similar fashion, procedures can also be used to help specify the process, but each is t o be treated as a macro that is replaced i n the a c t u a l t r a n s i t i o n function by the set of instructions it contains. A s such, procedures cannot call each other recursively. FinaUy, the p r o g r a m given to the ceUular processor does not need an exphcit ' h a l t ' comm a n d , since it is assumed t h a t the processor is suspended (as weU as started) by an exphcit c o m m a n d from higher levels. 5.2 Algorithm for Rapid Recovery G i v e n the set of operations available to the cehular processor (section 5.1.3), it remains t o use these as the basis of an a l g o r i t h m capable of c a r r y i n g out the recovery process sketched i n section 4.3. A l t h o u g h the constraints on the recovery process a n d on the ceUular processor are not sufficient to specify a unique a l g o r i t h m , this is not i m p o r t a n t for the present p u r p o s e , w h i c h simply is to show t h a t such an algorithm can exist. T h e algorithm used here can be s u m m a r i z e d as foUows: F o r each c o n t r o l element: 1. O b t a i n f r o m the measurement elements the locations of aU Une terminations a n d the orientations of aU Une segments w i t h i n the ceU. Terminations include not only t r u e endpoints of the Unes, but also points at w h i c h the zone boundaries are crossed. As required by the specifications of section 4.3.1, orientation measurements are q u a n t i z e d i n units of 10Â°. 2. Determine the type of j u n c t i o n ( i f any) that is present, and make exphcit several of its properties, such as the values of the angles involved. 3. A s s i g n i n i t i a l labels to the hnes according to the rules described i n section 4.2.2. 4. F o r each subsystem of each stream, repeat the foUowing: a. R e a d the relevant values from any neighboring C E that shares one of the hnes. U p d a t e the current values v i a the priority mechanism described i n section 4.2.2. b. R e a d the relevant values from those streams containing a variable w i t h a definite i n t e r p r e t a t i o n , a n d update the current values according to the rules given i n figure 4.2. Since this transmission apphes only to zones at the same l o c a t i o n i n the image, only the i n t e r n a l memories of the C E are involved. c. A p p l y the intra-hne constraints according to the rules given i n figure 4.2.2. These eUminate any l o c a l inconsistencies that may have arisen i n the new set of values. 5. Stop i t e r a t i o n when the t i m e h m i t is reached. T h e final interpretations are determined f r o m the assignment of the 'possible', 'preferred', and 'impossible' labels i n each subsystem, according to the rules given i n section 4.2.2. T h e foUowing sections describe i n greater detail how these operations are carried out by the ceUular processor. 5.2.1 D e t e r m i n a t i o n of B a s i c I m a g e P r o p e r t i e s T h e measurement elements i n each ceh describe the image basic properties available to the c o n t r o l element. These include the locations of the hne terminations a n d the orientations of the hne segments i n the area subtended by the ceh. There are a variety of ways this can be done. Here, each M E is assumed to signal the existence of a hne centered at the corresponding l o c a t i o n i n the image array. L i n e segments of different orientation, h o r i z o n t a l l e n g t h , a n d v e r t i c a l length are represented by different sets of M E s , each signalling the presence or absence of its p a r t i c u l a r type of segment by a simple b i n a r y o u t p u t . A s required by the architecture specifications given i n section 4.3.1, these elements represent length a n d position to a h i g h degree of precision, w i t h orientation represented only coarsely. These outputs contain a complete (in fact, redundant) description of ah hne segments i n the ceh, a n d can therefore support the determination of a l l the image properties needed by the c o n t r o l element. Three properties are of p a r t i c u l a r interest, a l l of w h i c h are represented v i a a b a n k of Unite-state m e m o r y elements: N u m b e r of lines i n the cell: T h i s can be determined from a count of the number of M E o u t p u t s that are active. A m a x i m u m of three is assumed (section 4.3.1). T h e endpoints of each segment: These are calculated for each segment from the knowledge of the relevant center point and the h o r i z o n t a l and v e r t i c a l lengths. No more t h a n six endpoints need to be stored. T h e orientation of each segment: These can be taken directly from the orientation l a b e l of the appropriate M E . N o more t h a n three values need to be stored. 5.2.2 D e t e r m i n a t i o n of J u n c t i o n P r o p e r t i e s T h e next step is to o b t a i n those properties of the j u n c t i o n useful for subsequent i n t e r p r e t a t i o n . These only need to be calculated once, their values then stored i n an appropriate bank of m e m o r y elements. F i v e p a r t i c u l a r sets of properties are used here: j u n c t i o n position, j u n c t i o n angles, j u n c t i o n type, j u n c t i o n rectangularity, and an auxihary set of hne descriptions (two for each hne) used for the i n t e r p r e t a t i o n of contiguity. A s required by the recovery process, a l l quantities are finite. A . Junction Position Junctions are detected simply by finding the intersection point of the hnes i n the cell. It is assumed t h a t each ceU is sufficiently small that at most one j u n c t i o n (and therefore one intersection point) can exist w i t h i n the area it subtends. T h e existence of the intersection point is determined by testing for the identity of the endpoints. For the case of T - j u n c t i o n s , a shghtly different procedure is used, based on a unique zero distance from an endpoint of one hne segment to a different hne segment. If no intersection point is found, no j u n c t i o n exists w i t h i n the ceU. Note that this is possible even i f the j u n c t i o n contains several hnes, since these hnes may be non contacting. If an intersection point is f o u n d , its location is stored into an appropriate memory element. B . Junction Angles T h e angle 9ij between each pair of connected hnes Si and Sj is simply the absolute value of the difference of the two orientations. T h e only real difficulty here is to determine h o w the hnes are connected - as seen from figure 5.2, each pair of Unes can be combined i n t w o different ways, corresponding to acute a n d obtuse forms. These can be distinguished v i a the dot product of the two hnes, defined to be (see, e.g., [Tho72]) cos(^ij) = (aÂ» â€¢aj)/\ai\\aj\. T h e d i s a m b i g u a t i o n of acute a n d obtuse junctions can be based on the sign of the cosine: positive for acute angles, negative for obtuse. If the difference between two hne orientations actually corresponds to angle 9ij, it wiU therefore have a value between 0Â° â€” 90Â° for pairs w i t h a positive dot p r o d u c t , a n d between 90Â° - 180Â° for a negative dot product. If these conditions do not h o l d , the angle must be 180Â° minus this value (figure 5.2). Since o n l y the sign of the dot product is i m p o r t a n t , division by the magnitudes need not be p e r f o r m e d , and so can be readily carried out by the control element. N o t e also that the dot product is a true scalar q u a n t i t y (see, e.g., [Tho72]), so that no artifacts are i n t r o d u c e d by the selection of any p a r t i c u l a r co-ordinate system. A m o n g other things, this takes care of any problems introduced by the discontinuity i n orientations at 180Â°. It also means t h a t o r i e n t a t i o n can be taken w i t h reference to any co-ordinate system, the only requirement being that the same system is used locally for any j u n c t i o n . Figure 5.2: C a l c u l a t i o n of orientation differences. C. Junction Type Junctions are classified by a two-stage process. T h e i n i t i a l classification based on their arity, i.e., the number of hnes existing at the c o m m o n intersection point. T h i s value can be o b t a i n e d simply by counting the hnes that have an endpoint identical w i t h the intersection point. Junctions are then m a r k e d as foUows: N o junction: T-junction: N o intersection point exists One endpoint contacts the intersection point L - j u n c t i o n : T w o endpoints contact the intersection point Arrow-, Y-junction: Three endpoints contact the intersection point F u r t h e r disambiguation can be based on the values of the angles between the hnes: L - j u n c t i o n (acute): angle is less t h a n 90Â°) L - j u n c t i o n (obtuse): angle is greater t h a n 90Â°) Arrow-junction: Y-junction: angles s u m to less t h a n 360Â° angles sum to exactly 360Â° C o m p h c a t i o n s can arise when j u n c t i o n angles are nearly orthogonal, since the uncertainty i n the sign of the dot product makes it difficult to discriminate acute L-junctions f r o m obtuse ones i n a rehable way. It also becomes difficult to distinguish arrow- f r o m Y - j u n c t i o n s if two such angles are present i n a j u n c t i o n (i.e., one of the hne pairs is almost coUinear w i t h a n other). Various techniques can lend robustness to the recovery process under these conditions (cf. section 3.3.3), but i n the interests of simphcity, only a few are used here (section 4.3.2). In p a r t i c u l a r , L - j u n c t i o n s w i t h angles determined to be 90Â° (based on the quantized estimates i n m e m o r y ) are treated as a separate type of L - j u n c t i o n that has the constraints c o m m o n to b o t h obtuse a n d acute L - j u n c t i o n s . A s for the other kinds of L-junctions, right-angled L - j u n c t i o n s can be determined by a simple test of j u n c t i o n angles. D . Junction Rectangularity A n i m p o r t a n t basis for the recovery of slant magnitude is the assumption that the j u n c t i o n corresponds t o a rectangular corner i n the scene, w i t h slant magnitudes assigned only t o those edges belonging to a j u n c t i o n obeying P e r k i n s ' laws (section 4.3.2). Consequently, it is i m p o r t a n t t o indicate whether or not a j u n c t i o n can correspond to such a corner. T h i s can be done v i a a simple test based on the angles a n d type of the j u n c t i o n : L-junction: no T-junction: no A r r o w - j u n c t i o n : one angle > 90Â° a n d two angles < 90Â° Y - j u n c t i o n : three angles > 90Â° R e c t a n g u l a r i t y is flagged simply by assigning a zero value to aU angles i n junctions that do not pass this test. In the interests of robustness, this procedure must be extended to handle right angles as weU. N o t e t h a t since two angles of 90Â° i n a j u n c t i o n form a T - j u n c t i o n , only one right angle is allowed i n any arrow- or Y - j u n c t i o n , aUowing the extension to be done i n a simple way: A r r o w - j u n c t i o n : one angle > = 90Â° a n d and two angles < = Y - j u n c t i o n : three angles > = 90Â°. 90Â°. E . C o n t i g u i t y Lines In contrast to the other interpreted properties, the contiguity of Unes w i t h their flanking regions requires the assignment of two values per hne, one for each side (section 3.2.1). C o n t i g u i t y is therefore represented here by a pair of contiguity hnes ("c-hnes") obtained by Inner contiguity lines in same direction as adjacent junction line Contiguity lines on both sides of each junction line F i g u r e 5.3: D e t e r m i n a t i o n of contiguity relations. offsetting the original " p a r e n t " hne a few pixels on either side (figure 5.3). T h e value of each c-hne indicates whether its corresponding region is contiguous w i t h the parent hne i n the j u n c t i o n . In a l l respects, c-hnes are treated as regular j u n c t i o n hnes, t a k i n g on the states of 'possible', 'preferred', a n d 'impossible'. Because several constraints apply to c-hnes that share a common region, it is useful t o have a record of w h i c h c-hnes are connected to each other inside the j u n c t i o n . Since almost all connected c-hnes are the inside edges of adjacent hnes (cf. section 3.2.1), the test for connectedness reduces almost completely to a search for these inside edges.^ T h e p r o b l e m , t h e n , is t o determine w h i c h c-hnes are on the " i n s i d e " , i.e., which c-hne faces the j u n c t i o n hne opposite its parent (figure 5.3). A simple way t o solve this problem is based on the cross product, which for hnes a a n d b is defined as the determinant (see, e.g., [Tho72]) a X 6 = ex ey ez aa: ay a^ bx by bz = |a||6| sin(6'a6)ez, where the ej are unit vectors i n the x, y, and z directions. In the case of two dimensions, the cross product describes the area swept out by the two vectors, its sign depending on the sense of the r o t a t i o n required to ahgn a w i t h b. Consider first one of the j u n c t i o n hnes. T h e cross product of this hne w i t h an adjacent j u n c t i o n hne can be readily determined by the control element, the sign of this q u a n t i t y describing the sense (either clockwise or counterclockwise) in which this hne must be r o t a t e d ^ T h e only exception is for the outer edges of the arrow-j unction, and these can be handled straightforwardly. to line up w i t h the adjacent hne. Consider now the associated c-hnes, together w i t h the hnes formed by j o i n i n g their outer points to the j u n c t i o n intersection. O n l y the c-hne o n the inside edge (i.e., the c-hne facing the adjacent j u n c t i o n hne) can give rise to a hne i n t h e same r o t a t i o n a l direction as that of the adjacent j u n c t i o n hne (figure 5.3). R e p e a t i n g the same procedure for the c-hnes of the adjacent hne then yields the pair of inside edges. T h e cross product is a quantity independent of the co-ordinate system (see, e.g., [Tho72]). It therefore allows processing to be unaffected by the particular co-ordinates used i n representing the orientation of the hnes. 5.2.3 Initial Assignment of Interpretations Once the basic image a n d j u n c t i o n properties have been determined, the next step is to assign the i n i t i a l interpretations to the variables i n each of the four streams. T h e states of these variables aie held i n eight separate banks of memory elements, one element per subsystem. O n l y three possible values can be attached to any complementary variable, and only five are possible for slant m a g n i t u d e (section 4.3.1). Consequently, these m e m o r y elements only need to take on a few possible states. T h e assignment operation Itself is a straightforward procedure that sets the values of the relevant m e m o r y elements, the p a r t i c u l a r choice of values depending only on the j u n c t i o n t y p e (section 4.3). T h i s can be carried out by using a set of conditional statements. Together w i t h the values describing the structure of the junctions, the resulting set of C E m e m o r y states provides the i n i t i a l configuration for the iterative part of the interpretation process. 5.2.4 P r o p a g a t i o n of Interpretations G i v e n an array of i n i t i a l interpretations, it remains to transmit these values to neighboring locations a n d streams. A s discussed i n section 4.2.2, this is done by an iterative process t h a t at each i t e r a t i o n replaces values of low priority w i t h values of higher priority, i.e., 'preferred' replacing 'possible', a n d 'impossible' replacing 'possible' and 'preferred'. T h e constraints at each j u n c t i o n guide the l o c a l transmission of these values, resulting i n "waves" of higherp r i o r i t y states c i r c u l a t i n g a r o u n d the hnes i n the image. T h e propagation of these waves continues u n t i l an e q u i h b r i u m state is reached or u n t i l the process is t i m e d out. variables s h a r i n g constraints w i t h the " t a r g e t " variable are accessed, a n d if any of these has a value of higher p r i o r i t y t h a n the value of the target variable, the memory element is set t o this value. T h i s can be done even for the case of state-dependent constraints (section 4.2.2, since only an a d d i t i o n a l conditional construct is required to put the appropriate constraints into effect. I n f o r m a t i o n access occurs v i a four different avenues: neighboring cells, neighboring streams, i n t r a - j u n c t i o n constraints, a n d intra-hne constraints (section 4.3). Since transmis- sion is based o n a simple p r i o r i t y mechanism, the order of the access operations w i t h i n each stream is u n i m p o r t a n t . T h i s allows the propagation process to be carried out by a relatively simple set of operations. A . U p d a t e s f r o m N e i g h b o r i n g Cells T h e u p d a t i n g of values from sources outside the ceh can be done concurrently for each hne segment. To access the appropriate values from neighboring locations, first determine w h i c h neighbors contain a continuation of the relevant hne segment. T h i s is done by reading the set of endpoint locations stored i n each neighboring control element and testing for equality against the endpoints of the l o c a l hne segment. Since a hne crossing a ceU b o u n d a r y is divided into two segments that terminate at the same point (i.e., the b o u n d a r y ) , this test succeeds i f and only i f the segment continues into the neighboring ceh. For each ceU containing a continuation of the segment, access the relevant set of m e m o r y elements a n d compare their values against those of the current ceU. Since continuations are required to have the same values, u p d a t i n g foUows the rules for bijective constraints described i n section 4.2.2. B . Updates from Neighboring Streams Just as i n f o r m a t i o n is t r a n s m i t t e d from neighboring locations, it also is t r a n s m i t t e d f r o m neighboring streams. T h e only difference between the two situations is that whereas inter-ceU transmission is based simply on priority, inter-stream transmission usually has an a d d i t i o n a l dependence on the p a r t i c u l a r type of j u n c t i o n a n d on the p a r t i c u l a r Une i n that j u n c t i o n (section 4.2.2). T h i s dependence is fixed for each j u n c t i o n type, w i t h u p d a t i n g carried out by a set of c o n d i t i o n a l assignments between the appropriate variables. Since this u p d a t i n g is based o n simple priority, the order i n which streams are evaluated is u n i m p o r t a n t . c. Updates from Intra-junction Constraints A f t e r assigning new values to the hnes based on sources external to the j u n c t i o n , the next step is to impose the set of l o c a l constraints on the hnes of the j u n c t i o n itself. U p d a t i n g follows the rules described i n section 4.2.2, w i t h the p a r t i c u l a r constraints depending on j u n c t i o n t y p e . Consequently, it can be carried out by a set of conditional constructs. Since the f i n a l result depends only on p r i o r i t y of the values involved, the order of evaluation of the hnes is u n i m p o r t a n t , a n d can even be done i n parallel. D . U p d a t e s f r o m C o m p l e m e n t a r y Subsystems A final p a t h of Information transmission originates i n the constraint between the values i n complementary subsystems: i f any value i n a subsystem has been set to 'impossible', t h e value of its complement is upgraded to 'possible' (see section 4.2.2). Since this constraint holds for aU hnes at a l l times, it can be carried out v i a a conditional assignment i n c o r p o r a t e d i n t o the assignment mechanisms used i n the other access paths. 5.2.5 Final Assignment of R e s u l t s A f t e r the p r o p a g a t i o n of values has been h a l t e d , a final "postprocessing" phase can be used to t r a n s f o r m the states of the sets of complementary variables into a more " s t a n d a r d " representation that expresses the two definite interpretations, the inconsistent i n t e r p r e t a t i o n , a n d the ambiguous i n t e r p r e t a t i o n . T h e rules of this transformation are given i n section 4.2.2. T h e t r a n s f o r m a t i o n itself can be carried out straightforwardly on the relevant memory elements since only a simple r e m a p p i n g of values is involved. 5.3 Neural Implementation T h e final requirement of a c o m p u t a t i o n a l analysis is that it demonstrate the existence of a p h y s i c a l system capable of c a r r y i n g out the process â€” i n p a r t i c u l a r , one compatible w i t h the n e u r a l mechanisms beheved to underhe h u m a n vision [Mar82]. B u t the detailed knowledge about the neurophysiology of v i s i o n is h m i t e d mostly to processes that measure simple image properties such as contrast a n d orientation (see, e.g., [Bis84, Sch86]). A n detailed analysis of the n e u r a l i m p l e m e n t a t i o n of r a p i d recovery is therefore not possible at the present time. However, the ceUular processor developed i n section 5.1 is largely compatible w i t h w h a t is k n o w n about the a n a t o m y a n d physiology of p r i m a t e visual systems. T h e m a i n stream beheved to be involved i n form vision begins w i t h retinal ceUs that measure simple properties such as the contrasts and motions of luminance gradients i n the image. T h e outputs of these ceUs extend to the l a t e r a l geniculate nucleus, a n d the geniculate ceUs i n t u r n extend to the v i s u a l cortex, w i t h a spatiotopic order m a i n t a i n e d at aU points along the way (e.g., [Bis84]). T h e v i s u a l cortex serves as the location where the outputs of this stream are brought together w i t h those of the other streams. It contains ceUs sensitive to a variety of simple properties, i n c l u d i n g contrast, color, and hne orientation (e.g., [ d Y v E 8 8 ] ) . These f o r m a dense spatiotopic m a p , w i t h each point in the array containing a description of the various image properties at the corresponding point i n the i n p u t image. A s such, this map is an array similar i n m a n y respects to the measurement layer of the ceUular processor. T h e spatiotopic ordering of ceUs i n the p r i m a t e visual cortex is not quite p o i n t - t o - p o i n t . R a t h e r , it is " p a t c h - t o - p a t c h " , w i t h each set of ganghon ceUs i n a retinal p a t c h p r o j e c t i n g to a separate module (or " h y p e r c o l u m n " ) i n the visual cortex [ H W 7 4 , H u b S l , Bis84]. E a c h h y p e r c o l u m n is a v e r t i c a l section of the cortex w i t h an area of approximately 1 m m x 1 m m ; the p r i m a t e visual cortex is thought to have about 2000 such columns, each containing at least several t h o u s a n d ceUs [ H u b S l ] . T h e corresponding p a t c h of the visual field increases w i t h eccentricity f r o m the fovea, but around the fovea itself it has dimensions of about 10' arc (i.e., 1/6Â°) [ H u b S l ] . A U the measurements made over each patch are brought together i n the corresponding h y p e r c o l u m n , aUowing it to completely analyze its section of the visual field. A similarity w i t h the control layer of the ceUular processor is evident. T h i s similarity is reinforced by the observation that most connections between ceUs are v e r t i c a l ones w i t h i n the c o l u m n itseff, l a t e r a l connections to other areas being much sparser a n d shorter, often w i t h lengths of only 1-2 m m (i.e., extending only to nearest neighbors) [ H u b S l ] . H hypercolumns can be identified w i t h the control elements of the ceUular processor, it w o u l d i m p l y t h a t h y p e r c o l u m n operation is more sophisticated t h a n geheraUy beUeved. B u t such sophistication w o u l d not be implausible given the number of ceUs i n each h y p e r c o l u m n and the density of their i n t e r n a l connections. In this context it is i m p o r t a n t to note that h y p e r c o l u m n organization is extremely common, being found i n most parts of the cortex i n virtuaUy a l l m a m m a U a n species [ G J M 8 8 ] . T h u s , it is not absolutely essential that r a p i d recovery is carried out i n the hypercolumns of the visual cortex â€” the hypercolumns of the extra-striate visual areas (see, e.g., [MNS7]) could also be used for this purpose. Chapter 6 Tests of the Theory T h e final stage of the analysis is to test the theory on actual hne drawings of p o l y h e d r a l objects. T w o sets of issues are of interest here. The first is how weU the recovery process handles various kinds of hne drawings. T h e process is tested on drawings of objects t h a t violate the u n d e r l y i n g assumptions about scene structure, and on drawings that cannot be given a consistent global i n t e r p r e t a t i o n . It is shown that a substantial amount of threed i m e n s i o n a l structure can be recovered under a wide range of conditions. T h e second set of issues concerns the abihty of the theory to explain the recovery of three-dimensional structure at the preattentive level of h u m a n vision. It is shown that the theory can explain â€” at least i n b r o a d outhne â€” how early visual processing can recover three-dimensional orientation f r o m some kinds of hne drawings, and why it cannot do so for others. 6.1 Performance on Line Drawings T o examine the power and the h m i t a t i o n s of the recovery process, it is tested on a range of hne drawings, including those i n which ah underlying assumptions are obeyed as weh as those i n w h i c h various assumptions are violated. A l t h o u g h the resulting interpretations are not perfect indicators of the overaU effectiveness of the process, they do provide an idea of the relative ease or difhculty of Interpreting the various kinds of hne drawings. Since the speed of the process is determined p r i m a r i l y by the speed of information transm i s s i o n , the absolute size of the hne drawing has virtuaUy no influence apart from a rescahng of the time course (section 2.5.2).^ T h e effects of size are therefore ehminated by rescaling a h drawings so that their average hne length is the same. F o r the drawings considered here, the average hne length is set to 5 ceh w i d t h s . T r a n s m i s s i o n speed can be similarly factored out by measuring time i n terms of the number of transitions between adjacent ceUs, or equivalently, by the number of iterations. T h i s value is essentially a free parameter, which can have different values when recovery is used i n different situations or for different purposes. However, i n order to o b t a i n an i n d i c a t i o n of the relative difficulty of recovery for various kinds of hne drawings, it is useful to base comparison on one p a r t i c u l a r t i m e h m i t . A s a representative value, the number of transitions is such that information is propagated along a distance of twice the average hne length. T h i s allows enough time (on average) for the estimates f r o m each j u n c t i o n to be t r a n s m i t t e d to their nearest neighbors, and for any u p d a t e d values t o be t r a n s m i t t e d back. Since the average length Is 5 ceh w i d t h s , 10 transitions are used. 6,1.1 Rectangular Objects W h e n scenes contain only rectangular objects, a l l assumptions about the s t r u c t u r a l constraints (section 3.1.1) are true, g i v i n g the process the best chance to o b t a i n a globaUy consistent i n t e r p r e t a t i o n of a l l scene-based properties. T h e corresponding hne drawings therefore test the abihty of the process to o b t a i n such interpretations under Ideal conditions. i) Convex objects T h e objects most amenable to r a p i d recovery are simple convex rectangular blocks (figure 6.1), since these not only obey ah s t r u c t u r a l assumptions, but also obey the principle of m a x i m u m convexity that is used to select the i n i t i a l set of interpretations (section 4.2.2). A s figure 6.1 shows, almost ah the three-dimensional structure has been recovered, w i t h u n ambiguous preferred values assigned t o a l l the Unes i n a l l four streams, a n d w i t h almost a l l the alternatives ruled out as impossible. A remnant of uncertainty remains In the center of the d r a w i n g , where the alternative convexities and slant signs are not yet completely ruled out. T h e propagation of the 'impossible' ^Performance does change as the size of the entire object approaches the dimensions of a zone, since the assumption of no more than three Unes per cell (section 4.3.1) can no longer be held. However, drawings here are assumed to be large enough that this is of no concern. values f r o m neighboring ceUs does, however, provide these areas w i t h a definite i n t e r p r e t a t i o n after a few more iterations. T h i s iUustrates a c o m m o n feature of the process â€” a m b i g u i t y is typicaUy eUminated by proceeding from the outside of the drawing to the inside. This is largely due to the low a m b i g u i t y of the L-junctions, which are most often found on the outside border of the d r a w i n g . T h e other area of uncertainty is the assignment of contiguity to the outer edges of t h e drawing. T h i s is due to the inherent ambiguity of the Une drawing itself, which can be interpreted as a block attached to various surfaces (floor, waU, ceiUng) or as a block w i t h o u t any attachments at a l l . T h e recovery process has no means for preferring one over the other, a n d so the interpretation of these values remains ambiguous. ii) Nonconvex objects Nonconvex rectangular objects obey aU s t r u c t u r a l constraints, but contain nonconvex corners t h a t are initiaUy assumed to be convex (section 4.2.2). A s seen from figure 6.2, the i n i t i a l assignment of an incorrect set of values to the nonconvex j u n c t i o n does not seriously affect the final interpretation. C o n t i g u i t y is assigned unambiguously and correctly to almost all surfaces, w i t h the exception of the outer edges, which â€” as for the case of the convex block â€” cannot be given an unambiguous interpretation. Note that the preference for contiguity of the edges of Y - j u n c t i o n s (section 4.2.2) has caused the lower edge to be given a 'preferred' value, although the opposite interpretation has not been definitely ruled out. T h e other streams similarly contain edges that either have a definite interpretation or involve preferred interpretations. A m b i g u o u s convexity and slant sign interpretations exist on the edges of the concave Y j u n c t i o n i n the center of the d r a w i n g . T h i s is due to its i n i t i a l preference as a convex j u n c t i o n a n d to the subsequent assignment of preferred complementary values based on values from its neighbors. T h e corresponding a m b i g u i t y i n these neighbors (i.e., the convex Y - j u n c t i o n s ) is removed v i a the certainty i n the L - j u n c t i o n interpretations. T h i s again iUustrates that m a n y of the unambiguous interpretations are first formed on the outside of the drawing and then propagated i n w a r d . Because slant magnitude does not depend on the c o n v e x / c o n cave distinction, it is u n a m biguously assigned to aU hnes, U m i t e d only by the transmission distance. Magnitude â€¢â€¢â€¢â€¢Hmpossible wftW-Possible Narrow gray lines mark cell boundaries Preferred r...:...\Â«m. ..Mn. :...:..Â»]Â».:.Â«1.. Convexity (+) Nonconvexity (o) Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) â€¢Qc Magnitude iiiii Impossible Â«UfcK Possible Narrow gray lines mark cell boundaries I Preferred iii) Occluded objects W h e n several objects exist i n the scene, projection to the image plane often results i n t h e p a r t i a l occlusion of one object by another. Information f r o m the occluded junctions is l o s t , a loss w h i c h is only p a r t i a h y compensated for by the constraints from the T - j u n c t i o n s . A s figure 6.3 shows, however, the recovery process is fairly robust against the effects of occlusion. A s s i g n m e n t s of contiguity are as good as those for i n d i v i d u a l blocks; indeed, t h e y are somewhat less ambiguous, since the T-junctions have added e x t r a i n f o r m a t i o n to the crossbars. C o n v e x i t y a n d slant are almost u n i m p a i r e d , w i t h only a shght increase i n the a r e a of u n c e r t a i n t y a r o u n d the central Y - j u n c t i o n s . T h e only significant loss of i n f o r m a t i o n occurs i n the hne connected to an occluded a r r o w j u n c t i o n on one end, a n d to an L - j u n c t i o n on the other. T h e L - j u n c t i o n can provide an assignment of slant sign, but cannot transmit slant magnitudes. Consequently, the hne must r e m a i n uninterpretable w i t h i n this stream. 6.1.2 Nonconforming Objects A n o t h e r test of r a p i d recovery concerns its abihty to interpret hne drawings of "nonconformi n g " objects, i.e., those that do not conform to a l l the s t r u c t u r a l assumptions that underhe the recovery process. T h e abihty of the process to recover various scene properties under such conditions provides an i n d i c a t i o n of its robustness i n domains beyond those for w h i c h it is o p t i m a l (cf. section 2.3). i) Nonrectangular objects G i v e n the i m p o r t a n c e of rectangularity for the i n i t i a l assignment of slant magnitudes (section 3.2.4) a n d the constraints on convexity (section 3.2.2) a n d slant signs (section 3.2.3), it is i m p o r t a n t to determine how recovery is affected when these assumptions are no longer true of the scene. F r o m figure 6.4, it is seen that the process can stiU recover a fair amount of s t r u c t u r e . T h e inner edges of a l l hnes are interpreted unambiguously as contiguous. T h e outer edges of the d r a w i n g r e m a i n largely uninterpreted. W h e n more iterations are allowed the contiguity i n t e r p r e t a t i o n assigned to the acute L - j u n c t i o n spreads around the outside of the d r a w i n g . M o s t of the trihnear junctions have been given unambiguous interpretations i n the convexity a n d slant sign dimensions. A l t h o u g h a contradiction i n slant magnitude has been Convexity (+) Nonconvexity (o) Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) Narrow gray lines mark cell boundaries found for one of the edges, a n d cannot be assigned t o t w o others (since the junctions violate P e r k i n s ' laws), the r e m a i n i n g four edges have been assigned definite values. ii) Origami objects A n o t h e r class of objects that do not conform to the s t r u c t u r a l assumptions are the origami objects [KanSO], formed by j o i n i n g extremely t h i n p o l y g o n a l plates t o each other along their edges. A l t h o u g h they are similar t o p o l y h e d r a i n h a v i n g planar surfaces, origami objects are never sohd, a n d so their projections cannot be interpreted as sohd polyhedra. A n example of such a d r a w i n g is the chevron shown i n figure 6.5. A s seen f r o m figure 6.5, the interpretation process is fairly robust t o the violation of this assumption. M o s t of the outer edges are interpreted as contiguous, an interpretation at odds w i t h that given t o the convex block. B u t three of the four inner edges of the rectangles are stiU interpreted unambiguously as being contiguous. T h e results i n the other three streams are largely unaffected by the v i o l a t i o n of this assumption, w i t h the interpretations m a t c h i n g almost exactly w i t h those of the sohd convex block. iii) Nonplanar objects M u c h of the power of a hne interpretation process stems from a basic assumption t h a t the surfaces of the corresponding object are planar (see section 2.2.1). T h e drawing i n figure figure 6.6 violates this basic a s s u m p t i o n , the upper surface being uninterpretable as a plane. T h e l o c a l nature of the r a p i d recovery process, however, allows m u c h of the structure of n o n p l a n a r objects t o be recovered, since global consistency is not enforced. T h i s is iUustrated i n the interpretations shown i n figure 6.6. C o n t i g u i t y is assigned correctly almost everywhere, w i t h inconsistent interpretations assigned only to the inner edges of the notch i n the upper surface. S i m i l a r considerations apply t o convexity a n d slant sign. F u r t h e r m o r e , slant m a g n i tudes are unambiguously assigned t o a l l hnes, a result due to the absence of a check o n slant m a g n i t u d e at L - j u n c t i o n s (section 4.1.3). Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) nQ^Magnitude iiiii Impossible Â«ww-Possible Narrow gray lines mark cell boundaries Preferred Slant Magnitude (Value) ZTQ^ Magnitude iiiii Impossible Slant Magnitude (Confidence) *WWB: Possible Narrow gray lines mark cell boundaries Preferred Convexity (+) Nonconvexity (o) Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) :Qc Magnitude iiiii Impossible *isÃ®i?. Possible Narrow gray lines mark cell boundaries Preferred 6.1.3 Impossible Objects Objects are said to be " i m p o s s i b l e " if they cannot exist under the assumption that connecting hnes i n the image correspond to connecting edges i n the scene. If accidental ahgnments are allowed, connecting image hnes can correspond to disconnected edges i n the scene, so that a corresponding object can be found for any hne drawing [Kul87]. B u t the conditions required for this are extremely unstable, v i o l a t i n g the general viewpoint constraint (section 3.2), so that such interpretations are not generaUy aUowed. Instead, the drawing is interpreted as an impossible figure containing a set of globaUy inconsistent interpretations. A s a final test of its abihties, the r a p i d recovery process is apphed to drawings of these impossible objects. To keep the influence of other factors to a m i n i m u m , a l l junctions are such that they can be consistently interpreted as rectangular corners. T h e apphcation of the recovery process to these drawings consequently provides a good test of how weh it can handle g l o b a l inconsistency. i) Objects of inconsistent contiguity and convexity T h e first class of impossible objects are those that correspond to drawings that cannot be given a consistent set of contiguity a n d convexity labeUings. T h e example considered here is shown i n figure 6.7. Such drawings violate the basic assumption that a surface contiguous w i t h a given edge remains contiguous throughout its entire l e n g t h ; among other things, this ehminates the distinction between object and background [Kul87]. In a d d i t i o n , several of the hnes cannot be given a consistent convexity interpretation along their l e n g t h , p r o v i d i n g a second source of inconsistency. Because the interpretation process involves only local sections of the d r a w i n g , however, it is relatively robust to such inconsistencies. T h i s is ihustrated i n figure 6.7. Here, almost aU hnes are given an unambiguous contiguity interpretation that is correct locaUy. T h e only exceptions i n this stream are two h o r i z o n t a l hnes that have been interpreted as inconsistent. Inconsistencies i n convexity a n d slant sign are also picked u p , but these are restricted entirely to the inner hnes, the outer sections h a v i n g a completely unambiguous i n t e r p r e t a t i o n . Slant m a g n i t u d e is completely unaffected by the inconsistencies i n contiguity a n d convexity, w i t h unambiguous interpretations assigned to virtuaUy aU Unes. â€¢â€¢â€¢â€¢-T:;:.::::.:;:!!JJ.-,.;v/..... >^Mai*K B B B â€¢ â€¢ B B B '' Nonconvexity (o) Convexity C+) EEL !â€¢ \ - i Â« # (I'l-- Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) llQn Magnitude iiiiMmpossible Â«ww: Possible Narrow gray lines mark cell boundaries I Preferred ii) Objects of inconsistent slant A n o t h e r class of impossible objects give rise to drawings i n which the hnes cannot be given a consistent set of slant estimates. A n example is shown i n figure 6.8. Such inconsistency negates the basis for the p r o p a g a t i o n of slant estimates along common edges. T h e results of the recovery process are shown i n figure 6.8. A s seen from this figure, m u c h of the (local) three-dimensional structure is stiU recovered. C o n t i g u i t y is assigned correctly t o all hnes, the only uncertainty existing i n the outer edges. C o n v e x i t y also is largely unaffected, although inconsistencies have begun to appear i n the Y - j u n c t i o n s . These inconsistencies are more severe i n the slant sign s t r e a m , although the arrow-junctions a n d L - j u n c t i o n s r e t a i n unambiguous interpretations. Because the estimation of slant magnitudes is independent of slant sign, unambiguous magnitude estimates are assigned to a l l the hnes. iii) Objects of inconsistent depth P a r t of the reason for the speed of the r a p i d recovery process is that it avoids global checks of the resulting description, using the consistency of the w o r l d itself as the basis for coherent interpretations. One i m p o r t a n t example of this is the complete lack of any check on d e p t h i n f o r m a t i o n (section 3.1.1). T h i s renders the process susceptible to a number of " i h u s i o n s " o n drawings for w h i c h the corresponding surfaces have globally inconsistent depths. A n e x a m p l e of such a d r a w i n g is the Penrose triangle, shown i n figure 6.9. A s seen from this figure, a l l four streams result i n interpretations that are largely u n a m biguous for a l l hnes. T h e only exceptions are uncertain contiguity estimates for the outer hnes of the d r a w i n g , a n d uncertain slant estimates for the innermost hnes. B o t h of these are t o be expected, since the uncertainty i n outer contiguity occurs for almost a l l drawings, a n d the u n c e r t a i n t y i n slant estimates is a consequence of the inner hnes contacting only L - a n d T - j u n c t i o n s , neither of w h i c h can give rise to a magnitude estimate. V i r t u a l l y aU l o c a l struct u r e is therefore recovered, w i t h no inconsistencies being detected. T h e i n t e r p r e t a t i o n is a n iUusion of exactly the t y p e expected, w i t h v i r t u a l l y aU edges assigned definite interpretations even t h o u g h the corresponding object cannot be reahzed. Convexity C+) * : ; ; ; : Jt I ;* â€¢ I â€¢ Slant Sign (1) Slant Magnitude (Value) â€¢ Q c Magnitude i i i i : Impossible Slant Magnitude (Confidence) 4sww Possible Narrow gray lines mark cell boundaries I Preferred Convexity (+) Nonconvexity Co) Slant Sign (1) Slant Sign (2) Slant Magnitude (Value) Slant Magnitude (Confidence) â€¢ Q c Magnitude iiii: Impossible Â«was Possible Narrow gray lines mark cell boundaries Preferred 6.2 Preattentive Recovery of Scene Structure T h e u l t i m a t e goal of the theory developed here is to explain the r a p i d recovery of threedimensional structure i n h u m a n early vision. In p a r t i c u l a r , the goal is to explain why c e r t a i n kinds of hne drawings can be r a p i d l y detected i n visual search tasks, and why others cannot. F i g u r e 6.10 shows the set of results considered. T h e search items, together w i t h the search rates, are taken from [ER91] a n d [ER92]. In ah. cases, two search rates are presented - those for displays i n w h i c h the target is present, a n d those for which it is absent. T h e recovery r a t i o p is the measure developed i n section 6.2.1 to explain these rates. A l t h o u g h not exhaustive, this set is representative of what is k n o w n about search rates for various kinds of hne drawings. B y m a k i n g several relatively simple assumptions about the relation of recovered s t r u c t u r e to search rates, the theory is able to explain the relative difficulty of search for a l l cases e x a m i n e d . Because these assumptions are fairly general, they also aUow predictions t o be made for drawings not yet tested. 6.2.1 Basic Assumptions T i m e a n d Space P a r a m e t e r s T o c a r r y out the analysis, it is necessary to specify b o t h the size of the drawings a n d the a m o u n t of time to be allocated. In what foUows, drawings are scaled to have the same m a x i m u m extension. T h i s is done so that the relative sizes m a t c h those of the drawings used for the experiments described i n [ER91] a n d [ER92]. T h e extent of the drawings is taken t o be 5 ceUs. If ceUs are related to hypercolumns (section 5.3), this wiU correspond closely to the a c t u a l number of hypercolumns involved. T h e time h m i t is set at 5 iterations â€” enough for a one-time propagation of i n f o r m a t i o n across the m a x i m u m extent of the d r a w i n g . T h i s is only meant to be a representative value, useful as the basis for a comparison of the difficulty of i n t e r p r e t a t i o n for various kinds of drawings. R e l a t i n g s t r u c t u r e to search rates Since the goal of this work is to explain the relative preference for certain kinds of hne drawings over others, a n d not the phenomenon of r a p i d detection per se, no commitment is Search Items Condition larget Rates (ms/item) Distractor B. Present 51 Absent ji 12 oo 96 0.4 C. oo D. 22 31 0.0 E. 35 65 1.2 37 66 0.0 52 80 1.1 63 101 0.0 iCi Sl H. \\i j \ r i F i g u r e 6.10: Results explained by theory. T h e search items a n d search rates are taken f r o m [ER91] a n d [ E R 9 2 ] . T h e recovery ratio p, discussed i n section 6.2.1, describes the difference i n the recovered three-dimensional structure of the target a n d distractor items. T h e correlation between p a n d search rate is evident. made here to any p a r t i c u l a r m o d e l of visual attention or visual search. Instead, a set of four relatively general assumptions is used to relate recovered structure to search rates: 1. S e a r c h rates increase w i t h greater target-distractor distinctiveness. This assumes t h a t search rates are largely governed by a signal-to-noise r a t i o that compares the relative number of distinctive features i n the target to the number of features it shares w i t h the distractors. T h i s is a widely-accepted assumption used to e x p l a i n search rates for m a n y kinds of v i s u a l stimuh (e.g., [ T G 8 8 , D H 8 9 ] ) . 2. T a r g e t - d i s t r a c t o r distinctiveness is based on differences in slant. It is assumed t h a t the slant sign a n d slant magnitude of each hne i n the interpreted d r a w i n g are combined into a single quantity that acts as an irreducible feature, capable of being detected almost immediately when sufficiently distinct (section 2.1.2). T h i s ass u m p t i o n is supported by the finding that the speed of search for hne drawings can be better explained i n terms of three-dimensional rather t h a n two-dimensional orientation [ER90b]. 3. C o m m o n uninterpretable lines increase target-distractor similarity. U n i n t e r p r e t a b l e hnes are assumed to be part of the "noise" that interferes w i t h the process of distinguishing target f r o m distractor i n visual search tasks. Such interference could exist for a variety of reasons. If, for example, the rapid-recovery system acted only to eUminate impossible interpretations, Unes w i t h o u t a definite slant est i m a t e w o u l d be assigned a l l possible values. T h i s set of values w o u l d therefore be c o m m o n t o b o t h target and distractor. 4. Slant is represented as a d e p a r t u r e f r o m zero. T h i s takes slant to be a quantity hke two-dimensional o r i e n t a t i o n , which is represented as a departure f r o m the canonical orientations of v e r t i c a l or h o r i z o n t a l [TG88]. Here, the canonical value is assumed to be zero, i.e., a three-dimensional orientation perpendicular to the hne of sight. To o b t a i n a quantitative measure of target-distractor similarity, a d d i t i o n a l assumptions are needed to refine the original set: 1'. S e a r c h rates increase w i t h the recovery ratio p. T h e quantity p is defined here as the r a t i o of the target-distractor difference over the target-distractor similarity. A l t h o u g h this is a considerable simphfication that among other things completely ignores configurational effects among two-dimensional features, it nevertheless provides a rough quantitative measure that captures something of the trade-off between distinctiveness a n d similarity. 2'. Differences are based on unambiguous slant estimates. Unambiguous estimates are those for which a slant magnitude has been assigned to the hne (section) a n d for w h i c h one of the slant signs is preferred. In hght of assumption 4, only differences i n nonzero slants contribute to tbe distinctiveness measure â€” since target a n d distractor always differ b y a 180Â° r o t a t i o n i n the image, the slants of corresponding hnes differ i n their sign. Consequently, the slant difference is always twice the value of the slant itseff. Since the exact l o c a t i o n of a feature is not i m p o r t a n t i n v i s u a l search (section 2.1.2), ambiguity m a y also arise i f two hnes of the same orientation have different slants. 3'. Similarities are based o n ambiguous slant estimates. If an ambiguous inter- p r e t a t i o n exists for the slant sign, or i f a slant magnitude is not possible, the corresponding hne segment is considered t o a d d t o similarity i n the same way as uninterpreted hnes. 4'. Slant signals are p r o p o r t i o n a l to line length along each orientation. In effect, each s m a l l section of hne is assumed t o signal the value of the slant at i t s l o c a t i o n , a n d t o pass this value on to the mechanisms governing visual search. Since the l o c a t i o n of a feature is not i m p o r t a n t for this purpose (cf. section 2.1.2), a h signals f r o m a c o m m o n orientation can simply be summed together. T h e t o t a l signal is therefore p r o p o r t i o n a l t o the cumulative length along a particular direction. In order t o avoid specifying different weights for different slants, each nonzero slant a n d slant difference are assigned the same value. In s u m m a r y , t h e n , search rates are assumed to increase w i t h the recovery r a t i o p, defined as Yle Y^i {segment agj has unambiguous ^ Y^e I2j {segment nonzero agj has ambiguous slant) slant) ' where agj denotes a hne segment of orientation 6 i n the image. Because of the a s y m m e t r y between u p w a r d a n d d o w n w a r d slants [ER90b] (also see fig 1.1), this r a t i o is taken t o apply only t o cases where the object corresponding t o the target is slanted upward. A g a i n , it should be emphasized that the theory developed here is not addressed towards explaining such a n a s y m m e t r y , b u t rather is only intended t o explain the relative difficulty of search. 6.2.2 Context E x p l a n a t i o n of P s y c h o p h y s i c a l Results Effects T h e first test of the theory is to see if it can explain w h y different contexts influence t h e detectabihty of a Y - j u n c t i o n among a set of similar junctions r o t a t e d by 180Â°. T h e detectabihty of this j u n c t i o n is greatly affected by the presence a n d shape of the s u r r o u n d i n g outhne, as shown i n figure 6.10, taken from [ER91]. W h e n Y - j u n c t i o n s are surrounded by a Slant Sign (2) Slant Sign (1) â€¢OcMagnitude i i i i i Impossible wsfw Possible I M M Preferred Narrow gray lines mark cell boundaries F i g u r e 6.11: Slant estimates for C o n d i t i o n A . Slant angle (in degrees) obtained by m u l t i p l y i n g slant m a g n i t u d e number by 20. quasi-hexagonal frame ( C o n d i t i o n A ) they are detected quite r a p i d l y (7 m s / i t e m for target present; 12 m s / i t e m for target absent). B u t a square surround ( C o n d i t i o n B ) causes search to slow down considerably (51 m s / i t e m for target present; 96 m s / i t e m for target absent). A comparison of the interpretations for C o n d i t i o n A (figure 6.11) a n d for C o n d i t i o n B (figure 6.12) shows that this effect is readily explained i n terms of the recovered three-dimensional structure. T h e i n t e r p r e t a t i o n of C o n d i t i o n A contains no ambiguity i n regards to slant, w i t h a considerable difference between target and distractor. Since there are no nonzero slants i n c o m m o n , the recovery r a t i o p is infinite, accounting for the fast search that occurs for this condition. Slant Sign (1) Slant Sign C2) Slant Magnitude (Value) Slant Magnitude (Confidence) n Q z Magnitude liiÂ»^ Impossible â€¢Â»}Â«Â»; Possible Preferred Narrow gray lines mark cell boundaries F i g u r e 6.12: Slant estimates for C o n d i t i o n B . Slant angle (in degrees) obtained by m u l t i p l y i n g slant m a g n i t u d e n u m b e r by 20. A l t h o u g h the long stem of the Y - j u n c t i o n is assigned a unique slant, this is only o n e - t h i r d the " s i g n a l " obtained f r o m the drawing of C o n d i t i o n A . F u r t h e r m o r e , a considerable amount of uninterpretable structure exists. T h e recovery ratio therefore has a relatively low value {p = 0.4), which explains the m u c h lower search rate. Contiguity To determine whether the slow search found i n C o n d i t i o n B is due to the failure of the recovery process or s i m p l y due to the presence of T - j u n c t i o n s , consider the drawings of C o n d i t i o n C a n d C o n d i t i o n D , taken f r o m [ER92]. A s seen from the figure, search for the target i n C o n d i t i o n C is fast, w i t h about the same speed as for that of C o n d i t i o n A . Consider now the drawings of C o n d i t i o n D . Since targets composed of two items can be easily detected i n a b a c k g r o u n d of single items [TG88], the target should be easy to detect if the distractor is not segmented i n t o two groups. T h e target also differs i n overah shape i n the image, which can only help t o speed search. B u t the search rates (22 m s / i t e m for target present; 31 m s / i t e m for target absent) clearly show that search is relatively difficult. T h e i n t e r p r e t a t i o n of the drawing i n C o n d i t i o n C is shown i n figure 6.13. T h e T-junctions have p a r t i t i o n e d the drawing into two groups, each of these being interpreted as a complete block w i t h unambiguous slants assigned to a l l hnes. T h e high recovery r a t i o is therefore h i g h (p = oo), e x p l a i n i n g the h i g h speed of search. T h e distractors i n C o n d i t i o n D , being identical to those of C o n d i t i o n C , have hkewise been interpreted as a pair of blocks. Since a l l nonzero slants m a t c h those of the separate blocks i n the target i t e m , however, no slant differences exist, a n d so p is zero. Target a n d distractor differ only i n the relative l o c a t i o n of their parts, and since relative l o c a t i o n cannot be determined at early levels (e.g., [Jul84a, Tre88], search is to be expected to be relatively slow. A l t h o u g h search is faster t h a n i n C o n d i t i o n B , this can easily be a t t r i b u t e d to some weak effect resulting from the overall difference i n two-dimensional shape. Rectangularity C o n d i t i o n s E a n d F (taken f r o m [ER91]) provide a direct test of the rectangularity constraint. These drawings have been distorted so as to violate the assumption of rectangularity i n two different ways. In C o n d i t i o n E , the internal Y - j u n c t i o n has been altered so that the system of junctions cannot be consistently interpreted as rectangular; indeed, the top surface is no Slant Sign (2) Slant Sign (1) 55? m 1 Slant Magnitude (Value) Slant Magnitude (Confidence) Â® o Q ^ Magnitude iiH! Impossible WBWW Possible 1 Preferred Narrow gray lines mark cell boundaries F i g u r e 6.13: Slant estimates for C o n d i t i o n C . Slant angle (in degrees) obtained by m u l t i p l y i n g slant m a g n i t u d e number by 20. â€¢Qcf^^Snitude iiin Impossible Possible Preferred Narrow gray lines mark cell boundaries F i g u r e 6.14: Slant estimates for C o n d i t i o n E . Slant angle (in degrees) obtained by m u l t i p l y i n g slant m a g n i t u d e number by 20. longer even p l a n a r . T b i s condition leads to slow search. To control for the possibihty that paraUehsm rather t h a n rectangularity is the key property, C o n d i t i o n F uses a cube stretched vertically, so that parahel hnes r e m a i n parallel while b o t h the Y - j u n c t i o n a n d arrow-junction now violate P e r k i n s ' laws. Search is again slowed. T h e i n t e r p r e t a t i o n of the d r a w i n g i n C o n d i t i o n E is shown i n figure 6.14. Here, the distortions of the junctions have created conflicts i n b o t h slant sign a n d slant magnitude along several hnes. T h e low value of the recovery ratio (p = 1.2) then explains the slow search speeds found. T h e results of C o n d i t i o n F are also easily explained - since the junctions violate P e r k i n s ' laws, an i n i t i a l assignment of slant magnitude is not even a t t e m p t e d . p is zero a n d search is slow. Consequently, Connectedness T o test the possibihty that r a p i d recovery is based on the direct lookup of complete objects rather t h a n v i a the interaction of more l o c a l structures (cf. section 2.3), search rates were determined for the drawings of C o n d i t i o n s G and H (taken from [ER91]). Condition G corresponds t o the rectangular block of C o n d i t i o n A , w i t h gaps introduced m i d w a y along the lengths of the hnes. If lookup depends on the presence of local features alone, search rates should be similar to those for C o n d i t i o n A . However, search slows down d r a m a t i c a l l y for this condition (52 m s / i t e m for target present; 80 m s / i t e m for target absent). A similar s i t u a t i o n arises i n C o n d i t i o n H , where the junctions themselves have been removed, leaving only a set of isolated hnes i n place. A g a i n , search slows down considerably (63 m s / i t e m for target present; 101 m s / i t e m for target absent). These results show that junctions are necessary for three-dimensional orientation to be recovered, but that they are not sufiicient. A l t h o u g h difficult to account for by a process based on the lookup of complete objects, these results are readily explained by the rapid-recovery process developed here. T h e interp r e t a t i o n of the drawing i n C o n d i t i o n G is shown i n figure 6.15. T h e i n t r o d u c t i o n of the gaps results i n two m a j o r differences from the estimates for C o n d i t i o n A : (1) instead of a single object, the d r a w i n g gives rise to a number of smaller parts scattered about the image, a n d (h) the i s o l a t i o n of the L - j u n c t i o n prevents them from receiving any k i n d of slant estimate. T w o sources of slowdown therefore emerge: not only are there a larger number of items to be considered, but the recovery r a t i o itself has a low value (p = 1.1) due to the uninterpreted L-junctions.'^ A n even simpler e x p l a n a t i o n can be given for the results of C o n d i t i o n H . Here, the absence of j u n c t i o n s prevents any slant estimate from being assigned to the hnes. A s such, they are left as sets of simple two-dimensional objects, which require higher-level processing to be grouped into assembhes corresponding to three-dimensional objects. scatter in slant estimates would also result if lines in the drawings are sufficiently small that accurate orientation measurements cannot be made. This scatter could only reduce search rates further. Slant Sign (2) Slant Sign (1) â€¢ O ^ Magnitude iiii! Impossible www Possible BIIM Preferred Narrow gray lines mark cell boundaries F i g u r e 6.15; Slant estimates for C o n d i t i o n G . Slant angle (in degrees) obtained by m u l t i p l y i n g slant m a g n i t u d e number by 20. Chapter 7 Summary and Conclusions A c o m p u t a t i o n a l theory is developed to explain the r a p i d interpretation of hne drawings at early levels of h u m a n v i s i o n . T h i s is done by first extending the framework of M a r r [Mar82] to allow processes to be analyzed i n terms of hmits on their c o m p u t a t i o n a l resources. The p r o b l e m of r a p i d hne i n t e r p r e t a t i o n is then examined along two dimensions: (i) reducing the t o t a l amount of i n f o r m a t i o n to be t r a n s m i t t e d , a n d (h) m a k i n g effective use of the i n f o r m a t i o n that is processed. T h e first of these is addressed by developing constraints on the structure of the recovered object that aUow it to interpreted i n subhnear time. T h e second is h a n d l e d by constraints on the d y n a m i c operation of the recovery process so that it considers the most hkely interpretations first. It is shown that the resulting process can be implemented on a mesh of simple processing elements, and that it can recover a considerable amount of threedimensional structure i n very httle t i m e . It also is shown that such a process can explain the abihty of h u m a n vision to recover three-dimensional orientation at preattentive levels. These results are relevant to several areas of study. F i r s t , the extension of M a r r ' s framework developed i n section 2.4 provides a way to discuss the various factors involved when a process is t o be explained i n t e r m of h m i t e d c o m p u t a t i o n a l resources. T h i s extension has elements contained i n previous attempts to incorporate resource h m i t a t i o n s (e.g., [ F B 8 2 , Tso87]) into a c o m p u t a t i o n a l framework, but it also puts forward several new distinctions (e.g., extern a l vs. i n t e r n a l constraints, constraint vs. h m i t a t i o n ) , and treats these i n a more systematic way. A l t h o u g h still i n r u d i m e n t a r y f o r m , this framework can help guide the development of c o m p u t a t i o n a l theories for other resource-hmited processes. A n o t h e r , more concrete framework is the taxonomy of image mappings proposed in sect i o n 2.1.1. Here, mappings are grouped on the basis of information flow across the image, w h i c h i n t u r n is related to lower bounds on their c o m p u t a t i o n a l complexity. T h e structure of this framework remains conjectural at the moment. If proven, these results would be interesting extensions of the work of M i n s k y a n d P a p e r t [MP69] on the abihties of simple parallel architectures to carry out various kinds of operations on images. T h e developments i n chapter 3 provide several interesting results concerning the complexi t y of coUapsed constraint satisfaction problems. These results support earUer observations (e.g., [Mac74, Mal87]) that such systems can often be solved quite easily. T h e y also show t h a t careful selection a n d coordination of such "coUapsed" subsystems can lead to a p p r o x i m a t i o n s that are not only soluble i n subUnear t i m e , but that also retain much of the information i n the o r i g i n a l set of constraints. It w o u l d be interesting to see whether the approach developed here ( v i z . , separation into weakly interacting subsets of b i n a r y and bijective constraints) could be usefuUy apphed i n other domains. T h e complementary subsystems developed i n chapter 4 provide an interesting way t o handle l o c a l inconsistencies a n d ambiguities. In p a r t i c u l a r , their i n c o r p o r a t i o n into a p a i r of Uberal a n d conservative interpretation schemes suggests a general way to handle interpret a t i o n problems that require inconsistency and ambiguity to be exphcitly represented a n d treated i n a systematic fashion. F i n a l l y , the results of chapters 5-6 provide support for the view of early vision sketched i n section 2.3.2 â€” that the " h o r i z o n t a l " modules formed by different levels of processing can be complemented by " v e r t i c a l " columns capable of p r o v i d i n g interpretations that are locaUy consistent. T h i s has imphcations for the study of b o t h machine and biological v i s i o n systems. T h e algorithms developed i n chapter 5 show that this style of processing can be easily i n c o r p o r a t e d into a machine vision system, aUowing it to o b t a i n r a p i d estimates of scene-based properties at aU points i n the image. It is seen from the results of section 6.1 that a considerable amount of scene structure can often be recovered this way. Consequently, a r a p i d recovery process can greatly facihtate the overaU operation of a machine vision system. T h e results of section 6.2 h o l d a similar i m p h c a t i o n for biological vision systems â€” r a p i d recovery at early levels can be used to help quickly construct higher-level descriptions of the world. F u r t h e r m o r e , given that hne interpretation is relatively difficult at early levels (cf. section 1.1), the results of chapter 6 make it plausible that other kinds of r a p i d recovery processes may also exist at these levels. O p e n Questions a n d F u t u r e Directions M a n y of the results concerning the a c t u a l performance of the r a p i d recovery process are based o n t i m e a n d space parameters assumed t o be representative of early visual processing. A l t h o u g h suitable as a first a p p r o x i m a t i o n , the selection of these values is nevertheless somew h a t arbitrary. It w o u l d therefore be useful to carry out a set of psychophysical experiments t o examine the t i m e course of this process i n greater detail, and to see if these values t r u l y are representative. A m o n g other things, such experiments might be able to confirm or refute the theory i n regards t o the order i n which various properties are actually recovered. A related set of issues apphes to the recovery r a t i o of section 6.2, used to relate recovered structure to search rate. T h i s quantity is sufhcient for present purposes, but is only a r o u g h i n d i c a t o r of search difficulty, a n d ideally w o u l d be replaced by a more rehable measure. T h e general idea that a signal-to-noise ratio largely governs search speed is widely accepted (e.g., [ T G 8 8 , D H 8 9 ] ) , but a more precise measure is not currently k n o w n . A s such, this p r o b l e m is not h m i t e d to e x p l a i n i n g the results of search for hne drawings. B u t as d a t a accumulates f r o m more search experiments, it might at least be possible to refine the recovery r a t i o t o take Into account such possibihties as several canonical slant values, a n d different weights for different slant magnitudes. A more general set of concerns involves the way i n which r a p i d recovery Is related to object recognition. One of the m a i n roles assumed for r a p i d recovery is to provide early estimates of scene-based properties that facihtate later processes, i n c l u d i n g those Involved w i t h object recognition (section 2.3.2). It Is entirely possible, however, that object recognition proceeds by a l o o k u p mechanism that uses simple Image properties to retrieve a complete globahy-conslstent m o d e l of the object (e.g., [PE90]). If so, r a p i d recovery at early levels could be accounted for entirely i n this way. T h e results of section 6.2, however, show t h a t recovery is destroyed by nonrectangular corners a n d by the i n t r o d u c t i o n of gaps Into the drawings, something rather difficult to account for In terms of this mechanism. F u r t h e r m o r e , a theoretical objection can also be raised against the Indiscriminate use of lookup tables, since an enormous amount of memory w o u l d be required to store a h possible views of each object at ah possible angles (see section 2.3). L o o k u p for a h m i t e d number of objects, however. Is entirely possible. Indeed, the process developed here can itself be viewed as using a simple form of lookup (cf section 5.1.3), the i n i t i a l interpretations based on a smaU number of " l o c a l " models Invoked by the j u n c t i o n s a n d the resulting interpretations then weeded out by in situ constraints. consistency of global models w i t h each other must also be estabhshed Since even the i n some way, the issue is therefore one of determining the appropriate granularity of the models involved. A n interesting direction for future research is to ascertain the various levels of granularity that might be used, and to determine how models of different granularity might interact. In any event, it has been shown here that smaher-grained " l o c a l " models are sufficient to allow a s u b s t a n t i a l amount of three-dimensional structure to be recovered i n very httle t i m e . It has also been shown that the properties recovered i n this way can be used to explain why p a r t i c u l a r kinds of hne d r a w i n g are or are not interpreted at early levels of h u m a n vision. A s such, the central point of this work has been estabhshed â€” substantial amounts of scene structure can be recovered i n very httle time by sphtting a process into quasi-independent streams t h a t are each concerned w i t h a single aspect of scene structure. T h i s principle is, of course, not h m i t e d to r a p i d hne interpretation, and it wiU be interesting to see if it can be apphed to other forms of r a p i d perception and r a p i d cognition. Bibliography [AF69] F . A t t n e a v e and R . Frost. T h e determination of perceived tridimensional orientation by m i n i m u m criteria. Perception [Arb87] M . A . A r b i b . Brains, Machines, 6 B : 3 9 1 - 3 9 6 , 1969. & Psychophysics, (2nd ed.). New Y o r k and B e r h n : and Mathematics Springer, 1987. [Att54] F . A t t n e a v e . Some i n f o r m a t i o n a l aspects of v i s u a l perception. Psychological Re- view, 61:183-193, 1954. [Att72] F . A t t n e a v e . Representation of physical space. In A . W . M e l t o n and E . M a r t i n , editors. Coding Processes in Human Memory, pages 11-29. W a s h i n g t o n D C : V . H . W i n s t o n & Sons, 1972. [Att82] F . A t t n e a v e . P r a g n a n z and soap bubble systems: A theoretical exploration. In J . Beck, editor, Organization and Representation in Perception, pages 11-29. HiUs- dale, N J : E r l b a u m , 1982. [BA88] J . R . Bergen and E . H . A d e l s o n . E a r l y vision and texture perception. Nature, pages 363-364, 1988. [Baa78] S. Baase. Computer Algorithms: Introduction to Design and Analysis. Reading, M A : A d d i s o n - W e s l e y , 1978. [Bal91] D . H . B a l l a r d . A n i m a t e vision. Artificial [BB82] D . H . B a l l a r d and C M . B r o w n . Computer Intelligence, Vision. 48:57-86, 1991. Englewood Chffs, N J : Prentice- H a l l , 1982. [BCG90] A . C . B o v i k , M . C l a r k , and W . S . Geisler. locahzed spatial filters. Intelligence, [Bec66] J . Beck. Perception IEEE M u l t i c h a n n e l texture analysis using Transactions on Pattern Analysis and Machine 12:55-73, 1990. Effect of orientation and of shape similarity on perceptual grouping. & Psychophysics, 1:300-302, 1966. [Bec82] J . Beck. T e x t u r a l segmentation. In J . Beck, editor. Organization tion in Perception, [BGS91] J . Beck, N . G r a h a m , a n d A . Sutter. Lightness differences a n d the perceived seg1991. & Psychophysics, I. B i e d e r m a n . H u m a n image understanding: Recent research and a theory. puter Vision, [Bis84] Representa- pages 285-317. Hillsdale, N J : E r l b a u m , 1982. regation of regions a n d populations. Perception [Bie85] and Graphics and Image Processing, Com- 32:29-73, 1985. P . O . B i s h o p . Processing of visual information w i t h i n the retinostriate system. In J . M . B r o o k h a r t a n d V . B . M o u n t c a s t l e , editors. Handbook of Physiology, Vol. Ill, Sensory Processes, Part 1, pages 341-424. Bethesda, M D : A m e r i c a n Physiological Society, 1984. [BL86] F . Bosehe a n d E . L . J . Leeuwenberg. A test of the m i n i m u m principle requires a perceptual coding system. Perception, [Bla89] 15:331-354, 1986. A . B l a k e . C o m p a r i s o n of the efficiency of deterministic a n d stochastic algorithms for v i s u a l reconstruction. IEEE Trans. Pattern Analysis and Machine Intelligence, 11:2-12, 1989. [Bra77] A . B r a n d t . M u l t i - l e v e l adaptive techniques for p a r t i a l differential equations: Ideas a n d software. In J . R . Rice, editor. Mathematical Software III, pages 277-318. N e w Y o r k , N Y : A c a d e m i c , 1977. [Bri87] W . L . Briggs. A Multigrid Tutorial. P h i l a d e l p h i a , P A : Society for I n d u s t r i a l a n d A p p h e d M a t h e m a t i c s , 1987. [Bro81] R . A . Brooks. Intelligence, [BT78] S y m b o h c reasoning among 3d models and 2d images. Artificial 17:285-348, 1981. H . G . B a r r o w a n d J . M . Tenenbaum. Recovering intrinsic scene characteristics from images. In A . R . H a n s o n a n d E . M . R i s e m a n , editors. Computer Vision Systems, pages 18-24. New Y o r k , N Y : A c a d e m i c , 1978. [BWH75] K . B e r b a u m , N . Weisstein, and C . S . H a r r i s . Psychonomic [Cae84] A vertex-superiority effect. Bull. Society, 6:418, 1975. T . CaeUi. O n the specification of coding principles of visual image processing. In P. DodweU and T . C a e h i , editors. Figurai Synthesis, pages 153-184. HiUsdale, N J : E r l b a u m , 1984. [CAT90] P. C a v a n a g h , M . A r g u i n , a n d A . T r e i s m a n . Effect of surface m e d i u m on v i - sual search for orientation and size features. Journal 16:479-491, 1990. of Experimental Psychology, [CF82] P . R . C o h e n and E . A . Feigenbaum. V i s i o n . In E . A . Feigenbaum and P . R . C o h e n , editors. The Handbook of Artificial Intelligence (Vol. 3), pages 139-194. S t a n f o r d , C A : Heuris Tech Press, 1982. [CHY90] K . C u h k II, L . P . H u r d , and S. Y u . a u t o m a t a . Physica C o m p u t a t i o n theoretic aspects of cehular D, 45:357-378, 1990. [Clo71] M . B . Clowes. O n seeing things. Artificial [CW90] K . R . C a v e and J . M . Wolfe. M o d e h n g the role of parallel processing i n v i s u a l search. Cognitive 22:225-271, 1990. [DH89] Psychology, Psycho- 1989. S . W . D r a p e r . T h e use of gradient and d u a l space i n hne-drawing representation. Artificial [Dun89] 2:79-112, 1971. J . D u n c a n and G . W . H u m p h r e y s . V i s u a l search and stimulus similarity. logical Review, 96:433-458, [Dra81] Intelligence, Intelligence, 17:461-508, 1981. J . D u n c a n . B o u n d a r y conditions on parallel processing i n h u m a n vision. Percep- tion, 18:457-469, 1989. [dYvE88] E . A . de Y o e and D . C . van Essen. Concurrent processing streams i n monkey v i s u a l cortex. Trends in Neuroscience, [Ede87] S. E d e l m a n . L i n e connectivity algorithms for an asynchronous p y r a m i d computer. Computer [EIS76] 11:219-226, 1988. Vision, Graphics,and Image Processing, 1990. Science, 1:323-326, 1990. J . T . E n n s a n d R . A . Rensink. Preattentive recovery of three-dimensional orientat i o n f r o m hne drawings. Psychological [ER92] 5:691-703, 1976. J . T . E n n s a n d R . A . Rensink. Sensitivity to three-dimensional orientation i n v i s u a l search. Psychological [ER91] on Computing, J . T . E n n s and R . A . Rensink. Influence of scene-based properties on v i s u a l search. Science, 247:721-723, [ER90b] 1987. S. E v e n , A . I t a i , and A . S h a m i r . O n the complexity of timetable and m u l t i c o m m o d i t y flow problems. SI AM Journal [ER90a] 40:169-187, Review, 98:335-351, 1991. J . T . E n n s a n d R . A . Rensink. A m o d e l for the r a p i d interpretation of hne drawings i n early vision. In Visual Search II. L o n d o n : Taylor k Francis, 1992. [Fal72] G . F a l k . Interpretation of imperfect hne d a t a as a three-dimensional scene. ficial Intelligence, 3:101-144, 1972. Arti- [Fel85] J . F e l d m a n . Connectionist models and parallelism i n higli level vision. In A . Rosenfeld, editor, Human and Machine II, pages 86-108. New Y o r k , N Y : A c a - Vision demic, 1985. [Fly72] M . J . F l y n n . Some computer organizations and their effectiveness. B . F . Green a n d L . K . A n d e r s o n . C o l o r coding i n a v i s u a l search task. Journal Experimental [GA68] A . G u z m a n - A r e n a s . Decomposition of a v i s u a l scene into three-dimensional bodies. Fall Joint Computer 4:31-44, 1989. S. G e m a n and D . G e m a n . Stochastic relaxation, G i b b s distributions and the Bayesian restoration of images. IEEE M . R . G a r e y and D . S . Johnson. Theory of NP-Completeness. [GJM88] Transactions on Pattern Analysis and Ma- 6:721-741, 1984. chine Intelligence, [GJ79] Conf, pages 291-304, 1968. R . G u r n s e y and R . A . Browse. A s y m m e t r i e s i n visual texture d i s c r i m i n a t i o n . Spatial Vision, [GG84] of 51:19-24, 1956. Psychology, In Proc. AFIPS [GB89] Transac- 21:948-960, 1972. tions on Computers, [GA56] IEEE Computers and Intractability: A Guide to the San Francisco: W . H . F r e e m a n , 1979. I.I. Glezer, M . S . Jacobs, and P . J . Morgane. Imphcations of the ' i n i t i a l b r a i n ' concept for b r a i n evolution i n cetacea. Behavioral and Brain Sciences, 11:75-116, 1988. [Gla84] F . G l a z e r . M u l t i l e v e l relaxation i n low-level computer vision. In A . Rosenfeld, editor, Multiresolution Image Processing and Analysis, pages 312-330. New Y o r k and B e r h n : Springer, 1984. [Gol69] M . J . E . Golay. H e x a g o n a l parallel p a t t e r n transformations. IEEE Computers, [Gou89] Transactions on 18:733-740, 1969. S . J . G o u l d . Wonderful Life: The Burgess Shale and the Nature of History. New Y o r k : N o r t o n , 1989. [GR88] A . G i b b o n s and W . R y t t e r . Efficient Parallel Algorithms. Cambridge: Cambridge U n i v e r s i t y Press, 1988. [Gra85] N . G r a h a m . Detection and identification of near-threshold v i s u a l patterns. of the American Journal Optical Society, A, 2:1468-1482, 1985. [Har87] D . H a r e l . Algorithmics. [Hil84] W . D . HiUis. T h e Connection M a c h i n e : A computer architecture based on ceUular a u t o m a t a . Physica Reading, M A : Addison-Wesley, 1987. D, 10:213-228, 1984. [HM53] J . Hochberg and E . M c A l i s t e r . A quantitative approach, to figurai goodness. nal of Experimental Psychology, Jour- 46:361-364, 1953. [Hoc78] J . E . H o c h b e r g . Perception (2nd ed.). Englewood Chffs, N J : P r e n t i c e - H a h , 1978. [Hor86] B . K . P . H o r n . Robot Vision. C a m b r i d g e , M A : M I T Press; New Y o r k : M c G r a w - H i l l , 1986. [Hub81] D . H . H u b e l . E x p l o r a t i o n of the p r i m a r y visual cortex, 1955-78. Nature, 299:515- 524,1981. [Huf71] D . A . Huffman. Impossible objects as nonsense sentences. D . M i c h i e , editors. Machine Intelligence In B . M e l t z e r a n d 6, pages 295-323. E d i n b u r g h : E d i n b u r g h U n i v e r s i t y Press, 1971. [HW74] D . H . H u b e l a n d T . N . W i e s e l . Sequence regularity and geometry of o r i e n t a t i o n columns i n monkey striate cortex. J. Comparative [HZ83] Neurology, 158:267-294, 1974. R . A . H u m m e l a n d S . W . Zucker. O n the foundations of relaxation labehng p r o cesses. IEEE Transactions on Pattern Analysis and Machine 5:267- Intelligence, 287,1983. [JB83] B . Julesz a n d J . R . Bergen. Textons, the fundamental elements i n preattentive vision a n d perception of textures. Bell System Technical Journal, 62:1619-1645, 1983. [Joh90] D . S . J o h n s o n . A catalog of complexity classes. In J . van Leeuwen, editor. of Theoretical Computer Science, Handbook Vol. A, pages 6 9 - 1 6 1 . A m s t e r d a m : E l s e v i e r , 1990. [Jul81] B . Julesz. Textons, the elements of texture perception, a n d their interactions. Nature, [Jul84a] 290:91-97, 1981. B . Julesz. A brief outhne of the t e x t o n theory of h u m a n v i s i o n . Trends in Neuro- science, 7:41-45, 1984. [Jul84b] B . Julesz. T o w a r d an axiomatic theory of preattentive vision. In G . M . E d e k n a n , W . E . G a l l , a n d W . M . C o w a n , editors. Dynamic Aspects of Neocortical Function, pages 585-612. Neurosciences Research F o u n d a t i o n , 1984. [Jul86] B . Julesz. T e x t o n gradients: T h e t e x t o n theory revisited. Biological Cybernetics, 54:245-261, 1986. [Jul87] B . Julesz. Preattentive h u m a n v i s i o n , a h n k between neurophysiology and psychophysics. In F . P l u m et a l . , editor. Handbook P h y s i o l o g y Society, 1987. of Physiology, Vol V. A m e r i c a n [KanSO] T . K a n a d e . A theory of o r i g a m i w o r l d . Artificial [Kan90] K . K a n a t a n i . Group-Theoretical Intelligence, 13:279-311, 1980. Methods in Image Understanding. New Y o r k a n d B e r h n : Springer, 1990. [Kar84] N . K a r m a r k a r . A new p o l y n o m i a l time algorithm for hnear p r o g r a m m i n g . binatorica, [Kar90] Com- 4:373-395, 1984. J . K a r l . Reversibihty of 2d cehular a u t o m a t a is undecidable. Physica D, 4 5 : 3 7 9 - 385, 1990. [Kha79] L . G . K h a c h i y a n . A p o l y n o m i a l a l g o r i t h m for hnear p r o g r a m m i n g . Soviet Math. Dokl., 20:191-194, 1979. [KI85] J . K i t t l e r and J . Ihingworth. R e l a x a t i o n labelling algorithms - a review. and Vision [KP85] 3:206-216, 1985. Computing, L . M . K i r o u s i s and C H . P a p a d i m i t r i o u . T h e complexity of recognising p o l y h e d r a l scenes. In 26th FOCS, [KP88] 1985. L . M . K i r o u s i s and C H . P a p a d i m i t r i o u . T h e complexity of recognizing p o l y h e d r a l scenes. Journal [KU84] Image C and System Sciences, 37:14-38, 1988. of Computer K o c h and S. U h m a n . Selecting one among the m a n y : A simple network i m - plementing shifts i n selective v i s u a l attention. Technical Report A . I . M e m o 770, M I T , 1984. [Kul87] Z. K u l p a . P u t t i n g order i n the impossible. Perception, [Lak78] I. L a k a t o s . The Methodology of Scientific Research 16:201-214, 1987. Programmes. Cambridge: C a m b r i d g e University Press, 1978. [LAN89] W . L i m , A . A g r a w a l , and L . Nekludova. A fast parahel a l g o r i t h m for labehng connected components i n image arrays. In P . M . D e w , R . A . E a r n s h a w , and T . R . H e y w o o d , editors. Parallel Processing for Computer Vision and Display, pages 169-179. R e a d i n g , M A : Addison-Wesley, 1989. [LBC89] J . J . L i t t l e , G . E . Blehoch, and T . A . Cass. A l g o r i t h m i c techniques for vision o n a fine-grained paraUel machine. IEEE and Machine [Lee71] Intelligence, E . L . J . Leeuwenberg. terns. American [Lev86] 11:244-257, Transactions Analysis 1989. A perceptual coding language for v i s u a l and auditory p a t - Journal of Psychology, 84:307-350, 1971. H . J . Levesque. M a k i n g behevers out of computers. Artificial 108, 1986. on Pattern computer Intelligence, 30:81- [Low85] D . Lowe. Perceptual Organization and Visual Recognition. Boston: Kluwer A c a - demic, 1985. [Low87] D . Lowe. T h e viewpoint consistency constraint. International Vision, [Mac73] Journal of Computer 1:57-72, 1987. A . K . M a c k w o r t h . Interpreting pictures of p o l y h e d r a l scenes. Artificial Intelligence, 4:121-137, 1973. [Mac74] A . K . M a c k w o r t h . On the Interpretation of Drawings as Three-Dimensional Scenes. D . P h i h thesis. U n i v e r s i t y of Sussex, J a n u a r y 1974. [Mac76] A . K . M a c k w o r t h . M o d e l - d r i v e n interpretation i n inteUigent vision systems. Per- ception, 5 ( 3 ) : 3 4 9 - 3 7 0 , 1 9 7 6 . [Mac77] A . K . Mackworth. Consistency i n networks of relations. Artificial Intelligence, 8:99-118, 1977. [Mac91] A . K . M a c k w o r t h . T h e logic of constraint satisfaction. Technical Report T R - 9 1 - 2 6 , Department of C o m p u t e r Science, University of B r i t i s h C o l u m b i a , 1991. [Mal87] J . M a h k . Interpreting hne drawings of curved objects. Int. J. Computer Vision, 1:73-103, 1987. [Mar79] D . M a r r . Representing and computing v i s u a l i n f o r m a t i o n . In Artificial An MIT Perspective, Intelligence: pages 15-80. C a m b r i d g e , M A : M I T Press, 1979. [Mar82] D . M a r r . Vision. San Francisco: W . H . Freeman, 1982. [MD90] J . M u l d e r and R . J . M . D a w s o n . Reconstructing p o l y h e d r a l scenes f r o m single two-dimensional images: T h e orthogonahty hypothesis. In P . K . Patel-'Schneider, editor, Proceedings [MF85] of the 8th Biennial Conf. of the CSCSI, pages 238-244, 1990. A . K . M a c k w o r t h and E . C . Freuder. T h e complexity of some p o l y n o m i a l network consistency algorithms for constraint satisfaction problems. Artificial Intelligence, pages 65-74, 1985. [ M M H 8 5 ] A . K . M a c k w o r t h , J . A . M u l d e r , and W . S . Havens. H i e r a r c h i c a l arc E x p l o i t i n g structured domains i n constraint satisfaction problems. Intelligence, [MN87] Computational 1:71-79, 1985. J . H . M a u n s e U and W . T . Newsome. V i s u a l processing i n monkey extrastriate cortex. Ann. [MP69] consistency: Rev. Neurosci., 10:363-401,1987. M . M i n s k y and S. P a p e r t . The Perceptron: C a m b r i d g e , M A : M I T Press, 1969. Principles of Computational Geometry. [MS87] R . M i l l e r a n d Q . F . S t o u t . D a t a movement techniques for the p y r a m i d architecture. SI AM [Nei63] Journal 16:38-60, 1987. U . Neisser. Decision time w i t h o u t reaction time: Experiments i n v i s u a l scanning. American [NKP87] of Computing, Journal of Psychology, 76:376-385, 1963. L . N i , C . T . K i n g , and P. P r i n s . Parallel algorithm considerations for hypercube management. In Proceedings of the 1987 Intl. Conf. on Parallel Processing, pages 717-720, 1987. [Not91] H . C . N o t h d u r f t . Different effects from spatial frequency masking i n texture segregation and t e x t o n detection tasks. Vision Research, 31:299-320, 1991. [NS86] K . N a k a y a m a a n d G . H . Silverman. Serial and parahel processing of v i s u a l feature conjunctions. Nature, [PC80] 320:264-265, 1986. D . N . Perkins and R . G . Cooper J r . How the eye makes up what the hght leaves out. In M . A . H a g e n , editor. The Perception Vol II, pages 95-130. of Pictures: N e w Y o r k , N Y : A c a d e m i c , 1980. [PD84] K . P r e s t o n , J r a n d M . J . B . Duff. Modern cations. [PDL+79] Cellular Automata: Theory and K . P r e s t o n , J r . , M . J . B . Duff, S. L e v i a l d i , P . E . N o r g r e n , and J . - I . T o r i w a k i . Basics of cehular logic w i t h some apphcations i n medical image processing. Proc. 67:826-856, [PE90] IEEE, 1979. T . Poggio and S. E d e l m a n . A network that learns to recognize three-dimensional objects. Nature, [Per68] Appli- New Y o r k : P l e n u m , 1984. 343:263-266, 1990. D . N . P e r k i n s . C u b i c corners. Technical Report Quarterly Progress Report N o . 89, Research L a b o r a t o r y of Electronics, M I T , 1968. [Per72] D . N . P e r k i n s . V i s u a l discrimination between rectangular and nonrectangular p a r aUelopipeds. Perception & Psychophysics, 12:396-400, 1972. [Per76] D . N . P e r k i n s . H o w good a bet is good form? [Per82] D . N . P e r k i n s . T h e perceiver as organizer and geometer. In J . Beck, editor. nization and Representation in Perception, Perception, 5:393-406, 1976. Orga- pages 73-93. Hihsdale, N J : E r l b a u m , 1982. [PTK85] T . Poggio, V . T o r r e , and C . K o c h . C o m p u t a t i o n a l vision and regularization theory. Nature, 317:314-319,1985. [Rab84] P . R a b b i t t . T h e control of attention i n visual search. In Varieties of Attention, pages 2 7 3 - 2 9 1 . New Y o r k , N Y : A c a d e m i c , 1984. [Ram85] V . S . R a m a c h a n d r a n . T h e neurobiology of perception. Perception, [Ram88] V . S . Ramachandran. Perceiving shape from shading. Sci. 14:97-103,1985. Am., 259:76-83, A u g 1988. [Ree84] A . P . Reeves. Vision, [Res82] P a r a h e l computer architectures for image processing. Graphics, and Image Processing, Computer 25:68-88, 1984. F . Restle. C o d i n g theory as an integration of gestalt psychology a n d i n f o r m a t i o n theory. In J . Beck, editor, Organization and Representation pages in Perception, 31-56. Hillsdale, N J : E r l b a u m , 1982. [Ric88] W . R i c h a r d s . Image i n t e r p r e t a t i o n : Information at contours. editor. Natural [RM89] Computation, pages 17-36. C a m b r i d g e , M A : M I T Press, 1988. R . Reiter a n d A . K . M a c k w o r t h . A logical framework for depiction a n d image i n t e r p r e t a t i o n . Artificial [Rob65] In W . R i c h a r d s , Intelligence, pages 125-155, 1989. L . G . Roberts. M a t c h i n g perception of three-dimensional sohds. Electro-optical Information pages 159-197. Processing, In Optical and Cambridge, M A : M I T Press, 1965. [RobSO] J . G . R o b s o n . N e u r a l images: T h e physiological basis of s p a t i a l vision. In C . S . H a r ris, editor. Visual Coding and Adaptability, pages 177-214. Hihsdale, N J : E r l b a u m , 1980. [Ros83] A . Rosenfeld. P a r a l l e l image processing using ceUular arrays. Computer, 16:14-20, 1983. [Ros86] A . Rosenfeld. P y r a m i d algorithms for perceptual organization. Behavior Methods, [Ros87] & Computers, 18:595-600, 1986. A . Rosenfeld. Recognizing unexpected objects: A proposed approach. In Proc. the DARPA [RP91] Instruments, Research Image Understanding of Workshop, pages 620-627, 1987. R . A . Rensink a n d G . P r o v a n . T h e analysis of resource-hmited vision systems. In Proc. Thirteenth Annual Conf. of the Cognitive Science Society, pages 311-316, 1991. [Sal85] S . N . Salthe. Evolving 1985. Hierarchical Systems. New Y o r k : C o l u m b i a U n i v e r s i t y Press, [SBG89] A . S u t t e r , J . B e c k , and N . G r a h a m . Contrast and spatial variables i n t e x t u r e segregation: Testing a simple spatial-frequency channels model. Perception & 46:312-332, 1989. Psychophysics, [Sch86] P . H . SchlUer. T h e central visual system. Vision [She81] R . N . Shepard. P s y c h o p h y s i c a l complementarity. In M . K u b o v y and J . R . P o m e r a n t z , editors, Perceptual Research, 26:1351-1386,1986. pages 279-341. Hihsdale, N J : E r l b a u m , Organization, 1981. [She83] G . M Shepherd. Neurobiology. Newr Y o r k a n d O x f o r d : O x f o r d U n i v e r s i t y P r e s s , 1983. [Sim81] H . A . S i m o n . The Sciences of the Artificial (2nd ed.). C a m b r i d g e , M A : M I T Press; N e w Y o r k : M c G r a w - H i U , 1981. [Smi90] M . A . S m i t h . Representations of geometrical a n d topological quantities i n cehular a u t o m a t a . Physica [Ste78] D, 45:271-277, 1990. K . A . Stevens. C o m p u t a t i o n of locaUy parallel structure. Biological Cybernetics, 29:19-28, 1978. [Sto87] Q . F . S t o u t . P y r a m i d algorithms o p t i m a l for the worst case. Parallel [Sto88] In L . U h r , e d i t o r . Vision, pages 147-168. N e w Y o r k , N Y : A c a d e m i c , 1987. Computer Q . F . S t o u t . M a p p i n g vision algorithms onto parahel architectures. Proc. IEEE, 76:982-995, 1988. [Sug86] K . S u g i h a r a . Machine Interpretation of Line Drawings. Cambridge, M A : M I T Press; N e w Y o r k : M c G r a w - H i h , 1986. [SV82] Y . Shiloach a n d U . V l s h k i n . A n O(log n) paraUel connectivity a l g o r i t h m . of Algorithms, [Tan84] S.L. Tanimoto. 3:57-67,1982. S o r t i n g , h i s t o g r a m m i n g , and other statistical operations on a p y r a m i d machine. In A . Rosenfeld, editor, Multiresolution Analysis, [TB90] Journal Image Processing and pages 136-145. New Y o r k a n d B e r h n : Springer, 1984. J . T . T o d d a n d P . Bressan. T h e perception of 3-dimensional affine structure f r o m m i n i m a l apparent m o t i o n sequences. Perception & Psychophysics, 48:419-430, 1990. [TG79] J . P . T h o m a s a n d J . G i h e . B a n d w i d t h s of orientation channels i n h u m a n v i s i o n . Opt. Soc. Am. A, 5:652-660, 1979. [Tho72] G . B . T h o m a s . Calculus and Analytic Geometry. Reading, M A : A d d i s o n Wesley, 1972. [TM87] T . Toffoh and N . H . M a r g o l u s . Cellular [TM90] Automata Machines: A New Environment C a m b r i d g e , M A : M I T Press, 1987. for Modeling. T . TofFoh and N . H . M a r g o l u s . Invertible ceUular a u t o m a t a : A review. Physica D, 45:229-253,1990. [Tow72] J . T . Townsend. Some results concerning the identifiabihty of paraUel and serial processes. British [Tre82] J. Math. Statist. A . T r e i s m a n . P e r c e p t u a l grouping and attention in v i s u a l search for features a n d for objects. J. of Experimental [Tre88] A . Treisman. Quart. [Tre91] J. of Experimental Human J . K . Tsotsos. of Experimental Psychology: Human Perception Perception and Performance, of and Experimental Journal of 1:346-355, 1987. A n a l y z i n g V i s i o n at the C o m p l e x i t y Level. Behavioral and Brain 13, 1990. J . F . T r a u b , G . W . WasiUcowski, and H . W o z n i a k o w s k i . Information-Based plexity. Perfor- 16:459-478,1990. A "complexity L e v e l " analysis of vision. International Vision, J . K . Tsotsos. Sciences, [Uhr87] 4 0 A : 2 0 1 - 2 3 7 , 1988. A . Treisman and S. Sato. C o n j u n c t i o n search revisited. Journal Computer [TWW88] Psychology, 17:652-676,1991. Psychology: [Tso90] 8:194-214, 1982. A . T r e i s m a n . Search, similarity, and integration of features between and w i t h i n mance, [Tso87] Psychology, Features and objects: T h e fourteenth bartlett m e m o r i a l lecture. dimensions. Journal [TS90] 25:168-199, 1972. Psychol., Com- New Y o r k , N Y : A c a d e m i c , 1988. L . U h r . H i g h l y paraUel, hierarchical, recognition cone perceptual structures. In L . U h r , editor. Parallel Computer Vision, pages 249-292. N e w Y o r k , N Y : A c a - demic, 1987. [UU84] S. U U m a n . V i s u a l routines. Cognition, [Vol82] R . V o U m a r . Some remarks about the efficiency of p o l y a u t o m a t a . Journal [VP88] of Theoretical Physics, 18:97-159, 1984. 21:1007-1015, International 1982. H . Voorhees and T . Poggio. C o m p u t i n g texture boundaries f r o m images. 333:364-367,1988. Nature, [Wal72] D . L . W a l t z . G e n e r a t i n g semantic descriptions from drawings of scenes w i t h shadows. Technical R e p o r t A I - T R - 2 7 1 , M I T , 1972. [Wal87] D . W a l t e r s . Selection of image primitives for general purpose visual processing. Computer [Wat87] Graphics and Image Processing, 37:261-298, 1987. A . B . W a t s o n . Efficiency of a m o d e l image code. J. Opt. Soc. Am. A, 4:2401-2417, 1987. [Wat88] R . J . W a t t . Visual Processing. Hihsdale, N J : E r l b a u m , 1988. [WB82] J . M . Woodhouse a n d H . B . B a r l o w . Spatial and t e m p o r a l resolution a n d analysis. In H . B . B a r l o w a n d J . D . M o h o n , editors. The Senses, pages 133-164. C a m b r i d g e : C a m b r i d g e U n i v e r s i t y Press, 1982. [WH74] N . Weisstein a n d C . S . H a r r i s . V i s u a l detection of hnes: A n object-superiority effect. Science, 186:752-755, 1974. [W1190] F . W i l k i n s o n . Texture segmentation. In W . C . Stebbins and M . A . Berkley, editors. Comparative Perception - Vol II: Complex Signals, pages 125-156. New Y o r k : J o h n W i l e y a n d Sons, 1990. [WM78] N . Weisstein and W . M a g u i r e . C o m p u t i n g the next step: Psychophysical measures of representation a n d i n t e r p r e t a t i o n . In A . R . Hansen a n d E . M . R i s e m a n , editors. Computer [WM85] Vision Systems, pages 243-260. N e w Y o r k , N Y : A c a d e m i c , 1978. R . J . W a t t and M . J . M o r g a n . A theory of the p r i m i t i v e s p a t i a l code i n h u m a n vision. Vis. Res., 25:1661-1674, 1985. [Woo81] R . J . W o o d h a m . A n a l y s i n g images of curved surfaces. Artificial Intelligence, 9:117- 140,1981. [ZSS83] S . W . Zucker, K . A . Stevens, and P. Sander. T h e relation between p r o x i m i t y a n d brightness similarity i n dot patterns. Perception & Psychophysics, 34:513-522, 1983. [Zuc86] S. Zucker. E a r l y orientation selection: Tangent fields a n d the dimensionahty of their support. In A . Rosenfeld, editor. Human and Machine Vision II, pages 335-364. New Y o r k , N Y : A c a d e m i c , 1986. [Zuc87a] S. Zucker. T h e diversity of perceptual grouping. In M . A . A r b i b and A . R . H a n s o n , editors, Vision, Brain, and Co-operative Computation, pages 231-261. C a m b r i d g e , M A : M I T Press, 1987. [Zuc87b] S. Zucker. E a r l y vision. In S . C . Shapiro, editor, Encyclopedia of Artificial gence, pages 1131-1152. New Y o r k : J o h n W i l e y and Sons, 1987. Intelli-
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- The rapid recovery of three-dimensional structure from...
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
The rapid recovery of three-dimensional structure from line drawings Rensink, Ronald Andy 1992
pdf
Page Metadata
Item Metadata
Title | The rapid recovery of three-dimensional structure from line drawings |
Creator |
Rensink, Ronald Andy |
Date Issued | 1992 |
Description | A computational theory is developed that explains how line drawings of polyhedral objects can be interpreted rapidly and in parallel at early levels of human vision. The key idea is that a time-limited process can correctly recover much of the three-dimensional structure of these objects when split into concurrent streams, each concerned with a single aspect of scene structure. The work proceeds in five stages. The first extends the framework of Marr to allow a process to be analyzed in terms of resource limitations. Two main concerns are identified: (i) reducing the amount of nonlocal information needed, and (ii) making effective use of whatever information is obtained. The second stage traces the difficulty of line interpretation to a small set of constraints. When these are removed, the remaining constraints can be grouped into several relatively independent sets. It is shown that each set can be rapidly solved by a separate processing stream, and that co-ordinating these streams can yield a low-complexity "approximation" that captures much of the structure of the original constraints. In particular, complete recovery is possible in logarithmic time when objects have rectangular corners and the scene-to-image projection is orthographic. The third stage is concerned with making good use of the available information when a fixed time limit exists. This limit is motivated by the need to obtain results within a time independent of image content, and by the need to limit the propagation of inconsistencies. A minimal architecture is assumed, viz., a spatiotopic mesh of simple processors. Constraints are developed to guide the course of the process itself, so that candidate interpretations are considered in order of their likelihood. The fourth stage provides a specific algorithm for the recovery process, showing how it can be implemented on a cellular automaton. Finally, the theory itself is tested on various line drawings. It is shown that much of the three-dimensional structure of a polyhedral scene can indeed be recovered in very little time. It also is shown that the theory can explain the rapid interpretation of line drawings at early levels of human vision. |
Extent | 10129176 bytes |
Genre |
Thesis/Dissertation |
Type |
Text |
File Format | application/pdf |
Language | eng |
Date Available | 2008-12-17 |
Provider | Vancouver : University of British Columbia Library |
Rights | For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use. |
DOI | 10.14288/1.0051339 |
URI | http://hdl.handle.net/2429/3048 |
Degree |
Doctor of Philosophy - PhD |
Program |
Computer Science |
Affiliation |
Science, Faculty of Computer Science, Department of |
Degree Grantor | University of British Columbia |
Graduation Date | 1992-11 |
Campus |
UBCV |
Scholarly Level | Graduate |
Aggregated Source Repository | DSpace |
Download
- Media
- 831-ubc_1992_fall_rensink_ronald_andy.pdf [ 9.66MB ]
- Metadata
- JSON: 831-1.0051339.json
- JSON-LD: 831-1.0051339-ld.json
- RDF/XML (Pretty): 831-1.0051339-rdf.xml
- RDF/JSON: 831-1.0051339-rdf.json
- Turtle: 831-1.0051339-turtle.txt
- N-Triples: 831-1.0051339-rdf-ntriples.txt
- Original Record: 831-1.0051339-source.json
- Full Text
- 831-1.0051339-fulltext.txt
- Citation
- 831-1.0051339.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0051339/manifest