UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Understanding stochastic local search algorithms : an empirical analysis of the relationship between… Smyth, Kevin R.G. 2004

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2004-0632.pdf [ 10.76MB ]
Metadata
JSON: 831-1.0051395.json
JSON-LD: 831-1.0051395-ld.json
RDF/XML (Pretty): 831-1.0051395-rdf.xml
RDF/JSON: 831-1.0051395-rdf.json
Turtle: 831-1.0051395-turtle.txt
N-Triples: 831-1.0051395-rdf-ntriples.txt
Original Record: 831-1.0051395-source.json
Full Text
831-1.0051395-fulltext.txt
Citation
831-1.0051395.ris

Full Text

Understanding Stochastic Local Search Algorithms An Empirical Analysis of the Relationship Between Search Space Structure and Algorithm Behaviour by Kevin R. G. Smyth B.Sc, The University of British Columbia, 2002 A THESIS S U B M I T T E D I N P A R T I A L F U L F I L M E N T OF T H E R E Q U I R E M E N T S FOR T H E D E G R E E OF M A S T E R OF S C I E N C E in The Faculty of Graduate Studies (Department of Computer Science) We accept this thesis as conforming to the required standard T H E UNIVERSITY OF BRITISH C O L U M B I A September 27, 2004 © Kevin R. G. Smyth, 2004 U B C i THEJJ^JIVERSITY OF BRITISH COLUMBIA FACULTY OF GRADUATE STUDIES Library Authorization for scholarly purposes may be granted bv the H P . H NT M • Permission for extensive copying of this thesi Name of Author (please print) Date (dd/mm/yyyy) Title of Thesis: Degree: Year: Departmentof ^mm^^cz The University of British Columbia Vancouver, B C Canada pagel of 1 test* uodi Abstract ii Abstract Combinatorial optimisation problems are an important and well-studied class of problems, with applica-tions in most areas of the computing sciences. Because of their prominence, combinatorial optimisation problems and their related decision problems have been the focus of extensive research for several decades. The propositional satisfiability problem (SAT), in particular, has been the focus of a vast amount of research, and a class of algorithms known as stochastic local search (SLS) algorithms has emerged as the state-of-the art on a variety of SAT problem classes. Much of the recent progress in algorithm development has been facilitated by an improved understand-ing of the properties of SAT instances and of high-performance SAT algorithms. This thesis studies the search space features underlying the behaviour of stochastic local search algorithms for SAT, extending ex-isting results from the literature and providing novel contributions. Search space features such as plateaus and the interconnectivity between plateaus are defined and studied on a variety of SAT instances, and it is empirically demonstrated that features such as these are responsible for the wide range in instance hard-ness observed in distributions of syntactically identical SAT instances. Furthermore, the novel concept of the performance criticality of variables in SAT instances is introduced, and the connections between search space structure and performance criticality are investigated. Finally, methods for practically exploiting the knowledge gained by this search space analysis are briefly explored, with the goal of improving the state-of-the-art in SAT solving. Contents i i i Contents Abstract • 1 1 Contents i i i List of Tables v i List of Figures v m List of Algorithms x Acknowledgements x i 1 Introduction 1 1.1 P r o b l e m D e f i n i t i o n a n d M o t i v a t i o n 2 1.2 E x i s t i n g W o r k 3 1.2.1 N u m b e r o f S o l u t i o n s 3 1.2.2 S o l u t i o n C l u s t e r i n g 4 1.2.3 B a c k b o n e F r a g i l i t y 5 1.2.4 L o c a l M i n i m a 6 1.2.5 E f f e c t o f S e a r c h S p a c e S t r u c t u r e o n A l g o r i t h m B e h a v i o u r 7 1.2.6 P l a t e a u D i s t r i b u t i o n 8 1.2.7 P r e d i c t i n g L o c a l S e a r c h C o s t 9 1.3 T h e s i s O u t l i n e H 2 Preliminaries 12 2.1 P r o p o s i t i o n a l F o r m u l a e 12 2 .2 S e a r c h S p a c e C o m p o n e n t s 1 4 2 .3 S L S A l g o r i t h m s f o r S A T 15 2 .4 E m p i r i c a l M e t h o d o l o g y 17 Contents iv 3 Plateau Characteristics 20 3.1 Definitions and Motivation 21 3.1.1 State Types 21 3.1.2 Plateaus • 22 3.2 Plateau characteristics of the UF-3-SAT distribution 23 3.2.1 Experimental Methodology 24 3.2.2 Fraction of Closed Plateaus 24 3.2.3 Plateau Size 25 3.2.4 Discussion 27 3.3 Plateau characteristics of individual UF-3-SAT instances 28 3.3.1 Experimental Methodology 28 3.3.2 Number of Plateaus 32 3.3.3 Fraction of Closed Plateaus 33 3.3.4 Plateau Size 35 3.4 Plateau characteristics of structured instances 37 3.4.1 Number of Plateaus 37 3.4.2 Plateau Size 39 3.5 Internal Plateau Structure 41 3.5.1 Branching Factor 46 3.5.2 Diameter 48 3.5.3 L M I N State Fraction 48 3.5.4 Exit Distance 50 3.6 Summary 52 4 Plateau Connectivity 54 4.1 Definitions and Motivation 54 4.2 P C G Properties 60 4.2.1 Out-Degree 62 4.2.2 Target Depth 63 4.3 Solution Reachability 65 4.4 Properties of Traps in the Search Space 69 4.4.1 Attractivity 70 4.4.2 Escape Difficulty 72 4.4.3 Connection With Mixture Models 75 4.5 Summary 77 Contents v 5 Performance Critical Variables 7 9 5.1 M o t i v a t i o n 8 0 5 .2 O r a c l e s a n d P e r f o r m a n c e C r i t i c a l i t y 81 5.3 P e r f o r m a n c e C r i t i c a l V a r i a b l e s f o r S L S A l g o r i t h m s 8 4 5.4 P e r f o r m a n c e C r i t i c a l V a r i a b l e s f o r D P L L A l g o r i t h m s 8 9 5.5 D i s c u s s i o n 91 5.5.1 C o n n e c t i o n w i t h S e a r c h S p a c e F e a t u r e s 9 2 5 .5 .2 A n A p p r o x i m a t e P C V O r a c l e 9 5 5.6 S u m m a r y 9 7 6 Conclusions and Future Work 9 8 Bibliography 1 0 2 A Test Suite Description 1 0 7 B Binary Decision Diagrams 110 List of Tables v i List of Tables 3.1 Number of plateaus in UF-3-SAT instances, counting all plateaus 32 3.2 Number of plateaus in UF-3-SAT instances, counting only open plateaus 34 3.3 Number of plateaus in UF-3-SAT instances, counting only closed plateaus 34 3.4 Size of closed plateaus in UF-3-SAT instances 35 3.5 Fraction of states contained in largest plateau for the UF3SAT test sets 35 3.6 Size of open plateaus in UF-3-SAT instances 36 3.7 Number of plateaus in structured instances 37 3.8 Fraction of closed plateaus in structured instances 38 3.9 Size of closed plateaus in structured instances 39 3.10 Fraction of states contained in largest plateau for the structured instances 40 3.11 Size of open plateaus in structured instances 41 3.12 Summary statistics of the distribution of branching factors for the structured instances . . . . 46 3.13 Lower bound on plateau diameters for the structured instances 47 3.14 L M I N state fraction for the structured instances 50 3.15 Exit distance statistics for the structured instances 51 4.1 Summary statistics of the distribution of the out-degree of P C G nodes i n UF-3-SAT instances 62 4.2 Summary statistics of the distribution of the out-degree of P C G nodes in structured instances 62 4.3 Summary statistics of the distribution of the average target depth of P C G nodes in UF-3-SAT instances 64 4.4 Summary statistics of the distribution of the maximum target depth of P C G nodes in UF-3-SAT instances 64 4.5 Summary statistics of the distribution of the average target depth of P C G nodes in structured instances 65 4.6 Summary statistics of the distribution of the maximum target depth of P C G nodes in struc-tured instances 65 4.7 Correlation between p(S | A) and Isc for the UF-3-SAT test sets 68 List of Tables v i i 4.8 p(S | A) values for the structured instances 69 4.9 Statistics from the distribution of ocmnx values for the UF-3-SAT test sets 71 4.10 Correlation between ocmax and Isc for the UF-3-SAT test sets 72 4.11 Summary statistics for the distribution of emax values for the UF-3-SAT test sets 74 5.1 P C V results for SLS algorithms 86 5.2 P C V results for DPLL-type algorithms 90 A . l Description of the random benchmark sets studied in this paper 108 A.2 Description of the structured benchmark sets studied in this paper 108 List of Figures v i i i List of Figures 2.1 Example cumulative distribution functions 18 3.1 Fraction of closed plateaus sampled at each level 25 3.2 Distribution of plateau sizes (UF-3-SAT, 20 variables) 26 3.3 Distribution of plateau sizes (UF-3-SAT, 50 variables) 26 3.4 Distribution of plateau sizes (UF-3-SAT, 100 variables) 26 3.5 Distribution of the number of plateaus at each level over instances in the uf30 test set 32 3.6 Scaling of the average number of plateaus wi th the number of variables i n the UF3SAT test sets 33 3.7 Average fraction of closed plateaus at each level for the UF3SAT test suites 34 3.8 P%,ax from the instance in the uflO test set wi th median Isc 43 3.9 A n example level 3 plateau from the a i s 6 instance 44 3.10 A n example level 5 plateau from the p a r i t y 8 instance 45 3.11 Distribution of branching factors for the 20,50, and 100 variable UF-3-SAT test sets 46 3.12 Lower bound on plateau diameters for 20 variable UF -3 -SAT test set 47 3.13 Distribution of the fraction of L M I N states for the 20,50, and 100 variable UF-3-SAT test sets 49 3.14 Distribution of exit distances for the 20, 50, and 100 variable UF - 3 - S A T test sets 51 4.1 P C G for an easy uflO instance 58 4.2 P C G for the hardest ufZO instance 59 4.3 P C G for the structured anom instance 61 4.4 Correlation between p(S | A) and Isc for the uf30 test set 67 4.5 Distribution of ocmax values for the UF -3 -SAT test sets 70 4.6 Correlation between ocmax and both Isc and p(S | A) for the uf-30 test set 71 4.7 Shifted RLDs from instances from the uf30 test set 73 4.8 Distribution of emax values for the UF -3 -SAT test sets 74 4.9 Correlation between a m a x and emax for the uf30 test set 75 4.10 R L D for hard instance from the uf30 test set, and approximation by mixed exponential dis-tribution 76 List of Figures i x 5.1 D i s t r i b u t i o n o f c r i t i c a l i t y f a c t o r s w i t h r e s p e c t t o N o v e l t y + f o r v a r i a b l e s i n t h e h a r d e s t uflOO i n s t a n c e 8 5 5.2 C o r r e l a t i o n b e t w e e n s e a r c h c o s t a n d P C V f r a c t i o n f o r t he uflOO b e n c h m a r k se t u s i n g t he N o v e l t y " 1 " a l g o r i t h m 8 7 5.3 C o r r e l a t i o n o f c r i t i c a l i t y f a c t o r s w i t h r e s p e c t t o d i f f e r e n t S L S a l g o r i t h m s 8 8 5.4 A d d i t i v i t y o f P C V A s u s i n g d y n a m i c r a n k i n g f o r N o v e l t y + 8 8 5.5 C o r r e l a t i o n o f c r i t i c a l i t y f a c t o r s w i t h r e s p e c t to d i f f e r e n t a l g o r i t h m s o n t h e b w a i n s t a n c e . . . 9 0 5.6 C o r r e l a t i o n b e t w e e n a v e r a g e c r i t i c a l i t y f a c t o r a n d t w o s e a r c h s p a c e f e a t u r e s f o r N o v e l t y + o n t h e ufZ5 tes t s e t 9 2 5.7 P a r t i a l , w e i g h t e d P C G f o r t h e h a r d e s t ufl5 i n s t a n c e 9 3 5.8 E f f e c t o n t h e P C G w h e n i n s t a n t i a t i n g m o s t c r i t i c a l v a r i a b l e i n h a r d e s t uf!5 i n s t a n c e 9 4 5.9 C o r r e l a t i o n b e t w e e n 7 a n d c r i t i c a l i t y f a c t o r f o r N o v e l t y " * " o n i n s t a n c e b w a 9 6 B . l C o m p a r i n g T r u t h T a b l e s , B i n a r y T r e e s , a n d B i n a r y D e c i s i o n D i a g r a m s 110 List of Algorithms x List of Algorithms 2.1 GSAT(J", maxTries, maxSteps) 16 2.2 WalkSAT/SKCXJF, maxTries, maxSteps, p) 17 3.1 Levelset(Jr, maxLev) 29 3.2 PartitionfLeueZsef*) 31 Acknowledgements x i Acknowledgements T h i s t h e s i s w o u l d h a v e n e v e r b e e n c o m p l e t e d w i t h o u t t he s u p p o r t o f m a n y p e o p l e . F i r s t a n d f o r e m o s t , I w o u l d l i k e t o t h a n k L i s a W o n g , w h o m I l o v e d e e p l y a n d t r u l y , f o r b e i n g a c o n s t a n t s o u r c e o f m o t i v a t i o n , f o r r e m i n d i n g m e t o t a k e t i m e t o e n j o y t h e s m a l l t h i n g s i n l i f e , a n d f o r p u t t i n g u p w i t h a l l o f t h e l a t e n i g h t s a n d w e e k e n d h o u r s r e q u i r e d t o s e e t h i s t h e s i s t h r o u g h to c o m p l e t i o n . M y t h a n k s g o o u t to m y f a m i l y f o r t h e i r c o n s t a n t i n t e r e s t i n m y r e s e a r c h , e v e n t h o u g h I n e v e r t o o k t h e t i m e t o g i v e a s a t i s f y i n g e x p l a n a t i o n o f i t . M a n y t h a n k s a r e d u e to m y s u p e r v i s o r H o l g e r H o o s f o r h i s s u p p o r t a n d e n c o u r a g e m e n t t h r o u g h o u t m y y o u n g a c a d e m i c c a r e e r , f o r g i v i n g m e t h e f r e e d o m to p u r s u e a l l o f t h o s e w i l d i d e a s , a n d f o r t he m i d n i g h t o i l b u r n t d u r i n g j o i n t w o r k . T h a n k s t o b o t h H o l g e r H o o s a n d N a n d o d e F r e i t a s f o r i m p r o v i n g t h i s t h e s i s w i t h i n s i g h t f u l c o m m e n t s a n d s u g g e s t i o n s . F i n a l l y , t h e r e a r e m a n y p e o p l e i n t h e c o m p u t e r s c i e n c e d e p a r t m e n t w h o h a v e m a d e m y l i f e e a s i e r a n d m o r e e n j o y a b l e o v e r t h e y e a r s . I w o u l d l i k e t o t h a n k a l l o f t h e m e m b e r s o f t he /3-lab f o r t h e i r h e l p a n d f o r i n t e r e s t i n g d i s c u s s i o n s , i n p a r t i c u l a r D a v e T o m p k i n s , K e v i n L e y t o n - B r o w n , S t e p h D u r o c h e r , a n d D a n T u l p a n . F i n a l l y , I a m i n d e b t e d t o V a l e r i e M c R a e f o r a l l o f h e r i n v a l u a b l e a s s i s t a n c e . Chapter 1. Introduction 1 Chapter 1 Introduction Combinatorial optimisation problems are an important and well-studied class of problems, with applica-tions in most areas of the computing sciences [GJ79]. Combinatorial optimisation problems can generally be described as computational problems in which all variables are discrete, and the goal is to find the best pos-sible configuration among the set of variables, where quantifying what constitutes the "best" configuration is highly problem-dependent [PS82]. Because of their prominence, combinatorial optimisation problems have been the focus of extensive research for several decades. Unfortunately, many interesting combinato-rial optimisation problems are known to be AfV-hard. Therefore, under the widely held assumption that V != MV, any exact algorithm solving many of these problems must, in the worst case, require at least an exponential amount of time in the problem size. Combinatorial optimisation problems (finding solutions that minimise a cost metric) are closely related to the corresponding decision problems (deciding whether there exists a solution with a given cost) [SW02]. In this thesis, we study a conceptually simple decision problem: the propositional satisfiability problem (SAT). Over the past decade, much progress has been made in solving both large and hard instances of the propositional satisfiability problem, in spite of the fact that the problem is ./VP-complete. Two main classes of algorithms have emerged as the state-of-the art for SAT solving, stochastic local search (SLS) algorithms [HS04] and systematic search algorithms based on the Davis-Putnam procedure (DPLL) [DLL62]. The for-mer are generally incomplete, i.e. they cannot decide unsatisfiability. Beyond this fundamental difference, however, there are certain classes of problem instances which can be handled more efficiently by either of the two classes of algorithms. For example, it seems that state-of-the-art D P L L type algorithms have a significant performance advantage on certain classes of SAT-encoded structured instances [SBH02]. M u c h of the progress in algorithm development has been facilitated by an improved understanding of the properties of SAT instances and of high-performance SAT algorithms (see, e.g., [BS03, SBH02, KS03]). With this thesis, we hope to complement and extend existing approaches for understanding the complex interplay between properties of SAT instances and the behaviour of the algorithms used to solve them. The majority of the thesis focuses on the analysis of SLS algorithms, though we also discuss the implications of our findings on the design of systematic search algorithms and some of our analysis is done on both SLS and systematic algorithms (cf. Sections 5.3 and 5.4). Furthermore, though the thesis exclusively stud-Chapter 1. Introduction 2 ies the properties of SAT instances, most of the definitions and observations can be generalised to other combinatorial optimisation problems. The remainder of this chapter is structured as follows. We first define the central problem of this thesis, and motivate its study. We then review previous work that has been done in this area, and comment on how the work relates to this thesis. Finally, we conclude this introduction with an outline of the remainder of the thesis. 1.1 Problem Definition and Motivation The amount of research that has been undertaken towards an understanding of how and why SLS algorithms behave the way they do is surprisingly much less than the amount of work spent developing said algorithms. Much of the research has focused solely on pushing the bounds of the set of instances which may be feasibly solved, and on reducing the time required to solve those instances. While this is, and will continue to be, an important goal, many of the approaches have been fairly ad-hoc, with more emphasis on tweaking algo-rithms than in understanding the underlying reasons for their performance. The main problem addressed in this thesis is how to extend the current understanding the relationship between search space features and algorithm behaviour. The ultimate goal of this research is to use this knowledge to improve existing algorithms, and to provide insights into how better algorithms can be developed. In order to simplify our search space analysis and abstract away irrelevant details, we restrict our anal-ysis to the propositional satisfiability problem. The propositional satisfiability problem (SAT) can be stated as follows: Given a propositional formula T, find an assignment of truth values to the variables in T under which T is satisfied, i.e., equivalent to true, or determine that no such variable assignment exists. In the former case, T is called satisfiable, and in the latter case it is called unsatisfiable. Most widely studied SAT algorithms are restricted to formulae in conjunctive normal form (CNF), i.e., to formulae of the form AJli (vj=i hjj where the Z,; are literals, i.e., variables or their negations. As alluded to above, there are two main classes of state-of-the-art algorithms for solving SAT for CNF formulae: Stochastic local search (SLS) algorithms and systematic search algorithms based on DPLL-type algorithms. This thesis focuses almost exclusively on the study of SLS algorithms for SAT. Stochastic local search algorithms operate in the space of complete variable assignments, and iteratively make local modi-fications to the complete assignment while trying to optimise a given objective function.1 In the SAT case, complete variable assignments correspond to a mapping from the binary truth values {T, to the set of variables of the formula Var(!F) = {xi,*2< • • • ,*n} (« is the number of propositional variables appearing in the formula). The local modifications generally correspond to flipping the truth values of a small set ^ o t e that in this context it is useful to regard SAT as an optimisation problem rather than a decision problem. Chapter 1. Introduction 3 (typically of size 1) of variables. One of the simplest SLS algorithms for SAT fitting this framework is a simple random walk: starting from a randomly chosen assignment, iteratively choose a random variable and flip the truth value assigned to that variable, and repeat this process until a solution is found or a given time bound is exceeded. It should be clear that the expected running time of such an algorithm on an instance with a single solution would be on the order of 0 ( 2 " ) , which quickly becomes computationally intractable as n grows large. 2 In practice, SLS algorithms employ some type of heuristic guidance. The most natural (and common) way to quantify the "quality" of a variable assignment is simply to use the number of unsatisfied clauses under the assignment. Thus, we define the objective function of each SLS algorithm as a function over variable assignments g : { T , _ L } " i—> R . The triple consisting of the set of assignments, the neighbourhood relation, and the objective function comprises the search space of a problem instance wi th respect to a given SLS algorithm. Section 2 .3 presents more rigorous definitions of these (and related) concepts. Clearly, the behaviour of any SLS algorithm is highly related to underlying features of the search space. For example, if there are many local minima in the search space, this may (and in fact does) significantly impede the progress of SLS algorithms. We might expect that instances wi th many solutions are easier than instances wi th few — or that instances wi th a large number of solutions that are distributed uniformly throughout the space of all assignments are easier than instances wi th the same number of solutions, but in which al l of the solutions appear in a tight cluster. The central problem of this thesis is to understand how these types of search space features affect algorithm behaviour. 1.2 Existing Work We now briefly review the most relevant literature related to the search space analysis of SAT instances. M a n y different aspects of the search spaces have been analysed, so we attempt to group the research into general categories. Some of the research follows different approaches while attempting to answer the same questions, while other research is very closely related to the work done in this thesis. The remainder of this section describes the existing work, and characterises the relationship between the existing work and the work done in this thesis. 1.2.1 Number of Solutions One search space feature that is known to affect the performance of SLS algorithms for SAT is the number of solutions. In 1996, Clark et al. investigated the correlation between the number of solutions and the 2Note that non-trivial upper bounds on running time (i.e. bounds lower than 0(2")) have been established for other simple SLS algorithms. See, e.g., [Pap91, Sch99]. Chapter 1. Introduction 4 hardness of an instance [CFG + 96] . For test instances, they used 1000 soluble random 3-SAT problems with 100 variables and 430 clauses (for a description of random 3-SAT problems, refer to Appendix A ) , and one of the first SLS algorithms introduced for SAT, G S A T . 3 In order to quantify the hardness of each instance, Clark et al. ran G S A T 100 times on each instance and used the fraction of successful runs as a measure of local search cost. Note that because GSAT is a very greedy algorithm, it can get permanently "stuck" in local minima regions of the search space, which is why not all of the runs were successful. A strong but noisy positive correlation was found between the number of solutions and this estimated local search cost, indicating that (not surprisingly) instances with fewer solutions tend to be harder than instances with many. Hoos extended this analysis in 1998 using an improved methodology [Hoo98]. Hoos used the G W S A T algorithm [SK93], which is a more robust, better performing variant of GSAT, and which can be guaranteed to find a solution (if one exists) given enough time [Hoo98]. He measured average local search cost (Isc) by measuring the average number of G W S A T steps required to find a solution rather than simply measuring the fraction of successful runs (we w i l l use this definition of local search cost throughout this thesis). Hoos also found a strong positive correlation between the number of solutions and Isc. In addition, he noted that while for instances having high numbers of solutions the variability in Isc is very small, it increases considerably for instances with lower numbers of solutions. Hoos also notes that with growing problem size, the dependency of Isc on the number of solutions gets weaker. While these results are significant, using the number of solutions to predict local search cost has a num-ber of drawbacks. First, it is obviously very computationally expensive to count the number of solutions — counting all the solutions to a SAT instance is in the #V-C complexity class [Val79], making this approach suitable only for a posteriori analyses. Note that this observation applies to many of the other metrics dis-cussed later in the section. Second, and more problematic, we have found that even if we consider only single solution instances, there can still be a very large range in Isc. This agrees wi th Hoos ' findings for small numbers of solutions, and suggests that there must be other search space factors affecting the hard-ness of instances. This thesis extends these results by presenting a model of the search space which can account for this large variance in Isc, even when accounting for the number of solutions. 1.2.2 Solution Clustering In 1997, Parkes extended the work done by Clark et al. by discovering that, for random 3-SAT instances near the phase-transition region [CKT91], solution states are found in clusters [Par97]. He studied this clustering phenomenon indirectly, by counting the unary prime implicates, or UPI's (i.e. literals that are logically entailed by the formula) of the instances. Parkes observed that for clause-to-variable ratios ct less 3 GSAT was conceived by Selman, Levesque, and Mitchell [SLM92] in 1992, and is described in detail in Section 2.3. Chapter 1. Introduction 5 than the critical clause to variable ratio occ,4 most instances have few UPFs but as the transition is made to cc > occ, most instances have many UPFs. This phenomenon was observed for problem sizes ranging from 50 to 300 variables. Parkes then described how instances that have many UPTs have no solutions outside of the region of the search space that is compatible with the UPFs , and that in this region there are few constraints and the density of solutions is high. He refers to such a region as a cluster, and proceeds to show that WalkSAT performs badly on these "clustered instances" (WalkSAT is another we l l known SLS algorithm for SAT — see [SKC93, SKC94] as wel l as Section 2.3). Finally, Parkes conjectured that the addition of a single clause to a satisfiable instance could convert a "solution cluster" into a "failed cluster", which would be a very attractive region of the search space, from which it is impossible to reach a solution. This was strictly a conjecture, and he d id not provide a convincing argument for it. In this thesis, we confirm Parkes' conjecture by empirically demonstrating that such search space features do in fact exist, and that they have a very significant impact on the behaviour of local search algorithms. Parkes' findings seem to indicate that the number of solutions should not have such a dominating effect on Isc as was previously believed. If all of the solutions of most instances are found in a single cluster, then the difficulty faced by SLS algorithms is to find the solution cluster. Because the solution density is high i n these clusters, the number of solutions should not be particularly relevant. This apparent contradiction with earlier results is addressed in the following section. 1.2.3 Backbone Fragility Singer, Gent and Smaill studied the effect of the backbone of SAT instances on the Isc of WalkSAT [SGS00]. The backbone of a SAT instance is simply the set of UPFs . In this study, they used the WalkSAT algorithm and random 3-SAT instances wi th 100 variables and a clause to variable ratio varying from 4.0 to 4.5. Their findings are closely related to those presented in the previous section. Singer et al. found that the number of solutions is highly correlated wi th Isc for instances with small-backbones, but is much less relevant for large-backbone instances. To explain this they conjectured that for small-backbone instances, finding the backbone is straightforward and the main difficulty is encountering a solution once the backbone has been satisfied. In this case, the density of solutions in the region satisfying the backbone is then important. For larger backbone sizes, the main difficulty is satisfying the backbone and the number of solutions becomes less important. This result is also related to another of Hoos' observations in [Hoo98]. Hoos found that the correlation 4 The critical clause to variable ratio is more commonly known as the phase transition point for random 3-SAT instances, and is the point at which a randomly chosen instance is satisfiable with probability 0.5[CKT91]. Chapter 1. Introduction 6 between the number of solutions and local search cost becomes weak i n the over-constrained region (the over-constrained region consists of all soluble SAT instances having a clause-to-variable ratio a > 4.26). Singer et al. observed that large-backbone instances dominate the over-constrained region. Since the cor-relation between number of solutions and Isc is not strong for large-backbone instances, this implies that most instances in the over-constrained region w i l l not exhibit this correlation. Singer, Gent, and Smaill also introduced a novel metric that they demonstrated was highly correlated wi th Isc. For a fixed backbone size, they generated 1000 WalkSAT runs, which terminated as soon a level 5 or lower state was found (please refer to the next chapter for a definition of the level of a state — in short, the level of an assignment is the number of clauses which are unsatisfied by the assignment). For each of these low level states, they measure the Hamming distance to the nearest solution, s, and find the average, which they label hdns(Tj-5,s). They find a strong positive correlation between hdns(Tf5,s) and Isc when backbone size is fixed. Motivated by these observations, Singer et al. make some conjectures about the search spaces of the instances they were examining. They conjecture that the "quasi-solution area" (this corresponds in this thesis to the set of all low level plateaus, 5 defined in Chapter 3) form interconnected areas of the search space such that WalkSAT can always reach a solution from them without moving to higher-level states. Further, they conjecture that in high-cost instances, this quasi-solution area extends to parts of the search space that are Hamming-distant from solutions, whereas in lower cost instances the area is less extensive. Our results indicate that this is indeed the case. However, Singer, Gent, and Smaill claim that variance in hdns(Tf5, C) accounts for almost all of the variance in cost at three different backbone sizes. We do not believe that such a simple measure can account for all of the variance in search cost. We have found that the hardest instances seem to be those that have very attractive closed plateau regions in combination with few low-level plateau regions that are wel l connected to solutions. We believe that while hdns(T^, C) does seem to show promise as a method for estimating local search cost, a complete understanding of SLS algorithm behaviour requires a more detailed analysis of the respective search spaces. 1.2.4 Local Minima Yokoo was interested in how the search space of random 3-SAT instances changes through the phase transi-tion [Yok97]. In particular, he investigated the effect that adding clauses had on the number of local minima states in the search space. For his analysis, he used a deterministic, greedy, lexographical tie-breaking al-gorithm, and 20 variable random 3-SAT instances. In this analysis, Yokoo exhaustively explored the entire search space of the problem instances, which explains why his analysis was restricted to such small in-stances. 5Informally/ a plateau is a set of connected states all having the same objective function value. Chapter 1. Introduction 7 Yokoo pointed out a seemingly paradoxical situation: the average number of solutions to the problems decreases as clauses are added, but soluble problems at the phase transition are harder than those beyond it. To explain this, Yokoo measured the average ratio of "solution-reachable" states, where solution-reachable states are states having the property that if his (deterministic) algorithm was initialized in those states, it would eventually reach a solution. Yokoo found that by adding more clauses, the ratio of solution-reachable states actually increased be-yond the phase transition. Next, he found that the size of closed plateaus decreases as more clauses are added, while the number of closed plateaus increases as clauses are added up to the phase transition point, and then slowly decreased. Thus, Yokoo conjectured that the reason problems get easier as more clauses are added is that the added clauses either break up closed plateau regions, or eliminate them altogether. Finally, Yokoo repeated the above analysis on 12 variable 3-colouring problems, wi th similar results. 1.2.5 Effect of Search Space Structure on Algorithm Behaviour In 1993, Gent and Walsh performed an analysis of the the behaviour of G S A T on 500 variable random 3-SAT problems [GW93]. Their experiments consisted of measuring how the level of visited states and "poss-flips" (poss-flips is the number of neighbours of a state which all have the best possible score) vary during a GSAT try (a single try is a series of hill-climbing steps, and ends when a maximum number of steps have been completed or a solution is found). Gent and Walsh found that the level initially decreases rapidly, but then flattens out until much search is needed to improve the level. To explain this, they show that GSAT tries can be divided into a series of distinct phases, which they label Ho, • • • , Hm. Each phase H, corresponds to the portion of the search trajectory in which a typical variable flip results a net increase of i satisfied clauses. G S A T starts in phase Hm, then moves to phase H m _ i , etc. until the search is in the Ho phase. When the search is in the Ho phase, they refer to this as the plateau-search phase because almost every move is 'sideways', wi th no change in level. Early search phases have a short duration, but the level decreases rapidly. A s we w i l l see, the results from our plateau-based search space analysis agree wi th these results, and extend them in a non-trivial way. We find that the search space of random 3-SAT instances consists of a single, large plateau region which is very easy to escape from at high levels. In regions such as this, a greedy algorithm such as GSAT makes rapid progress and the level decreases very quickly. In addition, we find that at high levels in the search space, many of the exits from these plateau regions have neighbours which are a number of levels lower than the current state, but as the level decreases, this difference in the level of neighbours tends to decrease. These results are discussed further in Chapter 4. Chapter 1. Introduction 8 1.2.6 Plateau Distribution In early SLS algorithms for SAT, a restart mechanism was used to address the problem of getting stuck in closed plateau regions. A fixed number of greedy hill-climbing steps were performed, and then the algorithm would be reinitialised to a random assignment. In 1995, Hampson and Kibler investigated how much time should be spent searching plateau regions before restarting [HK95]. To do this, they used a greedy hill-climbing algorithm that tracked the number of consecutive sideways moves made on a given plateau region, and they also used breadth-first search (BFS) to exhaustively search plateau regions to a limit of 100 000 states. They ran their experiments on random 3-SAT instances wi th 64 to 512 variables and a clause to variable ratio cc = 4.3. Hampson and Kibler made a number of discoveries that are significant to this study. They demonstrated that the size of plateau regions increases with increasing level. However, they also note that the density of open states6 in the plateau regions increases rapidly with increasing level. These observations are confirmed by our analysis in Chapter 3. Finally, Hampson and Kibler found that the size of lower level plateau regions seemed to grow expo-nentially wi th the number of variables (tt), while the density of open states in them decreased inversely wi th n, and that the number of lower level plateau regions increased linearly wi th n. From this, they con-jectured that if the search could be initialized randomly from states in lower level plateau regions, then the average search time would increase linearly wi th n. To our knowledge, this conjecture was never tested, and remains an open question. In 1997, Frank, Cheeseman, and Stutz performed a fairly detailed analysis of the search space of Random 3-SAT instances, wi th a focus on finding the distribution of plateau region sizes, as wel l as a preliminary ex-amination of the structure of these plateau regions [FCS97]. They used G S A T and random 3-SAT problems wi th 100 variables and cc = 3.8 to 4.6 for their analysis. Frank et al. generated 1000 instances with cc = 3.8,3.9,. . . , 4.5,4.6, and ran G S A T once per instance until it encountered a state at each level 1.. .5. Then, each of these states were expanded using breadth-first-search (BFS) to find the entire plateau containing the starting state, and they report properties such as the number of plateaus and how large they are. This work is very closely related to our work in Chapter 3, where we attempt to replicate their experiments as wel l as extend their analysis in interesting directions. Interestingly, our data differs from theirs dramatically, though the experiments were intended to be iden-tical. Because of these differences, we draw different conclusions from our experiments, and are able to explain conflicting observations that Frank et al. were not able to. Further discussion of these results are left for Chapter 3. 6 A n open state is a state having at least one neighbours at a lower level. Chapter 1. Introduction 9 1.2.7 Predicting Local Search Cost There has been a significant amount of research invested in attempting to find search space features which are easy to measure, and which can also be used to predict the local search cost (Isc) of problem in-stances. We have already discussed measures which correlate wel l wi th Isc, such as the number of solutions [CFG + 96] or hdns{Tf$, C) [SGSOO], but these are a posteriori measures for explaining algorithm performance. The following measures are computationally cheap to compute, and can be unsed a priori to estimate the expected local search cost. In 1998, Holger Hoos contributed to the literature with a chapter of his Ph.D. Dissertation devoted to the search space structure of SAT instances [Hoo98]. The first section of the chapter is devoted to analysing how the number of solutions affects local search cost, and is discussed earlier in this paper. The remaining sections are devoted to novel metrics that Hoos developed for measuring search space features. The first metric that Hoos introduces is the standard deviation of the number of unsatisfied clauses, sdnclu. This is a measure of the variance of the objective function. The motivation behind this is the intuition that the ruggedness or smoothness of the objective function across the search space should have an effect on the behaviour of SLS algorithms. Hoos measures sdnclu by exhaustive sampling for Random 3-SAT problems of size n = 20, but for n = 50 and n = 100 he resorts to randomly sampling 100000 states from the search space. One concern that we have wi th this random sampling method is that randomly sampling states from large instance w i l l almost exclusively find open states at relatively high levels. It is wel l known that SLS algorithms spend very little time searching these higher level states, so it is not immediately clear why properties of these higher level states w i l l have a significant impact on SLS behaviour in general. Nevertheless, Hoos does report a noisy negative correlation between sdnclu and local search cost, as wel l as a positive correlation between sdnclu and number of solutions for 20,50, and 100 variable Random 3-SAT instances. Hoos also finds that the correlation is stronger for under-constrained instances, and gets weaker as the number of clauses increases. Thus, sdnclu must be capturing global search space features which affect SLS algorithm performance, even though only high level states are being sampled. Next, Hoos compares sdnclu, local search cost, and number of solutions of structured instances to Ran-dom 3-SAT instances of equivalent size. He finds that his observations seem to hold for Random Graph Colouring problems and Blocks World Planning problems, but he finds inconsistent results when he exam-ines All-Interval-Series problems. Note that this is one of the only pieces of existing work which has studied the search space properties of structured problem instances. In this thesis, we also give equal precedence to the study of random and structured instances whenever possible. Following this, Hoos introduces a set of state types and investigates their distributions. Hoos defines the following state types: B P L A T states are those that have neighbours at higher, lower, and equal levels. Chapter 1. Introduction 10 SLOPE states only have neighbours at higher or lower levels. L M I N states have neighbours at the same level or higher. L M A X states have neighbours at the same level or lower. S L M I N and S L M A X states are L M I N and L M A X states respectively, which do not have any neighbours at the same level. Finally, IPLAT states are those that only have neighbours at the same level. By exhaustive sampling of Random 3-SAT instance with n = 20, Hoos measures the exact state distribu-tions. He finds that almost all states are BPLAT, followed by a much smaller number of SLOPE states, and even fewer L M I N and L M A X states. Hoos found very few S L M I N or S L M A X states, and no IPLAT states. Because it is infeasible to exhaustively enumerate the states of larger instances and random sampling finds almost exclusively B P L A T states, Hoos then sampled the state distribution along G W S A T trajectories. The results of this analysis show that G W S A T is very strongly attracted to L M I N states (which is not sur-prising because G W S A T uses hill-climbing to move through the search space), and that the only types of states that significantly affect GWSAT's behaviour are B P L A T and L M I N states. Hoos also performed this analysis on structured instances, wi th similar results. Finally, Hoos analyses another novel metric, the branching factor of local minima states, blmin. The motivation here is that it is wel l known that local minima (or closed) states have a major impact on SLS performance, and blmin is a way to study these states. Because local minima states only account for a small fraction of the states in the search space, they cannot be randomly sampled so instead Hoos samples blmin along G W S A T trajectories. He again uses Random 3-SAT, Graph Colouring, Blocks World Planning, and All-Intervals-Series instances. Hoos finds a negative correlation between blmin and local search cost, and a positive correlation be-tween blmin and both sdnclu and number of solutions. Hoos notes that the blmin values are fairly low, indicating that the plateau structures are quite "brittle" and that most neighbours of local minimum states are at higher levels. A l o n g with his earlier observation that IPLAT states do not seem to exist, this can help to explain the effectiveness of simple escape strategies such as random walk. Finally, Nudelman et al. recently investigated how features of SAT instances can be used to build models useful for predicting search cost [NLBD+04]. Their approach is to use machine learning to automatically determine which of a set of given features are useful in building compact models for predicting local search cost. They study a large set of random 3-SAT instances with varying clause to variable ratios. Though they study only complete algorithms, a similar analysis could theoretically be performed using stochastic local search algorithms. Our work is related to Nudelman et al.'s in several ways. Firstly, while they currently only consider features of the instances which can be computed efficiently (the analyses we perform are very computationally expensive), it may be possible to incorporate some of the features coming out of our work in order to improve their models. Secondly, the features automatically determined to be useful in predicting the search cost of an instance are obviously interesting objects of study. It should be possible to take those Chapter 1. Introduction 11 i n t e r e s t i n g f e a t u r e s , a n d a n a l y s e t h e m c a r e f u l l y t o d e t e r m i n e t h e r e l a t i o n s h i p b e t w e e n t h o s e f e a t u r e s , a n d f e a t u r e s o f t h e s e a r c h s p a c e o f t h e i n s t a n c e . 1.3 Thesis Outline T h e r e m a i n d e r o f t h e t h e s i s i s s t r u c t u r e d as f o l l o w e d . I n t h e n e x t c h a p t e r , w e i n t r o d u c e n e c e s s a r y c o n c e p t s a n d d e f i n i t i o n s r e q u i r e d f o r t h e r e m a i n i n g c h a p t e r s . C h a p t e r 3 s t u d i e s t h e d i s t r i b u t i o n o f p l a t e a u s i n t h e s e a r c h s p a c e , a n d h o w t h e p r o p e r t i e s o f t h e s e p l a t e a u s a r e r e l a t e d t o S L S a l g o r i t h m b e h a v i o u r . C h a p t e r 4 d i s c u s s e s t h e c o n n e c t i v i t y o f p l a t e a u s , i l l u s t r a t e s h o w w e c a n b u i l d s i m p l i f i e d m o d e l s o f t h e s e a r c h s p a c e t h a t a i d i n t h e u n d e r s t a n d i n g o f b o t h s e a r c h s p a c e s t r u c t u r e , a n d d e m o n s t r a t e s h o w t h i s s t r u c t u r e a f fec t s t he p e r f o r m a n c e o f S L S a l g o r i t h m s . C h a p t e r 4 a l s o c h a r a c t e r i s e s a n d i d e n t i f i e s a n d s t u d i e s i m p o r t a n t s e a r c h s p a c e f e a t u r e s t h a t h a v e a n e g a t i v e i m p a c t o n S L S a l g o r i t h m p e r f o r m a n c e - s o c a l l e d " t r a p s " i n t h e s e a r c h s p a c e . C h a p t e r 5 t a k e s a s l i g h t l y d i f f e r e n t d i r e c t i o n , a n d i d e n t i f i e s p r o p o s i t i o n a l v a r i a b l e s t h a t p l a y a c r i t i c a l r o l e w h e n s o l v i n g S A T i n s t a n c e s . W e c h a r a c t e r i s e t he r e l a t i o n s h i p b e t w e e n c r i t i c a l v a r i a b l e s a n d s e a r c h s p a c e s t r u c t u r e , w h i c h g i v e s n o v e l i n s i g h t s i n t o t h e l o c a l s e a r c h p a r a d i g m . F i n a l l y , w e s u m m a r i s e o u r m a i n r e s u l t s a n d s u g g e s t i n t e r e s t i n g d i r e c t i o n s f o r f u t u r e w o r k i n C h a p t e r 6. T h e t w o a p p e n d i c e s a t t he e n d o f t h e t h e s i s g i v e a d e s c r i p t i o n o f t h e tes t - se t s u s e d i n t he t h e s i s , a n d a n i n t r o d u c t i o n t o b i n a r y d e c i s i o n d i a g r a m s . Chapter 2. Preliminaries 12 Chapter 2 Preliminaries In this chapter, we introduce many of the notations and definitions used throughout the thesis. Many of our definitions are meant to be functionally equivalent to those presented in [Hoo98, HS04], though they may be presented differently in this thesis. Throughout the thesis, the following conventions are followed. Predicates are capitalised and italicised (E.g. LMIN(s)). Sets are italicised, wi th the first letter of each word in the name capitalised, and optional subscripts and superscripts allowed (E.g. Levelset^). Functions are also capitalised and have the first letter of each word in the name capitalised, but are required to be followed by a list of parameters in parentheses and are not allowed to have subscripts or superscripts in the name (E.g. PickTwoClauses(J-)). Literals are italicised, and must be a single lowercase symbol with optional subscripts but only primes allowed as superscript (E.g. /••). Finally, we use S\P and #S to denote the set theoretical difference of S and P, {x | x £ S A x £ P}, and the cardinality of S, respectively. The remainder of this chapter is structured as follows. First, we formalise the definition of propositional formulae and related definitions such as variable assignments and satisfiability. Then, we formalise many of the definitions used to define the search space of SAT instances. Fol lowing this, we introduce Stochastic Local Search algorithms for SAT and describe two reference algorithms used throughout the thesis. Finally, we describe the empirical methodology used throughout the thesis. 2.1 Propositional Formulae A propositional formula is, in its most general sense, a well-formed formula defined over a number of Boolean variables {x\, %2, • • • , xn} and the standard operators {->, A , V , —>, • • • }. Throughout this thesis, we focus solely on propositional formulae of a syntactically restricted class, called Conjunctive Normal Form. The next definition formalises this notion. Definition 2.1. (Propositional Formula, Conjunctive Normal Form) A propositional variable is an element of the set V = {x, | i E N } , and can be assigned a value from the set of truth values C = {T,_L}. A literal is either a propositional variable xu or its negation - i * , - . Given a literal /, Variable(l) represents the associated variable. Chapter 2. Preliminaries 13 A propositional sentence is a sentence over the alphabet P := V U C U O U { ( , ) } , where O represent the propositional operators { - i , A, V } . 1 A propositional sentence F$ is i n Conjunctive Normal Form if it can be written as a conjunction(A) of disjunctions(V) of literals: m ( k, \ Here, each disjunction \Jkj'=1 hj is called a clausal sentence involving ki literals. Because conjunction and disjunction are associative, we can disregard the order of the clausal sentences and of literals within a single clausal sentence. Thus, for each clausal sentence VyLi hj we define the set C(i) := U j i i hj> and call this set a clause. The number of literals in clause C is given by Length(C) := #C. We also define the set T := \Jf=l{C{i)}, and call this set a CNF formula. The number of clauses in T is given by Clauses^) := #!F. Finally, a C N F formula J7 is in fc-CNF if and only if 3k e N VC G .F : Length(C) = k. Note that, according to this definition, clauses cannot contain duplicate literals, and C N F formulae cannot contain duplicate clauses — this is not a serious restriction, since both duplicate literals and clauses are redundant. • We should note that restricting our attention to this subset of the propositional formulae is without loss of generality, since any propositional formula may be transformed into an equivalent formula in C N F . In fact, it is possible to convert any propositional formula into an equi-satisfiable C N F formula with only a polynomial increase in the size of the formula [PG86, dlT90]. Given a propositional formula with n variables, it should be easy to see that there are exactly 2" different ways to assign binary values to the variables. The propositional formula is defined in such a way that each one of those assignments evaluates to either true or false; the goal of the SAT problem is to find an assignment that evaluates to true for a given formula. The following definition formalises the concepts of variable assignments and the method of evaluating the value of each assignment under a given formula. Definition 2.2. (Assignment) The variable set Var(F) of formula T is defined as the set of all variables appearing in T. More formally: Var{?) = {Xj | 3C € T : x, e C V -,jc;- £ C} A variable assignment of formula T is a'mapping a : Var[T) f-> { T , J_} of the variable set of T to the truth values. The set of all possible variable assignments is denoted by Assign(!F). !Note that O may be extended to include other operators such as —>, *-*, and <8>, but that these additional operators are redundant and are useful only for writing formulae more succinctly. Chapter 2. Preliminaries 14 The value Vala (F) of formula T under assignment a is def ined i n d u c t i v e l y : Val„{b) := b be {T,±} f T i f b = ± r Vala(^b) : = 4 be{T,±} [ ±i(b = T Vala{xi) := Vala(a(Xi)) X; E V Vah^Xi) := Vala{-*(Xi)) xt € V , s . T i f B / e C : Val„(l) = T Vala(C) : = { W C e f _L o therwise . , , T i f V C 6 ^ : V f l / f l ( C ) = T Vala(F) := { _L o therwise . N o t e that Vfl /„(^ r ) = T i f a n d o n l y if a l l clauses con ta in at least one l i t e ra l / that is consistent w i t h f l , i.e. Val„ (I) = T. • Fina l ly , w e are i n a p o s i t i o n to f o r m a l l y state the P r o p o s i t i o n a l Sat i s f iabi l i ty P r o b l e m . T h e P ropos i t i ona l Sat is f iabi l i ty P r o b l e m is to de te rmine for a g i v e n P r o p o s i t i o n a l f o r m u l a i f there exists a n ass ignment a that satisfies the f o r m u l a : Definition 2.3. (Satisfiability) A var iab le ass ignment fl is a model of formula T i f a n d o n l y if Vala{!F) = T; i n this case w e say that a satisfies T. A f o r m u l a T is satisfiable i f a n d o n l y i f there exists at least one m o d e l of T. • 2.2 Search Space Components G i v e n a f o r m u l a T, w e say that the set of va r iab le ass ignments , Assign^), forms the search space S(!F). We also define a neighbourhood relation o n >S(^ r), N{!F) C S^) x >5(Jr). F o r brevi ty , w e t y p i c a l l y d o not i n c l u d e the reference to T w h e n referr ing to <S(JF) a n d N{!F), bu t i m p l i c i t l y assert the dependence o n the p r o b l e m instance, a n d w r i t e S a n d JV, respectively. Taken together, S a n d Af f o r m the search graph = (S,jV). de te rmines the space of assignments that a g i v e n S L S a l g o r i t h m is searching, as w e l l as the t ransi t ions ava i l ab le to the a l g o r i t h m . Typica l ly , howeve r , S L S a lgor i thms d o not p e r f o r m r a n d o m w a l k s o n this g raph ; they are genera l ly g u i d e d b y some f o r m of g r e e d y heur is t ic . T h i s heur is t ic i n f o r m a t i o n is cap tu red i n the evaluation function g(T) : i—> K , m a p p i n g search states to rea l n u m b e r s w h i l e e n s u r i n g that g l o b a l o p t i m a i n g c o r r e s p o n d to sa t is fying ass ignments . In mos t S L S a lgor i thms for S A T , w e s i m p l y define g{T,a) : = # { c 6 T \ Vala(c) = _L}, that is, the e v a l u a t i o n func t ion v a l u e of an ass ignment is just the n u m b e r of clauses that are not satisfied b y that Chapter 2. Preliminaries 15 assignment. Minimis ing g therefore corresponds to satisfying more clauses, and when g(T, a) is 0 there are no clauses unsatisfied by a, and T is satisfied. Finally, combing S, H, and g results in the search landscape L — (S,j\f,g). The search landscape fully defines the search space of an instance, and can be conceptualised as a graph in which nodes represent search states, edges represent neighbouring states, and each state has a "fitness" values associated with it. The goal then, of any SLS algorithm, is to perform a (biased) walk in the search landscape attempting to find a state wi th a fitness value of zero. For a SAT instance involving n variables, the search landscape can be conceptualised as an n-dimensional Boolean hypercube, wi th the evaluation function value of each state associated with the corresponding vertex. Chapters 3 and 4 of this thesis exclusively study properties of the search landscape in an attempt to understand how and why SLS algorithms behave the way they do on these landscapes. 2.3 SLS Algori thms for SAT For our purposes i n this thesis, we w i l l consider only a small (but well-studied) subset of SLS algorithms for SAT. In order to simplify the analysis done later in the thesis, we restrict our attention to SLS algorithms that search the landscape defined.by S(!F) = Assign(!F), N(F) corresponding to all pairs of states differing in the value of only one variable, and g(^F,a) :— #{c G T \ Vala{c) — _L}. Most modern state-of-the-art SLS algorithms for SAT are compliant wi th these restrictions, so we are still studying a very interesting class of algorithms. Additionally, much of the theory that we w i l l develop still applies to other types of algo-rithms (for example, those which use larger local neighbourhoods). For a more general and comprehensive description of SLS algorithms in general, the reader is referred to [Hoo98, HS04]. In this section, we give a brief overview of two "canonical" local search algorithms for SAT. These algo-rithms are both very well-studied, and also form the basis for many other (better performing) algorithms. The two algorithms are GSAT [SLM92], and W a l k S A T / S K C [SKC93, SKC94]. Pseudo-code for the well-known GSAT algorithm is shown in Algor i thm 2.1. The algorithm is quite simple. Each run of the algorithm consists of a number of tries, each of which is commenced by choos-ing a random starting assignment (cf line 3). 2 Once the starting point has been chosen, a fixed num-ber of steps are performed. In each step, a variable is selected (cf line 8) and flipped (cf. line 9). The variable is chosen randomly from all variables that minimise the heuristic function scoreGSAT, defined as scoreGSAT(v) = gi^F, a') — g{F, a), where a is the current assignment, and a' is the assignment with variable v flipped. This process is repeated until either a solution is found, or the maximum number of steps has been reached. 2Note that throughout the thesis, unless states otherwise, random choices are made uniformly over the given set. Chapter 2. Preliminaries 16 Algorithm 2 . 1 : G S A T ( ^ " , maxTries, maxSteps) Input : C N F F o r m u l a T, I n t e g e r s maxTries a n d maxSteps Output: S a t i s f y i n g a s s i g n m e n t o r " t i m e o u t " begin for try :— 1, • • • , maxTries do a:— r a n d o m l y c h o s e n e l e m e n t o f Assign (J7) for step : = 1, • • • , maxSteps do if Vala {J7) = T then return a else B := {v e Var{F) \ W 6 Var{T) : scorefs:AT{v) < score%SAT{v')} v : = r a n d o m l y c h o s e n e l e m e n t o f B a ~ a w i t h v a r i a b l e v f l i p p e d end end end return 'timeout' 14 end A s p r e v i o u s l y m e n t i o n e d , t h e b a s i c G S A T a l g o r i t h m f o r m s t h e b a s i s f o r m a n y o t h e r S L S a l g o r i t h m s , i n c l u d i n g ( b u t b y n o m e a n s l i m i t e d to) G W S A T [ S K 9 3 ] , G S A T / T A B U [ M S K 9 7 , M S G 9 7 ] , H S A T [ G W 9 3 ] a n d H W S A T [ G W 9 5 ] , a s w e l l a s v e r y p o w e r f u l d y n a m i c l o c a l s e a r c h a l g o r i t h m s s u c h as G L S [ M T O O ] , D L M [ S W 9 7 ] , a n d S A P S [ H T H 0 2 ] . A n o t h e r i n t e r e s t i n g a n d i n f l u e n t i a l c l a s s o f a l g o r i t h m s i s b a s e d o n t h e W a l k S A T f r a m e w o r k . P s e u d o -c o d e f o r t h e a r c h e t y p i c a l a l g o r i t h m i n t h i s f r a m e w o r k , W a l k S A T / S K C , i s s h o w n i n A l g o r i t h m 2 .2 . S t r u c -t u r a l l y , t h e a l g o r i t h m i s q u i t e s i m i l a r t o G S A T ; b o t h a l g o r i t h m s i t e r a t e o v e r a f i x e d n u m b e r o f tries a n d steps, r a n d o m l y r e i n i t i a l i s i n g a t t h e s t a r t o f e v e r y t ry , a n d c h o o s i n g a v a r i a b l e t o b e f l i p p e d i n e v e r y s t e p . T h e m a j o r d i f f e r e n c e b e t w e e n t h e t w o a l g o r i t h m s , a n d t h e d e f i n i n g f e a t u r e o f t h e W a l k S A T f r a m e w o r k , i s t h a t W a l k S A T / S K C u s e s a t w o - t i e r e d v a r i a b l e s e l e c t i o n m e c h a n i s m (c.f.,lines 7 - 16) . T o s e l e c t a v a r i -a b l e t o f l i p , a c u r r e n t l y u n s a t i s f i e d c l a u s e C i s c h o s e n a n d t h e c a n d i d a t e v a r i a b l e i s t h e n r e s t r i c t e d t o t h o s e a p p e a r i n g i n C . T h i s g u a r a n t e e s t ha t a t t h e v e r y l e a s t , c l a u s e C w i l l b e s a t i s f i e d a f t e r t h e v a r i a b l e f l i p . T h e se t B (c.f. l i n e 10) c o n t a i n s a l l v a r i a b l e s t h a t m i n i m i s e t h e h e u r i s t i c f u n c t i o n scoreSKC, d e f i n e d as scoresnKC(v) = #{C £ T \ Val„(C) = T A Vala,(C) = 1 } , w h e r e a1 i s e q u i v a l e n t t o t h e a s s i g n m e n t a, w i t h t h e v a l u e o f v a r i a b l e v f l i p p e d . N o t e t h a t t h i s h e u r i s t i c f u n c t i o n o n l y c o u n t s a n e g a t i v e c o n t r i b u t i o n f o r t h e c l a u s e s t h a t b e c o m e u n s a t i s f i e d w h e n a v a r i a b l e i s flipped; i t d o e s n o t c o u n t a p o s i t i v e c o n t r i b u t i o n f o r t h e c l a u s e s t h a t b e c o m e s a t i s f i e d (i.e. i t o n l y a s s e s s e s t h e " d a m a g e " d o n e b y a v a r i a b l e flip). L i n e s 11 - 16 s h o w h o w a t h e v a r i a b l e i s a c t u a l l y c h o s e n f r o m a m o n g the se t s . I f a v a r i a b l e c a n b e c h o s e n t h a t d o e s n o t u n s a t i s f y a n y n e w c l a u s e s , i t i s a l w a y s c h o s e n (c.f. l i n e 11) . O t h e r w i s e , w i t h p r o b a b i l i t y p, t h e v a r i a b l e i s c h o s e n r a n d o m l y f r o m a l l o f t h o s e i n t he c l a u s e , a n d w i t h p r o b a b i l i t y 1 — p t h e v a r i a b l e i s c h o s e n r a n d o m l y f r o m B ( t h e se t o f h e u r i s t i c a l l y b e s t v a r i a b l e s ) . Chapter 2. Preliminaries 1 7 Algorithm 2.2: W a l k S A T / S K C ( . F , maxTries, maxSteps, p ) Input : C N F F o r m u l a T, I n t e g e r s maxTries a n d maxSteps, P r o b a b i l i t y p Output: S a t i s f y i n g a s s i g n m e n t o r " t i m e o u t " begin for try :— 1, • • • , maxTries do a : = r a n d o m l y c h o s e n e l e m e n t o f Assign(!F) for step :=!,•••, maxSteps do if Vala^) = T then return a else U : = { C e T I Vala{C) = 1} C : = r a n d o m l y c h o s e n e l e m e n t o f U Cv •= {v e Var(F) \3l eC:v = Variable(l)} B:={veCv\ W e Cv : scoresaKC(v) < scoresaKC(v1)} if V o 6 B : scoresaKC{v) = 0 then | v := r a n d o m l y c h o s e n e l e m e n t o f B else I 2 3 4 5 6 7 8 9 10 11 12 13 14 IS 16 17 18 19 20 v :— r a n d o m l y c h o s e n e l e m e n t o f end a:— a w i t h v a r i a b l e v f l i p p e d Cv w i t h p r o b a b i l i t y p B w i t h p r o b a b i l i t y 1 — p end end end return 'timeout" 2i end W a l k S A T / S K C h a s b e e n s h o w n to o u t p e r f o r m G S A T ( a n d v a r i a n t s o f i t ) o n m a n y p r o b l e m c l a s s e s [ S K C 9 4 , H o o 9 8 ] . A s w i t h G S A T , a l a r g e n u m b e r o f S L S a l g o r i t h m s h a v e b e e n b a s e d o n t h e W a l k S A T f r a m e -w o r k . S o m e n o t a b l e e x a m p l e s i n c l u d e W a l k S A T / T a b u [ M S K 9 7 , M S G 9 7 ] , a n d N o v e l t y + [ H o o 9 9 ] . 2.4 Empir ica l Methodology T h i s t h e s i s i s v e r y e m p i r i c a l i n n a t u r e . T h r o u g h o u t t he t h e s i s , w e p e r f o r m e x p e r i m e n t s , t y p i c a l l y b y r u n -n i n g S L S a l g o r i t h m s r e p e a t e d l y o n d i s t r i b u t i o n s o f tes t i n s t a n c e s , a n d r e p o r t e m p i r i c a l r e s u l t s . B e c a u s e S L S a l g o r i t h m s a r e i n h e r e n t l y r a n d o m i s e d , t h e i r p e r f o r m a n c e o n a g i v e n i n s t a n c e c a n o n l y b e c h a r a c t e r i s e d b y a d i s t r i b u t i o n o f r u n - t i m e s f o r t h a t p a r t i c u l a r i n s t a n c e . H o o s a n d S t u t z l e h a v e d e v e l o p e d a m e t h o d o l o g y f o r e m p i r i c a l l y a n a l y s i n g S L S a l g o r i t h m s [ H S 9 8 , H S 0 4 ] , a n d w e f o l l o w t h e i r a p p r o a c h i n t h i s t h e s i s . T o s t u d y t h e d i s t r i b u t i o n o f S L S a l g o r i t h m r u n - t i m e s , w e u s e c u m u l a t i v e d i s t r i b u t i o n f u n c t i o n s ( C D F s ) . A C D F i s d e f i n e d s i m p l y a s t h e p r o b a b i l i t y t h a t a g i v e n r a n d o m v a r i a b l e X t a k e s a v a l u e l e s s t h a n o r e q u a l to x, i.e. cdf(x) = Pr[X < x). F o r d i s c r e t e d i s t r i b u t i o n s ( w h i c h w e e x c l u s i v e l y e n c o u n t e r i n t h e e m p i r i c a l Chapter 2. Preliminaries 18 -10 -5 0 5 10 100 1000 10000 100000 1e+06 x search steps Figure 2.1: Example cumulative distribution function. Left: The standard normal probability density func-tion, labelled norm(x), and the associated CDF, labelled cdf(x). Right: A n empirical CDF, the RTD of W a l k S A T / S K C on a hard random instance, as wel l as the best fitting exponential distri-bution — see text for details. studies throughout this thesis), this becomes: cdf(x)= £ Pr[X = y] (2.1) {y \y<x) The left pane of Figure 2.1 illustrates a simple C D F for the standard normal probability density function. For any x, if we would like to find Pr[X < x], we simply find the intersection of the C D F with the desired x value. For example, at x = 0, we get Pr[X < x] = 0.5. We use CDFs throughout this thesis, and w i l l assume that the reader is familiar wi th them. CDFs are useful for analysing the distribution of run-times of an SLS algorithm on a given instance. To achieve this, we run a given SLS algorithm repeatedly on an instance and measure either the number of search steps or the C P U time required to solve the instance (we assume that the algorithm is given enough resources that it eventually finds a solution in every run). Given this run-time distribution (RTD), it is a simple matter to determine the corresponding CDF. The right pane of Figure 2.1 illustrates such an empirical CDF, obtained by running W a l k S A T / S K C 1000 times on a hard Random 3-SAT instance, and recording the number of search steps performed in each run. The right pane of Figure 2.1 also shows the exponential distribution ed(x,m) — 1 — 2~x^m, wi th m -58000. This exponential distribution fits the RTD very well , which is typical of the RTDs of SLS algorithms (for further discussion of this observation, the reader is referred to [HS98, HS04]). Although it is desirable to study the entire RTD, we often need to quantify the performance of an algorithm on an instance with a single value. Since the RTDs are typically exponentially distributed, summary statistics of the distribution can be meaningfully interpreted. In this thesis, we use the mean of the R T D to quantify the search cost of Chapter 2. Preliminaries 19 an algorithm on a given instance, and refer to this value as the local search cost (Isc) of the given algorithm on the given instance. Because this is an empirical study, we need some benchmark instance sets. Appendix A gives an overview of the instances that we use. When assembling these benchmark sets, we attempted to focus on widely studied sets of instances that were representative of the the instances being studied in the lit-erature. To this end, we include a number of Random 3-SAT test sets from the phase transition region of various sizes [CKT91, GW94]. We also include SAT-encoded instances from other problem domains, such as graph colouring, planning, logistics, and hardware verification and design. Due to the extremely high computational resources required for many of the experiments in this thesis, many of the instances that we study are smaller (and thus correspondingly easier) than those previously found in the literature. However, it is our hope that the properties we observe and the intuitions that we gain by studying these smaller in-stances carries over to larger instances. This is not without precedence; Yokoo studied the changes in search space across the phase transition for 20 variable Random 3-SAT instances, wi th very interesting results that d id indeed carry over to larger instances [Yok97]. Finally, we briefly mention that a large number of the algorithms that we develop and use in this the-sis are based on binary decision diagrams (BDDs). BDDs are essentially a compact method of modelling Boolean functions. In this thesis, we use BDDs to represent sets of search states, in the following way. A given state X = (x\, %i, • • • , xn) is treated as the input to a Boolean function / over n variables. The Boolean function returns true if and only if the state X belongs to the set represented by / . BDDs are a convenient method of modelling such functions because the BDDs are compact, and many set operations (such as in-tersection, union, etc.) can be performed very efficiently wi th the BDDs. Appendix B gives a more detailed overview of BDDs, as wel l as references to additional resources. Chapter 3. Plateau Characteristics 2 0 Chapter 3 Plateau Characteristics One of the most prominent search space features of SAT instances (and many other combinatorial optimi-sation problems involving discretised evaluation functions) are structures known as plateaus. Simply put, a plateau is a set of states in the search space that are all connected to one another, and are indistinguishable wi th respect to a given evaluation function. Thus an SLS algorithm has no heuristic guidance when mov-ing between states on a plateau, and must either make a non-improving move or choose uniformly from neighbours on the plateau with the hope of finding a state which has neighbours wi th a lower evaluation function value (such states are referred to as exit states). In this chapter we demonstrate that plateaus are indeed a very prevalent search space feature in a wide variety of SAT instances, and we study many of their properties. Plateaus are interesting for a variety of reasons. For example, one of the most interesting classes of plateaus are those which consist entirely of states having only neighbours wi th higher evaluation function values. These states are known as local minima states, and they impede the progress of greedy local search algorithms. If a plateau contains only local minima states, then the plateau is referred to as closed, and it is impossible for an SLS algorithm to improve the evaluation function value without first making at least one non-improving move. Thus, we would like to get a sense of how widespread such closed plateaus are, and what properties the closed plateaus have. Plateaus are also a natural object of study because they fully partition the search space - that is, every state belongs to exactly one plateau. Thus, plateaus allow us to simplify our view of the search space by grouping states together and measuring properties of the encompassing plateaus. This chapter is structured as follows. The next section presents all of the formal definitions and language needed throughout the chapter, and further motivates the study of plateaus. Next, we study plateaus by sampling the plateaus actually encountered when using an SLS algorithm to solve the instances and look at properties such as the number of plateaus, their size, and the various types of plateaus. Next we describe a method of exhaustively finding all of the plateaus in the search space, and report statistics over all plateaus. Following this, we study the properties of plateaus in the search spaces of structured SAT instances. Next, we study the internal structure of plateaus, and consider how the anatomy of plateaus affects SLS algorithm behaviour. Finally, the chapter concludes with a discussion and summary of the main results. Chapter 3. Plateau Characteristics 21 3.1 Definitions and Motivation 3.1.1 State Types A s p r e v i o u s l y d i s c u s s e d , t he s e a r c h s p a c e S o f a c o m b i n a t o r i a l o p t i m i s a t i o n p r o b l e m i s c o m p o s e d o f a f i n i t e ( o r c o u n t a b l e i n f i n i t e ) s e t o f s t a t e s . I n t h e c a s e o f S A T , g i v e n a f o r m u l a T, t h e s e a r c h s p a c e S(T) := Assign(T), w h e r e Assign^) i s s i m p l y t he se t o f a l l c o m p l e t e v a r i a b l e a s s i g n m e n t s . F o r c o n v e n i e n c e , w e t y p i c a l l y c a l l t h e s e a s s i g n m e n t s search space states o r , f o r t he s a k e o f b r e v i t y , states. I t s h o u l d b e o b v i o u s t h a t t h e r e a r e e x a c t l y 2" s t a t e s , w h e r e n = #Var(!F), s i n c e e v e r y s t a t e i n v o l v e s n v a r i a b l e s e a c h h a v i n g t w o p o s s i b l e v a l u e s . A s s u m i n g t h a t w e a r e c o n s i d e r i n g o n l y n e i g h b o u r h o o d r e l a t i o n s N{J-) C Assign(J-) x Assign(J-) i n w h i c h t w o s t a t e s a r e n e i g h b o u r s i f a n d o n l y i f t h e y d i f f e r i n t h e v a l u e o f e x a c t l y o n e v a r i a b l e , t h e s e a r c h s p a c e c a n b e c o n c e p t u a l i s e d a s a B o o l e a n h y p e r c u b e . I n t h e f o l l o w i n g , w e c l a s s i f y t h e s e a r c h s p a c e s t a t e s i n t o d i f f e r e n t c l a s s e s , d e p e n d i n g o n t h e i r l o c a l n e i g h -b o u r h o o d . T h e d e f i n i t i o n i s f r o m [ H o o 9 8 , H S 0 4 ] . Definition 3.1. (State Types) L e t L — (S,j\f,g) b e a s e a r c h l a n d s c a p e . F o r a s t a t e s £ S, w e d e f i n e t h e f o l l o w i n g f u n c t i o n s w h i c h d e t e r m i n e t h e n u m b e r o f u p w a r d s , s i d e w a y s , a n d d o w n w a r d s s t e p s f r o m s t o o n e o f i t s d i r e c t n e i g h b o u r s : upw(s) := #{s' G Af(s) | g(s') > g(s)} sidew(s) := #{s' G JV(S) | g{s') = g(s)} downw(s) := #{s' G N(s) | g(s') < g(s)} B a s e d o n t h e s e f u n c t i o n s , w e d e f i n e t h e f o l l o w i n g s t a t e t y p e s : SLMIN(s) downw(s) = sidew(s) = D LMIN(s) downw(s) = 0 A sidew(s) > 0 Aupw(s) > 0 IPLAT(s) downw{s) = upw(s) = 0 LEDGE(s) downw(s) > 0 A sidew(s] > 0 A upw(s) > 0 SLOPE(s) • o downw(s) > 0Asidew(s] = 0 A upio(s) > 0 LMAX{s) downw(s) > 0 Asidew(s] > 0 A upw{s) = 0 SLMAX(s) sidew(s) -- upw(s) — 0 T h e s t a t e s d e f i n e d b y t h e s e p r e d i c a t e s a r e c a l l e d strict local minima (SLMIN), local minima (LMIN), plateau interior (IPLAT), ledge (LEDGE), slope (SLOPE), local maxima (LMAX), a n d sfn'cf. local max-ima (SLMAX) s t a t e s . • Chapter 3. Plateau Characteristics 22 It s h o u l d b e o b v i o u s f r o m t h e d e f i n i t i o n t h a t e v e r y s t a t e f a l l s i n t o e x a c t l y o n e o f t h e a b o v e c l a s s e s , a n d t h u s t h e s t a t e t y p e s p a r t i t i o n t h e s e a r c h s p a c e . A d e t a i l e d d i s c u s s i o n o f t h e s t a t e t y p e d i s t r i b u t i o n i n v a r i o u s c l a s s e s o f S A T i n s t a n c e s c a n b e f o u n d i n [ H o o 9 8 ] . 3.1.2 Plateaus W h i l e t h e d i s t r i b u t i o n o f s t a t e s i s a n i n t e r e s t i n g t o p i c o f s t u d y i n a n d o f i t s e l f , w e a r e m o r e i n t e r e s t e d i n s t u d y i n g l a r g e r s e a r c h s p a c e s t r u c t u r e s t ha t , a s w e w i l l d e m o n s t r a t e t h r o u g h o u t t h i s t h e s i s , h a v e d r a m a t i c e f fec t s o n t h e b e h a v i o u r o f S L S a l g o r i t h m s . R a t h e r t h a t s t u d y i n g s i n g l e s t a t e s , w e s t u d y g r o u p s o f c o n n e c t e d s t a t e s . T h e f o l l o w i n g d e f i n i t i o n f o r m a l i s e s t h e n o t i o n o f a region o f t h e s e a r c h s p a c e . Definition 3.2. (Region) L e t L = (S, H, g) b e a s e a r c h l a n d s c a p e a n d — (S, N) b e t he c o r r e s p o n d i n g s e a r c h g r a p h . A region i n G^ i s a se t R C S o f s e a r c h s t a t e s t h a t i n d u c e s a c o n n e c t e d s u b g r a p h o f G ^ . F o r m a l l y , R i s a r e g i o n i n G N i f a n d o n l y i f Vs ' , s " e R 3 s 0 , S i , • • • , S j . 6 R Vz 6 {0, • • • ,k - 1} : So — s' A Sjt = s " A A / - ( s „ s , - + 1 ) . • R e g i o n s a r e i n t e r e s t i n g b e c a u s e t h e y c o r r e s p o n d to p o r t i o n s o f t h e s e a r c h s p a c e i n w h i c h i t i s p o s s i b l e f o r a n S L S a l g o r i t h m to m o v e b e t w e e n a n y t w o s t a t e s b e l o n g i n g t o t h e r e g i o n w i t h o u t l e a v i n g t h e r e g i o n . N o t e t h a t d e p e n d i n g o n t h e e v a l u a t i o n f u n c t i o n , i t m a y a c t u a l l y b e i m p r o b a b l e ( o r e v e n i m p o s s i b l e ) t h a t a n S L S a l g o r i t h m w i l l a c t u a l l y t r a v e l b e t w e e n t w o s t a t e s i n t he r e g i o n — b u t s u c h a t r a j e c t o r y i s n o t r u l e d o u t b y t h e n e i g h b o u r h o o d r e l a t i o n . H o w e v e r , r e g i o n s a r e n o t i n t e r e s t i n g i n a n d o f t h e m s e l v e s . I n d i v i d u a l s t a t e s c o n s t i t u t e t r i v i a l n o n -i n t e r e s t i n g r e g i o n s . T h e e n t i r e s e a r c h s p a c e t y p i c a l l y f o r m s a t r i v i a l r e g i o n a s w e l l ( t h o u g h t h i s m a y n o t b e t h e c a s e , d e p e n d i n g o n t h e n e i g h b o u r h o o d r e l a t i o n ) . H o w e v e r , t h e n e x t d e f i n i t i o n f o r m a l i s e s a v e r y i m p o r t a n t a n d i n t e r e s t i n g t y p e o f r e g i o n , t he plateau region, a s w e l l as a c l o s e l y r e l a t e d s e a r c h s p a c e s t r u c t u r e , t he plateau. Definition 3.3. (Plateau Region, Plateau, Level) L e t L = (S,Af,g) b e a s e a r c h l a n d s c a p e a n d G^ =• (S,Af) b e t h e c o r r e s p o n d i n g s e a r c h g r a p h . A r e g i o n R i n G N i s a plateau region i n L i f a n d o n l y i f a l l s t a t e s i n R h a v e t h e s a m e e v a l u a t i o n f u n c t i o n v a l u e , i.e., 31 e IR Vs' e R : g(s') = I. I n t h i s c a s e , d e f i n e Level(R) := I t o b e t he level o f p l a t e a u r e g i o n R ( w e o c c a s i o n a l l y r e f e r t o t h e l e v e l o f a s t a t e s i m i l a r l y ) . A plateau i n L i s a m a x i m a l l y c o n n e c t e d p l a t e a u r e g i o n , i.e. a p l a t e a u r e g i o n P i s a p l a t e a u i f a n d o n l y i f - i 3 s € P 3s' G S\P : J\f(s,s') A g(s) = g(s'). W e d e n o t e t h e ( u n i q u e ) p l a t e a u c o n t a i n i n g a s t a t e s € S w i t h Plateau(s). • Chapter 3. Plateau Characteristics 23 The preceding definition was the central definition of the chapter, and we w i l l spend the majority of the remainder of the chapter studying plateaus and their properties. A s we have alluded to earlier, every state in the search space belongs to exactly one plateau and so plateaus form a natural partition of the search space. Furthermore, because the evaluation function value is constant for all states in a plateau, the evaluation function cannot be used by SLS algorithms to guide the search when choosing between states in the same plateau. Also , since all states in the same plateau are connected, an SLS algorithm can move between any two states on the plateau without leaving the plateau (though it may, of course, require multiple steps). A l l of these plateau properties enable us to group the states in a plateau together and treat the entire plateau as a single entity. A s we w i l l demonstrate, understanding the properties of plateaus allows us to more fully understand how search space features can affect algorithm behaviour. We next define two important classes of plateaus — open and closed. Because local search algorithms have no heuristic guidance (w.r.t. the evaluation function) while searching a plateau, it is not obvious how much time should be invested in searching a given (potentially closed) plateau. The answer to this question depends on many factors, but it essentially boils down to determining the expected time that it would take to find a state on that plateau from which the search can progress again; such a state is referred to as an exit. The following definition classifies plateaus into open plateaus, which are plateaus which contain at least one exit (and thus may be worth searching), and closed plateaus which do not contain exits. Definition 3.4. (Exits, Open and Closed Plateau Regions) Let L = (S,j\f,g) be a search landscape and P a plateau in L. A state s G P is an exit of P if and only if s has a neighbour at a lower level than P, i.e. EXIT(s) holds if and only if 3s1 G S\P : Af(s,s') A g(s) > g(s'). In this context, s' is called a target of exit s. A plateau P is called an open plateau if and only if it contains at least one exit, otherwise P is called a closed plateau. • The remainder of this chapter is dedicated to an empirical investigation of the properties of plateaus in the search space. We would like to provide answers to questions such as: H o w many plateaus are there? What is the ratio of open to closed plateaus? H o w big are the plateaus? What does the internal structure of these plateaus look like? 3.2 Plateau characteristics of the UF-3-SAT distr ibution In this section, we study plateaus in the search spaces of instances from the UF-3-SAT distribution (a de-scription of this distribution is given in Appendix A ) . Rather than study properties of individual instances, Chapter 3. Plateau Characteristics 24 we attempt to probabilistically sample plateaus uniformly and at random from the entire UF-3-SAT dis-tribution, and measure their properties. Thus, this analysis gives insight into what type of plateaus one would expect to encounter in a typical instance from the UF-3-SAT distribution, but does not give any sense of what the search spaces of individual instances look like. The experimental analyses in this section are intended to replicate those of [FCS97] exactly. We repli-cated the instance distributions and experimental methodology as accurately as possible from the descrip-tions in the literature, and performed the same analyses. However, our results differ dramatically from the previously published results, and we draw correspondingly different conclusions. These discrepancies w i l l be discussed throughout this chapter, but we note at this time that the results presented in this section have been rigorously checked, and corroborated by several different methods. 3.2.1 Experimental Methodology Ideally, we would like to sample plateaus at low levels uniformly and at random from the UF-3-SAT dis-tribution. Unfortunately, this is not possible unless the number of variables is very low. To overcome this problem, we follow the approach of [FCS97] and use GSAT (c.f. Section 2.3) to sample plateaus. For each instance in the distribution (1000 instances total), we ran GSAT once for each level from 0 to 5 and output the first state encountered by G S A T at that level. If G S A T d id not encounter a state at the desired level during the run, it was m n again. This process gave us a distribution of states for each level 0 to 5. Because we used G S A T to sample the states, the distributions are biased — nonetheless, since we are interested in SLS algorithm behaviour, the states form an interesting sample. Also , since G S A T is initialized randomly, the bias is due only to the greedy descent mechanism in GSAT, and not by the starting state. Each state s was then expanded into Plateau(s) by using breadth-first search. Each state was stored with very low memory overhead in a bit vector, and double-counting of states was avoided by storing them in a hash-table. Because the plateaus were frequently very large, a maximum of 10 6 states were explored (compare this wi th a maximum of 10000 states in [FCS97]). We recorded statistics such as the size of the plateau and the number of exits, as wel l as other statistics which w i l l be discussed in later sections. 3.2.2 Fraction of Closed Plateaus Figure 3.1 shows the proportion of plateaus that are closed at each level. It is interesting to note that the shape of the curves is similar for all number of variables. The proportion of closed plateaus is (trivially) 1 at level 0, and decreases rapidly as the level is increased — there are vanishingly few closed regions at higher levels. The proportion of closed plateaus decreases faster than exponentially wi th the level; for example, for the uf50 test set, the best-fit function was f(x) = a~xb, wi th d « 2 and b « 1.35, and similar functions fit Chapter 3. Plateau Characteristics 25 0 1 2 3 4 5 level Figure 3.1: Fraction of closed plateaus sampled at each level. the data well for the other test sets. Perhaps most interesting is the relatively low proportion of closed plateau regions at level 1 — in fact, over half of the plateaus sampled from the uf20 and uf50 distributions were open, and only slightly less than half of those from uflOO were open as well. This indicates that a solution can be reached from a large fraction of the plateaus sampled at level 1 simply by exploring the plateau until an exit is found. While this may make it sound like these instances should be trivial to solve, many other details must be considered. For example, even though a plateau is open, it is not necessarily the case that it is easy to find an exit — the plateau may be very large and there may be only a few exits, or the exits may be clustered together in a small, fairly unconnected region of the plateau. This is studied in depth in Section 3.5. As hinted at earlier, there are discrepancies between these results and results of Frank, Cheeseman and Stutz that this experiment was supposed to replicate exactly [FCS97]. Frank et al. also show a monotonic decrease in the proportion of closed plateaus with increasing level. However, the values that they report are much higher in some cases. For example, for the uflOO distribution, Frank et al. report that approximately 0.9 of the plateaus samples at level 1 were closed, down to approximately 0.2 at level 5. Their results for smaller test sets are more comparable to ours. 3.2.3 Plateau Size Figures 3.2, 3.3, and 3.4 show the distribution of the sizes of the plateau regions encountered by GSAT on the uflO, uf50, and uflOO test sets, respectively. Our first observation is that plateaus in these distributions are very large, and the size is directly proportional to both the level and the number of variables. The Chapter 3. Plateau Characteristics 26 plateau size plateau size plateau size Figure 3.2: Distribution of plateau sizes (UF-3-SAT, 20 variables). Left: all plateaus, Centre: open plateaus, Right: closed plateaus. 1 10 100 1000 10000 10OOOO 1 e+06 1 10 100 1000 10000 100000 1 e+06 1 10 100 1000 10000 100000 16406 plateau size plateau size plateau size Figure 3.3: Distribution of plateau sizes (UF-3-SAT, 50 variables). Left: all plateaus, Centre: open plateaus, Right: closed plateaus. 100 1000 10000 100000 1e+06 plateau size 100 1000 10000 100000 le+06 plateau size J ly '/ /// 0 1 2 3 4 5 " ) 1000 10000 100000 le+06 plateau size Figure 3.4: Distribution of plateau sizes (UF-3-SAT, 100 variables). Left: all plateaus, Centre: open plateaus, Right: closed plateaus. Chapter 3. Plateau Characteristics 27 variance in the distributions decreases substantially with the level which is quite interesting, but not partic-ularly surprising considering the random nature of the instances. Recall that the plateau exploration was terminated after encountering 106 states, explaining why many of the CDFs are truncated. We first consider the sizes of the closed plateaus, found in the right panes of the figures. The closed plateaus are relatively small, and the sizes increase with the number of variables. There is a slight tendency for higher level closed plateaus to be smaller than those at lower levels. Recall that there are also fewer closed plateaus at higher levels, which is why there are so few data points in the CDFs for the high-level distributions. Thus, we observe that closed plateaus are larger and more abundant at lower levels. The situation looks quite different for open plateaus, which are shown in the centre panes of the figures. We find that the vast majority of the sampled open plateaus are very large, with the size increasing with level. Because a maximum of 106 states were explored, the full CDFs are visible only for the uflO distri-bution. In fact, for the uflOO distribution, only 55% of the level 1 open plateaus, 10% of the level 2 open plateaus, and very few higher level plateaus had less than 106 states. The vast majority of the sampled plateaus for the ufl00 test set were very large. These results are not at all consistent with those of Frank et al. [FCS97]. For example, Frank et al. report that the median size of level 1 closed plateaus sampled from the uflOO distribution was 48 states, and that only approximately 4% of the level 1 closed plateaus contained more than 1000 states. In contrast, we find a median size of 3580 states and fully 68% of the level 1 closed plateaus contained more than 1000 states. In fact, 4% of the level 1 closed plateaus that we sampled from the uflOO had more than 106 states. Our results for the open plateaus also differ dramatically from those of Frank et al. For the uflOO distribution, Frank et al. report that, although they found that open plateaus were typically 10-30 times larger than closed plateaus, the vast majority of open plateaus at level 1 consisted of fewer than 1000 states and very few had more than 10000. In contrast, less than 2% of the level 1 open plateaus that we sampled from the uflOO distribution were smaller than 1000 states, and fully 45% of them were larger than 106 states. 3.2.4 Discussion It is somewhat troubling that our results from this section differ so dramatically from those of Frank et al. [FCS97], which they were intended to exactly replicate. However, we reiterate that all of our results presented in this section were carefully checked, and we are confident that they are correct. A l l of the al-gorithms employed were corroborated using completely different methods (the methods from succeeding sections): Because we are reporting much larger plateaus than those from the literature, one might suspect that we are somehow over-representing plateaus — that is, counting states that should not be included in a plateau or double-counting states. However, it is very simple to confirm that a set of states are all elements of a single plateau, and we have performed this check for a large number of the plateaus we sampled. Chapter 3. Plateau Characteristics 28 It is unc lea r w h a t c o u l d cause the difference i n results. O n e poss ib le source of the d i sc repancy is that, as w e w i l l see i n the next sect ion, there are i n fact many, m a n y s m a l l p la teaus i n the search space. H o w e v e r , w h e n w e ac tua l ly s a m p l e d the plateaus that were encounte red b y G S A T over a large n u m b e r of runs , n o n e o f these s m a l l p la teaus w e r e encountered . It is poss ib le that F r a n k et al. s o m e h o w used a n u n s o u n d i m p l e m e n t a t i o n of G S A T that encountered a large n u m b e r of these s m a l l pla teaus. W h i l e the analys is i n this sect ion pa in t a n in teres t ing p ic ture of w h a t p la teaus are t y p i c a l l y f o u n d i n an entire U F - 3 - S A T d i s t r i bu t i on , it g ives n o i n d i c a t i o n of w h a t the search spaces of i n d i v i d u a l instances l ook l i ke . Fo r example , w e are mos t interested i n h o w the search spaces of, say, the easiest a n d hardest instances f rom a d i s t r i b u t i o n differ. W e w o u l d l i ke to k n o w w h a t search space structures are present i n the h a r d instances that cause diff icul t ies for S L S a lgor i thms . 3.3 Plateau characteristics of individual UF-3-SAT instances In this sect ion, w e s t u d y the search space of i n d i v i d u a l U F - 3 - S A T instances i n de ta i l , rather than s a m p l i n g search space features f r o m the ent i re instance d i s t r i bu t i on . F o r each instance, w e f i n d e v e r y l o w l eve l state, a n d then g r o u p the states in to their respect ive plateaus. T h u s , w e get a de t a i l ed l o o k at the comple te l o w -l e v e l r eg ion of the search space for each instance. It is c r i t i ca l to s t u d y the search space o n an instance-by-instance basis i n o rder to unde r s t and the be-h a v i o u r of S L S a lgor i thms . S u c h an a p p r o a c h makes it poss ib le to s t u d y the search space features re-spons ib le for m a k i n g a g i v e n instance easy or h a r d for loca l search to so lve , rather than s i m p l y m a k i n g observat ions about the entire instance d i s t r ibu t ion . 3.3.1 Experimental Methodology In o rder to s t u d y the comple te l o w - l e v e l search space of these instances, w e need a m e t h o d to f ind every s ingle l o w l e v e l p la teau . T h i s is quite a di f f icul t task. S a m p l i n g cannot be u sed , because this w i l l not ensure that the entire search space is s a m p l e d . The comple te search space cannot be enumera t ed because it is too large. O u r a p p r o a c h to the p r o b l e m i n to first f i n d a l l of the states at a g i v e n l eve l , a n d then to pa r t i t i on this set of states in to its c o m p o n e n t plateaus. The f o l l o w i n g def in i t ion formal ises the n o t i o n of the comple te set of states at a g i v e n l eve l , w h i c h w e refer to as a levelset . Definition 3.5. (Levelset) L e t L = (S,N,g) be a search landscape . T h e levelset at level k, deno ted Levelsetk is the set of a l l states h a v i n g l e v e l k. F o r m a l l y , Levelsetk :— {s e S | g(s) = k}. N o t e that Levelsetk is c o m p r i s e d of a l l of the states f rom a l l p la teaus at l e v e l k. • Chapter 3. Plateau Characteristics 29 Algorithm 3.1: L e v e l s e t ^ , maxLev) Input : C N F Formula T, Integer maxLev Output: Set of BDDs representing Levelset begin //Build the partial levelsets zo.r.t. each clause LS ^ 0 foreach c G T do pis0 <- BDDSatSet(c) pis1 <- ^BDDSatSet(c) for k <— 2 • • • maxLev do | p/s fc <— ± end LS <- L S U { p / s } end / / Merge r«e partial levelsets until only one remaining while #LS > 1 do {pls\,pls2) <— ChooseTwoPartialLevelsets(LS) LS <- L S \ { p / s i , p / s 2 } for fc <— 0 • • • maxLev do | P / S l c 2 <~ V{!,;|;+;=fc}(P'4 A p/ S ; 2) end LS <— LS U {p /si o 2 } end / / On/y one set of partial levelsets left — it represents the set of complete levelsets return Is, where Is is the single remaining element of LS 36 end Because the sets of states that we are exploring are so large, it is infeasible to store them using conven-tional means such as simple lists of bit vectors. To avoid these problems, we use Binary Decision Diagrams (BDDs) to store the levelsets and the plateaus. BDDs allow us to represent the sets of states by Boolean functions: a state is an element of the set if and only if it is a minterm of the Boolean function represented by the B D D . Appendix B gives a brief introduction to BDDs, and a description of the B D D functions that we use. The pseudo-code shown in Algori thm 3.1 elucidates our method for calculating the low level levelsets Levelsetk, for k — 0, • • • , maxLev. For all of the experiments in this section, we set maxLev to 5. The input to the algorithm is a C N F Formula, and the maximum level that we wou ld like to calculate the levelset for. Note that the procedure calculates all of the levelsets simultaneously. A key concept for understanding this process is that of a partial levelset. This is simply a levelset wi th respect to a subset of the clauses. In the for loop declared at line 5, we bui ld partial levelsets wi th respect to each clause. A t level 0, we simply have all states that satisfy the clause. A t level 1, we have all states that do not satisfy the clause. Every state in the search space falls into exactly one of these classes, so for all levels Chapter 3. Plateau Characteristics 30 greater than 1 we set the BDDs to ± , representing the empty set. Note that each pis is a set of maxLev + 1 BDDs, and that LS is a set of sets of BDDs. Once we have these partial levelsets for each clause, we begin to combine them. A t line 22, we call a procedure to heuristically choose two partial levelsets to be combined. A n y two choices are correct, but the size of the BDDs explodes if the order of combination is not made carefully. We have found that choosing two partial levelsets according to a heuristic that both maximises the overlap of the support of the two BDDs while simultaneously minimising the B D D sizes gives the best results. The two chosen partial levelsets are removed from the collection LS, and then recombined into a new partial levelset. The levelset combination is shown in line 26. It should be clear that, given two disjoint sets of clauses Q and C 2 and their corresponding partial levelsets pls\ and pls-i, it is fairly straightforward to bui ld the partial levelset p / s j o 2 for C = C\ U C 2 . In order for a state s to be at level k wi th respect to C, it w i l l have to be at level i wi th respect to C\ and level / wi th respect to C 2 , where i and are chosen such that i + j = k. This is exactly what is accomplished in line 26 (note that the A and V operations are being performed on BDDs, and correspond to the B D D and and or operations, respectively). Finally, p/s l 02 is added to the collection LS. Note that p / s i o 2 is the partial levelset wi th respect to all of the clauses covered by pls\ and p/s 2. Since we start wi th a set of partial levelsets, each corresponding to a single clause in the formula, when all of the partial levelsets have been recombined so that there is only one remaining, this last remaining partial levelset is actually the (complete) levelset wi th respect to the entire formula. Once we have collected all of the low level levelsets, we must partition each levelset into its component plateaus. The pseudo-code of Algori thm 3.2 illustrates our partitioning process. The input to the algorithm is a B D D representation of Levelsetk, for some k, and the output is the set of all plateaus at level k (also represented as BDDs). In line 4, we choose a single cube from the remaining partition, and add it to the B D D P, representing the explored portion of the plateau. Starting from those states, we iteratively expand the plateau using a set-wise breadth-first-search until all of the states in the plateau have been added. Lines 9 - 1 3 illustrate how the plateau is expanded. For every variable v, the function representing the plateau but wi th v complemented is calculated, and restricted to level k. The disjunction of all of those functions, Nbr, represents all neighbours of the partial plateau that are also elements of the plateau being explored (note that many of the states may already be in P). Nbr is then added to P and removed from Remaining. It is trivial to see that when no more new states are generated, the entire plateau has been explored. Using the two procedures described above, we can find and represent every plateau in the entire search space, or just all plateaus at low levels in the search space. It is no longer necessary to resort to sampling the search space — we get a complete picture of the entire low level portion of the search space. However, the computation is exceptionally expensive (some of the experiments used to collect data for the succeeding Chapter 3. Plateau Characteristics 31 Algorithm 3.2: Partition(Leue/sef'c) Input : BDD Representation of Levelsetk Output: Set of BDDs representing each plateau begin Remainder <— Levelsetk while Remainder ^ J_ do P <- BDDPickOneCube(Remainder) fixed Point <— false //expand the plateau containing that cube (setwise-BFS) while Remainder ^ J_ and fixedPoint — false do //find all neighbours on the current plateau foreach v G Var(!F) do Nbrv <- BDDNegate(P,v) NbrLSv *— Nbrv A Remainder end Nbr <- \lv&Var{T) NbrLSv //check for fixed point if P A Nbr = P then | fixedPoint <— true else end end fixedPoint <— false P f - P V Nbr Remainder <— Remainder A ^Nbr 1/ add the complete plateau to the set of all plateaus PlateauSet <— PlateauSet U {P} end return PlateauSet 27 end //Start with entire levelset 1/ choose one BDD cube to start from Chapter 3. Plateau Characteristics 32 1 10 100 1000 10000 n u m b e r of p l a t e a u s Figure 3.5: Distribution of the number of plateaus at each level over instances in the uf30 test set. level 0 1 2 3 4 5 Test Set x ax/x x crx/x X crx/x x ax/x x crx/x x crx/x uf20 uf25 uf30 uf35 uf40 uf45 1.7 (0.65) 1.7 (0.56) 2.2 (0.73) 2.3 (0.80) 2.4 (0.78) 2.6 (0.73) 8.4 (0.43) 10.5 (0.41) 13.7 (0.46) 16.9 (0.51) 18.6 (0.53) 22.1 (0.48) 22.8 (0.41) 32.0 (0.36) 46.2 (0.36) 63.8 (0.42) 79.1 (0.45) 98.6 (0.39) 47.8 (0.36) 76.2 (0.33) 127.1 (0.33) 193.1 (0.38) 270.6 (0.45) 357.0 (0.37) 90.2 (0.30) 168.1 (0.32) 315.9 (0.32) 510.8 (0.35) 794.2 (0.44) 1134.2 (0.34) 155.6 (0.27) 339.3 (0.30) 703.1 (0.33) 1239.1 (0.34) 2088.2 (0.43) 3210.9 (0.33) Table 3.1: Statistics of the number of plateaus in UF-3-SAT instances, counting all plateaus. For every test set and level we show the mean number of plateaus as wel l as the coefficient of variation. sections ran for over four C P U weeks), so the analysis is restricted to fairly small instances. Nevertheless, we hope that the intuitions and hypotheses developed from studying these smaller instances w i l l also hold for larger instances — and we w i l l show that this is the case in later chapters of the thesis. 3.3.2 Number of Plateaus Figure 3.5 shows the distribution of the number of plateaus (of al l types) for the uf30 test set. The number of solution plateaus is quite low — in any given instance, there are less than 10 solution plateaus — and the C D F for level 0 looks quite jaggy simply because of the discretised scale on the x axis of the plot. As the level is increased, the typical number of plateaus also increases, though the relative rate of increase decreases wi th level. Interestingly, the coefficient of variation also decreases wi th level. Table 3.1 shows summary statistics of the number of plateau regions found at each level for all of the UF3SAT test sets. We see that the observations we made for the uf30 test set generalize wi th the number of Chapter 3. Plateau Characteristics 33 10000 20 25 30 35 40 45 # variables Figure 3.6: Scaling of the average number of plateaus wi th the number of variables for the UF3SAT test sets. Each point represents the average number of plateaus at the given level, for the distribution with the given number of variables. The error bars show the standard deviation of each distribution. variables. In all cases, the average number of plateaus increases wi th level, while the coefficient of variation decreases. We also observe that the number of plateaus increases rapidly wi th the number of variables. Figure 3.6 shows how the number of plateaus scales with the number of variables. There are too few data points for an accurate assessment of the scaling behaviour, but the scaling seems to be sub-exponential (note the log scale on the y axis of the plot). 3.3.3 Fraction of Closed Plateaus Tables 3.2 and 3.3 show the same data as Table 3.1, except that the plateaus have been separated into open and closed plateaus, respectively. We observe that the vast majority of plateaus are in fact open — especially at higher levels. Interestingly, while the number of open plateaus increases wi th level, the number of closed plateaus decreases. Thus, closed plateaus exist mainly at lower levels and are not only relatively rare (compared to open plateaus) at higher levels, they simply do not occur. Note that the number of closed plateaus does increase wi th the number of variables. Figure 3.7 shows the average proportion of plateaus that are closed at each level for all of the UF3SAT test suites. This figure is surprisingly similar to Figure 3.1. The ratio decreases rapidly wi th the level; the function f(x) = a~%b fits these curves wel l . The ratios tend to be higher for larger number of variables, though this is not true, for example, for the ufl.5 test set, which has a higher than expected ratio for lower levels, and lower than expected for higher levels. The coefficient of variation in the distribution of plateau counts is significantly higher for the closed Chapter 3. Plateau Characteristics 34 level 1 2 3 4 5 Test Set x <rx/x X <Tx/x x ax/x x crx/x x crx/x uf20 uf25 uf30 uf35 uf40 uf45 6.0 (0.56) 7.0 (0.53) 9.4 (0.57) 11.3 (0.73) 12.2 (0.71) 14.3 (0.65) 20.7 (0.45) 28.5 (0.41) 40.9 (0.42) 56.5 (0.48) 69.6 (0.52) 86.3 (0.44) 46.7 (0.37) 73.8 (0.35) 123.7 (0.35) 187.3 (0.40) 262.1 (0.47) 343.8 (0.39) 89.8 (0.31) 167.4 (0.32) 314.2 (0.33) 507.5 (0.36) 788.5 (0.44) 1123.6 (0.35) 155.6 (0.27) 339.1 (0.30) 702.6 (0.33) 1237.7 (0.34) 2085.1 (0.43) 3205.2 (0.33) T a b l e 3 .2 : S t a t i s t i c s o f t h e n u m b e r o f p l a t e a u s i n U F - 3 - S A T i n s t a n c e s , c o u n t i n g o n l y o p e n p l a t e a u s . F o r e v -e r y tes t se t a n d l e v e l w e s h o w t h e m e a n n u m b e r o f p l a t e a u s a s w e l l a s t h e c o e f f i c i e n t o f v a r i a t i o n . level 0 1 2 3 4 5 Test Set x crx/x x crx/x x ax/x X CTX/X x uxlx x crx/x uf20 uf25 uf30 uf35 uf40 uf45 1.7 (0.65) 1.7 (0.56) 2.2 (0.73) 2.3 (0.80) 2.4 (0.78) 2.6 (0.73) 2.4 (0.69) 3.5 (0.59) 4.3 (0.61) 5.6 (0.67) 6.5 (0.62) 7.8 (0.61) 2.1 (1.00) 3.6 (0.67) 5.3 (0.59) 7.3 (0.58) 9.5 (0.54) 12.3 (0.48) 1.1 (1.03) 2.4 (0.83) 3.4 (0.77) 5.8 (0.68) 8.6 (0.62) 13.2 (0.49) 0.4 (1.92) 0.7 (1.47) 1.7 (1.05) 3.3 (0.83) 5.7 (0.78) 10.7 (0.72) 0.1 (3.55) 0.1 (3.03) 0.5 (1.73) 1.4 (1.07) 3.1 (0.91) 5.7 (0.93) T a b l e 3 .3 : S t a t i s t i c s o f t h e n u m b e r o f p l a t e a u s i n U F - 3 - S A T i n s t a n c e s , c o u n t i n g o n l y c l o s e d p l a t e a u s . F o r e v -e r y tes t se t a n d l e v e l w e s h o w t h e m e a n n u m b e r o f p l a t e a u s as w e l l a s t h e c o e f f i c i e n t o f v a r i a t i o n . F i g u r e 3.7: A v e r a g e f r a c t i o n o f c l o s e d p l a t e a u s a t e a c h l e v e l f o r t he U F 3 S A T tes t s u i t e s . B o t h p l o t s s h o w the s a m e d a t a , b u t t h e p l o t o n t h e r i g h t i s a s e m i - l o g p l o t w h i c h i l l u s t r a t e s t h e ( s u p e r - ) e x p o n e n t i a l d e c r e a s e o f f r a c t i o n c l o s e d w i t h t he l e v e l . Chapter 3. Plateau Characteristics 3 5 level 0 1 2 3 4 5 Test Set x crx/x X crx/x X crx/x x crx/x x crx/x x crx/x uf20 uf25 uf30 uf35 uf40 uf45 6.7 (2.66) 8.0 (1.25) 13.8 (1.35) 35.0 (2.00) 76.3 (3.38) 68.4 (1.47) 5.3 (0.82) 7.1 (0.85) 17.4 (1.60) 24.7 (1.03) 37.4 (1.47) 48.9 (0.83) 4.9 (1.04) 5.3 (0.82) 9.2 (0.86) 13.3 (0.79) 23.4 (1.26) 25.1 (0.69) 2.5 (0.69) 3.6 (0.68) 5.2 (0.99) 8.4 (0.80) 11.6 (0.76) 19.7 (0.89) 2.1 (0.58) 2.3 (0.80) 4.3 (1.14) 5.6 (0.62) 6.8 (0.74) 11.5 (0.69) 2.4 (0.94) 2.0 (0.44) 3.2 (1.15) 4.4 (0.85) 4.8 (0.65) 7.3 (0.63) T a b l e 3 .4: S t a t i s t i c s o f t h e s i z e o f c l o s e d p l a t e a u s i n U F - 3 - S A T i n s t a n c e s . F o r e v e r y tes t se t a n d l e v e l w e s h o w t h e m e a n o f t h e a v e r a g e s i z e o f c l o s e d p l a t e a u s i n e a c h i n s t a n c e , a s w e l l a s t h e c o e f f i c i e n t o f v a r i a t i o n o f t he d i s t r i b u t i o n o f a v e r a g e s i z e s . level 1 2 3 4 5 Test Set fmax ax/x fmax vx/x fmax ax/x fmax ax/x fmax crx/x uf20 0.64 (0.3) 0.74 (0.3) 0.91 (0.1) 0.96 (0.0) 0.98 (0.0) uf25 0.64 (0.4) 0.78 (0.3) 0.95 (0.1) 0.98 (0.0) 0.99 (0.0) uf30 0.70 (0.3) 0.87 (0.2) 0.96 (0.1) 0.99 (0.0) >0.99 (0.0) uf35 0.71 (0.3) 0.88 (0-2) 0.98 (0.0) >0.99 (0.0) >0.99 (0.0) uf40 0.74 (0.3) 0.89 (0.2) 0.99 (0.0) >0.99 (0.0) >0.99 (0.0) uf45 0.75 (0.3) 0.91 (0.1) 0.98 (0.0) >0.99 (0.0) >0.99 (0.0) T a b l e 3 .5 : F r a c t i o n o f s t a t e s c o n t a i n e d i n t h e l a r g e s t p l a t e a u f o r t h e U F 3 S A T tes t se t s . F o r e v e r y test se t a n d l e v e l k w e s h o w t h e m e a n a n d c o e f f i c i e n t o f v a r i a t i o n o f t h e d i s t r i b u t i o n o f f^ax v a l u e s . r e g i o n s t h a n t h e o p e n , a n d t h i s v a l u e i n c r e a s e s w i t h t h e l e v e l . T h i s i s v e r y i n t e r e s t i n g , s u g g e s t i n g t h a t w h i l e s o m e s e a r c h s p a c e f e a t u r e s a r e f a i r l y u n i f o r m b e t w e e n i n s t a n c e s , t h e n u m b e r o f c l o s e d p l a t e a u s i s n o t . O b v i o u s l y , t h e n u m b e r o f c l o s e d p l a t e a u s i s c l o s e l y r e l a t e d to t h e n u m b e r o f L M I N s t a tes , w h i c h i s k n o w n t o b e c o r r e l a t e d w i t h p r o b l e m h a r d n e s s i n o t h e r c o m b i n a t o r i a l o p t i m i s a t i o n p r o b l e m s [Yok97]. T h u s , i t i s f e a t u r e s s u c h as t h i s t h a t m a y h e l p to e x p l a i n t h e l a r g e v a r i a n c e i n l o c a l s e a r c h c o s t w i t h i n a d i s t r i b u t i o n o f h o m o g e n e o u s i n s t a n c e s . 3.3.4 Plateau Size T a b l e 3 .4 s h o w s s t a t i s t i c s o f t h e s i z e s o f c l o s e d p l a t e a u s i n t h e U F - 3 - S A T tes t se t s . N o t s u r p r i s i n g l y , t he s i z e o f t h e c l o s e d p l a t e a u s g r o w s w i t h t h e n u m b e r o f v a r i a b l e s . W h a t i s m o r e i n t e r e s t i n g , h o w e v e r , i s t h a t t h e c l o s e d p l a t e a u s a r e t y p i c a l l y v e r y s m a l l , a n d t h a t t h e s i z e a c t u a l l y d e c r e a s e s w i t h t h e l e v e l . T h u s , c l o s e d p l a t e a u s a r e b o t h v e r y r a r e a n d v e r y s m a l l a t h i g h e r l e v e l s . S i n c e w e b e l i e v e t h a t c l o s e d p l a t e a u s a r e o n e o f the m o s t p r o m i n e n t s e a r c h s p a c e f e a t u r e s t h a t i n t e r f e r e w i t h S L S a l g o r i t h m s , t h i s i s q u i t e i n t e r e s t i n g — i t i s c r u c i a l f o r a h i g h - p e r f o r m a n c e S L S a l g o r i t h m to e f f e c t i v e l y a v o i d c l o s e d p l a t e a u s a t l o w l e v e l s o f t he s e a r c h s p a c e . T h e s i t u a t i o n w i t h o p e n p l a t e a u s i s m a r k e d l y d i f f e r e n t . W e f i n d t h a t t h e d i s t r i b u t i o n o f s i z e s o f o p e n p l a t e a u s i s q u i t e i r r e g u l a r . R e c a l l t h a t t h e r e a r e a l a r g e n u m b e r o f o p e n p l a t e a u s i n t h e s e a r c h s p a c e , e s p e -Chapter 3. Plateau Characteristics 36 level 1 2 3 4 5 Test Set med max med max med max med max med max uf20 2.5 63.8 1.9 428.6 1.7 1913.1 1.4 5891.4 1.1 14266.2 uf25 3.7 141.0 2.7 1389.0 2.1 7779.1 1.8 29658.1 1.6 88846.3 uf30 4.6 524.0 3.3 5753.5 2.5 34524.6 2.1 150860.1 2.0 523251.1 uf35 7.4 1409.8 4.4 15082.2 3.4 101773.3 2.6 512087.0 2.1 2.07 x 106 uf40 10.8 3675.0 6.2 46339.7 4.1 351692.3 3.3 1.98 x 106 2.6 8.99 x 106 uf45 16.9 5547.1 8.3 91009.9 5.8 851473.2 4.4 5.73 x 106 3.6 3.04 x 107 Table 3.6: Statistics of the size of open plateaus in UF-3-SAT instances. For every test set and level we show the mean of the distribution of median and maximum sizes of open plateau regions. da i ly at higher levels. With this in mind, consider the data presented in Table 3.5. In this table, we show statistics of the value: #Pk rk 1 7 1 max J m a x ~ #Levelsetk where k is the level, and Pmax is the open plateau at level k having maximal size. Intuitively, fmax gives an indication of how much of the level is contained in the largest plateau — it w i l l soon become clear why this is an interesting value to study. Table 3.5 illustrates a very interesting phonomenon: even though there are many open plateaus (up-wards of 3000 for the largest test sets at level 5), most of the states at higher levels belong to a single, large, open plateau. This phenomenon is present for all of the test sets, and appears stronger for the larger sets. Starting at roughly level 3, every level has one large plateau region that contains the bulk of the states at that level. A t higher levels, there are (relatively) very few states that do not belong to Pmax- There is also virtually no variance in the distribution of fmax values for large k, indicating that the phenomenon is present in all of the instances. Due to the irregular distribution of open plateau sizes, it is meaningless to consider, for example, the average plateau size at a given level. In order to illustrate the typical size of open plateau regions, we look at the distribution of median and max sizes, and present statistics of those distributions in Table 3.6. Not surprisingly (considering the results from the preceding paragraph), we find that the median open plateau size tends to be quite small. Surprisingly, the median size decreases wi th the level — typical open plateaus at level 5 contain less than 5 states. Closer examination reveals that at higher levels these very small open plateaus are typically degenerate in that they consist entirely of exit states — these plateaus would have very little effect on the behaviour of SLS algorithms. The median size of open plateaus increases with the number of variables. The size of Pmax is also shown in Table 3.6. In contrast to the median open plateau size, the maximum open plateau size increases rapidly with level. The maximum size also increase rapidly wi th the number of variables so that at level 5, the largest open plateau in the 45 variable test set has over 30 mil l ion states. Chapter 3. Plateau Characteristics 37 level 0 1 2 3 4 5 Instance # o • # o • # o • # o • # o • # o • anom 1 0 1 4 2 2 13 12 1 56 56 0 183 171 12 574 565 9 medium 2 0 2 5 2 3 37 33 4 221 188 33 986 871 115 4652 4431 221 flat20-easy 48 0 48 73 73 0 86 86 0 494 494 0 3023 3023 0 9014 9014 0 flat20-med 168 0 168 133 133 0 355 355 0 2371 2371 0 5662 5662 0 15841 15841 0 Hat20-hard 6 0 6 43 19 24 38 38 0 320 320 0 1238 1238 0 3458 3458 0 par8-l-c 1 0 1 1 1 0 19 15 4 65 18 47 414 229 185 1794 1152 642 Table 3.7: N u m b e r of plateaus i n s t ruc tured instances. F o r each instance a n d l eve l , w e s h o w the total n u m -ber of plateaus, the total n u m b e r of o p e n plateaus, a n d the total n u m b e r of c losed plateaus (# , o a n d • , respect ively) . H a p p i l y , the p r o p o r t i o n of exits f rom these immense plateaus is v e r y h i g h (for v e r y h i g h levels , these large pla teaus ac tua l ly consist o n l y of exits), so they d o not s ign i f ican t ly i m p e d e the progress of S L S a lgor i thms . 3.4 P l a t e a u characterist ics o f s t ruc tured ins tances In this sect ion, w e tu rn ou r at tent ion to another class of instances — s t ruc tured instances. These instances (descr ibed i n A p p e n d i x A ) are not r a n d o m l y generated, bu t rather are S A T - e n c o d e d instances of other p rob lems , s u c h as p l a n n i n g , g r a p h c o l o u r i n g , a n d pa r i t y check ing . A g a i n , w e examine the characteristics of p la teaus i n the search spaces of these instances. T h e a lgo r i thms w e use a n d the da ta w e measure are iden t i ca l to those of the p r e c e d i n g sect ion, a l l o w i n g the results to be contras ted . W e first s t u d y the n u m b e r of p la teaus d i s t r i bu t ed i n the l o w levels of the search space, a n d measure the f ract ion of the plateaus that are c losed . F o l l o w i n g this, w e present results of the e m p i r i c a l l y m e a s u r e d d i s t r ibu t ions of the sizes of the plateaus. 3.4.1 Number of Plateaus We first t u rn ou r at tent ion to Table 3.7, w h i c h s h o w s the total n u m b e r of p la teaus at each l eve l i n the s t ruc tured instances that w e s tud i ed . The total n u m b e r of plateaus is further b r o k e n d o w n in to the n u m b e r of o p e n a n d c losed plateaus. N o t e that i n this table the entries represent the ac tua l p la teau counts for the instances, u n l i k e s i m i l a r tables i n p r e v i o u s sections w h i c h d i s p l a y e d statistics across a d i s t r i b u t i o n o f instances. T h e results are v e r y interest ing. We first note that the total n u m b e r of pla teaus at each l eve l increases . w i t h the l e v e l — this is consistent w i t h ou r results for the U n i f o r m R a n d o m 3 - S A T test suites. There are too few data po in t s to m a k e a r igorous c l a i m , bu t the sca l ing of the total n u m b e r of pla teaus appears to be exponen t i a l w i t h the l e v e l , e spec ia l ly at h i g h e r levels . T h e total n u m b e r o f o p e n a n d c lo sed p la teau regions also seem to increase w i t h the l eve l for a l l of the s t ruc tured instances, w i t h the excep t ion of the g r a p h Chapter 3. Plateau Characteristics 38 level Instance 0 1 2 3 4 5 anom 1.00 0.50 0.08 0.00 0.07 0.02 medium 1.00 0.60 0.11 0.15 0.12 0.05 flat20-easy 1.00 0.00 0.00 0.00 0.00 0.00 flat20-med 1.00 0.00 0.00 0.00 0.00 0.00 flat20-hard 1.00 0.56 0.00 0.00 0.00 0.00 par8-l-c 1.00 0.00 0.21 0.72 0.45 0.36 Table 3.8: Fraction of plateaus that are closed at each level in structured instances. colouring instances which have no closed plateau regions at higher levels. Additionally, there does not seem to be a direct relationship between the size of the instances and the number of plateaus. The blocks-wor ld planning instance, medium, is the largest of the instances and while it does have more plateaus than the smaller planning instance anom and the smaller parity-checking instance p a r - 8 - l - c , the flat graph colouring instances have the most plateaus of all the structured instances. Due to the differences in problem size, it is difficult to draw a direct comparison between these results and those for the Uniform Random 3-SAT test suites. However, the smallest structured instance, anom, is approximately the same size as the largest Uniform Random 3-SAT instances — though the instances are obviously syntactically quite different. When comparing the results of these instances, we observe that the anom instance has significantly fewer plateaus (and fewer open plateaus) than the average number of plateaus (and average number of open plateaus) in the w/45 test set. A t lower levels the anom instance has fewer closed plateau regions than instances in the uf45, but this relationship is reversed at higher levels. Table 3.8 shows the fraction of plateaus at each level that are closed in the structured instances. For the planning instances we see that the fraction of plateaus that are closed tends to decrease rapidly with the level, indicating that the major obstacles to local search algorithms are encountered at low levels. The flat graph colouring instances are quite unusual in that, although they have a large number of plateau regions, all of the plateaus are open. The only exception to this is the hardest colouring instance, in which 56% of the level 1 plateaus were closed. Given this information about the search space of the graph colouring instances, it should not come as a surprise that these instances are indeed the easiest of the structured instances studied here. It is interesting that the hardest instance was the only one to have closed plateaus that were not solutions — this clearly suggests that these closed regions may be the reason why this instance, which is syntactically similar to the other colouring instances, is so much more difficult for SLS algorithms. This w i l l be studied much more closely in the following chapter of this thesis. Lastly, we observe that the parity instance is quite different than all of the other instances studied in this chapter (including the Uniform Random 3-SAT instances) i n that a large proportion of its plateaus are closed, even at higher levels in the search space. Thus, local search algorithms may have a difficult time even reaching the lower levels of the search space. This offers a partial explanation why parity instances (more specifically, larger versions of Chapter 3. Plateau Characteristics 39 level 0 1 2 3 4 5 Instance X (7XIX x. crx/x X (Jx/x x. crx/x x crx/x x trx/x anom medium flat20-easy flat20-med flat20-hard par8-l-c 1.0 (0.00) 1.0 (0.00) 65530.5 (2.64) 58.4 (1.21) 945.0 (0.00) 1.0 (0.00) 1.0 (0.00) 1.0 (0.00) (-) (-) 1701.0 (1.04) (-) 1.0 (0.00) 1.0 (0.00) (-) (-) (-) 2.8 (0.47) H 2.3 (1.08) (-) (-) (-) 19.7 (1.26) 2.9 (0.89) 4.1 (1.20) H H (-) 46.8 (2.17) 4.0 (1.04) 5.2 (1.62) H H (-) 35.0 (1.67) Table 3.9: Statistics of the size of closed plateaus in structured instances. For every instance and level we show the mean size of closed plateaus, as well as the coefficient of variation of the distribu-tion. Many of the instances (most notably the graph colouring instances) do not have any closed plateaus at some levels; these entries are labelled with - . syntactically similar instances) are among the hardest known instances for modern SLS algorithms. 3.4.2 Plateau Size We now study the distribution of plateau sizes in structured instances. First, Table 3.9 presents statistics of the distributions of sizes of closed plateaus at each level for the structured instances. We note that the mean size of closed plateaus is quite small and has little variance for the planning instances, though the average size does increase slowly with the level. As was mentioned in the previous section, the easy and median difficulty graph colouring problems have no closed plateaus that are not solutions. The solution plateaus tend to be quite large (especially for the easiest instance), again helping to explain why those instances are so easy for local search algorithms. We also see that the average size of the closed regions at level 1 in the hardest graph colouring instance is by far the largest that we have observed in any of the instances (though still only an average of 1701 states). The average size of the closed plateaus in the parity instance increases with the level and is larger than those of the planning instances. We observe a fair amount of variation in this distribution, and closer inspection reveals that there is indeed a wide range of sizes. To illustrate, at level 5 there are 642 closed plateau regions. The smallest has a single state, and the largest has 630 states. These results are quite different than those for the Uniform Random 3-SAT instances. While the closed plateaus tended to get smaller with the level in the Uniform Random 3-SAT instances, the size of closed plateaus increases with level in the structured instances. Thus, while closed plateaus were only a problem at very low levels for Uniform Random 3-SAT instances, they may significantly affect the behaviour of SLS algorithms even at high levels in the search spaces of structured instances. This is consistent with the relatively poor performance of current SLS algorithms on many structured instances. Next, we consider the size of open plateau regions in the structured instances. A n immediate first question that we would like to resolve is whether or not the structured instances exhibit the "one-big-plateau" phenomenon we observed in the Uniform Random 3-SAT test suites. Table 3.10 shows the f^ax Chapter 3. Plateau Characteristics 40 Instance 1 2 level 3 4 5 anom 0.43 0.19 0.11 0.05 0.04 medium 0.36 0.12 0.06 0.02 0.01 flat20-easy 1.00 0.99 0.98 0.98 0.97 flat20-med 0.99 1.00 1.00 1.00 0.99 flat20-hard 0.96 1.00 0.99 0.99 0.98 par8-l-c 1.00 0.66 0.29 0.13 0.16 Table 3.10: Fraction of states contained in the largest plateau for the structured instances. For every instance and level k we show the (single) fkax value. values, which can be contrasted wi th those in Table 3.5. We observe somewhat mixed results. The planning instances do not have large fknx values and, in fact, the value decreases with level. The parity instance shows a similar trend, though the values are significantly larger. These results are diametrically opposed to those that we obtained for the Uniform Random 3-SAT test suites — the search spaces of these instances do not seem to be dominated at higher levels by a single large plateau. The fkwx values for the colouring instances, however, are exceptionally large, even at fairly low levels in the search space. Given that we know that there are a large number of plateaus at these levels, it must be the case that the vast majority of (open) plateaus in the search space of these instances are very small (relative to the largest plateau). Table 3.11 shows statistics of the sizes of the open plateau regions for the structured instances. A s expected, both the median and the maximum sizes of the open plateaus tend to increase wi th the level. The planning and parity instances seem to have fairly small open plateaus compared with the other instances (including the Uniform Random 3-SAT instances), even at fairly high levels of the search space. The graph colouring instances are again the exception. While the median open plateau sizes are all fairly small (less than 1260 states for all levels less than 5), the maximum sizes are extremely large. A t the highest level measured, level 5, the largest open plateaus in all of the graph colouring instances consisted of hundreds of billions of search states. In fact, these results are consistent wi th our observations for the random instances. In all of the cases that we studied (both random and structured), the median plateau size at each level were generally quite small, while the maximum sizes could be very large. This suggests that it may be possible to partially explore plateaus encountered during search, as long as the maximum number of states explored is limited to a fairly small value — for example, the median plateau size (an idea which has been proposed before, e.g. in [FCS97, HK95]). A n approach such as this opens up many possibilities for augmenting existing SLS algorithms wi th plateau searching capabilities. Chapter 3. Plateau Characteristics 4 1 level 1 2 3 4 5 Instance med max med max med max med max med max anom 3.0 3.0 3.0 7.0 2.0 19.0 3.0 44.0 3.0 164.0 medium 4.0 4.0 3.0 15.0 3.0 54.0 4.0 177.0 4.0 604.0 flat20-easy 21.0 9.04 x 107 35.0 1.27 x 109 189.0 1.23 x IO 1 0 273.0 9.45 x IO 1 0 266.0 5.87 x IO 1 1 flat20-med 21.0 9.78 x 105 10.0 3.93 x 107 45.0 8.7 x 10s 49.0 1.25 x IO 1 0 42.0 1.26 x IO 1 1 flat20-hard 945.0 1.43 x 106 135.0 9.78 x 107 945.0 2.84 x 109 1260.0 4.7 x IO 1 0 1215.0 5.03 x IO 1 1 par8-l-c 4.0 4.0 6.0 162.0 24.0 498.0 18.0 3585.0 24.0 42886.0 T a b l e 3 . 1 1 : S t a t i s t i c s o f t h e s i z e o f o p e n p l a t e a u s i n s t r u c t u r e d i n s t a n c e s . F o r e v e r y i n s t a n c e a n d l e v e l w e s h o w b o t h t h e m e d i a n a n d m a x s i z e o f t h e o p e n p l a t e a u s . 3.5 Internal Plateau Structure U p to t h i s p o i n t , t h e d i s t r i b u t i o n o f p l a t e a u s h a s b e e n s t u d i e d — t h e s i z e , n u m b e r , f r a c t i o n o f p l a t e a u s t h a t a r e c l o s e d , e t c . I n t h i s s e c t i o n , w e s t u d y t h e a c t u a l s t r u c t u r e o f t h e p l a t e a u r e g i o n s , w h i c h c a n o b v i o u s l y h a v e a l a r g e i m p a c t o n t h e b e h a v i o u r o f S L S a l g o r i t h m s . W e w i l l s t u d y p r o p e r t i e s s u c h as t h e d i a m e t e r o f p l a t e a u s , t h e b r a n c h i n g f a c t o r o f t h e s t a t e s t h a t c o m p o s e t h e p l a t e a u s , a n d w e l l a s a m e a s u r e o f h o w d i f f i c u l t i t i s f o r S L S a l g o r i t h m s t o f i n d a n e x i t f r o m t h e p l a t e a u . T h e f o l l o w i n g d e f i n i t i o n s f o r m a l i s e t h e s e c o n c e p t s . Definition 3.6. (Diameter) L e t L = (S,jV,g) b e a s e a r c h l a n d s c a p e a n d GN = ( £ , A f ) b e t he c o r r e s p o n d i n g s e a r c h g r a p h . G i v e n a p l a t e a u P , t h e diameter o f P i s s i m p l y t h e d i a m e t e r o f t h e c o r r e s p o n d i n g s u b g r a p h o f GN, i.e. t h e m a x i m a l g r a p h g e o d e s i c o f G' = ( P , N p ) w h e r e Np(sj,S2) <^ > N(si,S2) A s i , S 2 G P . • Definition 3.7. (Branching Factor) L e t L = (S,J\f,g) b e a s e a r c h l a n d s c a p e , a n d GN = {S,j\f) b e t h e c o r r e s p o n d i n g s e a r c h g r a p h . G i v e n a p l a t e a u P , t h e plateau branching factor o f a s t a t e s i n P i s d e f i n e d a s t h e f r a c t i o n o f n e i g h -b o u r s o f s a l s o i n P : , . r, , >. #{s' G Plateau(s) I JV(S,S')} BranchingFactor(s) := # { s , , e 5 W s > 8 , , ) } S i m i l a r l y , w e d e f i n e t h e branching factor of plateau P a s t h e a v e r a g e o f BranchingFactor(s) o v e r a l l s t a t e s s G P . • T h e b r a n c h i n g f a c t o r a n d d i a m e t e r o f a p l a t e a u g i v e a s e n s e o f h o w t h e p l a t e a u i s a c t u a l l y s t r u c t u r e d . P l a t e a u s w i t h a h i g h b r a n c h i n g f a c t o r a n d l o w d i a m e t e r w o u l d b e " s q u e e z e d " i n t o a s m a l l r e g i o n o f t he s e a r c h s p a c e . S u c h a p l a t e a u w o u l d b e h i g h l y c o n n e c t e d ( i t w o u l d b e f a i r l y e a s y t o m a n e u v e r a r o u n d Chapter 3. Plateau Characteristics 42 the plateau), and we might expect that most of the propositional variables wou ld have a constant value across the entire plateau. Conversely, plateaus wi th low branching factor and high diameter would be more "brittle" and would snake their way throughout a larger portion of the search space. Definition 3.8. (Exit Distance) Given an open plateau P, the plateau exit distance of a state s e P i s simply the length of the shortest path in P from s to an exit state. We define the exit distance of plateau P as the average plateau exit distance of all states SEP. • Exit distance is a very interesting measure — it helps to quantify the intuition that even open plateaus may be difficult to escape from. If there are few exits distributed in a large plateau, they could be difficult for an SLS algorithm to find — especially considering that there is no heuristic guidance from the objective function on a plateau. Orthogonally, if there are many exits but they are all clustered together and not well connected wi th the rest of the plateau, it may still be difficult to reach an exit. With the goal of aiding in the conceptualisation of internal plateau structure, Figures 3.8, 3.9 and 3.10 show sample plateaus from three very different instances. Each node in the graphs represents a single state. Neighbouring states are connected by edges. Exit states are rendered i n green diamonds, while non-exit states are rendered in red octagons (note that all three of the plateaus are open). The graphs were generated by the excellent graph layout program yEd,1 and laid out using the "smart organic" method, which is based on the force directed layout paradigm. The plateaus were chosen so that they each contain roughly 1000 states; small enough to study the details within the plateaus, but containing enough states to ensure that large scale structures are observable. The plateaus are quite fascinating and beautiful. The differences between plateaus from different in-stances, though not surprising, are quite dramatic. After looking at many samples of plateaus from in-stances of various types, it becomes apparent that plateaus belonging to a given instance tend to share many features. To borrow a colourful term from Douglas Hofstadter, plateaus from the same family of instances seem to share the same "spirit" [Hof85]. To illustrate this point, first consider the plateau shown in Figure 3.8. This is P^mx (i.e. the largest open plateau region at level 3) from the instance in the uflO test set wi th median Isc. The plateau has 1222 states, 1047 of which are exits. The spirit of this plateau could be described as disarray — which is not surprising considering that the plateau comes from a randomly generated instance. It is interesting to note that the branching factor of the plateau is fairly low (approximately 0.185). A s we w i l l discuss shortly, all of the plateaus we studied have low branching factors. Also interesting is the distribution of non-exit states in the plateau. These states are distributed throughout the plateau, but they also appear in large clusters. 1yEd is freely available for download at www. yworks . com. Figure 3.8: Pmax from the instance in the uflO test set with median Isc. Chapter 3. Plateau Characteristics 44 t 4 Figure 3.9: An example level 3 plateau from the ais6 instance. Because there are so many exits, the exit distance of this plateau is quite low — but there do exist states having plateau exit distance of 6. Finally, we note that the diameter of the plateau is actually larger than the number of variables. We now turn our attention to the plateau shown in Figure 3.9, which has a very different spirit. This plateau is from an All-Interval-Series instance, and has level 3. It was chosen arbitrarily from all plateaus having approximately 1000 states — it contains 848 states, 800 of which are exits. This plateau is very beautiful — in fact, we found all of the large plateaus from the ais instance to be quite beautiful. The branching factor of the plateau is very low (approximately 0.04), and the plateau is perfectly symmetric.2 Again, there are a large number of exits from this plateau resulting in a small exit distance, and the diameter is larger than the number of variables. The last example plateau is shown in Figure 3.10. This level 5 plateau is from a highly structured parity learning instance — and this inherent structure is dramatically manifested in the plateau's spirit. Plateaus from the parity test set are characterised by large, very regular grids. The branching factor of this plateau is again quite low (approximately 0.09), and the plateau is comprised of multiple symmetric, well connected regions which are loosely coupled by a small number of edges. The plateau is open, but unlike the other sample plateaus, the fraction of exits is quite low (235 of the 969 states are exits), making this plateau a 2Note that it is not true that all plateaus in ais instances share this symmetry — many do, and all low level plateaus seem to have at least some symmetry, but they are not all as perfect as the example shown in Figure 3.9. Chapter 3. Plateau Characteristics 45 Figure 3.10: A n example level 5 plateau from the p a r i t y 8 instance. potential obstacle to local search algorithms. In fact, many of the low level plateaus in this instance look similar to this one, which helps to explain why these types of instances are so difficult for SLS algorithms. In contrast wi th the other example plateaus, the diameter of this plateau is less than the number of vari-ables. Also note that in this instance, the exits appear in tightly connected clusters rather than uniformly distributed throughout the plateau — this makes it significantly more difficult to find an exit from these plateaus. We should note now that these plateaus are somewhat atypical. A s we have seen in the previous sec-tions, the majority of plateaus are smaller than these. Nonetheless, there are many large plateaus - and as we w i l l see in the next chapter, the larger plateaus often dominate the space actually encountered by SLS algorithms - and studying these larger plateaus has illustrated how dramatically different the search spaces of different instance types can be. The remainder of this section studies the distributions of these properties over multiple plateaus from the same instances as wel l as over distributions of instances. In the following, the data for the UF-3-SAT test sets was collected in the same manner as in Section 3.2. We similarly sample plateaus for the structured instances, but since the structured test suites are composed of single instances, we sample 1000 plateaus at Chapter 3. Plateau Characteristics 4 6 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 branching factor branching factor branching factor Figure 3.11: Distribution of branching factors for the UF-3-SAT test sets. Left to Right: uf20, uf50, uflOO. level 0 1 2 3 4 5 Instance * <?x/x ^ x o~x/x x crx/x X crx/x X <7X/X x crx/x anom medium ais6 flat20-easy flat20-med flat20-hard par8-l-c 0 (0) 0 (0) . 0 (0) 0.209 (0.23) 0.069 (0.22) 0.136 (0) 0 (0) 0.007 (1.65) 0.002 (2.30) 0.006 (1.70) 0.245 (0.07) 0.120 (0.04) 0.178 (0.06) 0.023 (0) 0.019 (0.80) 0.009 (0.87) 0.031 (0.45) 0.257 (0.07) 0.167 (0.04) 0.232 (0.04) 0.043 (0.30) 0.025 (0.81) 0.012 (0.72) 0.045 (0.41) 0.276 (0.08) 0.205 (0.05) 0.272 (0.06) 0.056 (0.26) 0.036 (0.58) 0.019 (0.48) 0.058 (0.32) 0.298 (0.09) 0.240 (0.06) 0.310 (0.06) 0.072 (0.25) 0.053 (0.47) 0.025 (0.44) 0.064 (0.36) 0.321 (0.09) 0.272 (0.06) 0.345 (0.07) 0.086 (0.25) Table 3.12: Summary statistics of the distribution of branching factors for the structured instances. For each instance and level, we should the mean and variation coefficient of the distribution. each level for each structured instance. 3.5.1 Branching Factor Figure 3.11 shows the distribution of branching factors for plateaus sampled from the uf20, uf50, and uflOO test sets. The plateaus were sampled by using the same sampled states from Section 3.2. Starting from each sampled state, the plateau was expanded and the plateau branching factor was measured for each state, and then averaged to give the branching factor of each plateau. The entire distributions of branching factors are shown for each level in the figure. The branching factors tend to be lower for lower level plateau regions; this is likely a consequence of the smaller plateau size at lower levels (it should be obvious that the branching factor w i l l increase monoton-ically wi th plateau size, all else being equal). There is also more variance in the distribution of branching factors at lower levels; again, this is likely a consequence of the increased variance in the distribution of plateau sizes at lower levels. The branching factor also tends to decrease wi th the number of variables (an exception is the distribution for level 0; the average of the level 0 distribution increases slightly with the number of variables). Considering that the branching factors are normalised by the number of neigh-bouring states (i.e. the number of variables for SAT), this indicates that the rate of growth of the average branching factor is slower than linear in the number of variables. Chapter 3. Plateau Characteristics 47 Figure 3.12: Distribution of lower bound on plateau diameter for the uflO test set. Left: A l l plateaus, Centre: open plateaus, Right: closed plateaus. level 0 1 2 3 4 5 Instance X max X Cx/S max X crx/x max X erx/x max X uxlx max X crx/x max anom 0 (0) 0 0.009 (1.75) 0.04 0.022 (0.88) 0.06 0.033 (1.04) 0.12 0.050 (0.75) 0.19 0.089 (0.64) 0.27 medium 0 (0) 0 0.002 (2.36) 0.02 0.011 (0.95) 0.04 0.016 (0.85) 0.06 0.027 (0.61) 0.09 0.038 (0.56) 0.11 ais6 0 (0) 0 0.008 (1.75) 0.03 0.264 (0.98) 1.30 0.764 (1.05) 3.20 0.521 (0.49) 1.97 0.403 (0.68) 2.95 flat20-easy 0.619 (0.35) 1.02 0.264 (0.18) 0.40 0.207 (0.13) 0.33 0.178 (0.10) 0.25 0.160 (0.10) 0.23 0.148 (0.10) 0.27 flat20-med 0.161 (0.44) 0.33 1.315 (0.08) 1.55 0.354 (0.09) 0.53 0.236 (0.20) 1.55 0.187 (0.16) 0.97 0.164 (0.10) 0.33 flat20-hard 0.189 (0.10) 0.22 0.930 (0.28) 1.23 0.263 (0.12) 0.38 0.183 (0.09) 0.25 0.153 (0.09) 0.27 0.139 (0.08) 0.22 par8-l-c 0 (0) 0 0.039 (0.20) 0.05 0.202 (0.60) 0.36 0.129 (0.95) 0.66 0.198 (0.69) 0.66 0.448 (0.91) 1.45 Table 3.13: Summary statistics of the distribution of lower bounds on plateau diameter for the structured instances. For each instance and level, we show the mean, variation coefficient, and max of the distribution. Table 3.12 shows summary statistics of the distribution of branching factors for the structured test suites. Because the instances are so different, there is no clear picture emerging from the data, though we can make some observations. A s we observed for the UF-3-SAT test sets, the branching factor increases wi th the level for the structured instances (again, likely a consequence of the increasing plateau size). We also note that the branching factors are much lower than observed for the random instances, with the exception of the graph colouring instances (recall that the search spaces of these instances consists of a single very large plateau). The branching factor of zero at level zero for all of the instances except the graph colouring instances is due to the fact that these instances have a single solution — and thus no neighbouring states. Chapter 3. Plateau Characteristics 48 3.5.2 Diameter Figure 3.12 shows the distribution of a lower bound on the plateau diameter for the 20 variable UF-3-SAT test sets. This data is measured using the same sampled states from Section 3.2. From each sampled state s, Plateau(s) is searched i n a breadth-first manner, and the maximal length of the shortest path from s to any other state in Plateau(s) is recorded. Because this value is only calculated for one starting state, this value may not equal the true plateau diameter, but it is clearly a lower bound of the true diameter. Additionally, because a maximum of 10 6 states are explored, this bound may not be tight for the larger test sets (this is because the plateaus encountered are very large, and the breadth-first search restricts the explored states to those wi th short path lengths from the starting state s), so we present data only for the w/20 test set. A s we can see from the figure, there are a large number of cases in which the diameter is larger than the number of variables. This is interesting because it means that many of the plateaus are not contained in small sub-regions of the search space in which many of the variables have constant value for all states in the plateau. The larger plateaus at higher levels really are "interlaced" throughout the search space. Interestingly, the modes of the distributions at higher levels (level 3 and above) are exactly equal to the number of variables. Not surprisingly, the smaller closed plateaus have much smaller diameters as well . We also study the lower bound on plateau diameter for the structured instances. Table 3.13 shows the mean, variation coefficient, and maximum of the sampled distributions. Because the number of variables is different between each of the structured instances, we show the lower bound on plateau diameter nor-malised by the number of variables. Thus, values greater than one indicate that the diameter is at least as large as the number of variables. Not surprisingly, the diameter data essentially reflects the plateau sizes (c.f. Tables 3.9, 3.10, and 3.11). The instances that had large plateaus, namely the all-interval series and flat graph colouring instances, d id have plateaus whose diameter was larger than the number of variables. Recall that we expand a maximum of one mil l ion search states per plateau. This bound was exceeded for ais6 at level 4, f l a t 2 0 - e a s y at level 1, f l a t 2 0-med at level 2, and f l a t 2 0 - h a r d at level 1. Because the plateaus are so much larger than the cutoff, and we perform a breadth-first search, the measured lower bounds on the plateaus cannot be expected to be tight for the previously mentioned instances at higher levels, and thus we cannot extract any sense of the scaling behaviour of the diameter wi th level for these instances. 3.5.3 L M I N State Fraction We now turn to a very interesting measure — the fraction of L M I N states in a given plateau. This value is interesting because even though a plateau may be open (i.e. it may have an exit), if most of the states are non-exit states then it may be very difficult for an SLS algorithm to find the exit state. In particular, Chapter 3. Plateau Characteristics 49 Figure 3.13: Distribution of the fraction of L M I N states for the UF-3-SAT test sets. Left to Right: uf20, uf50, uflOO. greedy algorithms that do not al low non-improving moves could become temporarily trapped in these open plateaus. Note that we only measure these values for open plateaus, since the fraction is one for closed plateaus, by definition. Furthermore, since every open plateau consists of L M I N , possibly IPLAT, and exit states — and since IPLAT states are exceedingly rare [Hoo98] — there is a (near) perfect inverse relationship between the fraction of L M I N states and the exit density of a plateau. Figure 3.13 shows the distribution of L M I N state fractions for the UF-3-SAT test sets. For the smallest set, w/20, we see that at higher levels there are very few L M I N states in the plateaus meaning that most (or in some cases all) of the states are exit states — at high levels, it is easy to improve on the objective function. A s the level decreases, we observe a monotonic increase in the fraction of L M I N states until at level one the range of the distribution essentially covers the interval (0,1). A t level one, the mean L M I N state fraction is 0.58 — many of the plateaus have a high proportion of non-exit states. The situation becomes even more interesting for the uf50 and uflOO test sets. A s the number of variables increases, the L M I N state distributions shift toward one. Thus, as the problem size grows, even the open plateaus at low levels are made up of mostly of L M I N states. For the largest test set, uflOO, the mean fraction of L M I N states at level five is 0.41 and this increases to 0.77 at level one, with the mode of the distribution approaching one. It should not come as a surprise that the fraction is so high, or that it increases with problem size given the (relative) difficulty of these instances. Table 3.14 shows summary statistics from the corresponding distribution for the structured instances. We observe the same general trends in the structured instances as we do for the UF-3-SAT test sets. The fraction of L M I N states is higher at low levels, and tends to decrease with the level. One exception is the parity learning instance, for which the ratio stays relatively high even at higher levels. Interestingly, the rate of decrease for the planning (and to some extent the parity learning) instances is lower than for the all-Chapter 3. Plateau Characteristics 50 level 1 2 3 4 5 Instance x crx/x x. <ixlx x. crx/x X. (Tx/X x ax/x anom medium ais6 flat20-easy flat20-med flat20-hard par8-l-c 0.613 (0.13) 0.750 (0) H 0.403 (0.43) 0.681 (0.02) 0.896 (<0.01) 0.750 (0) 0.587 (0.20) 0.667 (0.25) 0.781 (0.20) 0.235 (0.63) 0.464 (0.11) 0.695 (0.13) 0.908 (0.17) 0.458 (0.24) 0.573 (0.30) 0.267 (0.78) 0.151 (0.81) 0.266 (0.26) 0.442 (0.28) 0.666 (0.26) 0.407 (0.41) 0.616 (0.31) 0.021 (0.64) 0.077 (1.09) 0.141 (0.48) 0.243 (0.50) 0.610 (0.28) 0A72 (0.47) 0.580 (0.37) 0.001 (0.35) 0.035 (1.34) 0.056 (0.74) 0.099 (0.83) 0.723 (0.25) Table 3.14: Summary statistics of the distribution of the fraction of L M I N states i n open plateaus for the structured instances. For each instance and level, we show the mean and variation coefficient of the distribution. The entries are marked ' - ' for level one of the a i s 6 instance because there are no open plateaus at level one for this instance. interval-series and flat graph colouring instances (and also seems to be slower than for the UF-3-SAT test sets). A t higher levels, the all-interval-series and flat graph colouring instances have relatively few L M I N states, indicating that it is easy to get to lower levels in these instances. In addition to these observations, we note also that all-interval-series and graph colouring instances seem to be much easier for SLS algorithms than the planning or parity instances (this is a very general statement, but it can be backed up with solid empirical evidence given for example in [Hoo98, HS04]). Thus, it seems reasonable to conjecture that there is a direct correlation between high L M I N state fractions at low levels and the intrinsic hardness of structured instances. Furthermore, if we consider only the flat graph colouring instances, we make an interesting observation that seems to corroborate our previous conjecture. A t each and every level, the L M I N state fraction increases monotonically as we go from the easiest to the hardest instance. Intuitively, this must be the case because, as we discussed in Section 3.4.1, the search spaces of the flat graph colouring instances are dominated by a single large open plateau. If it were possible to easily find an exit from each of these large open plateaus, then the graph colouring instances wou ld be trivial to solve. Since they are not trivial, there must be other sources of hardness such as the one we just discussed. 3.5.4 Exit Distance Figure 3.14 shows the distribution of exit distances for the UF-3-SAT test sets. This data is very interesting, because it gives an indication of the typical number of steps an SLS algorithm must take on an open plateau before reaching an exit. Obviously, the higher the exit distance, the more difficulties a plateau w i l l pose for a local search algorithm. This data was also measured using the same sampled states from Section 3.2. Starting from a sampled state s, Plateau(s) was explored using breadth-first search, and the distance to the closest exit was recorded. For obvious reasons, the distribution is taken only over open plateaus. Chapter 3. Plateau Characteristics 51 Figure 3.14: Distribution of exit distances for the UF-3-SAT test sets. Left to Right: uf20, uf50, uflOO. II level 1 2 3 4 5 Instance x Ojc/i x <rx/x x crx/x x <rx/x i crx/x anom medium ais6 flat20-easy flat20-med flat20-hard par8-l-c 0.777 (0.54) 0.811 (0.48) (-) 1.551 (0.90) 3.168 (0.68) 11.476 (0.64) 2.127 (0.42) 0.910 (0.70) 0.986 (0.72) 2.917 (1.01) 0.709 (1.11) 1.250 (0.75) 2.087 (0.69) 9.003 (0.58) 0.771 (0.81) 1.036 (0.76) 0.430 (1.44) 0.392 (1.42) 0.661 (0.98) 0.879 (0.87) 2.071 (0.96) 0.964 (0.82) 1.383 (0.86) 0.045 (4.61) 0.187 (2.11) 0.369 (1.37) 0.468 (1.20) 2.123 (1.03) 1.385 (0.96) 1.392 (0.90) 0.002 (25.10) 0.086 (3.29) 0.148 (2.40) 0.179 (2.16) 4.411 (1.13) Table 3.15: Summary statistics of the distribution of exit distances in open plateaus for the structured in-stances. For each instance and level, we show the mean and variation coefficient of the distri-bution. The entries are marked ' - ' for level one of the a is6 instance because there are no open plateaus at level one for this instance. The results are very interesting. At higher levels, even though the size of the sampled plateau is typically very large, the exit distances are very small - meaning that they should be easy to find. As the level decreases, the expected exit distance increases, indicating that even open plateau regions at lower levels may pose a problem for SLS algorithms. We do note, however, that the exit distance distributions at all levels are skewed toward shorter distances and the expected exit distances are small, even at lower levels. Thus, even though the plateaus encountered by GSAT are larger at higher levels, they are typically open with very small exit distances. The troublesome plateaus are those smaller plateaus at lower levels that are either closed, or have high exit distances. Table 3.15 shows summary statistics for the distribution of exit distances measured for the structured instances. There are some very interesting differences between these results and those for the distributions of LMIN state fractions in the previous section. The expected exit distance increases with the level for the planning instances, even though the LMIN state ratio decreases. The mean exit distances for the planning Chapter 3. Plateau Characteristics 52 instances were very low for all levels, indicating that, for these instances, it must be closed plateau regions that are the major source of hardness since open plateaus seem quite easy to escape from. The all-interval-series instance is quite interesting. There are no open plateaus at level one, and the exit distance at level two moderately high (2.917). Note however, that the exits from level two must lead to solutions. A t higher levels, the exit distance is negligible. When we combine this information with the fact that the L M I N state ratio is very low at higher levels, increasing to 0.781 at level two, it should not be surprising that this instance is quite easy for SLS algorithms. The data from the parity learning instance is also quite interesting. A l o n g wi th the high L M I N state ratio mentioned in the previous section, the p a r 8 -1 - c instance also has very high exit distances, even at higher levels. Thus, even though the parity learning instance has a large fraction of open plateaus at each level, it is very difficult to find an exit from many of the open plateaus. Figure 3.10 gives a good visual indication of the internal plateau structure underlying these observations. Finally we consider the flat graph colouring instances. The mean exit distance decreases with level for these instances, as the L M I N state fraction did . Also , corresponding to our observations for the L M I N state fraction, the mean exit distance monotonically increases from the easiest to the hardest instance at each level, demonstrating nicely which search space features make instances from the same distribution more difficult to solve than syntactically similar instances. 3.6 Summary In this chapter, we have introduced the basic search space features that we study in this thesis, in particular the plateau. We studied the plateau characteristics of the UF-3-SAT instance distribution, attempting to replicate and extend the results of Frank et al. [FCS97] by studying the plateaus encountered by the GSAT algorithm. We studied plateau properties such as size of plateaus, and the fraction of closed plateaus. We observed that the average fraction of closed plateaus decreases rapidly wi th the level, and that open plateaus tend to be much larger than closed plateaus, especially at higher levels. Our results differ from those of [FCS97] in that we report much larger plateau sizes. We have ensured that our results are correct, and provide some possible explanations for the discrepancies. Next, we studied the plateau characteristics of individual UF-3-SAT instances by exhaustively exploring the low level regions of the search space. This allows us to study the number of plateaus at each level, and we note that the number of plateaus increases rapidly with the level and that the majority of high level plateaus are open. We also studied the size of plateaus, observing that the size of closed plateaus decreases with level — thus, closed plateaus are only prominent at very low levels in the search space. Interestingly, we observe that while there are many open plateaus at high levels, the majority of these plateaus are very Chapter 3. Plateau Characteristics 53 small (and consist entirely of exits), while one single large plateau contains most of the states at higher levels. This "one-big-plateau" phenomenon is very interesting because it indicates that the entire low level search space is connected, and thus an algorithm never has to go to very high levels to reach any point in the low level search space. We performed a similar analysis on individual structured instances, wi th similar results. We observe that the number of plateaus increases (exponentially) wi th the level, and that the number of open as wel l as closed plateaus increases with the level (recall that for the random test sets, the number of closed plateaus decreases wi th level). When studying the fraction of closed plateaus at each level, we observed that, while the fraction of closed plateaus tended to decrease with level, the harder structured instances tended to have a higher proportion of closed plateaus at each level. We also noted that closed plateaus were typically quite small, and that they were largest for the hardest instance. We observed that the size of open plateaus increases wi th level, and that the median sizes were fairly small. The only structured instances to exhibit the "one-big-plateau" phenomenon were the graph colouring instances. Finally, we studied the internal structure of plateau regions. We observed very low branching factors for all of the instances studied, indicating that plateaus are rather "brittle" (this has been previously reported in [Hoo98]), though the branching factors increase with level. Next, we study the diameter of plateaus and observe that there are many cases, especially at higher levels, where the diameter is greater than the number of variables. In general, the diameter seemed to be proportional to the plateau size. We next studied the fraction of L M I N states from a plateau, and found that at high levels there were very few L M I N states (indicating that it is very easy to find an exit to a lower level state from high level plateaus), and that the L M I N state fraction decreases wi th level, though there were typically a substantial number of exits from plateaus at all levels. Lastly, we studied the exit distances of plateaus, and observed that at high levels the exit distance was typically very low, while at low levels the exit distances were occasionally very high. Chapter 4. Plateau Connectivity 54 Chapter 4 Plateau Connectivity The previous chapter studies the distribution of plateaus in the search space and properties of the plateaus, which is crucial to gain an understanding of what these components of the search landscape look like. This chapter takes an alternative direction, and focuses rather on the connections between plateau regions, and how the model consisting of these interconnected plateaus accurately characterises the search space of the instances that we study. We w i l l demonstrate how understanding properties of this model of the search space can aid in the understanding of algorithm behaviour. This understanding provides insight into how existing local search algorithms can be improved and new algorithms can be developed. This chapter is structured as follows. We first formalise the definitions of concepts that are central to this chapter, and motivate the research. Then we begin our study of Plateau Connectivity Graphs by undertaking an empirical study of their basic properties. Next, we demonstrate how Plateau Connectivity Graphs can be used to approximate the likelihood that an iterated improvement algorithm w i l l reach a solution (rather than get caught in a local minimum). Following this, we introduce a novel method for classifying search space traps, and empirically .demonstrate how these traps adversely affect local search algorithms. Finally, we summarise and discuss the main results from the chapter. 4.1 Definitions and Motivation We begin wi th the central definition of the chapter; that of a Plateau Connectivity Graph. A Plateau Con-nectivity Graph (PCG) is a directed graph in which each node represents a plateau, and there are edges between nodes if and only if the corresponding plateaus are connected. Two plateaus are considered to be connected if they each contain at least one state that neighbours a state in the other plateau. The following definition formalises this concept: Definition 4.1. (Plateau Connectivity Graph) A Plateau Connectivity Graph (PCG) is a directed graph G = (V, E). Each node v G V represents an entire plateau in the search landscape L = (<S, Af, g). There is an edge (u, v) G E from u to v if and only if there exists at least one pair of states su G u and sv G v such that level(su) > level(sv) Chapter 4. Plateau Connectivity 55 a n d s „ a n d sv a r e n e i g h b o u r s u n d e r t h e g i v e n n e i g h b o u r h o o d r e l a t i o n Af. S u c h a p a i r o f s t a t e s i s c a l l e d a n exit-target pair f r o m u t o v. A Partial PCG i s a P C G i n w h i c h n o t a l l o f t h e p l a t e a u s ( o r c o r r e s p o n d i n g e d g e s ) i n t h e s e a r c h s p a c e a r e r e p r e s e n t e d . U n l e s s o t h e r w i s e n o t e d , w e w i l l a s s u m e t h r o u g h o u t t h i s t h e s i s t h a t w e a r e r e f e r r i n g t o p a r t i a l P C G s . • I n t u i t i v e l y , P C G s f o r m a f a i r l y r e p r e s e n t a t i v e , b u t s t i l l s m a l l a n d s i m p l e , m o d e l o f t h e f e a t u r e s o f t h e s e a r c h s p a c e t h a t h a v e a l a r g e i m p a c t o n S L S a l g o r i t h m b e h a v i o u r . F o r e x a m p l e , w e c a n c o n s i d e r a s i m p l e l o c a l s e a r c h a l g o r i t h m w h i c h a l w a y s a l w a y s m a k e s e i t h e r s i d e w a y s o r d o w n w a r d s m o v e s , a n d c h o o s e s r a n d o m l y b e t w e e n a l l e x i t t a r g e t s . S u c h a n a l g o r i t h m w o u l d a l w a y s e i t h e r m o v e w i t h i n a p l a t e a u , o r f o l l o w o n e o f t h e P C G e d g e s — t h u s , t h e a l g o r i t h m w o u l d w a l k s t r i c t l y a l o n g t h e P C G e d g e s . T h e i n t u i t i o n i s s i m i l a r f o r m o r e c o m p l i c a t e d a l g o r i t h m s — b e c a u s e m o s t S L S a l g o r i t h m s f o r S A T a r e g u i d e d b y the s a m e o b j e c t i v e f u n c t i o n , w e c o n j e c t u r e t h a t P C G s w i l l c a p t u r e t h e i m p o r t a n t a s p e c t s o f t h e s e a r c h s p a c e , a n d t h u s w i l l a d e q u a t e l y m o d e l t h e b e h a v i o u r o f e v e n c o m p l e x a l g o r i t h m s . I n o r d e r t o b e t t e r u n d e r s t a n d h o w P C G s a c c u r a t e l y m o d e l t h e s e a r c h s p a c e , w e m a k e d i s t i n c t i o n s b e -t w e e n d i f f e r e n t t y p e s o f P C G n o d e s , d e p e n d i n g o n the n u m b e r o f i n c o m i n g a n d o u t g o i n g e d g e s . T h e f o l l o w i n g d e f i n i t i o n f o r m a l i s e s o u r c l a s s i f i c a t i o n . Defini t ion 4.2. ( P C G Node Types) G i v e n a ( p a r t i a l ) P C G G = (V, £ ) , w e c a n d e f i n e t h e f o l l o w i n g se ts : R = {veV\^3u:(u,v)i=E} (4.1) I = {u e V\->3v : (u,v) G E} (4.2) M = {ue L\level(u) > 0 } C L (4.3) S = {ue L\level(u) = 0 } C L (4.4) R i s t h e se t o f a l l r o o t s o f t h e g r a p h ; t h a t i s , t h e se t o f n o d e s w i t h n o i n c o m i n g e d g e s . L i s t h e se t o f a l l l e a v e s o f t h e g r a p h ; t h e se t o f n o d e s w i t h n o o u t g o i n g e d g e s . O b v i o u s l y , e l e m e n t s o f L c o r r e s p o n d t o c l o s e d p l a t e a u s . M i s t h e s e t o f c l o s e d p l a t e a u s t h a t a r e n o t s o l u t i o n p l a t e a u s ( c l e a r l y M C L , a n d M = L i f a n d o n l y i f t h e r e a r e s o l u t i o n s ) . S i s t h e s e t o f s o l u t i o n p l a t e a u s ( c l e a r l y S C L , a n d S = L i f a n d o n l y i f t h e r e a r e n o c l o s e d p l a t e a u s t h a t a r e n o t s o l u t i o n s ) . • Chapter 4. Plateau Connectivity 56 N o t e t h a t t h e c u r r e n t d e f i n i t i o n o f P C G s c a n o n l y p o r t r a y w h e t h e r t w o p l a t e a u s a r e c o n n e c t e d ( m e a n i n g t h a t i t i s p o s s i b l e f o r a l o c a l s e a r c h a l g o r i t h m t o m o v e d i r e c t l y b e t w e e n t h e t w o p l a t e a u s ) . I n t u i t i v e l y , h o w e v e r , w e w o u l d e x p e c t t h a t c e r t a i n c o n n e c t i o n s m a y b e m o r e i m p o r t a n t t h a n o t h e r s . F o r e x a m p l e , c o n s i d e r a p l a t e a u P ] h a v i n g a l a r g e n u m b e r o f e x i t s l e a d i n g t o a n o t h e r p l a t e a u Pi, b u t o n l y a f e w l e a d i n g t o a t h i r d p l a t e a u P3. T h e c o n n e c t i o n b e t w e e n P\ a n d Pi s e e m s s o m e h o w m o r e i m p o r t a n t , s i n c e i n p r a c t i c e w e w o u l d e x p e c t a n S L S a l g o r i t h m to f o l l o w t ha t e d g e m o r e t h a n t h e o t h e r . T o c a p t u r e t h i s n o t i o n o f r e l a t i v e i m p o r t a n c e , w e i n t r o d u c e e d g e w e i g h t s f o r P C G s . T h e f o l l o w i n g d e f i n i t i o n f o r m a l i s e s t h i s c o n c e p t . Definition 4.3. (Weighted PCG) G i v e n a s e a r c h l a n d s c a p e L = (S,M ,g), a weighted PCG i s a P C G a u g m e n t e d w i t h a f u n c t i o n w : E r—> [0,1] h a v i n g t h e f o l l o w i n g p r o p e r t y : V w e V : w(u,v) = l (4.5) {vev\{u,v)eE} T h a t i s , t h e s u m o f t h e w e i g h t s o f a l l o u t g o i n g e d g e s m u s t s u m t o 1 f o r e v e r y P C G n o d e . T h u s , a w e i g h t e d P C G G i s r e p r e s e n t e d b y a t r i p l e : G = (V, E, w), w h e r e V a n d £ a r e t h e v e r t e x a n d e d g e se t s o f t h e c o r r e s p o n d i n g u n w e i g h t e d P C G , a n d w i s t h e w e i g h t f u n c t i o n . • T h e r e a r e i n d e f i n i t e l y m a n y w a y s to c o n s t r u c t t h e f u n c t i o n w, b u t t h e r e a r e t w o m a j o r f a c t o r s t h a t w s h o u l d a t t e m p t t o c a p t u r e . F i r s t l y , a s h i n t e d a t a b o v e , w(u,v) s h o u l d b e p r o p o r t i o n a l to t he n u m b e r o f n e i g h b o u r i n g s t a t e s i n u a n d v. S e c o n d l y , w s h o u l d r e f l e c t t h e b e h a v i o u r o f t h e a l g o r i t h m i t i s a t t e m p t i n g t o m o d e l . F o r e x a m p l e , i f w e w o u l d l i k e to m o d e l a b e s t i m p r o v e m e n t a l g o r i t h m w i t h w(u, v), t h e n f o r e v e r y e x i t s t a t e i n u, o n l y t h e t a r g e t s t a t e h a v i n g t he b e s t s c o r e s h o u l d c o n t r i b u t e to t he w e i g h t ( r e f l e c t i n g t h e f a c t t h a t t h e o t h e r t a r g e t s w o u l d n e v e r b e s e l e c t e d b y t h e a l g o r i t h m ) . N o t e t h a t w o n l y a p p r o x i m a t e s t h e e x p e c t e d a l g o r i t h m b e h a v i o u r o n the p l a t e a u — i n p a r t i c u l a r , w e a r e m a k i n g t h e s i m p l i f y i n g a s s u m p t i o n t h a t t h e S L S a l g o r i t h m b e i n g m o d e l l e d i s a b l e t o c h o o s e u n i f o r m l y a n d r a n d o m l y f r o m a l l e x i t s t a t e s o n t h e p l a t e a u , w h i c h i s o b v i o u s l y n o t t h e c a s e i n e x i s t i n g a l g o r i t h m s . 1 I n t h i s c h a p t e r , w e o p t f o r a f a i r l y g e n e r a l w e i g h t f u n c t i o n t o e n s u r e t h a t w e a r e n o t m o d e l l i n g f e a t u r e s t h a t a r e o v e r l y a l g o r i t h m s p e c i f i c . O u r w e i g h t f u n c t i o n b a s i c a l l y g i v e s e q u a l p r e c e d e n c e to e v e r y e x i t - t a r g e t p a i r f r o m e v e r y s o u r c e p l a t e a u . F o r m a l l y , w e f i r s t d e f i n e t h e s e t o f a l l e x i t - t a r g e t p a i r s b e t w e e n t w o p l a t e a u s u a n d v: Extuv = {(s UiSp) I Su euAsve vAAf(su,sv) Ag(su) > g(sv)} (4.6) 1The exit taken by an SLS algorithm from a given plateau depends on many factors, including the state at which the algorithm entered the plateau, possible clustering effects which could make some exits inaccessible from neighbours in the plateau, and even higher-order effects such as search history (which could affect the exits chosen by, for example, a tabu search algorithm). Chapter 4. Plateau Connectivity 57 Next, we define the set of all exit-target pairs having u as the target: Ext„ = [ J Extm (4-7) v€V Finally, using these sets we are able to define our weight function: —. . #Extuv , . W { U ' V ) = JEXI; ( 4 - 8 ) Note that w gives equal precedence to every exit-target pair leaving a plateau. This ensures that we are not making the function algorithm-specific by only including certain exit-target pairs, but it also ensures that stronger connections are reflected by higher weights. In practice, we have found that this weight function achieves a good compromise between simplicity, generality, and predictive power. N o w that we have fully defined PCGs, we w i l l aid the intuitive understanding of these structures through examples. Figures 4.1 and 4.2 show partial PCGs for an easy instance and the hardest instance from the uflO test set, respectively. A l l nodes are labelled x.y, where x is the level of the plateau and y is simply an identifier that is unique to the level. Open plateaus are rendered in white ovals, closed plateaus in red squares, and solution plateaus in green triangles. The edge labels are the revalues, and dashed edges represent those edges wi th weight less than 0.05. Edges with weight greater than 0.3 are rendered wi th bold lines. In order to clearly illustrate these graphs we had to omit some details. First, plateaus are only shown if they were encountered at least one time over the course of 1000 runs of G W S A T (the graphs remain unchanged if other algorithms, such as WalkSAT or N o v e l t y + are used). This results in only Pmax being shown at higher levels {cf. Section 3.3.4). Clearly this w i l l not affect the conclusions that we can draw from studying the PCGs, since if the small degenerate plateaus at higher levels are very rarely encountered then they cannot have a significant impact on the behaviour of the SLS algorithms. The second detail omitted is that edges with weights lower than 0.01 are not displayed. 2 This is simply because again we assume that very weak edges w i l l have little impact on SLS algorithm behaviour but they clutter up the graphs and make it difficult to see the more important structures. We now turn our attention to Figure 4.1, which represents an easy instance from the uflO test set. The main feature of note is that the search space appears to be structured essentially as a large funnel directing all search trajectories toward the solution. Given the structure of the P C G , it is not at al l surprising that the instance is easy. There are a fair number of closed plateaus, but, wi th the exception of the plateau labelled 1.1, none of them are strongly connected to higher-level plateaus — meaning that we would not expect an 2 A n exception to this rule is made if removing these edges would leave a node with no incoming edges; in this case the incoming edge with the highest weight (still less than 0.01) is left in the graph. These weak edges are rendered in dashed grey lines. Chapter 4. Plateau Connectivity Figure 4.1: Weighted, partial PCG for an easy uflO instance. Chapter 4. Plateau Connectivity 59 F i g u r e 4 .2 : W e i g h t e d , p a r t i a l P C G f o r t h e h a r d e s t uf20 i n s t a n c e . Chapter 4. Plateau Connectivity 6 0 S L S a l g o r i t h m t o b e a t t r a c t e d t o t h o s e c l o s e d p l a t e a u s . F i g u r e 4 . 2 , r e p r e s e n t i n g t h e h a r d e s t o f t h e uflO i n s t a n c e s , i s n o t a b l y d i f f e r e n t . W h i l e t h e P C G s e e m s s i m i l a r a t h i g h e r l e v e l s ( o n l y P m a x w a s e n c o u n t e r e d , a n d e a c h h i g h l e v e l p l a t e a u i s s t r o n g l y c o n n e c t e d to t h e l a r g e p l a t e a u d i r e c t l y b e l o w ) , t h e r e a r e s u b s t a n t i a l d i f f e r e n c e s a t l o w e r l e v e l s . F i r s t l y , t h e r e a r e c o n s i d e r a b l y m o r e l o w - l e v e l p l a t e a u s t h a t w e r e e n c o u n t e r e d d u r i n g the 1 0 0 0 r u n s , a n d m a n y o f t h e s e a r e e i t h e r c l o s e d , o r a r e w e l l - c o n n e c t e d to c l o s e d p l a t e a u s . S e c o n d l y , a n d m o r e i m p o r t a n t l y , i f w e f o l l o w t h e h i g h e s t - w e i g h t e d e d g e s d o w n f r o m t h e h i g h l e v e l p l a t e a u s , t h e m a j o r i t y o f t h e e d g e s l e a d t o t h e c l o s e d p l a t e a u 1 . 1 , n o t t o a s o l u t i o n . C l e a r l y t h i s i s w h e r e t h e d i f f i c u l t y o f t h i s i n s t a n c e c o m e s f r o m . A s s u m i n g t h a t t h e e d g e w e i g h t s a r e r e p r e s e n t a t i v e o f h o w a r e a l S L S a l g o r i t h m w o u l d b e h a v e o n t h i s i n s t a n c e , w e c a n s e e t h a t t h e v a s t m a j o r i t y o f t r a j e c t o r i e s l e a d t o p l a t e a u s 2 . 1 , 2 . 3 , a n d 2 . 4 . B o t h 2 . 1 a n d 2 . 3 a r e o n l y c o n n e c t e d t o t h e c l o s e d p l a t e a u 1 . 1 , a n d p l a t e a u 2 . 4 h a s a n e d g e w i t h w e i g h t 0 .84 l e a d i n g t o 1 . 1 . It i s i n t e r e s t i n g t o n o t e t h a t t h e s e t w o i n s t a n c e s h a v e a s i m i l a r n u m b e r o f c l o s e d p l a t e a u s , a n d t h e y b o t h o n l y h a v e o n e w e l l - c o n n e c t e d c l o s e d p l a t e a u , y e t t h e l o c a l s e a r c h c o s t o f t h e i n s t a n c e s i s d r a s t i c a l l y d i f f e r e n t . A s w e h a v e a n e c d o t a l l y e x p l a i n e d , t h e k e y d i f f e r e n c e l i e s i n t h e c o n n e c t i o n s b e t w e e n t h e v a r i o u s p l a t e a u s . T h e r e m a i n d e r o f t h i s c h a p t e r w i l l f u r t h e r d e v e l o p t h e s e i d e a s a n d e m p i r i c a l l y d e m o n s t r a t e t h a t h a r d i n s t a n c e s c o n s i s t e n t l y h a v e s u c h s e a r c h s p a c e s t r u c t u r e s . F i n a l l y , F i g u r e 4 .3 s h o w s t h e P C G f o r t h e s m a l l e s t s t r u c t u r e d i n s t a n c e , a n o m . A s w e c a n see , t h i s P C G is s u b s t a n t i a l l y m o r e c o m p l e x t h a n t h o s e o f t h e r a n d o m i n s t a n c e s . B e c a u s e o f t h i s c o m p l e x i t y , w e w e r e u n a b l e t o a d d e d g e w e i g h t s w i t h o u t c l u t t e r i n g u p t h e f i g u r e . I n s t e a d , t h e c o l o u r o f e v e r y e d g e i s d i r e c t l y p r o p o r t i o n a l t o t h e w e i g h t ; d a r k e r e d g e s i n d i c a t e h i g h e r e d g e w e i g h t s . T h e P C G i s b r o k e n i n t o 13 d i s j o i n t c o n n e c t e d c o m p o n e n t s , 3 s o m e c o n t a i n i n g m a n y p l a t e a u s s o m e o n l y t w o . T h i s i n s t a n c e h a s a s i n g l e m o d e l , a n d t h u s o n l y o n e s o l u t i o n p l a t e a u , l a b e l l e d 0 . 1 . T h e l a r g e s t c o n n e c t e d c o m p o n e n t a c t u a l l y c o n t a i n s t h e s o l u t i o n . A l l p a t h s i n a l l o f t h e o t h e r c o m p o n e n t s l e a d t o c l o s e d p l a t e a u r e g i o n s , i n d i c a t i n g t h a t t h e r e a r e l a r g e p o r t i o n s o f t h e s e a r c h s p a c e f r o m w h i c h i t i s i m p o s s i b l e t o r e a c h a s o l u t i o n e x c e p t b y f i r s t g o i n g t o h i g h e r l e v e l s . B e c a u s e t h e y a p p e a r t o b e s o a t t r a c t i v e a n d w e l l - c o n n e c t e d , w e w o u l d e x p e c t t h a t t h e c l o s e d p l a t e a u s l a b e l l e d 1 . 1 a n d 1 . 3 w o u l d b e m a j o r f a c t o r s c o n t r i b u t i n g to t h e d i f f i c u l t y o f t h i s i n s t a n c e . T h i s w i l l b e i n v e s t i g a t e d f u r t h e r i n l a t e r s e c t i o n s o f t h i s c h a p t e r . 4.2 P C G Properties I n t h i s s e c t i o n , w e s t u d y s o m e b a s i c g r a p h - t h e o r e t i c p r o p e r t i e s o f P C G s i n o r d e r t o p a i n t a n e m p i r i c a l p i c t u r e o f w h a t t y p i c a l P C G s l o o k l i k e . W e u s e t he s a m e tes t s u i t e s a s i n t h e p r e v i o u s c h a p t e r , a l l o w i n g u s to c o m p l e m e n t t h e e x i s t i n g r e s u l t s w . r . t . p l a t e a u p r o p e r t i e s w i t h t h e s e n e w h i g h e r - l e v e l P C G r e s u l t s . I n t h i s 3Note that this disjointness is an artifact resulting from the omission of plateaus from the partial PCG — if all encountered plateaus up to level 9 are included in the partial PCG then the resulting partial PCG is connected. Chapter 4. Plateau Connectivity 61 Figure 4.3: Weighted, partial P C G for the structured anom instance. This figure shows a single P C G , but the P C G consists of multiple disjoint components. The vertical ordering of the nodes within each component represents the relative levels the corresponding plateaus. Nodes representing plateaus at the same level are aligned vertically where possible. Chapter 4. Plateau Connectivity 62 level 1 2 3 4 5 Test Set x crx/x x. <rx/x x crx/x x. cxlx X LTX/X uf20 uf25 uf30 uf35 uf40 uf45 1.10 (0.16) 1.08 (0.12) 1.09 (0.17) 1.09 (0.14) 1.09 (0.11) 1.08 (0.08) 1.94 (0.17) 1.89 (0.16) 1.90 (0.13) 1.81 (0.15) 1.77 (0.14) 1.77. (0.14) 2.72 (0.13) 2.57 (0.11) 2.49 (0.10) 2.41 (0.11) 2.30 (0.10) 2.32 (0.10) 3.32 (0.11) 3.07 (0.09) 2.97 (0.09) 2.90 (0.09) 2.78 (0.08) 2.77' (0.08) 3.84 (0.10) 3.50 (0.08) 3.40 (0.07) 3.28 (0.08) . 3.16 (0.07) 3.15 (0.07) Table 4.1: Summary statistics of the distribution of the out-degree of P C G nodes from the UF-3-SAT in-stances. For every test set and level we show the mean and coefficient of variation of the distribu-tion (over all instances) of the average out-degree of all open plateau nodes (over all open plateau nodes in each instance). level 1 2 3 4 5 Instance X <Tx/x x crx/x x. crx/x x. ax/x x o-x/x anom medium flat20-easy flat20-med flat20-hard par 8-1 -c 1.000 (0) 1.000 (0) 1.644 (3.32) 2.256 (6.40) 1.263 (0.88) 1.000 (0) 1.250 (0.35) 1.061 (0.22) 2.744 (3.35) 1.913 (8.31) 2.579 (2.61) 1.067 (0.23) 1.607 (0.37) 1.537 (0.39) 2.656 (1.81) 2.301 (3.32) 1.794 (1.89) 2.889 (1.19) 2.216 (0.41) 1.886 (0.44) 2.970 (0.90) 3.471 (1.96) 2.942 (0.84) 2.022 (0.64) 2.696 (0.45) 2.307 (0.49) 4.104 (0.80) 4.194 (1.69) 3.770 (0.86) 2.168 (1.28) Table 4.2: Summary statistics of the distribution of the out-degree of P C G nodes from the structured in-stances. For every test set and level we show the mean and coefficient of variation of the distri-bution (over all open plateau nodes in the instance) of the out-degree of the open plateau nodes. section, we measure the distribution of out-degrees of P C G nodes corresponding to open plateaus. We also measure the distribution of average and maximum target depths for the same P C G nodes (these terms w i l l be defined shortly). The PCGs were constructed from the plateau data presented in Section 3.3. To generate the PCGs, we started wi th the highest level plateaus and iteratively calculated Extu for the source plateau, and Extuv between u and all low level plateaus. Finally, zv(u, v) was calculated according to Equation 4.8. 4.2.1 Out-Degree Given an P C G G = (V,E) and an open plateau u at level k, the out-degree is simply: #{v £ V | 3s„ £ u,sv £ v : N(su,sv) A level(u) > level(v)} (4.9) For each instance and level, we calculated the out-degree of each node and then averaged these out-degrees to give an average out-degree for the given instance and level. Table 4.1 shows summary statistics of the distribution of out-degrees of P C G nodes for the UF-3-SAT instances. There are some interesting trends present in the table. The out-degree tends to increase with level, which is not surprising considering that there are fewer plateaus at low levels. More interesting is the fact Chapter 4. Plateau Connectivity 63 that the average out-degrees tend to decrease with problem size. This trend is present at all levels (including level 1), and the relationship is monotonic with the single exception of level 3 of uf40 being slightly lower than level 3 of uf45. The trend is particularly puzzling considering that the number of plateaus increases with problem size. Also notable from the table is the very low coefficient of variation in the distributions — because this is the coefficient of variation of the average out degrees, it indicates that the average values are similar across all instances in each test set. Finally, we observe that the coefficient of variation decreases both wi th level and problem size. We also consider similar data for the structured instances, and show this in Table 4.2. Note that the data in this table have not been double-averaged, and so the variation reported is the variation of the distribu-tion over all open plateau nodes at each level. We see that the data is quantitatively similar to that for the structured instances. The out-degrees of the nodes tend to increase with the level. Interestingly, the magni-tude of the out-degrees for the anom, medium, and p a r 8 - l - c instances are lower than the corresponding values from the random test sets (with the exception of level 3 of the p a r 8 - l - c instance, which has a higher than expected average out-degree). Close examination of these instances reveals that when there are a large number of plateaus present at a given level, many of these are very small and only connected to a single lower level plateau — resulting in an out-degree of one for many of the plateaus. The data presented in this section has interesting implications for algorithm mobility. Because the out-degrees of many of the plateaus are quite low, especially at lower levels, it becomes difficult for local search algorithms to directly move around the low level search space. Especially considering that many plateaus are connected to only one lower level plateau, it may be necessary for SLS algorithms to occasionally "re-treat" to significantly higher levels of the search space in order to avoid being restricted to certain regions of the search space. This is illustrated quite dramatically in Figure 4.3 (the P C G for the anom instance), where we see large disjoint subgraphs in the P C G between which it is impossible for an SLS algorithm to move without moving to higher search space levels. 4.2.2 Target Depth Tables 4.3 and 4.4 shows statistics of the average and maximum target depth, respectively, for the UF-3-SAT test sets. Given a P C G edge (u,v), the target depth is simply the difference in level between the two plateaus. Target depth gives a sense of the expected improvement we expect to observe when exiting a plateau — high target depth values indicate that it is possible to descend multiple levels by following a single edge. Interestingly, there is quite a range i n the target depth values. Both the average and maximum target depths are (trivially) 1 at level 1, and increase with the level. The rate of increase appears to be linear with the level, but the maximum values are only 1.51 (w/20, level 5), and 2.12 [ufZO, level 5) for the average and Chapter 4. Plateau Connectivity 64 level 1 2 3 4 5 Test Set X ax/x X ax/x X <7X/X X ax/x X uf20 1.00 (0) 1.17 (0.10) 1.29 (0.06) 1.40 (0.04) 1.51 (0.03) uf25 1.00 (0) 1.12 (0.06) 1.23 (0.04) 1.33 (0.03) 1.43 (0.03) uf30 1.00 (0) 1.11 (0.05) 1.21 (0.04) 1.30 (0.03) 1.39 (0.03) uf35 1.00 (0) 1.10 (0.05) 1.19 (0.04) 1.27 (0.03) 1.36 (0.03) uf40 1.00 (0) 1.09 (0.05) 1.16 (0.03) 1.24 (0.03) 1.32 (0.03) uf45 1.00 (0) 1.09 (0.05) 1.16 (0.03) 1.23 (0.02) 1.30 (0.02) Table 4.3: Summary statistics of the distribution of the average target depth of P C G nodes from the UF-3-SAT instances. For every test set and level we show the mean and coefficient of variation of the distribution (over all instances), of the average target depth of the open plateau nodes (over all open plateaus at the given level). level 1 2 3 4 5 Test Set X ax/x X crx/x X <rxlx X uxlx X crxlx uf20 1.00 (0) 1.33 (0.12) 1.61 (0.09) 1.87 (0.07) 2.12 (0.06) uf25 1.00 (0) 1.26 (0.10) 1.50 (0.07) 1.74 (0.06) 1.98 (0.05) uf30 1.00 (0) 1.24 (0.08) 1.46 (0.07) 1.68 (0.06) 1.91 (0.05) uf35 1.00 (0) 1.21 (0.08) 1.41 (0.07) 1.63 (0.06) 1.84 (0.05) uf40 1.00 (0) 1.19 (0.08) 1.37 (0.06) 1.57 (0.06) 1.77 (0.06) uf45 1.00 (0) 1.18 (0.07) 1.36 (0.07) 1.54 (0.05) 1.73 (0.05) Table 4.4: Summary statistics of the distribution of the average target depth of P C G nodes from the UF-3-SAT instances. For every test set and level we show the mean and coefficient of variation of the distribution (over all instances) of the maximum target depth of the open plateau nodes (over all open plateaus at the given level). maximum, respectively. Because the values are so low, we can expect that SLS algorithms typically progress by going down in one or two levels only, rather than jumping down multiple levels. Both the average and maximum target depths decrease with the problem size, which was unexpected but is likely related to the similar trend in out-degree discussed in the previous section. Finally, we note that the variance in the distributions is very low and tends to decrease with both the level (with the exception of level 1, for which the coefficient of variation is 0) and the problem size. The corresponding data for the structured instances is shown in Tables 4.5 and 4.6. Surprisingly, the data is quite quantitatively similar to the data for the random test sets. The target depths increase with level, and are fairly similar between the structured instances. One notable difference occurs at level 2. The anom, med, and p a r 8 -1 - c instances have a large number of level 2 plateaus with a target depth of 2 — indicating that it is fairly easy to reach solutions from level 2 in these instances. In contrast, the graph colouring and random test sets have relatively few connections between solutions and any states not at level 1. Chapter 4. Plateau Connectivity 6 5 level 1 2 3 4 5 Instance X C?x/X X. <TX/X x crx/x X. <Tx/x x crx/x anom medium flat20-easy flat20-med flat20-hard par8-l-c 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.417 (0.32) 1.758 (0.23) 1.008 (0.07) 1.002 (0.03) 1.004 (0.02) 1.967 (0.06) 1.705 (0.32) 1.644 (0.33) 1.335 (0.16) 1.296 (0.18) 1.266 (0.20) 1.399 (0.14) 1.657 (0.32) 1.530 (0.35) 1.453 (0.19) 1.327 (0.17) 1.335 (0.17) 1.652 (0.27) 1.635 (0.35) 1.557 (0.34) 1.452 (0.19) 1.371 (0.17) 1.379 (0.19) 1.489 (0.29) T a b l e 4 . 5 : S u m m a r y s t a t i s t i c s o f t h e d i s t r i b u t i o n o f t he a v e r a g e t a r g e t d e p t h o f P C G n o d e s f r o m t h e s t r u c -t u r e d i n s t a n c e s . F o r e v e r y i n s t a n c e a n d l e v e l w e s h o w t h e m e a n a n d c o e f f i c i e n t o f v a r i a t i o n o f t h e d i s t r i b u t i o n ( o v e r a l l o p e n p l a t e a u s a t t he g i v e n l e v e l ) o f a v e r a g e t a r g e t d e p t h s f r o m t h e o p e n p l a t e a u n o d e s . level 1 2 3 4 5 Instance x vx/x X. <TX/X x crx/x X (Tx/X x. crx/x anom medium flat20-easy flat20-med flat20-hard par8-l-c 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.000 (0) 1.500 (0.33) 1.788 (0.23) 1.012 (0.11) 1.003 (0.05) 1.026 (0.16) 2.000 (0) 2.036 (0.35) 1.989 (0.40) 1.765 (0.24) 1.639 (0.29) 1.534 (0.33) 1.889 (0.24) 2.199 (0.39) 1.848 (0.47) 2.114 (0.27) 1.896 (0.28) 1.886 (0.27) 1.729 (0.28) 2.258 (0.43) 2.002 (0.44) 2.313 (0.28) 2.021 (0.28) 2.082 (0.31) 1.659 (0.32) T a b l e 4 .6 : S u m m a r y s t a t i s t i c s o f t h e d i s t r i b u t i o n o f t h e m a x i m u m t a r g e t d e p t h o f P C G n o d e s f r o m t h e s t r u c -t u r e d i n s t a n c e s . F o r e v e r y i n s t a n c e a n d l e v e l w e s h o w t h e m e a n a n d c o e f f i c i e n t o f v a r i a t i o n o f t he d i s t r i b u t i o n ( o v e r a l l o p e n p l a t e a u s a t t h e g i v e n l e v e l ) o f m a x i m u m t a r g e t d e p t h s f r o m t h e o p e n p l a t e a u n o d e s . 4.3 Solution Reachability N o w t h a t w e h a v e d e f i n e d P C G s a n d s t u d i e d s o m e o f t h e i r b a s i c g r a p h - t h e o r e t i c p r o p e r t i e s , w e b e g i n t o t r e a t P C G s a s a c t u a l m o d e l s o f t h e s e a r c h s p a c e a n d d e m o n s t r a t e P C G s c a n b e u s e d t o m o d e l a n d u n d e r -s t a n d S L S a l g o r i t h m b e h a v i o u r . T h e i n t u i t i v e i d e a i s t h a t a c t u a l S L S a l g o r i t h m t r a j e c t o r i e s w i l l c o r r e s p o n d t o p a t h s i n t h e P C G s , u n d e r v a r i o u s s i m p l i f y i n g a s s u m p t i o n s . F i r s t , w e a r e u n a b l e t o m o d e l n o n - i m p r o v i n g m o v e s u s i n g P C G s . H o w e v e r , i t i s w e l l - k n o w n t h a t t he p e r f o r m a n c e o f m a n y S L S a l g o r i t h m s ( i n c l u d i n g a l g o r i t h m s t h a t d o a n d d o n o t m a k e n o n - i m p r o v i n g m o v e s ) i s h i g h l y c o r r e l a t e d o n t h e tes t s e t s s t u d i e d h e r e (see , f o r e x a m p l e , [ H o o 9 8 ] ) . T h u s , t he i n t r i n s i c h a r d n e s s o f a n i n s t a n c e i s r e l a t i v e l y a l g o r i t h m - i n d e p e n d e n t , s o a c c u r a t e l y m o d e l l i n g t h e b e h a v i o u r o f t h e s i m p l e r a l g o r i t h m s i s d e s i r a b l e b e c a u s e i t a b s t r a c t s a w a y t he s u b t l e t i e s i n t r o d u c e d b y m o r e c o m p l e x a l g o r i t h m s . S e c o n d , t h e t i m e s p e n t s e a r c h i n g p l a t e a u s i s n o t c a p -t u r e d b y P C G s , a n d w e i n s t e a d a s s u m e t h a t a n a l g o r i t h m i s a b l e t o i m m e d i a t e l y c h o o s e r a n d o m l y a n d i n -d e p e n d e n t l y f r o m t h e se t o f e x i t - t a r g e t p a i r s f r o m a g i v e n p l a t e a u . W h i l e t h i s a s s u m p t i o n m a y b e v i o l a t e d b y p h e n o m e n o n s u c h a s t he c l u s t e r i n g o f e x i t s w i t h i n p l a t e a u s ( w h i c h h a s b e e n o b s e r v e d a n d d i s c u s s e d b r i e f l y i n C h a p t e r 3 ) , a n d b y t h e f a c t t ha t , g i v e n a n e x i t , m o s t S L S a l g o r i t h m s a r e b i a s e d t o w a r d s c h o o s i n g Chapter 4. Plateau Connectivity 6 6 higher-quality targets, we believe that it is a reasonable first-order approximation of the behaviour of actual SLS algorithms. Formally, given a weighted P C G G = (V, E, w) we model algorithm behaviour by defining a Markov chain M = {Xo, X i , Xi, • • • }, where each X , is a random variable wi th domain V. The transition matrix, P, is derived from the P C G edge weights: Puv = w(u, v) wherever W(u, v) is defined, Puu = 1 for all u € L, and all other entries are zero. The self loops for all of the closed plateaus make those Markov chain states into "sinks", ensuring that all of the probability mass is eventually accumulated into those states. This transition matrix is of a special form: because it contains no cycles, the maximum number of transitions that can be made before encountering one of the "sink" states is bounded above by the maximum path length in the P C G , / (note that / is usually defined to be a constant for the partial PCGs that we study, but in all cases is bounded above by the number of propositional clauses). This immediately implies that, for t > I, we have: p(') = p . p C - 1 ) = pi*-1) = p( ;) (4.10) Now, given an initial distribution A over the Markov chain states, we can calculate the distribution of Xt, the state of the Markov chain at time t. By definition, this is just p^ = A • P W . For all t > /, this becomes n = p(0 = A • p O . Thus, the Markov chain converges to steady state n i n time polynomial in the size of the P C G . Obviously, the only states which w i l l be assigned any probability mass in n w i l l the those in L — i.e. the closed plateaus (including solutions). M gives us a model wi th which we can estimate the probability that a greedy SLS algorithm we wi l l end up in each of the closed plateau regions, given an arbitrary initial distribution over the starting states. In order to model the behaviour of actual SLS algorithms as accurately as possible, we let A(u) be directly proportional to the number of states in u — corresponding to randomly and uniformly choosing a starting state. Given our model, we would like to evaluate how accurately it captures the interesting features of the search space, and how accurately it models the behaviour of actual local search algorithms. In order to quantify this, we define the "solution reachability" of an instance as the probability mass assigned by M. to al l solution plateaus in the steady state. Intuitively, we expect that the hardness of an instance should be (negatively) correlated with the probability that the Markov chain simulation ends up in a solution. The next definition demonstrates how we (efficiently) calculate the desired probability, and introduces our notation of "solution reachability". Definition 4.4. (Solution Reachability) Let G = (V, E, w) be a weighted P C G , and A be a probability distribution over V. We define the Chapter 4. Plateau Connectivity 67 ...4+.*j&.... 0.4 0.6 p(S|A) 0.8 10000 CD 100 t 0.2 0.4 0.6 0.8 p(S|A) Figure 4.4: Correlation between p(S \ A) and two measures of Isc for the uf30 test set. Left: fraction of successful GSAT runs. Right: G W S A T Isc (measured i n search steps). conditional reachability p(v | A) of each node v e V recursively: p(v | A) := A(v) iiveR YJU P(U I A) • w(u, v) + A(v) otherwise (4.11) Given the conditional reachability of each node, the solution reachability p(S \ A) of the P C G is then simply defined as: p ( S | A ) : = £ p ( s | A ) (4.12) ses Note that the set S in the equation represents the set of solution plateaus in the P C G . • Figure 4.4 demonstrates how effectively our model captures the search space features underlying in-stance hardness. In the left pane of the figure, we show the correlation between the solution reachability p(S | A) and the fraction of successful runs of the GSAT algorithm. The plot also shows the function y — 0.105102 * IO 0 - 8 7 0 1 6 6 ' * , which is the best least squares fit through the points of an equation of the form y = c • 10bx (i.e. a straight line in the semi-log plot). To measure the fraction of successful GSAT runs, we ran G S A T 1000 times (without restart) wi th a cutoff high enough to ensure that all runs that did not encounter a closed plateau successfully completed (a cutoff of 1000000 was used for all experiments, whereas all suc-cessful runs took less than 10000 steps). There is a surprisingly strong correlation between the logarithm of the success fraction and the solution reachability (correlation coefficient r = 0.91). While we were expecting the correlation, the exponential relationship between p(S | A) and the fraction of successful runs is surpris-ing. Note, however, that the exponential function has a sum of squared residuals ssr = 0.864 (rms of residu-als = 0.094, variance of residuals = 0.009), while the best fit linear function has ssr = 1.138 (rms of residuals = 0.108, variance of residuals = 0.012), so the exponential function more accurately models the relationship. Chapter 4. Plateau Connectivity 68 Test Set p(S | A) Isc crx/x r ts uf20 0.73 (0.31) 124.60 (1.50) -0.79 12.8 uf25 0.73 (0.33) 200.07 (0.75) -0.85 16.2 uf30 0.75 (0.31) 309.92 (1.05) -0.86 16.5 uf35 0.74 (0.34) 429.50 (0.95) -0.85 16.0 uf40 0.75 (0.35) 628.30 (0.99) -0.84 15.4 uf45 0.80 (0.28) 713.63 (1.09) -0.65 8.5 Table 4.7: Correlation between p(S | A) and G W S A T Isc for the UF-3-SAT test sets. p(S \ A) and /sc are the mean solution reachability and local search cost for the entire test sets, respectively. The columns labelled crx/x are the coefficient of variation of the associated distributions. Finally, the columns labelled r and ts show the correlation coefficient and test statistic of each regression analysis, respectively. Note that a test statistic ts > 2.262 indicates a statistically significant correlation wi th confidence level a = 0.05. The right pane of Figure 4.4 shows the correlation between p(S | A) and the local search cost of a more robust variant of GSAT, G W S A T [SK93]. This plot also shows the best least squares fit equation of the form y = \Qax+b) which in this case is the function y = i o - 1 - 3 1 0 4 *+ 3 - 3 2 3 6 2 . G W S A T is able to make upwards moves (which are not captured by our simple model), but this does not seem to have a significant negative effect on the predictive power of our model. We observe a strong negative correlation (r — —0.86) between the logarithm of G W S A T Isc and p(S | A ) . In this case, the exponential relationship is less surprising since given a fixed success probability in our model, it may still require an SLS algorithm an exponential number of steps to actually find a solution. Table 4.7 shows the correlation coefficient of the regression analysis performed on each of the test sets, in order to study how the predictions of the model scale wi th problem size. The results demonstrate clearly that p(S | A) is highly (inversely) correlated with the local search cost of G W S A T for all of the UF-3-SAT test sets studied here, and the results seem to scale wel l . It is slightly troubling that the w/45 test set had a relatively low correlation coefficient. Closer examination reveals that this is caused by a few outliers having high p(S | A) and Isc values. Further investigation indicates that in each of these outlying instances there was at least one open plateau region that had an uncharacteristically high exit distance — making the instance relatively hard, while not being factored into the calculation of p(S | A ) . Given our simplifying assumptions, it is not surprising that there exist cases for which the model gives poor predictions — overall, however the results are very positive. In order to get even more accurate predictions, the simplifying assumptions could be relaxed by including information in the model about, for example, the fraction of exits in plateaus and/or the exit distance, and incorporating this into the model by adding suitably defined self-loops for open plateaus representing the expected number of steps required to find an exit from the plateau. We also note that this correlation analysis was repeated wi th other local search algorithms such as WalkSAT, Novelty 4 ", and SAPS, wi th similar results. Finally, Table 4.8 shows the solution reachability values for each of our structured instances. Although Chapter 4. Plateau Connectivity 69 Instance P(S I A) flat20-easy flat20-med flat20-hard par8-l-c anom medium 0.566 0.552 1.000 1.000 0.969 0.298 Table 4.8: p(S | A) values for the structured instances. we do not have a distribution of instances available to study the correlation between p(S | A) and Isc for the structured instances, we do note that the values that we observe are negatively correlated with Isc between all of the instances. The graph colouring instances, wi th the very high solution reachability values, are also the easiest instances (GWSAT Isc of 50, 96, and 710 for the easy, median, and hard instance, respectively), and the solution reachability of the hardest graph colouring instance is lower than the others. The planning instances, which have mid-range solution reachability values, are also only moderately difficult instances (GWSAT Isc of 1096 and 2573 for anom and medium, respectively). Finally, the parity instance, which has the lowest solution reachability value is also by far the most difficult instance (GWSAT Isc 14248). This indicates that our model is informative for the structured instances as wel l as the random ones. 4.4 Properties of Traps in the Search Space In the previous section, we studied a novel method of modelling the search space of SAT instances and the connection between a feature of this model — namely the probability of reaching a solution plateau — and the hardness of an instance. In this section, we study the extent to which a single closed plateau can affect the behaviour of local search algorithms. Intuitively, if a closed plateau is both attractive and difficult for a local search algorithm to escape from, then we would expect that plateau to have a significant negative impact on the local search algorithm. Informally, we refer to such plateaus as traps. The remainder of this section is broken into two major subsections, one dealing wi th the attractivity of traps, and the second with their escape difficulty. We w i l l show that there do exist closed plateaus that are exceptionally attractive for SLS algorithms, and that this attractivity is closely related to the difficulty of the UF-3-SAT instances. In fact, we w i l l show that in the hardest instances, an accurate measure of the attractivity of the single most attractive trap is enough to explain the difficulty of the instance — indicating that these traps are the single most influential search space feature that makes SAT instances hard. Chapter 4. Plateau Connectivity 70 Figure 4.5: Distribution of amax values for the UF-3-SAT test sets. The y value at every a value along the x axis of the graph gives the fraction of instances having ocmax > cc. 4.4.1 Attractivity The previous section gave us the foundations for a definition of the attractivity of a trap. Intuitively, we define the attractivity of a trap as the reachability of the trap versus the reachability of all solutions. The following definition formalises this notion. Defini t ion 4.5. (Attractivity Factor) Given a P C G G = (V, E) and a closed plateau u e V, the attractivity factor a of u is defined as: «(w) : = p ( S | A ) (4.13) This is simply the conditional reachability of the plateau normalised by the solution reachability, and gives an indication of how likely we expect the trap would be encountered compared to solutions. For brevity, we let Pmax denote the plateau with the largest attractivity factor ccmax : = maxu€M a(u). • Figure 4.5 shows the distribution of ocmax values for all of the UF-3-SAT test sets, and Table 4.9 shows the same data numerically for specific a thresholds. It is somewhat surprising how low the fraction of instances having ocmax > 1 is, ranging between 0.06 and 0.12. This means that less than 12% of instances had closed plateaus that were more attractive than solutions, in all of the test sets. Also interesting is that the fractions tend to decrease wi th problem size for low thresholds, but for a. larger than 0.5 there does not seem to be a well-defined relationship between the various problem sizes. This indicates that as the problems get larger, Chapter 4. Plateau Connectivity 71 Test Set a > 0.05 K > 0.1 a > 0.25 a > 0.5 a > 0.75 « > 1 a > 2 uf20 0.78 0.58 0.30 0.21 0.09 0.07 0.03 uf25 0.68 0.52 0.26 0.20 0.14 0.10 0.02 uf30 0.69 0.47 0.28 0.15 0.10 0.08 0.03 uf35 0.58 0.45 0.28 0.18 0.15 0.12 0.07 uf40 0.56 0.43 0.30 0.19 0.14 0.10 0.06 uf45 0.50 0.35 0.21 0.11 0.09 0.06 0.02 Table 4.9: Statistics from the distribution of amax values for the UF-3-SAT test sets. Every column gives the proportion of instances having ocmax greater than the given value. 10000 1000 100 10 I 0.0001 iii: j..^..y.;ii»v*:...*.+iiiii...H:!. §-^^mm \ :;-St::t:!:»i:;!!i::::::!:::!:x:::!:!iK::::::!: Figure 4.6: Correlation between ccmax and Isc (left) as wel l as p(S | A) (right) for the uf-30 test set. fewer instances tend to have very attractive single plateau regions, suggesting that the hardness of larger instances might come either from multiple attractive closed plateaus, or from plateaus that are open, but still pose an obstacle for local search by having, for example, high exit-distances and/or few exits. In order to understand the extent to which Pmax is responsible for the hardness of UF-3-SAT instances, we examine the correlation between ocmax and Isc. Figure 4.6 shows this correlation for the uf30 test set (cf Figure 4.4, showing the correlation between p(S | A) and Isc for the same test set). The results are quite surprising — the correlation is very strong (correlation coefficient r = 0.81). This indicates that simply measuring the attractivity of the single most attractive trap gives a very good prediction of the hardness of the instance. Also note that the relationship between ccmax and Isc is polynomial, as opposed to the exponential relationship between p(S | A) and Isc. Table 4.10 summarises the correlation data for all of the UF-3-SAT test sets. Very interestingly, the correlation coefficients that we observe are only slightly worse that those in Table 4.7, and the values scale wel l wi th the problem size. Because ccmax and p(S | A) are so closely related, and both are highly correlated with Isc, we must determine if the effects of one subsumes the other. Figure 4.6 (right) shows the correlation between ccmax and p(S | A) on the uf30 test set (similar correlations were observed for the other UF-3-SAT test sets). There is an obvious (inverse) correlation between the two measures, which should not be surprising considering their Chapter 4. Plateau Connectivity 72 Test Set crx/x Isc <rx/x r fs uf20 0.39 (2.23) 128.91 (1.49) 0.81 13.7 uf25 0.33 (1.73) 201.42 (0.75) 0.81 13.6 uf30 0.38 (2.85) 310.39 (1.04) 0.81 13.4 uf35 0.39 (1.97) 431.26 (0.96) 0.83 14.9 uf40 0.60 (3.70) 628.29 (0.98) 0.89 19.1 uf45 0.30 (3.12) 712.61 (1.07) 0.80 13.1 T a b l e 4 . 1 0 : C o r r e l a t i o n b e t w e e n ctmltx a n d G W S A T Isc f o r t he U F - 3 - S A T tes t se t s . ocmax a n d Isc a r e t he ocmax a n d l o c a l s e a r c h c o s t f o r t h e e n t i r e tes t se t s , r e s p e c t i v e l y . T h e c o l u m n s l a b e l l e d crx/x a r e t he c o e f f i c i e n t o f v a r i a t i o n o f t h e a s s o c i a t e d d i s t r i b u t i o n s . F i n a l l y , t h e c o l u m n s l a b e l l e d r a n d f s s h o w t h e c o r r e l a t i o n c o e f f i c i e n t a n d test s t a t i s t i c o f e a c h r e g r e s s i o n a n a l y s i s , r e s p e c t i v e l y . r e l a t e d d e f i n i t i o n s . W h a t i s s u r p r i s i n g , h o w e v e r , i s t h e t i g h t n e s s o f t h e c o r r e l a t i o n . It s e e m s t ha t , f o r t he U F -3 - S A T tes t se t s , t h e e f fec t s o f P m a x a l o n e a r e e n o u g h t o e x p l a i n t he v a r i a t i o n i s h a r d n e s s b e t w e e n i n s t a n c e s f r o m t h e s a m e tes t se t . T h u s , r a t h e r t h a n t h e r e b e i n g m a n y s m a l l e r c l o s e d p l a t e a u s c a u s i n g t h e d i f f i c u l t y , t h e r e s e e m s t o b e o n e m a j o r t r a p i n e a c h i n s t a n c e . T h i s s u g g e s t s t h a t s e a r c h s t r a t e g i e s t h a t e x p l i c i t l y d e t e c t a n d a v o i d s u c h t r a p s m a y b e v e r y e f f e c t i v e . T h e s e r e s u l t s a r e p a r t i c u l a r l y i n t e r e s t i n g c o n s i d e r i n g t h e r e s u l t s f r o m C h a p t e r 3 . I n p a r t i c u l a r , w h i l e w e o b s e r v e t h a t t h e U F - 3 - S A T i n s t a n c e s c o n t a i n a s i g n i f i c a n t n u m b e r o f c l o s e d p l a t e a u s a t l o w l e v e l s , m a n y o f t h e s e p l a t e a u s d o n o t c o n t r i b u t e t o t h e h a r d n e s s o f t h e i n s t a n c e . F u r t h e r m o r e , s i n c e t h e n u m b e r o f c l o s e d p l a t e a u s b e c o m e s v a n i s h i n g l y s m a l l a s t h e l e v e l i n c r e a s e s , i t a p p e a r s t h a t t h e t a s k o f d i s c o v e r i n g P * „ r s h o u l d b e f e a s i b l e . 4.4.2 Escape Difficulty A s w e a l l u d e d t o e a r l i e r i n t h i s s e c t i o n , w e w o u l d l i k e t o c h a r a c t e r i s e a " t r a p " a s a c l o s e d p l a t e a u t h a t i s b o t h a t t r a c t i v e a n d d i f f i c u l t to e s c a p e f r o m . T o s o m e e x t e n t , t h e a t t r a c t i v i t y f a c t o r o f a c l o s e d p l a t e a u c a p t u r e s b o t h a t t r a c t i v i t y a n d e s c a p e d i f f i c u l t y . H o w e v e r , w e n o w a t t e m p t t o e x p l i c i t l y c h a r a c t e r i s e t he e s c a p e d i f f i c u l t o f c l o s e d p l a t e a u s . W e c h a r a c t e r i s e t h e e s c a p e d i f f i c u l t y o f a c l o s e d p l a t e a u b y m e a s u r i n g t h e s h i f t i n t h e R L D t h a t o c c u r s w h e n a l o c a l s e a r c h a l g o r i t h m i s i n i t i a l i s e d u n i f o r m l y a n d r a n d o m l y f r o m s t a t e s w i t h i n t h e c l o s e d p l a t e a u . Definition 4 . 6 . (Escape Difficulty) G i v e n a P C G G = (V, E) a n d a c l o s e d p l a t e a u u e V, a n d a n y l o c a l s e a r c h a l g o r i t h m A, t h e e s c a p e d i f f i c u l t y e o f u i s d e f i n e d as : W h e r e Isc i s s i m p l y t h e m e a n o f t h e R L D o b t a i n e d b y r u n n i n g A o n t h e i n s t a n c e , a n d / s c ( u ) i s Chapter 4. Plateau Connectivity 73 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 •> 1 ( 1 • ' ' ''Ap~~*— /// : /' f:"i \ r / •' , f original RTD / ' QRTD a=0.17 " QRTD a=0.41 V QRTD a=1.04 -QRTDo=9.91 ' i 10 1000 Search Steps 100000 Figure 4.7: Shifted RLDs obtained by initialising G W S A T from various closed plateau regions. Shown are the instance wi th median and maximum local search cost (left and right, respectively) from the uf30 test set. The RLDs are labelled by the cc values of the closed plateaus from which G W S A T is initialised. See text for further details. the mean of the the qualified R L D obtained by first randomly and uniformly initialising ,4 from states in u, and then running A as usual (without restart). For brevity, we define e m a x :— e{Pmax), i.e. the escape difficulty of the most attractive plateau. • Figure 4.7 shows example shifted RLDs for the instances with median and maximum local search cost for the G W S A T algorithm from the M/30 test set. In this figure, we initialised G W S A T from each of the closed plateaus with highest cc values (i.e. the most attractive plateaus). We first turn our attention to the left pane of the figure, which shows the shifted RLDs for the instance with median search cost. We show shifted RLDs for the three most attractive closed plateaus. Most surprising in this figure is that initialising from the most attractive plateau (cc = 0.223) actually makes the instance slightly easier for GWSAT. While surprising, this is likely an indication of why this instance is not more difficult; though Pmax is quite attractive, being situated i n the plateau is not detrimental to the search trajectory. The two other plateaus (cc — 0.06 and 0.09) result in a shift to the right in the RLDs, indicating that starting from these plateaus is detrimental to the search. However, in these cases, the plateaus are not very attractive so the instance is still easy to solve. The right pane of Figure 4.7 shows shifted RLDs for the hardest instance from the w/30 test set. This instance has fewer surprises. Initialising from the most attractive plateaus (cc — 9.91,1.04, and 0.41) results in a shift to the right, and in fact e monotonically increases with cc for each of these plateaus. We also observe one plateau for which the initialisation makes the instance easier, but this plateau is not very attractive. Given these observations, it should be evident why this instance is difficult — the search space contains closed plateaus which are both very attractive and difficult to escape from. Figure 4.8 shows the distribution of e m a x values for all of the UF-3-SAT test sets, and Table 4.11 shows summary statistics from the same distributions. Chapter 4. Plateau Connectivity 74 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 —^ 1 1 1 Uf20 uf25 uf30 -uf35 u(40 uf45 " %v^'1 : v A Y.v. . ^V • A \ ; \ ; '• i ' ^ v v . : i v,\V, • i j i 0 0.5 1 1.5 2 2.5 Figure 4.8: Distribution of emax values for the UF-3-SAT test sets. The y value at every e value along the x axis of the graph gives the fraction of instances having emax > e. Test Set X min med max Cmax ^ 1 uf20 1.259 (0.27) 0.489 1.201 2.529 0.786 uf25 1.199 (0.31) 0.343 1.155 2.410 0.690 uf30 1.212 (0.32) 0.273 1.221 2.136 0.727 uf35 1.193 (0.36) 0.187 1.154 2.625 0.690 uf40 1.154 (0.34) 0.328 1.120 2.357 0.660 uf45 1.101 (0.34) 0.195 1.054 2.220 0.570 Table 4.11: Summary statistics for the distribution of emax values for the UF-3-SAT test sets, x, crx/x, min, med, and max are the mean, coefficient of variation, min, median, and maximum of the distribu-tion, respectively. The column labelled emax > 1 shows the fraction of instances from each test set for which initialising from Pmax increases the search cost. Chapter 4. Plateau Connectivity 75 2.2 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0.0001 + + + + + i + , : +... + ! + + ++ f i + + i + V , i + I ++ *+++! ± +u + i + + + : + + + + I + :+ + + 0.001 F i g u r e 4 . 9 : C o r r e l a t i o n b e t w e e n ccmax a n d emax f o r t h e uf30 tes t set . E a c h p o i n t r e p r e s e n t s a s i n g l e i n s t a n c e . I n g e n e r a l , w e o b s e r v e t h a t ocmax a n d emax a r e n o t c o r r e l a t e d , a s i l l u s t r a t e d f o r t h e uf30 tes t s e t i n F i g u r e 4 .9 . T h e c o r r e l a t i o n c o e f f i c i e n t i n t h i s c a s e i s r — 0 .06 . T h i s i s n o t t e r r i b l y s u r p r i s i n g , a n d i l l u s t r a t e s tha t , w h i l e emax i s a n i n t e r e s t i n g a n d u s e f u l s e a r c h s p a c e f e a t u r e , i t i s n o t d i r e c t l y r e l a t e d t o Isc. F o r e x a m p l e , c o n s i d e r a n i n s t a n c e t h a t h a s a h i g h Isc c a u s e d b y a v e r y a t t r a c t i v e t r a p . I n t h i s c a s e , i t i s l i k e l y t h a t emax w i l l a c t u a l l y b e s m a l l b e c a u s e t h e S L S a l g o r i t h m i s d r a w n r a p i d l y i n t o t h e t r a p — t h u s , t h e d i f f e r e n c e b e t w e e n t h e o r i g i n a l R L D a n d t h e R L D w h e n i n i t i a l i s e d f r o m i n t h e t r a p i s s m a l l . T h i s i n t u i t i o n i s c o n f i r m e d b y F i g u r e 4 . 9 , w h e r e a l l i n s t a n c e s w i t h a m a x > 1 h a v e o n l y m o d e r a t e emax ( t h o u g h i n t e r e s t i n g l y , emax i s g r e a t e r t h a n 1 i n a l l o f t h e s e c a s e s ) . I n c o n t r a s t , w e h a v e s e e n c a s e s i n w h i c h c l o s e d p l a t e a u s h a d a l o w ex. v a l u e , b u t i f t h e s e a r c h w a s i n i t i a l i s e d i n t he p l a t e a u , t h e l o c a l s e a r c h c o s t i n c r e a s e d d r a m a t i c a l l y (c.f. F i g u r e 4 .7 ) . A g a i n , t h i s i n t u i t i o n i s s u p p o r t e d b y F i g u r e 4 . 9 , w h i c h i l l u s t r a t e s m a n y c a s e s i n w h i c h a n i n s t a n c e h a s a l o w ocmax v a l u e , b u t h i g h emax. 4.4.3 Connection With Mixture Models I n [ H o o 0 2 ] , H o o s i n t r o d u c e s a " m i x t u r e m o d e l " f o r t h e b e h a v i o u r o f S L S a l g o r i t h m s f o r S A T . H e o b s e r v e s tha t , w h i l e t h e R L D s o f S L S a l g o r i t h m s o n S A T i n s t a n c e s a r e t y p i c a l l y e x p o n e n t i a l l y d i s t r i b u t e d , t h e R L D s o f h a r d i n s t a n c e s m a y e x h i b i t s u b s t a n t i a l d e v i a t i o n s f r o m e x p o n e n t i a l d i s t r i b u t i o n s . H o o s d e m o n s t r a t e s t h a t t h e R L D s o f t h e s e i r r e g u l a r i n s t a n c e s c a n t y p i c a l l y b e f i t b y t w o c o m p o n e n t m i x e d e x p o n e n t i a l d i s t r i b u t i o n s , i.e. f u n c t i o n s o f t h e f o r m med[p, nil, m2\ = V • ed[m\] + (1 — p ) • ed\mi\ (4 .15) Chapter 4. Plateau Connectivity 76 Search Steps Figure 4.10: R L D for hard instance from the uf30 test set, and approximation by mixed exponential distri-bution where ed[m](x) = 1 — 2~x^m is the cdf of an exponential distribution with median m. The intuitive ex-planation is that there are two competing factors contributing to the irregular R L D : the solution(s) and a trap. Our search space models and the mixture models agree very wel l wi th this earlier result. A s an example, we consider Figure 4.10 which shows the R L D of G W S A T on the hardest instance from the u/30 test set. Also shown in this figure is the best fit med function merf[0.19,92,1742], which approximates the empirical R L D almost perfectly.4 We also show the qualified R L D when initialising G W S A T from states in Pmax. The R L D is fairly sig-nificantly shifted (e = 1.27) and, not surprisingly, differs substantially from an exponential distribution. However, ed[l742] matches the right tail of the shifted R L D wel l . Interestingly, ed[1742] also matches the right tail of the original R L D wel l (the tails of the original R L D and the shifted R L D coincide). Given our hypotheses about the effect of traps (such as Pmax in this instance) on run-time behaviour, it should not be surprising that the right tails of these distributions coincide; it the trap is very attractive and difficult to escape from, then the algorithm w i l l l ikely fall into the trap quickly and the long term behaviour of the tra-jectory w i l l not be substantially different than if the algorithm had been initialised into the trap. However, because the original R L D is located substantially to the left of the shifted R L D , this indicates that there is at least some chance of reaching a solution without encountering the trap. This observation is supported by the solution reachability of this instance, which is 0.08. 4Note that there is a slight deviation in the left tail. This is caused by the initial search phase after initialisation, where the algorithm is able to improve the evaluation function rapidly, but has little chance of finding a solution. For an in-depth explanation of this deviation, refer to [Hoo98]. Chapter 4. Plateau Connectivity 77 This interpretation agrees wel l wi th the best-fit mixed exponential med[0.19,92,1742].5 The first compo-nent of this distribution, weighted 0.19, is an exponential distribution wi th a low median of only 92 search steps. This component represents the trajectories which are " lucky" and do not encounter any major traps, and thus find a solution quite quickly. The second component, wi th the higher weight of 0.81, is an expo-nential wi th a higher median of 1742 search steps. Most interestingly, this exponential distribution matches the right tails of both the original R L D and the R L D after initialising from the Pmax. This component repre-sents trajectories which encounter the trap. 4.5 Summary In this chapter, we introduced a novel a posteriori method of modelling and studying the search space fea-tures underlying the behaviour of SLS algorithms. This model gives insights into the search space features responsible for the hardness of instances, and also gives new insights into existing results from the litera-ture. The central definition of this chapter extends the notion of plateaus, defined in the previous chapter, wi th information about the connectivity of the plateaus. By modelling the search space with a graph in which plateaus are nodes, and nodes are connected by an edge if there exists at least one exit-target pair between the nodes — called a plateau connectivity graph — we are able to construct a compact model of the search space, while retaining many interesting properties. We then improve this model by introducing weighted PCGs, in which the edge weights correspond to the degree to which the corresponding plateaus are connected. With this additional information, weighted PCGs capture the relevant properties of the corresponding search spaces in a compact way that facilitates our later analyses. Our first empirical analysis of the chapter is of basic graph theoretic properties of the PCGs. We measure the average out-degree of P C G nodes, and the average and maximum target depth of P C G nodes. This analysis gives insight into high-level search space features that affect, for example, the mobility of SLS algorithms. We next introduced one of the major contributions of this thesis — the notion of solution reachability. We demonstrated how it is possible to construct a Markov chain over the weighted P C G which effectively simulates SLS algorithm behaviour on the corresponding instance. Next, we proved that the Markov chain reaches the steady state in time polynomial in the size of the instance, and that in the steady state all of the probability mass is assigned to closed plateaus (including solutions). Solution reachability was then defined as the probability mass over all solution plateaus in the steady state. We demonstrated that solu-5 T h i s function was computed using the function fitting procedure i n g n u p l o t which , given the function med\p,m\,mi\ = p • ed[m{\ + (1 — p) • ed[nt2] and the empirical R L D , finds the parameters p, m\, and ni2 that minimise the s u m of squared errors. Chapter 4. Plateau Connectivity 78 t i o n r e a c h a b i l i t y c o r r e l a t e s v e r y w e l l w i t h Isc, a n d t h u s o u r m o d e l e f f e c t i v e l y c h a r a c t e r i s e s S L S a l g o r i t h m b e h a v i o u r . F o l l o w i n g t h i s , w e i n t r o d u c e d a n d s t u d i e d the p r o p e r t i e s o f t r a p s i n t h e s e a r c h s p a c e . I n t u i t i v e l y , a t r a p i s a c l o s e d p l a t e a u t h a t i s b o t h a t t r a c t i v e a n d d i f f i c u l t t o e s c a p e f r o m . W e d e f i n e d t h e a t t r a c t i v i t y o f a c l o s e d p l a t e a u a s t h e a m o u n t o f p r o b a b i l i t y m a s s a s s i g n e d t o t h e c l o s e d p l a t e a u r e l a t i v e t o t h e p r o b a b i l i t y m a s s as -s i g n e d t o a l l s o l u t i o n p l a t e a u s . I n t e r e s t i n g l y , w e d e m o n s t r a t e t h a t t h e a t t r a c t i v i t y o f t h e m o s t a t t r a c t i v e t r a p c o r r e l a t e s w e l l w i t h Isc a s w e l l as s o l u t i o n r e a c h a b i l i t y . G i v e n the r e s u l t s f r o m t h e p r e v i o u s c h a p t e r (i.e. t h e s m a l l s i z e o f c l o s e d p l a t e a u s ) , t h i s s u g g e s t s t h a t i t m a y b e p o s s i b l e a n d e f f e c t i v e i n p r a c t i c e to d y n a m i c a l l y d e t e c t a n d a v o i d t h e s e a t t r a c t i v e t r a p s , p o t e n t i a l l y i m p r o v i n g t h e p e r f o r m a n c e o f S L S a l g o r i t h m s . N e x t , w e d e f i n e d t h e n o t i o n o f t h e e s c a p e d i f f i c u l t y o f a t r a p , w h i c h i s e s s e n t i a l l y a m e a s u r e o f h o w m u c h w o r s e o f f a n S L S a l g o r i t h m i s w h e n i n i t i a l i s e d i n t o t h e g i v e n t r a p . W e f i n d t h a t a l a r g e p r o p o r t i o n o f t h e c l o s e d p l a t e a u s h a v e a n e s c a p e d i f f i c u l t y g r e a t e r t h a n o n e ( i n d i c a t i n g t h a t i t i s d e t r i m e n t a l t o b e i n i t i a l i s e d i n t o t h e s e c l o s e d p l a t e a u s ) , a n d t h a t t h e a v e r a g e e s c a p e d i f f i c u l t y i s g r e a t e r t h a n o n e f o r a l l o f t h e tes t s e t s c o n s i d e r e d . F i n a l l y , w e n o t e t h a t t h e a t t r a c t i v i t y f a c t o r o f t h e m o s t a t t r a c t i v e t r a p a n d t h e e s c a p e d i f f i c u l t y o f t h e s a m e t r a p a r e u n c o r r e l a t e d , a n d w e g i v e a n e x p l a n a t i o n o f w h y t h i s i s n o t s u r p r i s i n g , a n d h o w i t i l l u s t r a t e s a p o t e n t i a l w e a k n e s s o f o u r m e a s u r e o f e s c a p e d i f f i c u l t y . O u r f i n a l a n a l y s i s o f t h e c h a p t e r c o n n e c t s p l a t e a u c o n n e c t i v i t y w i t h t h e " m i x t u r e m o d e l s " d e f i n e d b y H o o s [Hoo02]. W e p r o v i d e a l i m i t e d a m o u n t o f e m p i r i c a l e v i d e n c e d e m o n s t r a t i n g t h a t m i x t u r e m o d e l s a r e t h e r e s u l t o f h i g h l y a t t r a c t i v e c l o s e d p l a t e a u s , t h a t h a v e a h i g h e s c a p e d i f f i c u l t y . W e d e m o n s t r a t e t h a t Pmax (i.e. t h e m o s t a t t r a c t i v e c l o s e d p l a t e a u ) i s r e s p o n s i b l e f o r t h e l a r g e c o m p o n e n t o f t h e m i x e d e x p o n e n t i a l d i s t r i b u t i o n , w h i c h i s c o n s i s t e n t w i t h t he m o t i v a t i o n g i v e n b y H o o s w h e n h e i n t r o d u c e d t h e s e m o d e l s . Chapter 5. Performance Critical Variables 79 Chapter 5 Performance Critical Variables and the Value of Information in SAT Solving In this chapter, we introduce the notion of performance criticality of variables in solving satisfiable SAT instances. Intuitively, a variable is performance critical if and only if it has a truth assignment under which the run-time of a given SAT algorithm is substantially reduced. Understanding the properties of PCVs (and the connection between PCVs and the search space features studied in the previous chapters) is useful for understanding what makes some SAT instances harder than others. P C V s are tightly related to many of the concepts presented in earlier chapters; in particular, P C V s reflect asymmetry in the search space. We present a bound on the amortisable cost of realising an oracle for performance critical variables (PCVs) and show that P C V s exist for a number of high-performance SAT algorithms and a range of SAT instances. P C V s are algorithm dependent, yet, particularly for structured instances, the P C V sets of differ-ent SAT solvers often overlap. We also provide results on the additivity of the effects of P C V assignments and on the correlation between the number of PCVs and the hardness of SAT instances. We also discuss the relationship between P C V s and the established concepts of backdoor and backbone variables. Most inter-estingly, we discuss the relationship between PCVs , plateau connectivity, and traps in the search space. This chapter is structured as follows. First, we motivate the study and introduce the important con-cepts used throughout the chapter. Next, we give formal definitions for performance critical variables and variable assignments and discuss how these notions of performance criticality are related to the theoretical idea of oracles for variable and value selection in SAT and their practical approximation using heuristics. Following this, we present the results of our empirical studies of performance critical variables and variable assignments for high-performance stochastic local search and systematic search algorithms for SAT. Finally, we discus the relationship between P C V s and related concepts from previous chapters of this thesis as wel l as some existing concepts from the literature, and conclude by summarising our main results and potential extensions to this work. Chapter 5. Performance Critical Variables 80 5.1 Mot iva t ion T h e w o r k p r e s e n t e d i n t h i s c h a p t e r i s b a s e d o n t h e i n t u i t i v e p r e m i s e t h a t i n t y p i c a l s a t i s f i a b l e i n s t a n c e s o f S A T , n o t a l l v a r i a b l e s a r e e q u a l w i t h r e s p e c t t o t h e i r p o t e n t i a l f o r r e d u c i n g t h e r u n - t i m e o f a g i v e n S A T a l g o r i t h m w h e n a s s i g n e d a p a r t i c u l a r t r u t h v a l u e . T h i s g i v e s r i s e t o t h e n o t i o n o f performance criticality; i n t u i t i v e l y , performance critical variable assignments (PCVAs) a r e t r u t h a s s i g n m e n t s o f i n d i v i d u a l v a r i a b l e s t h a t l e a d t o s u b s t a n t i a l r e d u c t i o n s o f t h e r u n - t i m e o f a g i v e n S A T a l g o r i t h m ; a n d performance critical variables (PCVs) a r e t h e v a r i a b l e s f o r w h i c h s u c h a s s i g n m e n t s e x i s t . T h e m a g n i t u d e o f t h e r u n - t i m e r e d u c t i o n s c a u s e d b y a P C V A o r P C V i s m e a s u r e d b y a s u i t a b l y d e f i n e d criticality factor. N o t e t h a t t h e n o t i o n o f a p e r f o r m a n c e c r i t i c a l v a r i a b l e d i f f e r s s u b s t a n t i a l l y f r o m t h e w e l l - k n o w n c o n c e p t o f a b a c k b o n e v a r i a b l e [ P a r 9 7 ] ; i n p a r t i c u l a r , a s w e w i l l s ee l a t e r , P C V s e x i s t f o r i n s t a n c e s t ha t h a v e n o b a c k b o n e s a n d b a c k b o n e v a r i a b l e s a r e n o t n e c e s s a r i l y p e r f o r m a n c e - c r i t i c a l . T h e i d e a s b e h i n d P C V s a r e c l o s e l y r e l a t e d to t h a t u n d e r l y i n g the r e c e n t l y i n t r o d u c e d n o t i o n o f a m i n i m a l b a c k d o o r se t b u t d i f f e r i n s e v e r a l i m p o r t a n t a s p e c t s . I n p a r t i c u l a r , w h i l e b a c k d o o r se t s c a p t u r e a p o t e n t i a l l y d r a s t i c r e d u c t i o n i n t he a s y m p t o t i c t i m e c o m p l e x i t y f o r s o l v i n g a g i v e n f a m i l y o f S A T i n s t a n c e s b a s e d o n t h e k n o w l e d g e o f a se t o f v a r i a b l e s , P C V s o f f e r a m o r e f i n e - g r a i n e d m e a s u r e o f r u n - t i m e r e d u c t i o n s f o r i n d i v i d u a l i n s t a n c e s t ha t a r e a c h i e v a b l e b a s e d o n k n o w l e d g e a b o u t i n d i v i d u a l v a r i a b l e s . P C V s a r e c l o s e l y r e l a t e d to t he s e a r c h s p a c e f e a t u r e s t h a t w e h a v e s t u d i e d i n p r e v i o u s c h a p t e r s . U n d e r -s t a n d i n g h o w s e a r c h s p a c e f e a t u r e s a r e r e l a t e d t o P C V s f a c i l i t a t e s a d e e p e r u n d e r s t a n d i n g o f t he r e a s o n s u n d e r l y i n g t h e p e r f o r m a n c e c r i t i c a l i t y o f v a r i a b l e s . I n p a r t i c u l a r , a s w e w i l l d e m o n s t r a t e l a t e r i n t he c h a p t e r , t h e r e i s a s t r o n g c o r r e l a t i o n b e t w e e n t h e a v e r a g e c r i t i c a l i t y f a c t o r o f v a r i a b l e s a n d b o t h s o l u t i o n r e a c h a b i l i t y a n d ctmax (cf. S e c t i o n s 4.3 a n d 4.4). F u r t h e r m o r e , w e d e m o n s t r a t e t h e e f fec t o f c r i t i c a l v a r i a b l e s o n p l a t e a u c o n n e c t i v i t y , w h i c h i l l u s t r a t e s h o w t h e s e a r c h s p a c e c h a n g e s w h e n P C V s a r e i n s t a n t i a t e d . P C V s a l s o h a v e i n t e r e s t i n g p r a c t i c a l a p p l i c a t i o n s : T h e e x i s t e n c e o f P C V s i n d i c a t e s a p o t e n t i a l f o r r e -d u c i n g t h e r u n - t i m e o f a g i v e n S A T a l g o r i t h m . T h i s r a i s e s t w o f u n d a m e n t a l q u e s t i o n s . D o P C V s e x i s t i n c o m m o n l y s t u d i e d t y p e s o f S A T i n s t a n c e s ? A n d h o w m u c h i s i t w o r t h t o i d e n t i f y t h e m , i n t e r m s o f t h e a m o r t i s a b l e c o s t o f a p r o c e d u r e t h a t d e t e r m i n e s t h e p e r f o r m a n c e c r i t i c a l i t y o f a g i v e n v a r i a b l e ? I n t h i s c h a p t e r , w e p r o v i d e a n s w e r s t o b o t h o f t h e s e q u e s t i o n s . R e g a r d i n g t h e f i r s t q u e s t i o n , w e p r o v i d e a n u m b e r o f p o s i t i v e r e s u l t s . W e f o u n d t h a t f o r s i x h i g h -p e r f o r m a n c e S A T a l g o r i t h m s a n d a b r o a d r a n g e o f S A T i n s t a n c e s w e s t u d i e d , t h e r e e x i s t se t s o f P C V s , w h i c h i n m a n y c a s e s c o n t a i n s u b s t a n t i a l l y m o r e t h a n 5% o f t h e v a r i a b l e s a p p e a r i n g i n t h e r e s p e c t i v e i n s t a n c e . F u r t h e r m o r e , f o r r a n d o m a s w e l l a s f o r s t r u c t u r e d i n s t a n c e s w e f o u n d a p o s i t i v e c o r r e l a t i o n b e t w e e n t h e a l g o r i t h m - s p e c i f i c h a r d n e s s o f t h e i n s t a n c e a n d t h e s i z e o f i t s P C V s e t — i n d i c a t i n g t h a t t h e r e i s t h e m o s t p o t e n t i a l f o r e x p l o i t i n g P C V s i n t h e h a r d e s t i n s t a n c e s . W e a l s o f o u n d e v i d e n c e t h a t t h e r u n - t i m e r e d u c t i o n Chapter 5. Performance Critical Variables 81 p o t e n t i a l o f i n d i v i d u a l P C V s i s a d d i t i v e , i.e., b y c o n s i d e r i n g se t s o f P C V s , o f t e n s u b s t a n t i a l l y l a r g e r r u n - t i m e r e d u c t i o n s c a n b e a c h i e v e d t h a n b y t h e r e s p e c t i v e i n d i v i d u a l P C V s . R e g a r d i n g t h e s e c o n d q u e s t i o n , w e p r e s e n t a s i m p l e b o u n d o n t h e a m o r t i s a b l e c o m p u t a t i o n a l c o s t f o r o b t a i n i n g P C V i n f o r m a t i o n , g i v i n g a s e n s e o f h o w m u c h v a l u e w e c a n a s s i g n t o t h e i n f o r m a t i o n t h a t a g i v e n v a r i a b l e i s a P C V . F u r t h e r m o r e , w e p r e s e n t p r e l i m i n a r y e m p i r i c a l r e s u l t s s u g g e s t i n g t h a t a t l e a s t i n c e r t a i n c a s e s , i n f o r m a t i o n o n P C V s f o r S L S a l g o r i t h m s c a n b e o b t a i n e d i n a c o m p u t a t i o n a l l y i n e x p e n s i v e w a y , b y c o l l e c t i n g s i m p l e s t a t i s t i c s o n t h e i n d i v i d u a l v a r i a b l e a s s i g n m e n t s e n c o u n t e r e d i n l o c a l m i n i m a o f t h e r e s p e c t i v e s e a r c h s p a c e s . It m a y b e n o t e d t h a t o u r n o t i o n o f p e r f o r m a n c e c r i t i c a l i t y i s a l g o r i t h m d e p e n d e n t . H o w e v e r , u n l i k e t h e s o m e w h a t r e l a t e d n o t i o n o f b a c k d o o r v a r i a b l e se t s , t h e c o n c e p t o f p e r f o r m a n c e c r i t i c a l i t y a p p l i e s d i r e c t l y t o a n y t y p e o f S A T a l g o r i t h m . F u r t h e r m o r e , w e h a v e f o u n d t h a t t h e P C V se t s f o r t h e d i f f e r e n t a l g o r i t h m s a p p l i e d t o t h e s a m e S A T i n s t a n c e t y p i c a l l y o v e r l a p t o s o m e e x t e n t ; t h i s i s p a r t i c u l a r l y t h e c a s e f o r s t r u c t u r e d i n s t a n c e s a n d s u g g e s t s t h a t t h e n o t i o n o f p e r f o r m a n c e c r i t i c a l i t y c a p t u r e s (a t l e a s t t o s o m e e x t e n t ) a l g o r i t h m i n d e p e n d e n t p r o p e r t i e s o f t h e s e i n s t a n c e s . 5.2 Oracles and Performance Crit ical i ty T h e i d e a u n d e r l y i n g p e r f o r m a n c e c r i t i c a l i t y i s r a t h e r s t r a i g h t f o r w a r d . G i v e n a S A T i n s t a n c e i n t h e f o r m o f a C N F f o r m u l a T, f i x i n g t h e t r u t h v a l u e a s s i g n e d t o o n e o f t h e v a r i a b l e s i n T s u c h t h a t s a t i s f i a b i l i t y i s p r e s e r v e d s h o u l d i n t u i t i v e l y l e a d t o a r e d u c t i o n i n t h e r u n - t i m e r e q u i r e d b y a S A T a l g o r i t h m f o r s o l v i n g T. W h e n m e a s u r i n g t h e s i z e o f t h e s e a r c h s p a c e i n t e r m s o f t h e c o m p l e t e s e t o f t r u t h a s s i g n m e n t s to u n a s s i g n e d v a r i a b l e s i n T, a s s i g n i n g o n e v a r i a b l e l e a d s t o a r e d u c t i o n i n s e a r c h s p a c e s i z e b y a f a c t o r o f t w o . C l e a r l y , t h e s a m e f a c t o r w o u l d b e o b t a i n e d f o r t h e r e d u c t i o n i n r u n - t i m e o f a n e x t r e m e l y s i m p l e ( a n d i n e f f i c i e n t ) s e a r c h a l g o r i t h m , s u c h a s u n i f o r m r a n d o m s a m p l i n g . H o w e v e r , t h e e f fec t o f s u c h a n a t o m i c a s s i g n m e n t o n m o r e p o w e r f u l a l g o r i t h m s c a n b e v e r y d i f f e r e n t , f o r e x a m p l e , b e c a u s e t h e s o l u t i o n d e n s i t y o f a s a t i s f i a b l e i n s t a n c e ( o r t h e p o t e n t i a l f o r p r o p a g a t i o n effects) i n c r e a s e s ( o r d e c r e a s e s ) s h a r p l y as a r e s u l t o f t he a t o m i c a s s i g n m e n t u n d e r c o n s i d e r a t i o n . I n t u i t i v e l y , w e c a l l a n a t o m i c a s s i g n m e n t p e r f o r m a n c e c r i t i c a l w . r . t . t h e g i v e n a l g o r i t h m A i f i t r e d u c e s t h e r u n - t i m e o f A b y m o r e t h a n a f a c t o r o f t w o , i.e., t h e r e d u c t i o n e f fec t f o r A e x c e e d s w h a t c o u l d b e e x p e c t e d d u e t o t he c o r r e s p o n d i n g r e d u c t i o n i n s e a r c h s p a c e s i z e . T h i s i s c a p t u r e d b y t h e f o l l o w i n g d e f i n i t i o n : Definition 5.1. (Performance-critical variable assignment (PCVA)) G i v e n a p r o p o s i t i o n a l f o r m u l a T, l e t Var{T) d e n o t e t h e se t o f v a r i a b l e s o c c u r r i n g i n T. L e t A b e a S A T a l g o r i t h m , a n d l e t sc{T,A) d e n o t e t h e s e a r c h c o s t o f A o n T, i.e., t h e ( e x p e c t e d ) Chapter 5. Performance Critical Variables 8 2 r u n - t i m e r e q u i r e d b y A f o r s o l v i n g T. F i n a l l y , l e t T\x := v] d e n o t e t h e f o r m u l a o b t a i n e d f r o m T b y s e t t i n g x t o t r u t h v a l u e v. A n a t o m i c t r u t h a s s i g n m e n t x := v, w h e r e x 6 Var(!F) a n d v 6 { -L , T } , i s c a l l e d a performance critical variable assignment (PVCA) o f T w i t h r e s p e c t t o A i f a n d o n l y i f : sc(T[x := v],A) < sc{J:,A)/2 (5.1) I n t h i s c o n t e x t , t h e criticality factor ofx := v i s d e f i n e d as : cf(x : = o ^ M ) := sc{Jr,A)/(2-sc(F[x := v],A)) (5 .2) a n d P C V A s a r e c h a r a c t e r i s e d b y a c r i t i c a l i t y f a c t o r > 1. • N o t e t h a t t h e d e f i n i t i o n o f P C V A s g e n e r a l i s e s e a s i l y t o m u l t i p l e v a r i a b l e a s s i g n m e n t s , b y e n s u r i n g t h a t t h e s e a r c h c o s t i s d e c r e a s e d b y a f a c t o r o f 2k, w h e n t h e r e a r e k v a r i a b l e a s s i g n m e n t s b e i n g m a d e . W e i n v e s t i g a t e t he b e h a v i o u r o f P C V A s i n t h i s g e n e r a l s e n s e l a t e r i n t he c h a p t e r w h e n w e s t u d y t h e a d d i t i v i t y o f P C V A s . T h e n o t i o n o f p e r f o r m a n c e c r i t i c a l i t y c a n a l s o b e e a s i l y g e n e r a l i s e d to v a r i a b l e s r a t h e r t h a n v a r i a b l e a s s i g n m e n t s . T h e i n t u i t i o n h e r e i s to r u n A o n T\x : = _L] a n d !F[x := T] i n p a r a l l e l u n t i l a s o l u t i o n i s f o u n d . A v a r i a b l e i s c a l l e d p e r f o r m a n c e c r i t i c a l i f a n d o n l y i f t h e c o m b i n e d r u n - t i m e o f t h e s e t w o p a r a l l e l r u n s i s s m a l l e r t h a n t h e r u n - t i m e o f A o n T, a s f o r m a l i s e d b y t h e f o l l o w i n g d e f i n i t i o n . Definition 5.2. (Performance-critical variable (PCV)) A v a r i a b l e x € Var(T) i s a performance critical variable (PCV) o f T w i t h r e s p e c t t o A i f a n d o n l y i f : m i n { s c ( T \ x := T], A),sc{T[x : = 1 ] , A)} < sc(iT, A) 12 (5.3) I n t h i s c o n t e x t , t h e criticality factor ofx i s d e f i n e d as : cf{x,F,A):=sc(F,A)/(2-mm{sc{F[x:=T),A),sc(F[x:= ±},A)}) (5.4) a n d P C V s a r e c h a r a c t e r i s e d b y a c r i t i c a l i t y f a c t o r > 1. • F r o m t h e s e d e f i n i t i o n s i t f o l l o w s i m m e d i a t e l y t h a t i f x := v i s a P C V A o f T w i t h r e s p e c t t o A, t h e n x i s a P C V o f T w . r . t . A, a n d c o n v e r s e l y , i f x i s a P C V o f T w . r . t . A, t h e n e i t h e r x : = T o r x : = _ L i s a P C V A o f T w . r . t . A. A l t h o u g h i n t h e l a t t e r c a s e , i t i s p o s s i b l e t h a t b o t h x := T a n d x := _L a r e P C V A s o f T w . r . t . A, o u r e m p i r i c a l r e s u l t s ( r e p o r t e d i n t h e n e x t t w o s e c t i o n s ) i n d i c a t e t h a t t h i s h a p p e n s e x t r e m e l y r a r e l y . C o n s e q u e n t l y , f o r a g i v e n S A T i n s t a n c e a n d a l g o r i t h m t h e r e i s a n e a r o n e - t o - o n e r e l a t i o n s h i p b e t w e e n P C V A s a n d P C V s . Chapter 5. Performance Critical Variables 8 3 B o t h n o t i o n s o f p e r f o r m a n c e c r i t i c a l i t y a r e c o n c e p t u a l l y m o t i v a t e d b y t h e i d e a o f p e r f e c t o r a c l e s (i.e., o r a c l e s w i t h o u t e r r o r ) . I n t h e c o n t e x t o f t h i s w o r k , a p e r f e c t o r a c l e c o r r e c t l y d e c i d e s t h e t r u t h o f a p r e d i c a t e w i t h o u t i n c u r r i n g a n y c o s t i n t e r m s o f r e s o u r c e c o n s u m p t i o n . A perfect PCVA oracle w o u l d d e c i d e f o r a g i v e n a l g o r i t h m A , f o r m u l a T, v a r i a b l e x e Var{T), a n d t r u t h v a l u e v £ { T , ±}, w h e t h e r x := v i s a P C V A o f T w . r . t . A. L i k e w i s e , a perfect PCV oracle w o u l d d e c i d e f o r a g i v e n a l g o r i t h m A, f o r m u l a T, a n d v a r i a b l e x € Var{T), w h e t h e r x i s a P C V o f T w . r . t . A. C o n s i d e r i n g t h e c l o s e r e l a t i o n s h i p b e t w e e n P C V s a n d P C V A s , i n t h e f o l l o w i n g w e o n l y c o n s i d e r P C V o r a c l e s . I n p r a c t i c e , t h e r e s u l t s o b t a i n e d f r o m a n o r a c l e m a y b e a p p r o x i m a t e d u s i n g h e u r i s t i c s . T y p i c a l l y , t h i s h a s t w o c o n s e q u e n c e s : t h e a n s w e r s p r o v i d e d b y t h e h e u r i s t i c m a y b e e r r o n e o u s , a n d c o m p u t i n g t h e s e a n s w e r s c o n s u m e s c o m p u t a t i o n a l r e s o u r c e s . T h i s r a i s e s t h e f o l l o w i n g c e n t r a l q u e s t i o n : U p to w h i c h p o i n t c a n the c o s t o f a h e u r i s t i c a p p r o x i m a t i n g a p e r f e c t P C V o r a c l e b e a m o r t i s e d t h r o u g h t h e r e d u c t i o n i n s o l u t i o n c o s t t h a t i s a c h i e v e d b y u s i n g s u c h a n o r a c l e , o r i n o t h e r w o r d s : w h a t i s t h e v a l u e o f t h e i n f o r m a t i o n o b t a i n e d b y t h e o r a c l e ? T h e a n s w e r t o t h i s q u e s t i o n d e p e n d s o n t w o f a c t o r s : (i) t h e a v e r a g e r e d u c t i o n i n s e a r c h c o s t t h a t c a n b e a c h i e v e d b y u s i n g a p e r f e c t P C V o r a c l e , a n d ( i i ) t he a s s u m e d q u a l i t y o f t h e h e u r i s t i c a p p r o x i m a t i o n , i.e., t he e r r o r r a t e o f t h e h e u r i s t i c . T h e a v e r a g e s e a r c h c o s t r e d u c t i o n c a n b e e a s i l y d e t e r m i n e d f r o m t h e c r i t i c a l i t y f a c t o r s i n t r o d u c e d a b o v e . I n p a r t i c u l a r , f o r t h e e r r o r - f r e e c a s e , w e o b t a i n t h e f o l l o w i n g r e s u l t : Theorem 5.1. Given a propositional formula T and a SAT algorithm A, let n denote the number of variables in T, and nc the number of PCVs ofT w.r.t. A; furthermore, let cf denote the average criticality factor of the PCVs of T with respect to A. Then the amortisable expected run-time of an error-free heuristic for deciding the criticality of variables in T is bounded from above by: sc{F,A)-nc/n-{l-l/cf). Proof: L e t H b e t h e h e u r i s t i c p r o c e d u r e , hc(x) b e t h e c o s t o f e v a l u a t i n g t h e h e u r i s t i c f o r v a r i a b l e x, a n d he b e t h e a v e r a g e h e u r i s t i c c o s t o v e r a l l o f t h e v a r i a b l e s i n T. A s s u m e t h a t w e u s e Ti. t o c o n s t r u c t a n e w a l g o r i t h m A' a s f o l l o w s : w e c o n s i d e r t h e v a r i a b l e s i n r a n d o m o r d e r , a n d f o r e a c h v a r i a b l e x w e e x e c u t e H t o d e t e r m i n e i f x i s a P C V . I f x i s a P C V w e r u n A o n F[x : = T ] a n d T\x : = ± ] i n p a r a l l e l , o t h e r w i s e w e c o n t i n u e t r y i n g t h e v a r i a b l e s i n o r d e r . T h e s e a r c h c o s t o f t h i s a l g o r i t h m , sc(F, A') i s j u s t t he e x p e c t e d h e u r i s t i c c o s t (he) t i m e s t h e e x p e c t e d n u m b e r o f t r i a l s b e f o r e c h o o s i n g a P C V (n/nc) p l u s t he e x p e c t e d c o s t o f s o l v i n g t h e t w o s i m p l i f i e d f o r m u l a e i n Chapter 5. Performance Critical Variables 84 parallel ( 1 / c f • sc(T, A)). Obviously, the cost w i l l be amortised if and only if: sc(F,A') < & (n/nc) •hc + (1/cf) • sc (^ r , ^ ) < (n/nc)-hc < sc(T,A) sc(F,A) sc(F,A)-(1-1/of) sc(iT,A)-nc/n-(l-l/cf) he < • For example, if we assume that nc/n = 0.08, and cf = 1.76 then the heuristic cost he is amortised if and only he < 0.035 • sc(J", .4). These values are actual data obtained from Novelty 4 " on the bwa instance — thus, in practice, a perfect heuristic for deciding if a variable is a P C V could be allocated a substantial amount (3.5%) of the computational resources allocated to solving the problem and the cost would be amor-tised. In general, P C V oracles w i l l likely be imperfect, i.e. have errors, but the cost can still be amortised in this case. 5.3 Performance Critical Variables for SLS Algorithms In this section, we study P C V s wi th respect to various high-performance SLS algorithms for SAT, and discuss the relationship between PCVs and algorithm behaviour. We show that a nontrivial number of P C V s exist for many different types of SAT instances, and that the corresponding criticality factors are reasonably high, which demonstrates the potential for practically exploiting P C V s . The algorithms we study are WalkSAT [SKC93], Novelty 4 " [Hoo99], and SAPS [HTH02]. We use the implementation of these algorithms from the SLS algorithm package U B C S A T [TH04]. WalkSAT and Novelty 4 " have been widely studied in the literature, and are often used as benchmarks to compare against, while SAPS is relatively new but has been shown to be the best performing SLS algorithm on various types of SAT instances. As opposed to previous chapters of this thesis, where we focused on simpler SLS algorithms for the sake of abstracting away algorithm details, in this chapter we use state-of-the art algorithms. The instances used in this chapter are described in Appendix A , and include larger instances than we were able to study in previous chapters. Again, we chose a variety of instance types and sizes, in order to study the effects of P C V s on many different instances. However, the complexity of the analysis, while less restrictive than i n previous chapters, still limited the size of the instances we could study. In order to measure the distribution of criticality factors over all P C V A s in a given SAT formula T, we do the following. First, a complete SAT algorithm (i.e. an algorithm that is guaranteed to decide whether an instance is satisfiable in a finite amount of time) is used to test whether the (restricted) formula !F[x := v] is Chapter 5. Performance Critical Variables 8 5 0.1 1 10 100 criticality (actor F i g u r e 5 .1 : D i s t r i b u t i o n o f c r i t i c a l i t y f a c t o r s w i t h r e s p e c t t o N o v e l t y " 1 " f o r v a r i a b l e s i n t h e h a r d e s t uflOO i n s t a n c e . E a c h p o i n t r e p r e s e n t s o n e v a r i a b l e , a n d c r i t i c a l i t y f a c t o r s a r e m e a s u r e d as i n D e f i n i t i o n 5 .2 . V a r i a b l e s w i t h a c r i t i c a l i t y f a c t o r g r e a t e r t h a n o n e ( i n t h i s c a s e 3 7 % o f a l l v a r i a b l e s ) a r e P C V s . s a t i s f i a b l e f o r e v e r y a t o m i c v a r i a b l e a s s i g n m e n t x := v (2 • n t es t s m u s t b e c o m p l e t e d , w h e r e n i s t he n u m b e r o f v a r i a b l e s i n T). T h e n , s c ( ^ r [ x : = v], A) i s m e a s u r e d a n d r e c o r d e d f o r e v e r y satisfiable f o r m u l a T\x := v] a n d S L S a l g o r i t h m A. I f T\x := v] i s u n s a t i s f i a b l e , t h e n i t s s e a r c h c o s t i s se t t o i n f i n i t y . T h e c r i t i c a l i t y f a c t o r o f e a c h a t o m i c a s s i g n m e n t a n d v a r i a b l e a r e c a l c u l a t e d f r o m t h e s e a r c h c o s t s a c c o r d i n g t o D e f i n i t i o n s 5.1 a n d 5 .2 , r e s p e c t i v e l y . W e e v a l u a t e t w o m e t h o d s o f c o m p u t i n g sc{!F[x :— v],A). B o t h o f t h e s e m e t h o d s i n v o l v e s u b s t i t u t i n g t h e t r u t h v a l u e v f o r x i n t h e f o r m u l a , t h e n s i m p l i f y i n g t h e f o r m u l a b e f o r e s o l v i n g i t u s i n g A. T h e f i r s t m e t h o d , partial, s u b s t i t u t e s v a r i a b l e x w i t h t r u t h v a l u e v a n d s i m p l i f i e s t h e f o r m u l a b y r e m o v i n g a l l l i t e r a l s t h a t b e c o m e f a l s e a n d b y r e m o v i n g a l l c l a u s e s t h a t b e c o m e t r u e a s a c o n s e q u e n c e o f t h e s u b s t i t u t i o n ( t h i s p r o c e s s i s k n o w n as u n i t p r o p a g a t i o n ) . T h e s e c o n d , full, d o e s a complete u n i t p r o p a g a t i o n o f t he a t o m i c a s s i g n m e n t x := v — t h a t i s , t h e a s s i g n m e n t x := v i s u n i t p r o p a g a t e d , b u t i f a n y n e w u n i t c l a u s e s a r e g e n e r a t e d a s a r e s u l t o f a p r o p a g a t i o n , t h e c o r r e s p o n d i n g a t o m i c a s s i g n m e n t i s a l s o p r o p a g a t e d . F i n a l l y , a s a b a s e l i n e m e a s u r e m e n t , w e a l s o c o n s i d e r a t h i r d m e t h o d , init, w h i c h s i m p l y i n i t i a l i z e s x to v a t t he s t a r t o f t h e s e a r c h , b u t d o e s n o t f u r t h e r r e s t r i c t t h e t r u t h v a l u e o f x. S i n c e init d i d n o t r e s u l t i n a n y m a j o r r e d u c t i o n s i n s e a r c h c o s t , w e d o n o t d i s c u s s r e s u l t s f o r t h i s v a r i a n t i n t h e f o l l o w i n g . I n a l l c a s e s a t l e a s t 1 0 0 r u n s w e r e p e r f o r m e d o n e a c h ( s i m p l i f i e d ) f o r m u l a , a n d t h e m e a n n u m b e r o f s e a r c h s t e p s r e q u i r e d to s o l v e t he i n s t a n c e i s r e p o r t e d a s t he s e a r c h c o s t . N o t e t h a t t e c h n i c a l l y , t h e f i r s t t w o m e t h o d s c a n b e s e e n as p r e p r o c e s s i n g s t a g e s t h a t a r e i n t e g r a t e d i n t o t he r e s p e c t i v e S L S a l g o r i t h m s . F i g u r e 5.1 s h o w s t h e d i s t r i b u t i o n o f c r i t i c a l i t y f a c t o r s w i t h r e s p e c t t o N o v e l t y + f o r t h e h a r d e s t r a n d o m -Chapter 5. Performance Critical Variables 86 Benchmark sc Walksat-SKC PCV frac / # cf(c.v.) sc Novelty"*" PCV frac / # cf (cv.) sc SAPS PCV frac / # c/(c.i>.) i(/50 uflOO uf200 682 3643 27249 0.06 / 2.87 0.04 / 4.20 0.03 / 6.20 1.24 (0.17) 1.26 (0.14) 1.29 (0.16) 416 3369 14529 0.09 / 4.30 0.08 / 8.06 0.04 / 8.38 1.37 (0.27) 1.44 (0.47) 1.28 (0.16) 291 1358 9423 0.06 / 2.89 0.03 / 2.71 0.01 / 2.02 1.21 (0.11) 1.16 (0.10) 1.16 (0.08) anomaly medium huge bwa 587 1182 21786 18073 0.29 / 14 0.17/20 0.07 / 31 0.06 / 27 42.91 (0) 149 (0) 1.66 (0) 1.53 (0) 177 446 12498 10362 0.42 / 20 0.16 / 18 0.08 / 37 0.08 / 36 10.04 (0) 62.98 (0) 1.86 (0) 1.76 (0) 137 325 3427 3203 0.40 / 19 0.19 / 22 0.05 / 25 0.05 / 24 8.29 (0) 37.97 (0) 1.51 (0) 1.44 (0) loga 139465 0.04 / 29 1.48 (0) 63276 0.05 / 43 1.38 (0) 11606 0.01 / 5 1.11 (0) 08 542 <0.01 / 1.5 2.24 (0.32) 8317 0.03 / 17.14 3.80 (0.25) 368 0.01 / 4.17 1.19 (0.07) par8-l-c 9251 0.25 / 16 3.46 (0) 2421 0.25 / 16 2.36 (0) 1577 0.17/11 1.73 (0) ssa7552-038 85319 0.06 / 83 4.90 (0) - - / - - H 4539 0/0 n/a Table 5.1: P C V results for thee prominent SLS algorithms, sc is the expected search cost for each algorithm on each benchmark set. P C V frac and # are the mean fraction of variables that are P C V s and number of variables that are PCVs , respectively, cf and cv. ave the mean and coefficient of vari-ation, respectively, of the criticality factors of variables that are PCVs . Note that the test sets containing more than one instance are presented in italics. Entries marked ' - ' indicate that the P C V measurement had not completed after 150 000 C P U seconds. 3-SAT instance from the uflOO test set (the distribution is over al l variables i n the instance). Note the large fraction (0.37) of variables that are PCVs . This shows that solving T\x :— T ] and !F[x := _L] in parallel for any of those variables would result in a speedup. Even more interesting is the range of criticality factors. The smallest criticality factor is 0.33, meaning that solving T\x :— T ] and F[x :— 1] in parallel would result in an approximately 3-fold slowdown, while some variables have criticality factors greater than 25, i.e., lead to a 25-fold speedup. This result is very encouraging — because there are so many PCVs , it may be possible in practice to find such variables and realise the speedup. Also note that to take advantage of a P C V it is only necessary that the identity of the variable be known, not the correct truth value of the variable. Table 5.1 shows the mean number and fraction of PCVs , as wel l as the mean criticality factors for all of the benchmark sets used in this study. The results show clearly that for all three algorithms, almost all of the instances studied here have a nontrivial number of PCVs . Generally, there seem to be a higher fraction of P C V s wi th respect to N o v e l t y + than the other two algorithms, and the fraction for WalkSAT seems higher than SAPS in most cases. The average criticality factors are also reasonably high, indicating that a substantial speedup is realisable for most of the PCVs , particularly for the structured instances. The fraction of variables that are performance critical decreases with increasing problem size for both random and structured instances, though the average criticality factors stay relatively constant as the problem size is increased for the UF-3-SAT test sets. We also note that for all of the instances studied there exist PCVs wi th criticality factors much higher than the average — in many cases, finding the "maximally critical" P C V renders the instance trivial to solve. In fact, the relationship between P C V s and algorithm behaviour is even more interesting than Table 5.1 alone would suggest. Figure 5.2 illustrates the correlation between search cost and the fraction of variables Chapter 5. Performance Critical Variables 87 0.3 o 0.1 0 7 — ! — ! ! ! ! ! ! ! ! ! [ ! ! M I | ; + i I N N ; 1 1 I ! l | 1 M n* i ! 1—T-TTTT III*-:: j if | + I I 1 + i i IIIII-*-; ; : ; + : • : : + : : : : : : ; : = : : : : : = + + +; i l l * ; ; ; ! ; ; ;+ I ! I i i i + ; ; ; • : : • + i i : + : : ! : : ! i + ; ; ! ; ; : ! • i + : : : ; ; ! ! ! + if ! + ; ; ; ; ; ; ; i -H-I+ i++ i i i i i ; +;+ i -f+i i ; : + ; : : ; i i ;+ ; ! i !I + + • :+•::•:: ;; * : + : : : : : ; ; ; + ;;;+ I l+l i I++ .1 ! .1 tUftit. * +++: 4* +: + i + • ! ; • ! : : + 1 1 1 1 1 1 1 1 1 H- 1 1 I 1 j 1 j 1 j ; ; ; i ; i i 100 1000 10000 100000 1e+06 search cost [steps] Figure 5.2: Correlation between search cost and PCV fraction for the uflOO benchmark set using the Novelty4" algorithm. that are PCVs for all of the instances in the uflOO benchmark set. Interestingly, we see that harder instances have more critical variables, which indicates that harder instances offer more potential for performance improvements than easier instances. We have confirmed that similar (though in some cases weaker) cor-relations exist for all of the algorithms studied in this paper (including the DPLL algorithms of the next section), on all of the random benchmark sets. Interestingly, the criticality factors with respect to different SLS algorithms are quite highly correlated. Figure 5.3 shows the correlation of criticality factors with respect to Novelty4" and SAPS for the hardest in-stance from the uflOO test set, as well as for the bwa instance. Furthermore, the areas of the plot containing variables that are critical for both algorithms consistently contains a large fraction of the critical variables for each algorithm. In particular, if a variable is performance critical for Novelty4", then it is almost always performance critical for SAPS. The correlations and overlap of critical variables is even stronger when com-paring the two algorithms to WalkSAT. Thus, it appears that for at least some instances, some variables are performance critical independent of the algorithm. We give evidence in the next section that this is also true between different DPLL-type algorithms, as well as between SLS and DPLL-type algorithms. Next, we study to which extent the run-time reductions achieved by correctly assigning PCVs are addi-tive in the sense that combining multiple PCVAs leads to run-time reductions greater than those achieved by individual PCVAs. For this purpose, for a given formula T, we generate a sequence of formulae T = TQ, • • • ,TN such that for 1 < i < n, TT := T\-\\x := v], where the atomic assignment x := v is chosen so that sc(J7 !_1[x := v],A) is minimized. Intuitively, this gives us an optimistic estimate on the performance improvement afforded by combinations of PCVAs. Chapter 5. Performance Critical Variables 88 n. < co 10 0.1 4ft 10 100 cf. (Novelty+) 10 0.1 0i 1 cf. (Novelty*) Figure 5.3: Correlation of criticality factors with respect to Novelty4" and SAPS. Left: correlation on the hardest instance from the uflOO test set. Right: correlation on the bwa instance. Each point represents one variable, and criticality factors are measured as in Definition 5.2. Figure 5.4: Additivity of PCVAs using dynamic ranking for Novelty4". nr/sc(P,A) is a normalized line on the plot indicating when the instance becomes trivial; see text for details. Left: Hard uflOO instance; right: bwa instance from the blocks benchmark set. Chapter 5. Performance Critical Variables 8 9 F i g u r e 5.4 s h o w s t h e n o r m a l i z e d s e a r c h c o s t sc(Jri,A)/sc(!F,A) tor e a c h o f t h e f o r m u l a e T Q , . . . , T n -T h e r e s u l t s s h o w t h e d r a m a t i c d e c r e a s e s i n s e a r c h c o s t o b t a i n e d w h e n m u l t i p l e P C V A s w i t h t h e l a r g e s t c r i t i c a l i t y f a c t o r s a r e u s e d t o s i m p l i f y t h e f o r m u l a . T h e l i n e l a b e l l e d nr / sc(F, A) i n t h e p l o t s i n d i c a t e s t he p o i n t a t w h i c h t h e s e a r c h c o s t o f t h e s i m p l i f i e d i n s t a n c e b e c o m e s l e s s t h a n t h e n u m b e r o f v a r i a b l e s i n T{. I n b o t h o f t h e e x a m p l e s s h o w n , t h e i n s t a n c e s b e c o m e t r i v i a l l y s o l v a b l e a f t e r o n l y a r e l a t i v e l y s m a l l f r a c t i o n o f t h e a t o m i c a s s i g n m e n t s h a v e b e e n m a d e . I n fac t , i n b o t h c a s e s w h e n u s i n g t h e full s i m p l i f i c a t i o n t e c h n i q u e , t h e u n i t p r o p a g a t i o n a l o n e s o l v e s t h e i n s t a n c e s a f t e r a t y p i c a l l y r e l a t i v e l y s m a l l n u m b e r o f P C V A s h a v e b e e n m a d e . It m a y b e n o t e d t h a t a s a s i d e - e f f e c t , t h i s m e t h o d f o r m e a s u r i n g t h e e f f ec t s o f P C V A a d d i t i v i t y p r o v i d e s a n e f f e c t i v e w a y f o r f i n d i n g s m a l l b a c k d o o r se t s f o r t h e g i v e n i n s t a n c e s ( t h i s w i l l b e d i s c u s s e d f u r t h e r i n t h e ' D i s c u s s i o n ' s e c t i o n o f t h i s c h a p t e r ) . 5.4 Performance Critical Variables for DPLL Algorithms W e n o w p r e s e n t r e s u l t s f o r P C V s f o r D P L L a l g o r i t h m s . T h e s e a l g o r i t h m s a r e c o m p l e t e ( g u a r a n t e e d t o f i n d a s o l u t i o n o r d e t e r m i n e t h a t n o n e e x i s t s i n f i n i t e t i m e ) , a n d h a v e b e e n s h o w n to o u t p e r f o r m S L S a l g o -r i t h m s o n m a n y p r o b l e m c l a s s e s , p a r t i c u l a r l y o h S A T - e n c o d e d i n s t a n c e s f r o m d o m a i n s s u c h as p l a n n i n g a n d h a r d w a r e v e r i f i c a t i o n . W e tes t t h r e e D P L L - t y p e a l g o r i t h m s : S A T Z - r a n d [ G S K 9 8 ] , K C N F S [ D D 0 1 ] , a n d z c h a f f [ M M Z + 0 1 ] . T h e s e t h r e e a l g o r i t h m s h a v e b e e n w e l l - s t u d i e d i n t h e l i t e r a t u r e , a n d a r e a m o n g the s t a te -o f - t h e - a r t c o m p l e t e a l g o r i t h m s o n the b e n c h m a r k se t s w e s t u d y i n t h i s p a p e r . T h e h e u r i s t i c s e m p l o y e d i n e a c h a l g o r i t h m a r e q u i t e d i f f e r e n t ; n e v e r t h e l e s s , w e d e m o n s t r a t e t h a t t h e r e a r e n o n t r i v i a l se t s o f P C V s w i t h r e s p e c t t o e a c h o f t h e D P L L a l g o r i t h m s . O u r e x p e r i m e n t a l p r o t o c o l f o r f i n d i n g P C V s w i t h r e s p e c t t o t h e s e s y s t e m a t i c S A T a l g o r i t h m s i s s i m i l a r t o t h a t f o r t h e p r e v i o u s l y d i s c u s s e d S L S a l g o r i t h m s . W e i t e r a t e t h r o u g h e v e r y a t o m i c a s s i g n m e n t x := v, a n d m e a s u r e sc(!F[x := v}) b y f i r s t d o i n g a f u l l u n i t p r o p a g a t i o n o f t h e a s s i g n m e n t t o o b t a i n t h e s i m p l i f i e d f o r m u l a T\x := v\ i n w h i c h v a r i a b l e x i s f o r c e d t o t a k e t h e v a l u e v, a n d t h e n r u n n i n g e a c h a l g o r i t h m o n T\x := v}. B e c a u s e S A T Z - r a n d i s a r a n d o m i z e d a l g o r i t h m , w e r u n i t 1 0 0 t i m e s o n e a c h f o r m u l a , a n d r e p o r t t h e m e a n s e a r c h c o s t . T h e t w o o t h e r a l g o r i t h m s a r e d e t e r m i n i s t i c , a n d c o n s e q u e n t l y n e e d t o b e r u n o n l y o n c e p e r i n s t a n c e . F o r a l l t h r e e a l g o r i t h m s w e u s e d m e a s u r e s o f s e a r c h c o s t t h a t c o r r e l a t e l i n e a r l y w i t h r u n t i m e (e.g. n u m b e r o f n o d e s i n t h e s e a r c h t r ee , o r t o t a l n u m b e r o f i m p l i c a t i o n s ) . H e n c e , t h e s e a r c h c o s t v a l u e s r e p o r t e d i n T a b l e 5 .2 c a n b e c o m p a r e d f o r t h e s a m e a l g o r i t h m b e t w e e n d i f f e r e n t i n s t a n c e s , b u t n o t b e t w e e n d i f f e r e n t a l g o r i t h m s . D P L L - t y p e a l g o r i t h m s t y p i c a l l y r u n a f u l l u n i t p r o p a g a t i o n a s a p r e p r o c e s s i n g s t e p ; e v e n i f t h i s w e r e n o t t h e c a s e , u n i t c l a u s e s a r e g e n e r a l l y d i s c o v e r e d a n d p r o p a g a t e d v e r y q u i c k l y d u r i n g t h e s e a r c h . F o r t h i s r e a s o n , o u r r e s u l t s f o r partial (i.e. d o i n g a p a r t i a l r a t h e r t h a n f u l l u n i t p r o p a g a t i o n t o o b t a i n T\x := v}) a r e Chapter 5. Performance Critical Variables 90 KCNFS SATZ-rand zchaff Benchmark sc PCV frac / # cf (c.v.) sc PCV frac / # c'f(c.v.) sc PCV frac / # cf (c.v.) uf50 67 0.03 / 1.38 1.24 (0.12) 302 0.23 / 11.61 1.86 (0.46) 279 0.22 / 11.06 1.24 (0.12) uflOO 587 0.07 / 6.80 1.17 (0.07) 2090 0.06 / 6.26 1.21 (0.23) 3528 0.25 / 25.13 1.83 (0.36) ufiOO 5186 0.22 / 44.8 1.76 (0.34) 42172 0.12 / 24.51 1.27 (0.08) 376187 0.24 / 47.18 2.66 (1.26) anomaly 3 0.04 / 2 1.5(0) 14 0.46 / 22 2.41 (0) 82 0.27 / 13 7.58 (0) medium 12 0.21 / 24 2.88 (0) 142 0.26 / 30 18.77 (0) 318 0.22 / 26 32.08 (0) huge 10 <0.01 / 2 1.25 (0) 1041 0.11 / 49 2.50 (0) 1805 0.05 / 21 1.60 (0) bwa 10 . 0.01 / 6 1.32 (0) 958 0.10 / 45 2.46 (0) 2163 0.06 / 26 1.70 (0) loga - - / - -(-) - - / - - H 45325 0.01 / 11 1.06 (0) U8 881419 0.07 / 4.78 1.51 (0) 4695 0.11 / 79.43 1.36 (0.24) 1944 0.11 / 62.29 1.92 (0.25) pnr8-l-c 20 0.23 / 15 2.95 (0) 111 0.42 / 27 4.40 (0) 581 0.36 / 23 2.12 (0) ssa7552-038 1628 0/0 n/a 440084 0.36 / 540 4.90 (0) 2677 <0.01 / 6 1.04 (0) Table 5.2: PCV results for DPLL-type algorithms.se is the expected search cost for each algorithm on each benchmark set. PCV frac and # are the mean fraction of variables that are PCVs and number of variables that are PCVs, respectively, cf and c.v. are the mean and coefficient of variation, respectively, of the criticality factors of variables that are PCVs. Entries marked ' - ' indicate that the PCV measurement had not completed after 150 000 CPU seconds. 10 ..... . . . : : + \ i \ r s r t t ; i >.;-.!.+•!•• ! M?f!!i-+ + H + + +; i i i i i ! EE:3::::::::E::i::1::i:s::E 10 + • • ::::: + + i+. ; j L..L.LLL. + ++ + ; + + f ! ] i ' T I T i " + ; ; ; ; ; ; : + " ^ T + : * * , nWfetli^ * r + * • • • +: ;....f+j..LL..L. .+ i J | \ i 1-H44- + 0.1 1 c.f. (Novelty+) 0.1 1 c.f. (zchaff) 10 Figure 5.5: Correlation of criticality factors with respect to different algorithms on the bwa instance. Left: Novelty"1" vs. SATZ-rand, Right: zchaff vs. SATZ-rand. Each point represents one variable, and criticality factors are measured as in Definition 5.2. virtually identical to those for full, and we report only the results for the latter. Table 5.2 shows our PCV results for the DPLL-type algorithms. As for the previously considered SLS algorithms, that there are a nontrivial number of PCVs for virtually all of the benchmark instances con-sidered. Interestingly, there often seem to be more PCVs for the DPLL-type algorithms than for the SLS algorithms when compared on a given test-set, most notably for the SATZ-rand and zchaff algorithms. The average criticality factors are also reasonably high for all of the test sets. As with the SLS algorithm results, most instances have PCVs with very high criticality factors with respect to the DPLL algorithms. Additional results indicate that there is a significant amount of overlap between the sets of PCVs for SLS algorithms and DPLL-type algorithms, as well as between PCV sets for different DPLL-type algorithms. For example, consider Figure 5.5, which shows the correlation between the criticality factor of each variable in the bwa instance for Novelty"1" and SATZ-rand (left) and zchaff and SATZ-rand (right). We see that Chapter 5. Performance Critical Variables 91 the region for which the criticality factor is greater than one for both algorithms consistently contains a large portion of the points that are greater than one for each algorithm. This is quite surprising, given the different nature of these algorithms and their underlying heuristics, and provides evidence that algorithm-independent performance critical variables exist for some instances. We also observe additivity effects for PCVAs with respect to the DPLL-type algorithms on many of the test sets that are similar to those for SLS algorithms. For example, zchaff on the ssa7552-038 instance has only 6 PCVs (out of 1501 variables). However, when the single PCVA with the highest criticality factor is used to simplify the formula, the entire formula becomes trivial. As another example, KCNFS shows excellent additivity on the hardest uflOO instance — the dynamic ranking found a small backdoor of only 6 PCVAs, which is surprisingly small considering the results from [Int03]. Furthermore, the search cost decreased by more than a factor of two with the addition of each PCVA, indicating that a speedup is possible even if all 26 assignments to the corresponding variables were run in parallel. 5.5 Discussion As previously mentioned, the notion of PCVs is conceptually related to that of backdoor variables [WGS03]. Both concepts are algorithm-dependent; but while the notion of performance criticality applies to arbitrary SAT algorithms, backdoor sets are relative to polynomial-time SAT algorithms, such as unit propagation. Furthermore, it should be noted that, unlike our concept of performance criticality, the notion of a backdoor inherently refers to a property of a set of variables. It is also more coarse-grained, since it only captures potential reductions from exponential to polynomial-time asymptotic complexity for certain classes. It should be noted that variables from (minimal) backdoor sets are not necessarily performance critical, since a reduction may only be obtained when several or all of the elements of a given backdoor set are assigned the correct truth value. Additionally, it is currently unclear whether small backdoors can be determined and exploited in an amortised way. Interestingly, our additivity for PCVAs resulted in new, substantially smaller backdoor sets than previ-ously known for the random SAT instances considered here. For example, at clauses-to-variables ratio 4.3, Interian [Int03] reports an expected backdoor size of approximately 0.5 n torn = 100 variables. Using our dynamic ranking scheme and Novelty4" on the uflOO set, we find expected backdoor sets of size 0.078 • n, with minimum and maximum values of 0.05 • n and 0.16 • n, respectively. We find similar results on this test-set when using the dynamic ranking scheme with all of the other algorithms. Another concept that is intuitively related to PCVs is that of the backbone; the backbone of a satisfiable SAT instance is the set of variables that have the same truth values in all satisfying assignments. Obviously, any PCVA x := v where x is a backbone variable x must set v to the correct value; hence, knowledge of Chapter 5. Performance Critical Variables 92 10 1 1 1 4 ! !• \ j i -i i j i + + : ~ v - - - - - t . . + r * i *..! |+ r - - — 4 . V * * t -+++: t ••f.-j)., • • f | ;• f^* i i i i i i i i i 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p(S | A) 0.0001 F i g u r e 5 .6: C o r r e l a t i o n b e t w e e n p(S | A ) a n d a v e r a g e c r i t i c a l i t y f a c t o r ( le f t ) , a n d ocmax a n d a v e r a g e c r i t i c a l i t y f a c t o r ( r i g h t ) w i t h r e s p e c t to N o v e l t y + o n the uf25 tes t se t . E a c h p o i n t r e p r e s e n t s o n e i n s t a n c e f r o m t h e set , a n d t h e c r i t i c a l i t y f a c t o r i s a v e r a g e d o v e r a l l v a r i a b l e s i n t h e i n s t a n c e . b a c k b o n e i s p o t e n t i a l l y u s e f u l i n n a r r o w i n g d o w n t h e se t o f c a n d i d a t e s f o r P C V A s . H o w e v e r , b a c k b o n e v a r i a b l e s a r e n o t n e c e s s a r i l y p e r f o r m a n c e c r i t i c a l , a n d P C V s m a y n o t n e c e s s a r i l y a p p e a r i n t h e b a c k b o n e o f a g i v e n i n s t a n c e . I n t h i s c o n t e x t , i t m a y b e n o t e d t h a t s o m e o f o u r t e s t - i n s t a n c e s (ii8*) h a v e n o b a c k b o n e s , b u t d o h a v e P C V s (cf T a b l e s 5.1 a n d 5 .2) . S t i l l , t h e r e m a y b e a n o i s y c o r r e l a t i o n b e t w e e n t he t w o c o n c e p t s ; c o n -s i d e r i n g r e c e n t a p p r o a c h e s f o r t h e e f f i c i e n t h e u r i s t i c d e t e r m i n a t i o n o f b a c k b o n e v a r i a b l e s [ Z R L 0 3 , D D 0 1 ] , t h i s i s s u e s h o u l d b e f u r t h e r i n v e s t i g a t e d . 5.5.1 Connection with Search Space Features W h i l e w e h a v e d e m o n s t r a t e d t h a t p e r f o r m a n c e c r i t i c a l v a r i a b l e s e x i s t , a n d t h a t t h e y h a v e a s u b s t a n t i a l i m p a c t o n t h e p e r f o r m a n c e o f S A T a l g o r i t h m s , s o f a r w e h a v e n o t d i s c u s s e d t h e u n d e r l y i n g c a u s e s o f p e r -f o r m a n c e c r i t i c a l i t y . W e n o w i n v e s t i g a t e t h e n a t u r e o f t he c o n n e c t i o n b e t w e e n t h e s e a r c h s p a c e f e a t u r e s s t u d i e d i n p r e v i o u s c h a p t e r s a n d p e r f o r m a n c e c r i t i c a l v a r i a b l e s . T h e l e f t p a n e o f F i g u r e 5 .6 s h o w s t h e c o r r e l a t i o n b e t w e e n s o l u t i o n r e a c h a b i l i t y p(S | A ) ( d e f i n e d i n S e c -t i o n 4 . 3 ) , a n d t h e a v e r a g e c r i t i c a l i t y f a c t o r o f e a c h i n s t a n c e i n t h e uf25 tes t se t . V e r y i n t e r e s t i n g l y , i n s t a n c e s w i t h a h i g h a v e r a g e c r i t i c a l i t y f a c t o r t e n d t o h a v e l o w s o l u t i o n r e a c h a b i l i t y ( t he c o r r e l a t i o n c o e f f i c i e n t i s r = —0.51) . T h i s s u g g e s t s t h a t f i x i n g c r i t i c a l v a r i a b l e s t o t h e i r c o r r e c t v a l u e s m a y i n c r e a s e t h e s o l u t i o n r e a c h a b i l i t y , t h u s r e n d e r i n g t h e i n s t a n c e s e a s i e r . T h e r i g h t p a n e o f F i g u r e 5 .6 s h o w s a s i m i l a r c o r r e l a t i o n b e t w e e n t h e m a x i m u m a t t r a c t i v i t y f a c t o r o f c l o s e d p l a t e a u s ccmax, a n d t h e a v e r a g e c r i t i c a l i t y f a c t o r o f e a c h i n s t a n c e . T h e s e t w o m e a s u r e s a r e p o s i t i v e l y c o r r e l a t e d ( c o r r e l a t i o n c o e f f i c i e n t r = 0 .49 ) , s u g g e s t i n g f u r t h e r t h a t f i x i n g c r i t i c a l v a r i a b l e s t o t h e i r c o r r e c t v a l u e s m a y r e d u c e t h e a t t r a c t i v i t y f a c t o r o f t h e m o s t a t t r a c t i v e t r a p , t h u s r e n d e r i n g t h e i n s t a n c e s ea s i e r . Chapter 5. Performance Critical Variables 93 Figure 5.7: Partial, weighted P C G for the hardest uf25 instance. Chapter 5. Performance Critical Variables 94 Figure 5.8: Effect on the PCG when instantiating the most critical variable in the hardest uf25 instance. Chapter 5. Performance Critical Variables 95 T o f u r t h e r t es t t h e s e c o n j e c t u r e s , w e c l o s e l y e x a m i n e t h e s e a r c h s p a c e o f t he h a r d e s t uft.5 i n s t a n c e , b o t h b e f o r e a n d a f t e r t h e P C V A w i t h t h e h i g h e s t c r i t i c a l i t y f a c t o r i s f i x e d ( the m o s t c r i t i c a l P C V A w i t h r e s p e c t to N o v e l t y 4 " h a s a c r i t i c a l i t y f a c t o r o f 2 .4 ) . F i g u r e 5.7 s h o w s t h e w e i g h t e d , p a r t i a l p l a t e a u c o n n e c t i v i t y g r a p h s f o r t h e o r i g i n a l f o r m u l a . It i s o b v i o u s w h y t h i s i n s t a n c e i s h a r d . T h e m a j o r i t y o f p a t h s w i t h s i g n i f i c a n t w e i g h t s l e a d d o w n to t h e p l a t e a u l a b e l l e d 2 . 1 . F r o m t h i s p o i n t , m o s t o f t h e p a t h s l e a d t o c l o s e d p l a t e a u s , w i t h o n l y a f e w o u t g o i n g p a t h s t ha t c a n r e a c h s o l u t i o n s . T h i s h a r d i n s t a n c e h a s s o l u t i o n r e a c h a b i l i t y p ( S | A ) = 0 . 1 5 . A f t e r t h e m o s t c r i t i c a l P C V A h a s b e e n a s s i g n e d , a n d t h e f o r m u l a c o r r e s p o n d i n g l y s i m p l i f i e d ( a l l c l a u s e s c o n t a i n i n g t h e l i t e r a l r e m o v e d f r o m t h e f o r m u l a , a n d a l l o c c u r r e n c e s o f t h e n e g a t e d l i t e r a l r e m o v e d f r o m a l l c l a u s e s ) , t h e s e a r c h s p a c e l o o k s d r a s t i c a l l y d i f f e r e n t , a s i l l u s t r a t e d i n F i g u r e 5 .8 . F i r s t l y , t h e r e a r e f e w e r p l a t e a u s a t l e v e l 1, a n d m o r e a t l e v e l 3 , a n d t h e r e a r e s i g n i f i c a n t l y f e w e r c l o s e d p l a t e a u s . T h e m o s t i n t e r -e s t i n g c h a n g e , h o w e v e r , c o n c e r n s t h e p l a t e a u c o n n e c t i v i t y . M o s t o f t h e p a t h s w i t h s i g n i f i c a n t w e i g h t s s t i l l l e a d d o w n to a l e v e l 2 p l a t e a u ( l a b e l l e d 2 . 5 ) , 1 b u t f r o m t h i s p o i n t o n t h e s i t u a t i o n i s d r a s t i c a l l y d i f f e r e n t . T h e r e a r e o n l y f o u r o u t g o i n g e d g e s f r o m p l a t e a u 2 . 5 , a n d t w o o f t h e m l e a d t o c l o s e d p l a t e a u r e g i o n s . H o w e v e r , o n e e d g e ( w i t h w e i g h t 0 .17) l e a d s d i r e c t l y t o t h e s o l u t i o n p l a t e a u a n d t h e l a s t e d g e ( a l s o h a v i n g w e i g h t 0 .17) l e a d s t o a p l a t e a u t h a t f e e d s d i r e c t l y i n t o t h e s o l u t i o n p l a t e a u . T h u s , f i x i n g t h e m o s t c r i t i c a l P C V A e f f e c t i v e l y m a d e m a n y o f t h e t r a p s m u c h l e s s a t t r a c t i v e , a n d t h e s o l u t i o n m o r e a c c e s s i b l e . T h e s o l u -t i o n r e a c h a b i l i t y o f t h e s i m p l i f i e d i n s t a n c e i s p ( S | A ) = 0 .36 , w h i c h i s n o w h i g h e r t h a n 1 5 % o f t h e o t h e r i n s t a n c e s i n t h e tes t se t . F u r t h e r m o r e , t h e s i m p l i f i e d i n s t a n c e i s n o w a t t h e 6 0 f ' ' q u a n t i l e o f t h e h a r d n e s s d i s t r i b u t i o n f o r N o v e l t y 4 " ( r a t h e r t h a n t h e 1 0 0 f , ! ) . T h e s e r e s u l t s p r o v i d e n e w i n s i g h t s i n t o w h y v a r i a b l e s a r e p e r f o r m a n c e c r i t i c a l w i t h r e s p e c t to S L S a l -g o r i t h m s . W e c o n j e c t u r e t h a t t he m o s t c r i t i c a l v a r i a b l e a s s i g n m e n t s a r e t h o s e a p p e a r i n g i n v e r y a t t r a c t i v e t r a p s , b u t h a v i n g a v a l u e t h a t i s i n c o n s i s t e n t w i t h t h e se t o f s o l u t i o n s . T h u s , t h e a t t r a c t i v e t r a p e f f e c t i v e l y r e s t r i c t s t h e g i v e n v a r i a b l e t o t h e ( i n c o r r e c t ) v a l u e c o m p a t i b l e w i t h t h e t r a p s . T h e n e x t s e c t i o n d i s c u s s e s t he p o s s i b i l i t y o f e x p l o i t i n g p e r f o r m a n c e c r i t i c a l v a r i a b l e s b a s e d o n t h i s c o n j e c t u r e . 5.5.2 A n Approximate P C V Oracle L i k e i n t h e c a s e o f b a c k d o o r se t s a n d b a c k b o n e v a r i a b l e s , i t i s u n c l e a r w h e t h e r t h e r e i s a r e a s o n a b l y e f f i c i e n t w a y o f d e t e r m i n i n g P C V s o r o f g u e s s i n g t h e m w i t h r e a s o n a b l e a c c u r a c y . B a s e d o n t h e i n t u i t i o n t h a t i n t h e c a s e o f S L S a l g o r i t h m s , t h e e x i s t e n c e o f P C V A s ( a n d h e n c e P C V s ) c o u l d b e t h e r e s u l t o f l o c a l m i n i m a a t t r a c t -i n g t he s e a r c h i n t o n o n - s o l u t i o n a r e a s o f t he r e s p e c t i v e s e a r c h s p a c e , w e p e r f o r m e d i n i t i a l e x p e r i m e n t s w i t h a s i m p l e h e u r i s t i c t h a t c o u n t e d t h e n u m b e r o f t i m e s e a c h v a r i a b l e w a s se t to T a n d _L, r e s p e c t i v e l y , i n t he l o -c a l m i n i m a e n c o u n t e r e d d u r i n g a s m a l l n u m b e r o f s h o r t i n d e p e n d e n t r u n s o f N o v e l t y 4 " o n a g i v e n i n s t a n c e , ^ote that there is no correspondence between the node ids of the two PCGs. Chapter 5. Performance Critical Variables 9 6 1 0 , 0 0.2 0.4 0.6 0.8 1 Y F i g u r e 5 .9 : C o r r e l a t i o n b e t w e e n 7 a n d c r i t i c a l i t y f a c t o r f o r N o v e l t y " * " o n i n s t a n c e b w a . T h e 7 v a l u e s w e r e c o l l e c t e d b y r u n n i n g N o v e l t y 4 " f i v e t i m e s f o r 1 0 0 s t e p s , a n d m e a s u r i n g t h e f r e q u e n c y o f l o -c a l m i n i m a i n w h i c h e a c h a t o m i c a s s i g n m e n t o c c u r r e d . E a c h p o i n t r e p r e s e n t s o n e b a c k b o n e -c o m p a t i b l e a t o m i c a s s i g n m e n t . A s i l l u s t r a t e d i n F i g u r e 5 .9 , f o r s o m e i n s t a n c e s w e o b s e r v e a n e g a t i v e c o r r e l a t i o n ( c o r r e l a t i o n c o e f f i c i e n t r = —0.80) b e t w e e n the c r i t i c a l i t y f a c t o r o f a t o m i c a s s i g n m e n t s x := v c o m p a t i b l e w i t h t h e b a c k b o n e a n d 7, t h e f r e q u e n c y o f l o c a l m i n i m a i n w h i c h x w a s a s s i g n e d v. T h u s , t h e P C V A s w i t h h i g h e s t c r i t i c a l i t y f a c t o r s t e n d e d t o h a v e l o w f r e q u e n c i e s — i n d i c a t i n g t h a t t h e c o r r e s p o n d i n g v a r i a b l e w a s se t t o t h e o p p o s i t e v a l u e i n t h e m a j o r i t y o f t h e l o c a l m i n i m a e n c o u n t e r e d d u r i n g the s h o r t r u n s . T h e n a t u r a l i n t e r p r e t a t i o n o f t h i s o b s e r v a t i o n i s t h a t , w h i l e m a n y v a r i a b l e a s s i g n m e n t s a p p e a r f r e q u e n t l y i n l o c a l m i n i m a , t h e c r i t i c a l o n e s a r e t h o s e t h a t a r e f r e q u e n t l y se t t o a v a l u e t h a t i s i n c o m p a t i b l e w i t h a n y s o l u t i o n s . T h i s a n a l y s i s s u p p o r t s o u r c o n j e c t u r e f r o m t h e l a s t s e c t i o n , a n d f u r t h e r e l u c i d a t e s t h e r e l a t i o n s h i p b e t w e e n s e a r c h s p a c e f e a t u r e s a n d a l g o r i t h m b e h a v i o u r . F i g u r e 5.9 s u g g e s t s t h a t i t m a y b e p o s s i b l e t o h e u r i s t i c a l l y d e t e r m i n e P C V s i n a n e f f i c i e n t w a y — a n d to t a k e a d v a n t a g e o f t h e m i n p r a c t i c a l a l g o r i t h m s . It m a y b e n o t e d t h a t o u r m e t h o d f o r d e t e r m i n i n g 7 v a l u e s i s s i m i l a r t h e t h e m e t h o d u s e d i n [ Z R L 0 3 ] f o r d e t e r m i n i n g p s e u d o - b a c k b o n e v a r i a b l e s . H o w e v e r , c o n t r a r y t o t h e i r r e s u l t s , w e f i n d t h a t s e t t i n g t h e s e v a r i a b l e s to t h e i r opposite values i s m o r e e f f e c t i v e t h a n s e t t i n g t h e m to t h e p r e v a l e n t v a l u e s i n l o c a l m i n i m a . U n f o r t u n a t e l y , i t i s b e y o n d t h e s c o p e o f t h i s t h e s i s to p r o v i d e a n i m p l e m e n t a t i o n a n d e v a l u a t i o n o f a p r a c t i c a l m e t h o d f o r e x p l o i t i n g P C V s . Chapter 5. Performance Critical Variables 97 5.6 Summary In this chapter w e have i n t r o d u c e d the n o t i o n of per formance c r i t i ca l va r iab les ( P C V s ) . P C V s are related to p r e v i o u s l y s t u d i e d concepts of backbone var iab les a n d m i n i m a l b a c k d o o r sets, bu t differ f r om these i n in teres t ing w a y s . B a s e d o n c o m p u t a t i o n a l exper iments w i t h severa l h igh -pe r fo rmance S A T a lgor i thms a p p l i e d to a v a r i o u s sets of b e n c h m a r k instances, w e have demons t ra t ed that a lmos t a l l instances have a s igni f icant n u m b e r of P C V s for b o t h stochastic loca l search a n d systemat ic search a lgor i thms . Pa r t i cu l a r ly for s t ruc tu red instances, w e obse rved a s ignif icant ove r l ap b e t w e e n the P C V sets for different a lgor i thms a p p l i e d to the same p r o b l e m instance as w e l l as some cor re la t ion b e t w e e n the respect ive c r i t i ca l i ty factors. We have presented ev idence for the a d d i t i v i t y of the effects of P C V s , as w e l l as for a pos i t ive cor re la t ion be tween s ize of P C V sets a n d the (algori thm-specif ic) hardness of instances. W e have also demons t ra t ed a clear connec t ion be tween per formance c r i t i ca l va r iab les a n d search space features. W e have s h o w n that the average c r i t i ca l i ty factor of an instance is cor re la ted w i t h p(S | A ) a n d Umax (c-f- Sections 4.3 a n d 4.4, respect ively) , a n d w e have i l lus t ra ted , t h r o u g h the use of p la teau connect iv-i t y g raphs , h o w the search space is affected w h e n the mos t c r i t i ca l P C V A is ins tant ia ted . Based o n these observat ions , w e conjecture that per formance c r i t i ca l i ty is the resul t of at t ract ive traps that force the cr i t ica l va r iab les to va lues that are inconsis tent w i t h a l l so lu t ions . These results raise a n u m b e r of in teres t ing quest ions. P o s s i b l y the mos t re levant of these concerns the existence of heur is t ics that app rox ima te P C V oracles i n an amor t i sed w a y a n d hence c a n be used to i m p r o v e ex i s t ing S A T a lgor i thms . We repor ted some p r e l i m i n a r y e n c o u r a g i n g results for S L S a lgor i thms , but further w o r k is r equ i r ed i n o rder to ob ta in more conc lus ive answers . O v e r a l l , w e be l ieve that a n u n d e r s t a n d i n g of the connec t ion b e t w e e n per formance c r i t i ca l i ty a n d search space features w i l l g ive rise to further interest ing results for S A T a n d other c o m b i n a t o r i a l p rob lems a n d w i l l u l t i m a t e l y l ead to i m p r o v e m e n t s i n ou r ab i l i ty to so lve these p rob lems . Chapter 6. Conclusions and Future Work 98 Chapter 6 Conclusions and Future Work In this thesis w e have s t u d i e d v a r i o u s search space proper t ies , a n d e m p i r i c a l l y demons t ra ted that these search space features are responsible for m a k i n g instances h a r d . W e u s e d the p r o p o s i t i o n a l sat isf iabi l i ty p r o b l e m as ou r m a i n a p p l i c a t i o n area because of its p rac t i ca l re levance, a n d concep tua l s i m p l i c i t y — but w e also note that mos t of the concepts a n d in tu i t ions d e v e l o p e d th roughou t the thesis general ise to other c o m b i n a t o r i a l p rob l ems . T h i s chapter summar i ses the m a i n con t r ibu t ions of the thesis, a n d p r o v i d e s inter-es t ing d i rec t ions for future w o r k . O u r con t r ibu t ions b e g i n i n C h a p t e r 3, w h e r e w e extend (and correct) resul ts f r om the l i terature concern-i n g the d i s t r i b u t i o n of p la teaus i n the search space. We f ind pla teaus that are s ign i f i can t ly larger than those p r e v i o u s l y r epor t ed — a n obse rva t ion that clears u p some apparen t ly con t r ad ic to ry results i n the l i terature. N e x t , w e ex tend the analys is of pla teaus b y i n t r o d u c i n g a c o m p u t a t i o n a l l y expens ive (but s t i l l feasible o n p r o b l e m s of a n o n t r i v i a l size) m e t h o d based o n b i n a r y d e c i s i o n d i a g r a m s for exhaus t i ve ly e x p l o r i n g the p la teau s t ructure of S A T instances. T h i s a l l o ws us to get a comple te p i c tu re of the l o w - l e v e l regions of the search space, w i t h o u t resor t ing to s a m p l i n g (as p r ev ious approaches w e r e fo rced to do) . We report a v e r y interest ing, n o v e l resul t for the u n i f o r m r a n d o m 3 - S A T instances. W e f i n d that at h i g h levels the vast major-i ty of states at a g i v e n l eve l a l l reside w i t h i n a s ingle , large, connec ted p la teau , a n d that o n l y at l o w levels does the search space rea l ly degenerate into more loca l l y connec ted regions . W e also report that c losed plateaus are t y p i c a l l y m u c h smal le r than o p e n plateaus, a n d that c losed pla teaus become v a n i s h i n g l y rare as the l e v e l is increased, i n d i c a t i n g that schemes w h i c h e x p l i c i t l y detect a n d a v o i d c losed plateaus m a y be he lp fu l . We also ex tend ou r p la teau analys is to i nc lude s t ruc tured instances. T h i s makes the results m u c h more p rac t i ca l ly re levant , s ince s t ruc tured instances are more in teres t ing f r o m a p rac t i t ioner ' s po in t of v i e w than r a n d o m l y genera ted instances, a n d strengthens the con t r i bu t i on of this thesis since m u c h of the ex is t ing l i terature focuses so le ly o n r a n d o m l y generated instances. We p e r f o r m a de ta i l ed analys is of the in te rna l s tructure of pla teaus, w h i c h has not been pe r fo rmed i n the past. W e measure p la teau proper t ies s u c h as the b r a n c h i n g factor of l oca l m i n i m a states, the p la teau diameter , the f ract ion of states i n the p la teau that are c losed , a n d the average dis tance to an exit f r om the p la teau . Chapter 6. Conclusions and Future Work 99 C h a p t e r 4 extends the results f r o m C h a p t e r 3 b y s t u d y i n g the interconnect iv i ty of plateaus i n the search space. W e demonstrate h o w w e c a n abstract a w a y irrelevant details o f the search space a n d construct a s imple , c o m p a c t m o d e l of the search space that is still representive e n o u g h to capture the features respon-sible for instance hardness . W e investigate s imple graph-theoret ic propert ies of this m o d e l , a n d discuss the consequences of these results o n a lgor i thm mobil i ty . Next , w e define a M a r k o v process o n the m o d e l , b o u n d the m i x i n g time of the M a r k o v cha in w i t h a p o l y n o m i a l (in the size of the S A T instance), a n d d e m o n -strate h o w this s impl i f i ed m o d e l of the search space can give insight into the b e h a v i o u r of S L S algori thms. F u r t h e r m o r e , w e e m p i r i c a l l y demonstrate the effectiveness of this m o d e l b y s h o w i n g h o w certain features of the m o d e l correlate w e l l w i t h instance hardness . W e also p r o v i d e a f o r m a l def ini t ion of "traps" i n the search space, a n d demonstrate that, for u n i f o r m r a n d o m 3 -SAT instances, the effects of the single mos t attractive p la teau if e n o u g h to account for the diff i -cu l ty of the instance. W e introduce a n o v e l def ini t ion of the escape diff iculty of traps, a n d demonstrate that i n the vast major i ty of cases it is detr imenta l to initialise the search f r o m the most attractive trap. Final ly , w e connect p la teau connect iv i ty w i t h H o o s ' "mixture models" , a n d p r o v i d e e m p i r i c a l ev idence that it is attractive traps that are responsible for the h a r d c o m p o n e n t of the i rregular R T D s . T h e f inal chapter descr ib ing the e m p i r i c a l analyses of the thesis, C h a p t e r 5, takes a s l ight ly different d irect ion that the p r e v i o u s chapters, a n d introduces the n o v e l concept of per formance criticality of v a r i -ables i n satisfiable S A T instances. Af ter formal ly de f in ing the no t ion of p e r f o r m a n c e criticality, w e p r o v i d e a b o u n d o n the amort isable r u n n i n g time of a perfect P C V oracle. O u r e m p i r i c a l analysis in this chap-ter demonstrates that there are a non- tr iv ia l n u m b e r of P C V for al l of the instances s tud ied , indicat ing a w i d e s p r e a d potent ia l for pract ical ly explo i t ing P C V s . W e e m p i r i c a l l y demonstrate that the fraction of P C V s that a n instance has is pos i t ive ly correlated w i t h local search cost — i.e. that h a r d instances tend to have m o r e P C V s . W e also s h o w that sets of P C V s are addi t ive i n the sense that the run- t ime i m p r o v e m e n t s realisable b y s imul taneous ly instantiat ing a n u m b e r of P C V s is amort isable . In fact, w e s h o w that i n a large n u m b e r of cases (part icularly for harder , a n d structured instances), o n l y a few P C V s n e e d be instantiated before the instance becomes comple te ly tr ivial to solve. A l s o , w e s h o w that there is a significant a m o u n t of o v e r l a p be tween the sets of P C V s w i t h respect to different a lgor i thms , ind ica t ing that some P C V s are a l g o r i t h m dependent . W e also s h o w that per formance criticality is closely related to the search space features s tud ied i n earlier chapters. In part icular , w e s h o w that the average criticality factor of a n instance is h i g h l y correlated w i t h b o t h so lut ion reachabi l i ty a n d the m a x i m u m trap attractivity of a n instance. F u r t h e r m o r e , w e demonstrate h o w the search space changes w h e n the most crit ical var iable is instantiated — not surpris ingly , this i n -creases so lut ion reachabi l i ty a n d decreases the attractivity of traps. F inal ly , w e descr ibe a m e c h a n i s m that m a y be useful for construct ing a pract ical P C V oracle. Chapter 6. Conclusions and Future Work 100 It s e e m s t h a t w i t h a n y s c i e n t i f i c e n d e a v o u r , r e s o l v i n g a f e w q u e s t i o n s r e v e a l s a h o s t o f n e w p r o b l e m s , a n d t h i s t h e s i s w a s n o e x c e p t i o n . T h e r e a r e m a n y p o s s i b l e d i r e c t i o n s i n w h i c h t h i s r e s e a r c h m a y b e e x -t e n d e d . F i r s t l y , w e n o t e a g a i n t h a t o u r a n a l y s e s t h r o u g h o u t t h e t h e s i s w e r e r e s t r i c t e d t o f a i r l y s m a l l S A T i n s t a n c e s d u e t o t he c o m p u t a t i o n a l l y e x p e n s i v e n a t u r e o f t h e e m p i r i c a l e x p e r i m e n t s b e i n g p e r f o r m e d . O n e p o s s i b l e w a y t o e x t e n d t h e a n a l y s i s t o l a r g e r i n s t a n c e s i s t o o n l y p a r t i a l l y e x p l o r e p l a t e a u s . I n f ac t , w e d e v e l -o p e d a m e t h o d f o r t h e p a r t i a l e x p l o r a t i o n o f p l a t e a u s w h i c h w o r k s b y s t o r i n g o n l y a s i n g l e s t a t e to r e p r e s e n t e a c h p l a t e a u . W h e n w e a r e s a m p l i n g s t a t e s i n o r d e r to m e a s u r e t h e d i s t r i b u t i o n o f p l a t e a u s e n c o u n t e r e d d u r i n g a s e r i e s o f S L S a l g o r i t h m r u n s , a s i n g l e s t a t e i s s t o r e d to r e p r e s e n t e a c h e n c o u n t e r e d p l a t e a u . W h e n w e n e e d t o d e t e r m i n e w h i c h e x i s t i n g p l a t e a u ( i f a n y ) a g i v e n s t a t e s b e l o n g s to , w e r u n a n u m b e r o f s i m p l e l o c a l s e a r c h a l g o r i t h m s i n p a r a l l e l , e a c h i n i t i a l i s e d to t h e p l a t e a u r e p r e s e n t a t i v e s t a t e s , r e s t r i c t e d t o s t a y at t h e s a m e l e v e l , a n d b i a s e d t o w a r d s r e d u c i n g t h e h a m m i n g d i s t a n c e t o s. I f a p a t h i s f o u n d f r o m a n y o f t h e r e p r e s e n t a t i v e s t a t e s t o s, t h e n s i s k n o w n to b e l o n g t o t h e c o r r e s p o n d i n g p l a t e a u , o t h e r w i s e w e u s e s a s t h e r e p r e s e n t a t i v e s t a t e f o r a n e w p l a t e a u . G i v e n t he se t o f r e p r e s e n t a t i v e s t a t e s , a p a r t i a l ( a p p r o x i m a t e ) P C G c a n b e c o n s t r u c t e d b y p e r t u r b i n g e a c h r e p r e s e n t a t i v e s t a t e , w h i l e r e s t r i c t i n g t h e p e r t u r b e d s t a t e t o t h e s a m e l e v e l , i n o r d e r t o s a m p l e a n u m b e r o f s t a t e s f r o m t h e p l a t e a u . A l l o f t h e l o w e r l e v e l n e i g h b o u r s o f t h e s e s t a t e s c a n b e f o u n d , a n d t h e c o n t a i n i n g p l a t e a u o f e a c h n e i g h b o u r c a n b e f o u n d u s i n g the t e c h n i q u e d e s c r i b e d a b o v e . P r e l i m i n a r y e x p e r i m e n t s i n d i c a t e t h a t t h i s t e c h n i q u e h o l d s a g r e a t d e a l o f p o t e n t i a l . W h i l e w e h a v e a t t e m p t e d t o m a k e o u r e x p e r i m e n t s a s c o m p l e t e a n d r e p r e s e n t a t i v e a s p o s s i b l e , t i m e c o n s t r a i n t s o c c a s i o n a l l y r e q u i r e d t h a t w e o n l y p e r f o r m e d o u r e x p e r i m e n t s o n a s i n g l e r e p r e s e n t a t i v e c a s e . E x a m p l e s o f t h i s i n c l u d e t h e a n a l y s i s o f t he c o n n e c t i o n b e t w e e n e s c a p e d i f f i c u l t y a n d m i x t u r e m o d e l s i n S e c t i o n 4 . 4 , t h e e f fec t o n t h e s e a r c h s p a c e o f i n s t a n t i a t i n g the P C V w i t h t h e h i g h e s t c r i t i c a l i t y f a c t o r i n S e c t i o n 5 .5 , a n d t h e e x a m p l e P C V o r a c l e h e u r i s t i c , a l s o i n S e c t i o n 5 .5 . W h i l e w e d o b e l i e v e t h e s e e x a m p l e s a r e r e p r e s e n t a t i v e , w e i n t e n d t o e x p a n d t h e s e a n a l y s e s t o m o r e i n s t a n c e s i n o r d e r t o f u l l y j u s t i f y t h e r e s p e c t i v e c l a i m s . T h e m a i n f o c u s o f t h i s t h e s i s w a s t h e a n a l y s i s o f t he r e l a t i o n s h i p b e t w e e n s e a r c h s p a c e s t r u c t u r e a n d a l g o r i t h m b e h a v i o u r , a n d w h i l e w e d i d s u g g e s t m e t h o d s o f e x p l o i t i n g t h e r e s u l t s w e w e r e n o t a b l e t o i m -p l e m e n t t h e m . S o m e d i r e c t i o n s t h a t w e b e l i e v e a r e p a r t i c u l a r l y p r o m i s i n g i n c l u d e s c h e m e s w h i c h e x p l i c i t l y d e t e c t a n d a v o i d c l o s e d p l a t e a u s , a n d t h e i n c o r p o r a t i o n o f a P C V h e u r i s t i c ( s u c h a s t h a t d e s c r i b e d i n S e c t i o n 5.5) i n t o a S A T s o l v e r . O t h e r d i r e c t i o n s f o r f u t u r e w o r k i n c l u d e t h e s t u d y o f s t r o n g p o l y n o m i a l S A T s o l v i n g t e c h n i q u e s i n t h e c o n t e x t o f p e r f o r m a n c e c r i t i c a l i t y (see , e.g., [ B W 0 3 ] ; s u c h t e c h n i q u e s c o u l d b e u s e d i n s t e a d o f u n i t p r o p a -g a t i o n a s a p r e p r o c e s s i n g s t a g e a f t e r a s s i g n i n g a t r u t h v a l u e t o a P C V ) a n d t h e i n v e s t i g a t i o n o f t h e s c a l i n g o f P C V se t s i z e a n d a v e r a g e c r i t i c a l i t y f a c t o r s o f P C V s w i t h p r o b l e m s i z e f o r f a m i l i e s o f s t r u c t u r e d a n d r a n d o m i n s t a n c e s . I t m a y a l s o b e i n t e r e s t i n g t o s t u d y t h e c o n c e p t o f P C V s f o r u n s a t i s f i a b l e S A T i n s t a n c e s ; Chapter 6. Conclusions and Future Work 101 t h i s r e q u i r e s a s l i g h t l y d i f f e r e n t d e f i n i t i o n , b u t s i m i l a r i n t u i t i o n s a n d a n a l y s i s t e c h n i q u e s a p p l y . F i n a l l y , i t s h o u l d b e n o t e d t h a t t h e c o n c e p t o f p e r f o r m a n c e c r i t i c a l i t y c a n b e e a s i l y a p p l i e d to o t h e r c o m b i n a t o r i a l p r o b l e m s . T h r o u g h o u t t h i s t h e s i s , w e h a v e u s e d S A T to i n t r o d u c e a n d s t u d y t h e s e c o n c e p t s b e c a u s e o f i t s c o n c e p t u a l s i m p l i c i t y a n d i t s p r o m i n e n t s t a t u s i n t h e t h e o r e t i c a l a n d e m p i r i c a l s t u d y o f a l -g o r i t h m i c c o m p l e x i t y a n d c o m b i n a t o r i a l p r o b l e m s o l v i n g . H o w e v e r , w e b e l i e v e i t w i l l b e f r u i t f u l to s t u d y t h e c o n c e p t o f p e r f o r m a n c e c r i t i c a l i t y f o r o t h e r c o m b i n a t o r i a l p r o b l e m s , p a r t i c u l a r l y f o r w i d e l y s t u d i e d o p t i m i s a t i o n p r o b l e m s , s u c h as M A X - S A T , T S P , a n d m o r e g e n e r a l c o n s t r a i n t s a t i s f a c t i o n p r o b l e m s . It i s o u r s t r o n g b e l i e f t h a t i n o r d e r t o d e s i g n h i g h - p e r f o r m a n c e S L S a l g o r i t h m s f o r S A T , i t i s n e c e s s a r y t o u n d e r s t a n d , a s c o m p l e t e l y as p o s s i b l e , p r o p e r t i e s o f t h e s p a c e w h i c h i s b e i n g s e a r c h e d . T h i s t h e s i s c o n t r i b u t e s to t h i s u n d e r s t a n d i n g , a n d i t i s o u r h o p e t h a t t h e c o n c e p t s d i s c u s s e d w i t h i n t h i s t h e s i s w i l l p r o v i d e i n s i g h t s l e a d i n g t o i m p r o v e m e n t s i n t h e s t a t e - o f - t h e - a r t i n S A T s o l v i n g . Bibliography 1 0 2 Bibliography [ B r y 9 2 ] R . E . B r y a n t . S y m b o l i c b o o l e a n m a n i p u l a t i o n w i t h o r d e r e d b i n a r y d e c i s i o n d i a g r a m s . ACM Computing Surveys, 2 4 ( 3 ) , 1 9 9 2 . [ B S 0 3 ] D . L e B e r r e a n d L . S i m o n . T h e e s s e n t i a l s o f t he S A T ' 0 3 c o m p e t i t i o n . I n Submitted to LNAI. J u n e 2 0 0 3 . [ B W 0 3 ] F a h i e m B a c c h u s a n d J o n a t h a n W i n t e r . E f f e c t i v e p r e p r o c e s s i n g w i t h h y p e r - r e s o l u t i o n a n d e q u a l i t y r e d u c t i o n . I n Proc. SAT 2003, p a g e s 1 8 3 - 1 9 2 , 2 0 0 3 . [ C A 9 6 ] J a m e s M . C r a w f o r d a n d L a r r y D . A u t o n . E x p e r i m e n t a l r e s u l t s o n t h e c r o s s o v e r p o i n t i n r a n -d o m 3 - S A T . Artificial Intelligence Journal, 8 1 ( l - 2 ) : 3 1 - 5 7 , 1 9 9 6 . [ C F G + 9 6 ] D . C l a r k , J . F r a n k , I. G e n t , E . M a c l n t y r e , N . T o m o v , a n d T . W a l s h . L o c a l s e a r c h a n d the n u m b e r o f s o l u t i o n s . I n E . F r e u d e r , e d i t o r , Principles and Practice of Constraint Programming - CP'96, v o l -u m e 1 1 1 8 o f Lecture Notes in Computer Science, p a g e s 1 1 9 - 1 3 3 . S p r i n g e r V e r l a g , B e r l i n , G e r m a n y , 1 9 9 6 . [ C K T 9 1 ] P e t e r C h e e s e m a n , B o b K a n e f s k y , a n d W i l l i a m M . T a y l o r . W h e r e t h e r e a l l y h a r d p r o b l e m s a r e . I n Proceedings of the Twelfth International Joint Conference on Artificial Intelligence (IJCAI-91), p a g e s 3 3 1 - 3 3 7 , 1 9 9 1 . [ D D 0 1 ] O l i v i e r D u b o i s a n d G i l l e s D e q u e n . A b a c k b o n e - s e a r c h h e u r i s t i c f o r e f f i c i e n t s o l v i n g o f h a r d 3 - S A T f o r m u l a e . I n Proc. IJCAI-01, p a g e s 2 4 8 - 2 5 3 , 2 0 0 1 . [ D L L 6 2 ] M a r t i n D a v i s , G e o r g e L o g e m a n n , a n d D o n a l d L o v e l a n d . A m a c h i n e p r o g r a m f o r t h e o r e m p r o v i n g . Communications of the ACM, 5 ( 7 ) : 3 9 4 - 3 9 7 , J u l y 1 9 6 2 . [ d l T 9 0 ] T . B o y d e l a T o u r . M i n i m i z i n g t he n u m b e r o f c l a u s e s b y r e n a m i n g . I n Proc. of the 10th Conference on Automated Deduction, p a g e s 5 5 8 - 5 7 2 , 1 9 9 0 . [ F C S 9 7 ] J e r e m y F r a n k , P e t e r C h e e s e m a n , a n d J o h n S t u t z . W h e n g r a v i t y f a i l s : L o c a l s e a r c h t o p o l o g y . Journal of Artificial Intelligence Research, 7 : 2 4 9 - 2 8 1 , 1 9 9 7 . Bibliography 103 [ G J 7 9 ] M . R . G a r e y a n d D . B . J o h n s o n . Computers and Intractability. A Guide to the Theory of NP-Completeness. W . H . F r e e m a n , N e w Y o r k , 1 9 7 9 . [ G S K 9 8 ] C a r l a P. G o m e s , B a r t S e l m a n , a n d H e n r y K a u t z . B o o s t i n g c o m b i n a t o r i a l s e a r c h t h r o u g h r a n -d o m i z a t i o n . I n Proc. AAAI-98, p a g e s 4 3 1 ^ 3 7 , M a d i s o n , W i s c o n s i n , 1 9 9 8 . [ G W 9 3 ] I a n G e n t a n d T o b y W a l s h . A n e m p i r i c a l a n a l y s i s o f s e a r c h i n g s a t . Journal of Artificial Intelligence Research, 1 : 4 7 - 5 9 , 1 9 9 3 . [ G W 9 4 ] I a n G e n t a n d T o b y W a l s h . T h e S A T p h a s e t r a n s i t i o n . I n Proceedings of the Eleventh European Conference on Artificial Intelligence (ECAI'94), p a g e s 1 0 5 - 1 0 9 , 1 9 9 4 . [ G W 9 5 ] I a n G e n t a n d T o b y W a l s h . U n s a t i s f i e d v a r i a b l e s i n l o c a l s e a r c h . I n A m s t e r d a m J . H a l l a m , I O S P r e s s , e d i t o r , Hybrid Problems, Hybrid Solutions, p a g e s 7 3 - 8 5 , 1 9 9 5 . [ H K 9 5 ] S . H a m p s o n a n d D . K i b l e r . P l a t e a u s a n d p l a t e a u s e a r c h i n b o o l e a n s a t i s f i a b i l i t y p r o b l e m s : W h e n to g i v e u p s e a r c h i n g a n d s t a r t a g a i n . I n DIMACS Challenge '95,1995. [ H o f 8 5 ] D o u g l a s R . H o f s t a d t e r . Metamagical Themas: Questing for the Essence of Mind and Pattern, c h a p t e r 13 : M e t a f o n t s , M e t a m a t h e m a t i c s , a n d M e t a p h y s i c s . B a s i c B o o k s , 1 9 8 5 . [ H o o 9 8 ] H o l g e r H . H o o s . Stochastic Local Search - Methods, Models, Applications. P h D t h e s i s , T e c h n i s c h e U n i v e r s i t a t D a r m s t a d t , 1 9 9 8 . [ H o o 9 9 ] H o l g e r H . H o o s . O n the r u n - t i m e b e h a v i o u r o f s t o c h a s t i c l o c a l s e a r c h a l g o r i t h m s f o r S A T . I n Proc. AAA1-99, p a g e s 6 6 1 - 6 6 6 , 1 9 9 9 . [ H o o 0 2 ] H o l g e r H . H o o s . A m i x t u r e - m o d e l f o r t h e b e h a v i o u r o f s i s a l g o r i t h m s f o r S A T . I n In Proc. AAAI'02, p a g e s 6 6 1 - 6 6 7 , 2 0 0 2 . [ H S 9 8 ] H o l g e r H . H o o s a n d T h o m a s S t i i t z l e . E v a l u a t i n g l a s v e g a s a l g o r i t h m s - p i t f a l l s a n d r e m e -d i e s . I n Proceedings of the 14th Annual Conference on Uncertainty in Artificial Intelligence (UAI-98), p a g e s 2 3 8 - 2 4 5 , S a n F r a n c i s c o , C A , 1 9 9 8 . M o r g a n K a u f m a n n P u b l i s h e r s . [ H S 0 0 ] H o l g e r H . H o o s a n d T h o m a s S t i i t z l e . S A T L I B : A n o n l i n e r e s o u r c e f o r r e s e a r c h o n S A T . I n SAT 2000. I O S P r e s s , A m s t e r d a m , T h e N e t h e r l a n d s , 2 0 0 0 . [ H S 0 4 ] H o l g e r H . H o o s a n d T h o m a s S t i i t z l e . Stochastic Local Search—Foundations and Applications. M o r g a n K a u f m a n n P u b l i s h e r s , U S A , 2 0 0 4 . [ H T H 0 2 ] F r a n k H u t t e r , D a v e A . D . T o m p k i n s , a n d H o l g e r H . H o o s . S c a l i n g a n d p r o b a b i l i s t i c s m o o t h i n g : E f f i c i e n t d y n a m i c l o c a l s e a r c h f o r S A T . I n Proc. CP 2002, p a g e s 2 3 3 - 2 4 8 , 2 0 0 2 . Bibliography 104 [ In t03] Y . I n t e r i a n . B a c k d o o r se t s f o r r a n d o m 3 - S A T . I n Proc. SAT 2003, p a g e s 2 3 1 - 2 3 8 , 2 0 0 3 . [ K K R R 9 2 ] A . P . K a m a t h , N . K . K a r m a r k a r , K . G . R a m a k r i s h n a n , a n d M . G . C . R e s e n d e . A c o n t i n u o u s a p -p r o a c h t o i n d u c t i v e i n f e r e n c e . Mathematical Programming, 5 7 : 2 1 5 - 2 3 8 , 1 9 9 2 . [ K M S 9 6 ] H e n r y K a u t z , D a v i d M c A l l e s t e r , a n d B a r t S e l m a n . E n c o d i n g p l a n s i n p r o p o s i t i o n a l l o g i c . I n Proc. KR'96, p a g e s 3 7 4 - 3 8 4 , 1 9 9 6 . [ K S 9 6 ] H e n r y K a u t z a n d B a r t S e l m a n . P u s h i n g the e n v e l o p e : P l a n n i n g , p r o p o s i t i o n a l l o g i c , a n d s t o c h a s t i c s e a r c h . I n Proc. AAAI'96, p a g e s 1 1 9 4 - 1 2 0 1 , 1 9 9 6 . [ K S 0 3 ] H e n r y K a u t z a n d B a r t S e l m a n . T e n c h a l l e n g e s r e d u x : R e c e n t p r o g r e s s i n p r o p o s i t i o n a l r e a s o n -i n g a n d s e a r c h . I n Ninth International Conference on Principles and Practice of Constraint Program-ming (CP 2003), 2 0 0 3 . [ M M Z + 0 1 ] M a t t h e w W . M o s k e w i c z , C o n o r F . M a d i g a n , Y i n g Z h a o , L i n t a o Z h a n g , a n d S h a r a d M a l i k . C h a f f : E n g i n e e r i n g a n E f f i c i e n t S A T S o l v e r . I n Proc. DAC'01,2001. [ M S G 9 7 ] B . M a z u r e , L . S a i s , a n d E . G r e g o i r e . T a b u S e a r c h f o r S A T . I n Proceedings of the 14th National Conference on Artificial Intelligence, p a g e s 2 8 1 - 2 8 5 . A A A I P r e s s / T h e M I T P r e s s , M e n l o P a r k , C A , U S A , 1 9 9 7 . [ M S K 9 7 ] D a v i d A . M c A l l e s t e r , B a r t S e l m a n , a n d H e n r y A . K a u t z . E v i d e n c e f o r i n v a r i a n t s i n l o c a l s e a r c h . I n Proc. AAAI-97, p a g e s 3 2 1 - 3 2 6 , 1 9 9 7 . [ M T 0 0 ] P. M i l l s a n d E . T s a n g . G u i d e d l o c a l s e a r c h f o r s o l v i n g S A T a n d w e i g h t e d M A X - S A T p r o b l e m s . I n I .P. G e n t , H . v a n M a a r e n , a n d T . W a l s h , e d i t o r s , SAT2000—Highlights of Satisfiability Research in the Year 2000, p a g e s 8 9 - 1 0 6 . 2 0 0 0 . [ N L B D + 0 4 ] E . N u d e l m a n , K . L e y t o n - B r o w n , A . D e v k a r , Y . S h o h a m , a n d H . H o o s . U n d e r s t a n d i n g r a n d o m sat : B e y o n d t h e c l a u s e s - t o - v a r i a b l e s r a t i o . I n International Conference on Principles and Practice of Constraint Programming (CP), 2 0 0 4 . [ P a p 9 1 ] C h r i s t o s H . P a p a d i m i t r i o u . O n s e l e c t i n g a s a t i s f y i n g t r u t h a s s i g n m e n t ( e x t e n d e d a b s t r a c t ) . I n Proceedings of the 32nd annual symposium on Foundations of computer science, p a g e s 1 6 3 - 1 6 9 . I E E E C o m p u t e r S o c i e t y P r e s s , 1 9 9 1 . [ P a r 9 7 ] A n d r e w J . P a r k e s . C l u s t e r i n g a t t he p h a s e t r a n s i t i o n . I n Proceedings of the 14th National Con-ference on Artificial Intelligence, p a g e s 3 4 0 - 3 4 5 . A A A I P r e s s / T h e M I T P r e s s , M e n l o P a r k , C A , U S A , 1 9 9 7 . Bibliography 105 [ P G 8 6 ] D . A . P l a i s t e d a n d S . G r e e n b a u m . A s t r u c t u r e p r e s e r v i n g c l a u s e f o r m t r a n s l a t i o n . Journal of Symbolic Computation, 2 : 2 9 3 - 3 0 4 , 1 9 8 6 . [ P S 8 2 ] C h r i s t o s H . P a p a d i m i t r i o u a n d K e n n e t h S t e i g l i t z . Combinatorial optimization: algorithms and complexity. P r e n t i c e H a l l , 1 9 8 2 . [ S B H 0 2 ] L a u r e n t S i m o n , D a n i e l L e B e r r e , a n d E d w a r d A . H i r s c h . T h e S A T 2 0 0 2 c o m p e t i t i o n , 2 0 0 2 . [ S c h 9 9 ] U w e S c h o n i n g . A p r o b a b i l i s t i c a l g o r i t h m f o r k - s a t a n d c o n s t r a i n t s a t i s f a c t i o n p r o b l e m s . I n Proceedings of the 40th Annual Symposium on Foundations of Computer Science, p a g e 4 1 0 . I E E E C o m p u t e r S o c i e t y , 1 9 9 9 . [ S G S 0 0 ] J . S i n g e r , I. G e n t , a n d A . S m a i l l . B a c k b o n e f r a g i l i t y a n d t h e l o c a l s e a r c h c o s t p e a k . Journal of Artificial Intelligence Research, p a g e s 2 3 5 - 2 7 0 , 2 0 0 0 . [ S K 9 3 ] B . S e l m a n a n d H . A . K a u t z . D o m a i n - i n d e p e n d e n t e x t e n s i o n s t o g s a t : S o l v i n g l a r g e s t r u c t u r e d s a t i s f i a b i l i t y p r o b l e m s . I n Proc. IJCAI-93,1993. [ S K C 9 3 ] B . S e l m a n , H . A . K a u t z , a n d B . C o h e n . L o c a l s e a r c h s t r a t e g i e s f o r s a t i s f i a b i l i t y t e s t i n g . I n Proc. 2nd DIMACS Challenge on Cliques, Coloring, and Satisfiability, 1 9 9 3 . [ S K C 9 4 ] B . S e l m a n , H . A . K a u t z , a n d B . C o h e n . N o i s e s t r a t e g i e s f o r i m p r o v i n g l o c a l s e a r c h . I n Proc. AAAI-94,1994. [ S L M 9 2 ] B . S e l m a n , H . J . L e v e s q u e , a n d D . G . M i t c h e l l . A n e w m e t h o d f o r s o l v i n g h a r d s a t i s f i a b i l i t y p r o b l e m s . I n Proc AAAI-92,1992. [ S W 9 7 ] Y . S h a n g a n d B . W . W a h . D i s c r e t e l a g r a n g i a n - b a s e d s e a r c h f o r s o l v i n g M A X - S A T p r o b l e m s . I n Proceedings of the 15th International Joint Conference on Artificial Intelligence, v o l u m e 1, p a g e s 3 7 8 - 3 8 3 . M o r g a n K a u f m a n n P u b l i s h e r s , S a n F r a n c i s c o , C A , U S A , 1 9 9 7 . [ S W 0 2 ] J o h n S l a n e y a n d T o b y W a l s h . P h a s e t r a n s i t i o n b e h a v i o r : f r o m d e c i s i o n t o o p t i m i z a t i o n . I n In the Proceedings of the Fifth International Conference on Theory and Applications of Satisfiability Testing (SAT 2002), 2 0 0 2 . [ T H 0 4 ] D a v e A . D . T o m p k i n s a n d H o l g e r H . H o o s . U B C S A T : A n i m p l e m e n t a t i o n a n d e x p e r i m e n t a t i o n e n v i r o n m e n t f o r S L S a l g o r i t h m s f o r S A T a n d M A X - S A T . I n In the Proceedings of the Seventh International Conference on Theory and Applications of Satisfiability Testing (SAT 2004), p a g e s 3 7 -4 6 , 2 0 0 4 . [ V a l 7 9 ] L . V a l i a n t . T h e c o m p l e x i t y o f e n u m e r a t i o n a n d r e l i a b i l i t y p r o b l e m s . SIAM Journal on Comput-ing, 8 ( 3 ) : 4 1 0 - 4 2 1 , 1 9 7 9 . Bibliography 106 [WGS03] Ryan Williams, Carla Gomes, and Bart Selman. Backdoors to typical case complexity. In Proc. IJCAI-03, Acaplco, Mexico, 2003. [Yok97] Makoto Yokoo. W h y adding more constraints makes a problem easier for hill-climbing algo-rithms: Analysing landscapes of csp's. In Proc. CP'97, volume 1330 of LNCS, pages 357-370, 1997. [ZRL03] W. Zhang, A . Rangan, and M . Looks. Backbone guided local search for maximum satisfiability. In Proc. IJCAI-03, pages 1179-1184,2003. Appendix A. Test Suite Description 107 Appendix A Test Suite Description The empirical nature of this thesis required that an interesting set of problem instances be selected for study. Choosing a representative set of SAT instances for an empirical study is a daunting, if not impossible, task. The instances must represent interesting problems that have been studied in the past and/or are relevant to practical applications. However, there are always restrictions on the number and size of instances that can be studied. Many of the empirical experiments performed in this thesis were exceedingly computationally inten-sive, both wi th respect to computation time and storage. Because of this, we were restricted to fairly small SAT instances that are considered quite easy by modern standards. Nonetheless, we hope that even the study of small instances is interesting, because many of the observations and intuitions gained by studying small instances generalise to larger instances. When possible, we studied larger instances and contrasted the results wi th those for the smaller instances. Even after restricting the test sets to small instances, some of the experiments still took multiple C P U months to complete. We study two main class of instances: random and structured. The random instances are interesting because they have been extensively studied in the past, so there is a wealth of literature to contrast results with. Another advantage of studying randomly generated instances is that distributions of syntactically identical instances can be generated and studied. The structured instances are generally much less avail-able, and must be studied on and instance-by-instance basis. One exception to this rule is that we may generate random distributions of instances from other problem domains, such as graph colouring, and then encode these instances into propositional satisfiability. The random instances that we study come from the Uniform Random 3-SAT distribution, near the phase-transition region [CKT91]. We generally refer to this entire class of instances as UF-3-SAT. Table A . l shows the UF-3-SAT test sets that we study. The first collection of test sets, named uf20-uf50, were generated explicitly for this thesis. There are 100 instances in each test set, and the number of variables ranges from 20 to 50 in steps of 5. The clause to variable ratio in each test set is determined by Crawford and Auton's formula C/V = 4.258 + 58.26i> - 5 / 3 , which was empirically demonstrated to fit the phase transition region well , even for small instances [CA96]. We also used four set of larger instances, taken from SATLIB [HS00]. These sets, wi th 20,50,100, and 200 variables, also have a clause to variable ratio near the phase transition, Appendix A. Test Suite Description 108 Benchmark n m # ufZO 20 93 100 uf25 25 113 100 uftO 30 133 100 uf35 35 154 100 uf40 40 175 100 uf45 45 196 100 uflO 20 91 1000 uf50 50 218 1000 uflOO 100 430 1000 uflOO 200 860 1000 Table A . l : Description of the random benchmark sets studied in this paper; n, m, and # are the number of variables, clauses and instances, respectively, in each benchmark set. Benchmark n m # U8 (66-1068) (186-8214) 14 anomaly 48 261 1 medium 116 953 1 huge 459 7054 1 bwa 459 4675 1 loga 828 6718 1 a i s6 61 581 1 f l a t 2 0 - e a s y 60 116 1 f la t20-med 60 116 1 f l a t 2 0 - h a r d 60 116 1 p a r 8 - l - c 64 254 1 ssa7552-038 1501 3575 1 Table A.2: Description of the structured benchmark sets studied in this paper; n, m, and # are the number of variables, clauses and instances, respectively, in each benchmark set. Note that most of the "sets" actually consist of a single instance. but the actual number of clauses was determined empirically for each set by experimentally determining the point at which approximately 50% of the instances in the set were satisfiable. Choosing interesting structured instances proved to be more challenging. We needed small instances, which were hard to come by and so already limited the set of potential instances. We also wanted instances syntactically different instances encoded from different problems. Table A.2 shows the structured instances used in the thesis, which are all available for download from SATLIB [HS00]. The U8 set consists of 14 in-stances of various sizes, and are from a SAT formulation of Boolean function synthesis problems [KKRR92]. The anomaly , medium, huge, and bwa sets al l consist of a single instance, and correspond to SAT-encoded instances of the well-known blocks wor ld problem domain. The l o g a instance is from a SAT formulation of a logistical planning problem. A more detailed description of the problem domains and the SAT encod-Appendix A. Test Suite Description 109 ing used to generate these instances can be found in [KS96, KMS96]. The a i s 6 instance a SAT formulation of the all-interval-series problem, inspired by a well-known problem in serial music composition [Hoo98]. The f l a t 2 0 - e a s y , f l a t 2 0 - m e d , and f l a t 2 0 - h a r d are SAT-encoded instances of the graph colouring problem. To obtain these instances, 1000 graph colouring instances were generated using Joe Culberson's flat graph generator,1 each having 20 nodes, 3 colours, and a connectivity of 0.245. The graph colouring problems were then converted into propositional formulae using Joe Culberson's converting utility. These three instances are those wi th the lowest, median, and maximum local search cost for the G W S A T algorithm (cf. Section 2.3) from the entire distribution of instances. The p a r 8 - l - c instance is a propositional version of a parity learning problem. 2 Finally, the s s a 7 5 5 2 - 0 3 8 instance is from a hardware verification problem testing for single-at-stuck faults. 3 There are quite a number of syntactically different structured instances represented in our test sets, though the number of instances is fairly low. Combined wi th the large number of randomly generated instances, we are confident that we have chosen an interesting set of instances for study, which are still small enough for our empirical experiments to be computationally feasible. Available online at http: / /web. cs .ualberta. ca/~joe/Coloring/Generators/f lat .html 2More details regarding the problem and the propositional encoding may be found on the DIMACS website ftp: / /dimacs . rutgers.edu/pub/challenge/satisfiability/contributed/crawford/README. 3More details regarding the problem and the propositional encoding may be found on the DIMACS website ftp: / /dimacs. rutgers.edu/pub/challenge/satisfiability/contributed/UCSC/instances/cnf-ssa/README. Appendix B. Binary Decision Diagrams 110 Appendix B Binary Decision Diagrams Reduced Ordered Binary Decision Diagrams (ROBDDs, or BDDs for short) are a data structure useful for compactly representing Boolean functions using Directed Acyclic Graphs (DAGs). They represent func-tions canonically and efficiently (both in terms of space and time complexity), making them popular in practical applications. For a thorough introduction to BDDs, the reader is referred to [Bry92]. For all of the experiments conducted in this thesis, we used the popular B D D package C U D D version 2.4.0, available online at h t t p : / / v l s i . Colorado . e d u / ~ f abio/CUDD. Throughout this thesis, we use BDDs to store very large sets of (related) search space states. In order to do this, we assume that states are binary vectors of the form X = (x\, xi, • • • , x„) (where n is the number of propositional variables). Such a vector can be interpreted as the input to a Boolean function / : {0,1}" t—• {0,1}. Thus, we can represent a set of states with such a Boolean function by adopting the convention that a state X is in the set defined by / if and only if / ( X ) = 1 {i.e. the output is interpreted as true). By using a B D D to represent / , we can efficiently store and manipulate the corresponding set of binary states. Figure B . l illustrates the relationship between Boolean functions, binary trees, and binary decision dia-grams. Each of the three data structures represents the same set of three-variable states {Oil , 110, 111},but the B D D does so using the least amount of space. The B D D uses two rules to reduce the number of nodes Figure B . l : Comparing three different representations of the same Boolean function. Left: Truth Table, Centre: Binary Tree, Right: Binary Decision Diagram. In the diagrams, dashed edges symbolise a value of 0 being assigned to the source variable, while solid edges symbolise a value of 1. Appendix B. Binary Decision Diagrams 111 u s e d . F i r s t , i f b o t h o f t h e o u t g o i n g e d g e s o f a n o d e n\. p o i n t t o t h e s a m e n o d e ni, t h e n n\ i s r e d u n d a n t a n d c a n b e r e m o v e d a n d a n y i n c o m i n g e d g e s c a n b e p o i n t e d d i r e c t l y a t « 2 - S e c o n d , e q u i v a l e n t n o d e s m u s t b e r e m o v e d . I n a d d i t i o n t o t h e s p a c e s a v i n g s , a f u r t h e r a d v a n t a g e o f B D D s i s t h a t i t i s p o s s i b l e ( a n d e f f i c i e n t ) t o c a l c u l a t e t h e i n t e r s e c t i o n a n d u n i o n o f se t s o f s t a t e s s i m p l y b y u s i n g t h e B D D and (A) a n d or o p e r a t i o n s . T h e s e o p e r a t i o n s w e r e c r u c i a l f o r e f f i c i e n t l y c a l c u l a t i n g Levelsetk a n d p a r t i t i o n i n g t h e se t i n t o i n d i v i d u a l p l a t e a u s ( r e c a l l t h a t b o t h Levelsetk a n d p l a t e a u s w e r e s t o r e d u s i n g B D D s ) . T h e p s e u d o - c o d e o f A l g o r i t h m s 3.1 a n d 3.2 r e f e r e n c e n o n - s t a n d a r d B D D o p e r a t i o n s t h a t w e w i l l n o w d e f i n e . BDDSatSet(LiteralSet c) t a k e s a s i n p u t a se t o f l i t e r a l s ( i n t e r p r e t e d a s a d i s j u n c t i o n , f o r m i n g a c l a u s e ) a n d r e t u r n s t h e se t o f s t a t e s c o n s i s t e n t w i t h t h e i n p u t c l a u s e . O f c o u r s e , t h e o u t p u t i s r e t u r n e d as a B D D . H e n c e , t h i s f u n c t i o n i s t r i v i a l : w e s i m p l y b u i l d a s m a l l B D D r e p r e s e n t i n g t h e d i s j u n c t i o n ( i t s h o u l d b e e a s y t o s e e t h a t t h e n u m b e r o f n o d e s i n s u c h a B D D m u s t b e l i n e a r i n t h e n u m b e r o f l i t e r a l s ) . T h e m i n t e r m s o f t h i s f u n c t i o n r e p r e s e n t t h e d e s i r e d s t a t e s . BDDPickOneCube(BDD f) r e t u r n s a n a r b i t r a r y c u b e o f t h e i n p u t B D D . A c u b e i s s i m p l y a c o n j u n c t i o n o f l i t e r a l s ( a l l v a r i a b l e s d o n o t n e c e s s a r i l y n e e d t o b e r e p r e s e n t e d i n t h e c o n j u n c t i o n ) t h a t i s c o n s i s t e n t w i t h a t l e a s t o n e m i n t e r m o f t h e B D D . I n o u r c o n t e x t , a c u b e i s a v e r y c o m p a c t r e p r e s e n t a t i o n o f a s u b s e t o f t h e s t a t e s r e p r e s e n t e d b y t h e o r i g i n a l B D D . F i n d i n g a c u b e o f a B D D i s s i m p l e ; s i m p l y f i n d a p a t h t o t h e n o d e l a b e l l e d "1", a n d t h e n c o n s t r u c t a n e w B D D o u t o f a l l l i t e r a l s a p p e a r i n g a l o n g t h a t p a t h . F i n a l l y , BDDNegate(BDD f,int var) e x c h a n g e s t h e o u t g o i n g e d g e s o f a l l n o d e s a s s o c i a t e d w i t h v a r i a b l e var. T h i s h a s t h e e f fec t o f s i m u l t a n e o u s l y f l i p p i n g v a r i a b l e var i n a l l o f t h e s t a t e s i n t h e i m p l i c i t l y r e p r e s e n t e d set . 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0051395/manifest

Comment

Related Items