UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Automating physical reorganizational requirements at the access path level of a relational database management.. 1980

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
UBC_1980_A6_7 W44.pdf [ 6.65MB ]
UBC_1980_A6_7 W44.pdf
Metadata
JSON: 1.0051815.json
JSON-LD: 1.0051815+ld.json
RDF/XML (Pretty): 1.0051815.xml
RDF/JSON: 1.0051815+rdf.json
Turtle: 1.0051815+rdf-turtle.txt
N-Triples: 1.0051815+rdf-ntriples.txt
Citation
1.0051815.ris

Full Text

AUTOMATING PHYSICAL REORG ANIZATIONAL BEQUIEEMENTS AT THE ACCESS PATH LEVEL OF A RELATIONAL DATABASE MANAGEMENT SYSTEM by GRANT EDWIN WEDDELL B.Sc*, U n i v e r s i t y of B r i t i s h Columbia, 1976 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF THE FACULTY OF GRADUATE STUDIES (Department of Computer Science) We accept t h i s t h e s i s as conforming t o the r e q u i r e d standard. THE UNIVERSITY OF BRITISH COLUMBIA MASTER OF SCIENCE i n February, 1980 1980 > In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of Brit ish Columbia, I agree that the Library shall make i t freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the Head of my Department or by his representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of C^^^/er J c ^ e e . The University of Brit ish Columbia 2075 Wesbrook Place Vancouver, Canada V6T 1W5 Date ) E-6 B P 75-5 1 I E A b s t r a c t Any design of an access path l e v e l of a database management system must make allowance f o r p h y s i c a l r e o r g a n i z a t i o n requirements* The f a c i l i t i e s p r o v i d e d f o r such requirements at the access path l e v e l have so f a r been p r i m i t i v e i n nature (almost always, i n f a c t , r e q u i r i n g c omplicated human i n t e r v e n t i o n ) T h i s t h e s i s begins t o e x p l o r e the n o t i o n of i n c r e a s i n g t h e degree of automation of such requirements a t the access path l e v e l ; t o c o n s i d e r the p r a c t i c a l b a s i s f o r s e l f - a d a p t i n g or s e l f - o r g a n i z i n g data management systems* C o n s i d e r a t i o n i s f i r s t g i v e n to the mo t i v a t i o n ( j u s t i f i c a t i o n ) o f such a n o t i o n . Then, based on a review o f the r e l e v e n t a s p e c t s of a number of e x i s t i n g data management systems, we present a complete design s p e c i f i c a t i o n and o u t l i n e f o r a proposed access path l e v e l . Regarding t h i s system we c o n s i d e r i n d e t a i l the automation o f two major asp e c t s of p h y s i c a l o r g a n i z a t i o n : the c l u s t e r i n g of records on mass storage media and the s e l e c t i o n of secondary i n d i c e s . The r e s u l t s o f our a n a l y s i s o f these problems p r o v i d e s a b a s i s f o r the u l t i m a t e demonstration of f e a s i b i l i t y of such automation* Table o f Contents Chapter I. I n t r o d u c t i o n 1 Chapter I I : P r e l i m i n a r i e s ................................. 8 A. J u s t i f i c at i o n F o r Automation ............<,............ 8 B. P r e v i o u s Systems .................................... 12 C. D e t a i l e d Eroblem D e f i n i t i o n .........................30 D. S i m p l i f y i n g Assumptions ............................. 45 Chapter I I I : The System .................................... 48 A. O v e r a l l Design 48 B. The Primary Index .... . . . . . . . . . . . . . . . . . . . o...........64 C. D i s c r i m i n a t o r S e l e c t i o n .....^....................... 84 D. .Secondary Indexing ..................................95 E. .Search S t r a t e g i e s .109 Chapter IV: Future D i r e c t i o n s And Con c l u s i o n s ........... 122 A. The Next Step .............122 B. C o n c l u s i o n s ...................127 References ..............................................131 Appendix A: Base R e l a t i o n s Osed By SORAAM .. 139 - i i - L i s t o f Tables II-1..BNF Syntax Of Legal Search P r e d i c a t e s For The ESS Of System R ....................................24 II-2* Example E f f e c t Of Change In Use Of A F o l i o In SODMS o........... 2 8 I I - 3. C l u s t e r Performance R e l a t i v e To A Query Group ..,,,,3 9 I I I - L i . . S p e c i a l R a l a t i c n s Used By SOEAAM .................. 54 I I I - 2 . Scan Cost Of A Base R e l a t i o n V ia A Secondary Index k ............. 116 - i i i L i s t o f F i g u r e s I I - 1 . A r c h i t e c t u r e Of System R ........................... 20 I I - 2 . H o r i z o n t a l Vs V e r t i c a l C l u s t e r i n g ....36 I I - 3. Example H o r i z o n t a l C l u s t e r i n g ,..,,.........38 I I I - 1. Storage S t r u c t u r e Of A Base R e l a t i o n In SOEAAM .,,,52 I I I - 2 . Example Operation Trees On SORAAH ................. 63 I I I - 3 . A K-d Tree Index To " c l u s t e r 3 " (from F i g u r e II-3) .70 II I - U . S orted Order I n s e r t i o n In A K-d Tree ..,.,,........73 I I I - 5 . Example Base R e l a t i o n With Primary And Secondary Indexing .............. ................................9 7 I I I - 6 . A C y c l i c Dependency In Secondary Index S e l e c t i o n .100 - i v - Acknowledgements I would l i k e t o thank my s u p e r v i s o r s Paul Gilmore and A l Fowler f o r t h e i r i n v a l u a b l e guidance and support. I would a l s o l i k e to thank Dave K i r k p a t r i c k and again Paul Gilmore f o r t h e i r c o l l a b o r a t i o n on the work r e s u l t i n g i n Se c t i o n C of Chapter I I I . Many thanks are a l s o due Bandy Goebel and the r e s t of my f e l l o w graduates f o r many l i v e l y d i s c u s s i o n s r e g a r d i n g the thesis;. F i n a l l y I would l i k e t o thank Paul and the N a t u r a l Science and E n g i n e e r i n g Research C o u n c i l f o r t h e i r f i n a n c i a l s u p p o r t i -v- I n t r o d u c t i o n 1 Chapter 1. I n t r o d u c t i o n Reference w i l l be made throughout t h i s t h e s i s to the "access path l e v e l " of a r e l a t i o n a l database* management system (RDMS). T h i s l e v e l i s d e f i n e d by T s i c h r i t z i s and Lochovsky 2 as the l e v e l r e s p o n s i b l e f o r p r o v i d i n g e f f i c i e n t access on r e l a t i o n s . T h e i r d e f i n i t i o n of access path l e v e l presumes the e x i s t e n c e of a " f i l e system" and, i n t u r n , provides f o r the implementation of the " r e l a t i o n a l language" l e v e l ( t h i s l a t t e r l e v e l r e p r e s e n t s the lowest l e v e l user i n t e r f a c e ) . In more a b s t r a c t terms the access path l e v e l r e p r e s e n t s the lowest l e v e l i n t e r f a c e between the co n c e p t u a l schema [ N i j s s e n 76aJ and p h y s i c a l schema of a RDMS. I t provides the f i r s t l e v e l of i n f o r m a t i o n r e p r e s e n t a t i o n a b s t r a c t i o n . C l e a r l y , the design o f t h i s l e v e l i s determined almost e n t i r e l y by the requirements o f the: RDMS data management language (DML) and data d e f i n i t i o n language (or DDL, both o f which comprise the user i n t e r f a c e to the r e l a t i o n a l language l e v e l ) . T s i c h r i t z i s and Lochovsky are very s p e c i f i c about the r e s p o n s i b i l i t i e s of the access path l e v e l f o r an RDMS. E s s e n t i a l l y two f a c i l i t i e s are c a l l e d f o r . The f i r s t i n v o l v e s iThe s i n g l e word "database" i n s t e a d of "data base" or " d a t a - base" i s used throughout the t h e s i s * zsee pages 243-245 of [ T s i c h r i t z i s and Lochovsky 77] I n t r o d u c t i o n 2 e f f i c i e n t access w i t h i n a r e l a t i o n . . T h i s , i n essence* p r o v i d e s f o r a s s o c i a t i v e access to re c o r d s . The second i n v o l v e s e f f i c i e n t access between r e l a t i o n s and i s r e a l i z e d , u s u a l l y , with a l i n k i n g convention* Such f a c i l i t i e s might* f o r example, be used t o implement, r e s p e c t i v e l y , the s e l e c t i o n and j o i n o p e r a t o r s of the r e l a t i o n a l a l g e b r a [Codd 70]. In Chapter I I (where, among other t h i n g s , a review w i l l be made of some e x i s t i n g r e l a t i o n a l database systems) we w i l l f i n d the r e s p o n s i b i l i t i e s of some e x i s t i n g access path l e v e l s somewhat more e x t e n s i v e than s e l e c t i o n and l i n k f a c i l i t i e s . A d d i t i o n a l d u t i e s have i n c l u d e d : * (a) management of l o c k i n g p r o t o c o l s , (b) p r o v i s i o n f o r a v a r i e t y of recovery methodology, (c) automatic t r a n s a c t i o n deadlock d e t e c t i o n and recovery and even (d) complex query e v a l u a t i o n . To a very l i m i t e d extent p r o v i s i o n a t the access path l e v e l has a l s o been made f o r : (e) automatic p h y s i c a l c l u s t e r i n g of records on secondary storage. We can see, t h e r e f o r e , t h a t the d e f i n i t i o n o f access path l e v e l i s by no means c l e a r c u t . N e v e r t h e l e s s , many e x i s t i n g r e l a t i o n a l database systems (e^g. System R [Astrahan et a l . *The reader should not t h i n k from t h i s l i s t t h a t l i t t l e remains to be handled by the " r e l a t i o n a l language" l e v e l . . The R e l a t i o n a l Data System (RDS) of System R [Ast r a h a n e t a l . .76] provides f o r e v o l v i n g views (or l o g i c a l subschema), f a c i l i t i e s f o r i n t e g r i t y a s s e r t i o n s and an automatic t r a n s a c t i o n t r i g g e r i n g f a c i l i t y . I n t r o d u c t i o n 3 76], ZETA [ B r o d i e e t a l . .75], OMEGA [ Schmid e t a l . 75] and INGEES [Stonebraker e t al;. 76]) conform c l o s e l y to the h i e r a r c h y o f l e v e l s d e f i n e d by T s i c h r i t z i s and Lochovsky. Our concern i n t h i s t h e s i s i s t o extend t h e : f u n c t i o n a l i t y of the access path l e v e l t o the p o i n t of automating, as f a r as p o s s i b l e , the p h y s i c a l r e o r g a n i z a t i o n requirements of a EDMS. Such requirements ( i n order o f p r i o r i t y ) i n c l u d e : (a) mass storage space management. I t i s a s u r p r i s i n g f a c t t h a t some of the most popular commercial database management systems (DMS) a v a i l a b l e a t the time of t h i s w r i t i n g 1 r e q u i r e the e x p l i c i t a l l o c a t i o n and c o n t r o l o f f i x e d s i z e d i s k " e x t e n t s " t o t h e i r most a b s t r a c t conceptual u n i t s * . (b) p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by a d d i t i o n and d e l e t i o n of t u p l e s and r e l a t i o n s . In some commercial DMSs i t i s t h e user's r e s p o n s i b i l i t y to p e r i o d i c a l l y schedule " f i l e compaction" u t i l i t i e s f o r the purpose of r e c l a i m i n g space occupied by " d e l e t e d " records. (c) p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by changing p a t t e r n s of usage of e x i s t i n g r e l a t i o n s . . (d) p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by augmentation or a l t e r a t i o n of the data model i t s e l f . (e) p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by the d e t e c t i o n of f a u l t s i n the p h y s i c a l (as opposed to l o g i c a l ) o r g a n i z a t i o n of the t u p l e s and r e l a t i o n s . I am sure the reader w i l l agree t h a t (a) and (b) above can and should be automated a t some l e v e l of an EDMS. . A need to automate (c) , (d) and (e), however, i s not as e v i d e n t . Consider i n more d e t a i l what i s i m p l i e d by (c) . We can i n c l u d e *For example, TOTAL - a product of Cincom Systems, Inc [Datapro 79] i n use a t more than 2000 i n s t a l l a t i o n s . I n t r o d u c t i o n 4 i n t h i s area d e c i s i o n s with regard to the a r c h i v i n g o f i n f o r m a t i o n , a l l d e c i s i o n s r e g a r d i n g the c l u s t e r i n g of r e c o r d s and even d e c i s i o n s with regard to the s e l e c t i o n of such t h i n g s as i n d i c e s . } 2 The reader may be u n c l e a r about the i m p l i c a t i o n s of (d). To c l a r i f y ; such p h y s i c a l r e o r g a n i z a t i o n s might be i m p l i e d by the a d d i t i o n of an a t t r i b u t e t o a r e l a t i o n or the a d d i t i o n of a new r e l a t i o n . The automation of such p h y s i c a l r e o r g a n i z a t i o n would f u r t h e r c o n t r i b u t e t o d a t a independence. 3 There are some i n t e r e s t i n g thoughts with regard to item (e ) . Experience has shown that no matter what the system o r hardware something w i l l at some p o i n t i n time become very wrong with the database.* I t i s q u i t e safe to say t h a t s i g n i f i c a n t resource i n the form of f u l l time t e c h n i c a l p e r s onnel i s spent i n d e t e c t i n g and c o r r e c t i n g such problems i n almost a l l commercial environments using a DM S 1 E e c a l l t h a t the access path l e v e l must at l e a s t provide f o r automatic maintenance of such t h i n g s . 2 A t the time of w r i t i n g the author i s not aware of any e x i s t i n g DMS (experimental or commercial, r e l a t i o n a l or otherwise) t h a t f u l l y automates (or even comes c l o s e ) to p h y s i c a l r e o r g a n i z a t i o n requirements o f item (c) alone. 3 N o t having to recompile (rewrite!) a l l a p p l i c a t i o n programs u n a f f e c t e d by the change i n the l o g i c a l model i s c e r t a i n l y n i c e . Even more d e s i r a b l e i s r e l i e f from the n e c e s s i t y o f r e f o r m a t t i n g and r e l o a d i n g v a s t amounts of data due to a minor a l t e r a t i o n i n the l o g i c a l model. *I speak from experience when I say t h a t the mysterious ( i e . unsolved) disappearance of e n t i r e t r a c k s of data from a d i s k can and w i l l occur. I n t r o d u c t i o n 5 ( s u c c e s s f u l l y ) . C o n s i d e r i n g the growing complexity of the data s t r u c t u r e s i n v o l v e d a t the acc e s s path l e v e l of DMSs i t i s c l e a r t h a t automating recovery of p h y s i c a l data i n t e g r i t y i s very d e s i r a b l e . There i s , however, an o v e r r i d i n g m o t i v a t i o n f o r such degrees of automation;. T h i s concerns the growing tendency o f op e r a t i n g systems t o assume the r e s p o n s i b i l i t i e s of database management systems [Rodriguez and Echhouse 77]. Indeed many f e e l t h a t f i l e systems a r e , i n f a c t , nothing more than a p r i m i t i v e DMS.. We regard t h e d e s i r e of a "RDMS at the f i l e system l e v e l " ("phase f o u r " i n [Berryman and Fowler 79]) as the : prime m o t i v a t i o n ( j u s t i f i c a t i o n ) f o r the i n c r e a s e d automation of p h y s i c a l r e o r g a n i z a t i o n requirements (the f i r s t s e c t i o n o f Chapter I I d i s c u s s e s t h i s more f u l l y ) . Now t h a t we understand the n o t i o n s "access path l e v e l " and " p h y s i c a l r e o r g a n i z a t i o n requirements" we can c a t e g o r i z e the degree of automation of a given access path l e v e l : L1: no r e o r g a n i z a t i o n a l e f f o r t i s ever necessary.. L2: r e o r g a n i z a t i o n never i s a b s o l u t e l y necessary and can n e v e r t h e l e s s always be accomplished c o n c u r r e n t l y with r e t r i e v a l and a l t e r a t i o n . L3: r e o r g a n i z a t i o n i s never a b s o l u t e l y n e c e s s a r y but, once i n i t i a t e d , r e q u i r e s e x c l u s i v e access to any a f f e c t e d s t r u c t u r e s and data ( i . e . . no concurrency). L4: r e o r g a n i z a t i o n can become a b s o l u t e l y necessary but i s e n t i r e l y p r e d i c t a b l e and can always be accomplished c o n c u r r e n t l y with r e t r i e v a l and a l t e r a t i o n . L5: r e o r g a n i z a t i o n can become a b s o l u t e l y necessary and i s e n t i r e l y p r e d i c t a b l e , but once i n i t i a t e d , r e q u i r e s e x c l u s i v e access t o any a f f e c t e d I n t r o d u c t i o n 6 s t r u c t u r e s and data (almost a l l commercial DMSs are no higher than t h i s l e v e l ) . L6: r e o r g a n i z a t i o n can become a b s o l u t e l y necessary and i s not p r e d i c t a b l e and w i l l not permit any degree of concurrency of use of any a f f e c t e d s t r u c t u r e s and data (a garbage c o l l e c t o r f o r a l i s p i n t e r p r e t e r i s i n t h i s c a t e g o r y ) . . I t i s not c l e a r which of L3 o r L4 i s more d e s i r a b l e . With r e s p e c t t o t h i s ' h i e r a r c h y 1 i t i s c l e a r t h a t an implementation should a t t a i n a l e v e l as c l o s e to L1 as p o s s i b l e . * Indeed, s i n c e : l e v e l L6 above i s c l e a r l y unacceptable, an implementation must achieve a l e v e l a t l e a s t as good as L5. The. design s p e c i f i c a t i o n f o r the access path l e v e l presented i n t h i s t h e s i s accomplishes p h y s i c a l r e o r g a n i z a t i o n a l requirements concerning a type of r e c o r d c l u s t e r i n g ( i * e . " h o r i z o n t a l " c l u s t e r i n g 2 ) and index s e l e c t i o n . These are r e a l i z e d at a degree: of automation corresponding t o t h a t of L2 above. The system, i t s e l f , i s c a l l e d SOEAAM ( f o r S e l f - O r g a n i z i n g B e l a t i o n a l A s s o c i a t i v e Access Method). Chapter I I i s concerned, i n i t i a l l y * with a more d e t a i l e d d i s c u s s i o n of the moti v a t i o n s and j u s t i f i c a t i o n s f o r such automation* A review f o l l o w s concerning the r e l e v e n t a s p e c t s !There i s good reason to b e l i e v e t h a t a c h i e v i n g l e v e l L1 i t s e l f i s i m p o s s i b l e . As s h a l l be shown, the " o p t i m a l " arrangement of records w i t h i n pages makes sense only with regard to a s p e c i f i c c h a r a c t e r i z a t i o n of t a b l e or r e l a t i o n use (or query h i s t o r y ) . In a r e a l world environment such a c h a r a c t e r i z a t i o n i t s e l f may very from one day t o the next. .. 2 s e e " D e t a i l e d Problem D e f i n i t i o n " i n Chapter I I . I n t r o d u c t i o n 7 of a number of e x i s t i n g BDMSs. Some d e t a i l e d problem d e f i n i t i o n s f o r our access path l e v e l are then presented f o l l o w e d by an o u t l i n e of the s i m p l i f y i n g assumptions. Chapter I I I i s concerned with the design of SOEAAM. An i n i t i a l s e c t i o n o u t l i n e s the design s p e c i f i c a t i o n s and presents an o v e r a l l design of the system.. The next two s e c t i o n s are concerned with the primary i n d e x i n g and the l a s t two with the secondary i n d e x i n g and search s t r a t e g y . . Such c o n s i d e r a t i o n s of SOEAAM r e l a t e to the two major as p e c t s o f p h y s i c a l o r g a n i z a t i o n i m p l i e d by p a t t e r n s of usage of base r e l a t i o n s : t u p l e c l u s t e r i n g or b l o c k i n g and secondary index s e l e c t i o n . The f i r s t s e c t i o n o f Chapter IV presents an o u t l i n e of the "next s t e p " towards a demonstration of f e a s i b i l i t y : the a c q u i s i t i o n of some s p e c i f i c e m p i r i c a l evidence and the t e s t i n g environment i m p l i e d . The t h e s i s concludes with an o v e r a l l assessment of SOEAAM. In p a r t i c u l a r , advantages r e a l i z e d over o t h e r systems with regard t o s e l f - o r g a n i z a t i o n and automatic index s e l e c t i o n are given. Unresolved i s s u e s and sug g e s t i o n s f o r f u t u r e r e s e a r c h comprise the l a s t p art of the assessment* A. P r e l i m i n a r i e s 8 Chapter I I : P r e l i m i n a r i e s A. J u s t i f i c a t i o n f o r Automation Ample evidence e x i s t s t h a t suggests a tendency o f o p e r a t i n g systems t o assume more d i r e c t r e s p o n s i b i l i t y of such t h i n g s as DMSs. Consider, f o r example* the re c e n t i n t r o d u c t i o n of System/38 from IBM [Henry 78]..In t h i s system, support f o r a DMS was thought important enough to warrant the microcoding of a s i g n i f i c a n t percentage o f the r e q u i r e d software [Watson and Aberle 78]. Nevertheless, one may ask what we can expect of a DMS t h a t a more t r a d i t i o n a l f i l e system ( p o s s i b l y i n c l u d i n g access method u t i l i t i e s ) does not pr o v i d e . The obvious advantage of a DMS i s i t s g r e a t l y enhanced c a p a c i t y f o r " d i r e c t " 1 i n f o r m a t i o n r e p r e s e n t a t i o n . With r e s p e c t to t h i s an i n t e r e s t i n g analogy can be drawn between DMSs and f i l e systems on one s i d e and high l e v e l and low l e v e l programming languages on the other. One may even c l a i m t h a t , where o p e r a t i n g system f a c i l i t i e s f o r pro c e d u r a l s p e c i f i c a t i o n ( i n the form of command l a n g u a g e s / i n t e r p r e t e r s ) have ev o l v e d enormously, corresponding f a c i l i t i e s f o r i n f o r m a t i o n r e p r e s e n t a t i o n and manipulation have remained a t a p r i m i t i v e l e v e l * tThe degree of " d i r e c t " n e s s of v a r i o u s DMS's "conceptual schema" [ N i j s s e n 76a] i s o f t e n used, i n f a c t * as the.most s i g n i f i c a n t measure of adequacy ( f o r example, e v a l u a t i o n s concerning " r e l a t i o n a l " verses "network" a b s t r a c t i o n s i n [ McGee 76 ]) . A. P r e l i m i n a r i e s 9 In a DMS, f a c i l i t i e s f o r i n t e g r i t y and s e c u r i t y c o n s t r a i n t s p e c i f i c a t i o n are f a r more e l a b o r a t e . T r a d i t i o n a l f i l e systems have provided f o r a degree of s e c u r i t y but o n l y at a g r o s s f i l e l e v e l . D i r e c t support f o r t h e maintenance of p h y s i c a l i n t e g r i t y alone i n most f i l e systems i s minimal.. In e n f o r c i n g a conceptual schema at the o p e r a t i n g system l e v e l a g r e a t e r degree of d e v i c e independence of " a l l above the OS" i s achieved. Osers are f a r l e s s capable of t a k i n g advantage of device dependent knowledge i n implementing a p p l i c a t i o n s ( t h i s , of c o u r s e , being the concern of the DMS implementors). T h i s r e s u l t s i n an environment f a r l e s s s u s c e p t i b l e t o such t h i n g s as hardware upgrades and even c o n t r i b u t e s s i g n i f i c a n t l y t o the o v e r a l l p o r t a b i l i t y o f a l l the a p p l i c a t i o n s . 1 Recovery f a c i l i t i e s provided by a DMS are almost always more e x t e n s i v e than those of f i l e systems. The l a t t e r , f o r example* have, i n almost a l l cases* no support f o r recovery a t the t r a n s a c t i o n l e v e l . A DMS p r o v i d e s g r e a t e r degrees o f recovery at l a r g e r l e v e l s of concurrency to f i n e r g r a i n s o f data. There i s a more s u b t l e but extremely important advantage to the enforcement of a c o n c e p t u a l schema at the o p e r a t i n g system l e v e l ( i . e . a DMS i n s t e a d o f a f i l e system). T h i s 1 Should o p e r a t i n g system p r i m i t i v e s take advantage of a DMS t o perform some of t h e i r d u t i e s then such a DMS would c o n t r i b u t e d i r e c t l y t o the p o r t a b i l i t y of the o p e r a t i n g system i t s e l f . A. P r e l i m i n a r i e s 10 concerns the r e s u l t i n g i n t e g r a t i o n of a l l i n f o r m a t i o n r e p r e s e n t a t i o n used by a l l a p p l i c a t i o n s . Such a r e s u l t would b e n e f i t v i r t u a l l y a l l l e v e l s of o p e r a t i n g system support* C o n s i d e r , f o r example, t h a t the normal p r o l i f e r a t i o n of s p e c i a l purpose recovery u t i l i t i e s would e s s e n t i a l l y be checked** A good case e x i s t s t h a t contends such an i n t e g r a t i o n t o be p r e r e q u i s i t e t o the attainment of h i g h e r l e v e l s o f computing environments. 2 N a t u r a l l y , a DMS subsuming the r o l e of f i l e system must be a b l e to support a v a r i e t y of " t r a d i t i o n a l " o p e r a t i n g system f a c i l i t i e s . These i n c l u d e : (a) s p o o l i n g systems* (b) message passing systems, (c) d e v i c e and network i n t e r f a c e s and even (d) com p i l e r source code management. A remarkable design c h a r a c t e r i s t i c of the System/38 from IBM [IBM 78] i s t h a t the DMS of t h a t machine p r o v i d e s the b a s i c support f o r a l l of the above* I n [McDonell 77] many arguments are presented i n f a v o u r of a "homogeneous secondary s t o r a g e i n p u t - o u t p u t i n t e r f a c e " (based upon the CODASYL DDL/DML, [CODASYL 71], [CODASYL 73] ) . In an appendix t o h i s t h e s i s we are presented with an e n t i r e implementation design f o r a 1 T h i s i s t r u e s i n c e a l l t h a t would normally be r e q u i r e d by most a p p l i c a t i o n s would a l r e a d y be provided by the DMS. . 2A complete demonstration of t h i s , however, i s considered to be o u t s i d e the scope of the t h e s i s . A. P r e l i m i n a r i e s s p o o l i n g subsystem based on t h i s i n t e r f a c e ; In h i s t h e s i s , [ C a r g i l l 79] goes t o great l e n g t h s i n d i r e c t l y a p p l y i n g the i n f o r m a t i o n modeling c a p a b i l i t y of the UNIX f i l e system f o r the purposes of compiler source code maintenance. 1 I f we accept t h a t a DMS i n s t e a d o f a f i l e system i s both d e s i r a b l e and n o n - r e s t r i c t i v e then obvious j u s t i f i c a t i o n e x i s t s f o r a c h i e v i n g the h i g h e s t degree of automation p o s s i b l e f o r p h y s i c a l r e o r g a n i z a t i o n r e q u i r e m e n t s . . C l e a r l y , one cannot allow a s i t u a t i o n i n which an e n t i r e computing system i s f r e q u e n t l y u n a v a i l a b l e because of time-consuming r e f o r m a t t i n g or r e b u i l d i n g phases; We see* t h e r e f o r e , the design of our access path l e v e l a p p r o p r i a t e i n t h i s more g l o b a l environment. One l a s t p o i n t we s h a l l d e al with i n t h i s s e c t i o n concerns the idea of a r r i v i n g a t our "dream" by simply implementing our DMS on top of a c u r r e n t l y e x i s t i n g f i l e system. Experience has shown t h a t such an approach e v e n t u a l l y i n v o l v e s e i t h e r s i g n i f i c a n t l y a l t e r i n g the op e r a t i n g system i t s e l f o r adopting s u b s t a n t i a l design compromises..The System R people [ A s t r a h a n et a l . 76] found i t necessary, f o r example, to make s e v e r a l extensions to VM/370 (IBM's l a r g e - s c a l e v i r t u a l machine f a c i l i t y ) . Others concur [Berryman and Fowler 79], [Gray 78] t h a t t h i s i s t o be expected* On the other hand, c o n s i d e r a d e c i s i o n made by the desi g n e r s of INGRES: lone cannot help f e e l , when readi n g the t h e s i s , t h a t h i s l i f e would have been made much e a s i e r c o u l d he.have taken advantage of a more powerful i n f o r m a t i o n modeling c a p a b i l i t y . A. P r e l i m i n a r i e s 12 "In keeping with the design d e c i s i o n of not modifying UNIX these c o n s i d e r a t i o n s 1 were i n c o r p o r a t e d i n the design d e c i s i o n not to support c l u s t e r i n g . " 2 We s h a l l c o n s i d e r o u r s e l v e s , t h e r e f o r e , t o be unconstrained i n the design of our access path l e v e l by lower l e v e l s of software. L i k e w i s e , we s h a l l be f o r e v e r wary of our commitment to the r e l a t i o n a l model of a c o n c e p t u a l schema. B» Previous Systems A number of RDMSs have been developed at v a r i o u s r e s e a r c h centers* T h i s s e c t i o n f i r s t p r e s e n t s a general overview o f some of these. Emphasis i n t h i s overview i s placed on t h e i r r e s p e c t i v e access path l e v e l s (as we s h a l l see the d i s t i n c t i o n of an access path l e v e l i s s u r p r i s i n g l y w e l l - d e f i n e d i n most of these systems).. The p h y s i c a l r e o r g a n i z a t i o n a l aspects o f the access path l e v e l i s of p a r t i c u l a r concern.. There has been some work concerning s e l f - o r g a n i z a t i o n and the problem of r e c o r d (tuple) c l u s t e r i n g . The:second p a r t of t h i s s e c t i o n reviews t h i s work i n r e l a t i o n to our n o t i o n o f p h y s i c a l r e o r g a n i z a t i o n a l requirements. 1 H t h e s e c o n s i d e r a t i o n s " i n v o l v e the f a c t s t h a t f i l e s are r e a l i z e d i n UNIX by small (512 byte) p o s s i b l y randomly l o c a t e d p h y s i c a l b l o c k s ( c e r t a i n l y an environment i n h i b i t i n g c o n s i d e r a t i o n of c l u s t e r i n g ) . 2 T h i s passage i s taken from page 20 2 of [Stonebraker et a l . 76 J . B. P r e l i m i n a r i e s 13 ZETA [ B r o d i e et a l . 75] i s a EDMS ( r e l a t i o n a l database management system) being developed a t the U n i v e r s i t y o f Toronto t o run on IBM 370 type machines* The system i s composed of th r e e p r i n c i p a l l e v e l s termed MINIZ, the EXECUTOR and the "language f a c i l i t i e s " . The f i r s t o f these l e v e l s , MINIZ, performs the d u t i e s r e q u i r e d by our n o t i o n of an access path l e v e l . T h i s component implements r e l a t i o n s d i r e c t l y as f i l e s of data* That i s , each r e l a t i o n i n MINIZ i s re p r e s e n t e d as a s i n g l e f i l e . 1 B e s i d e s t h i s i n i t i a l l e v e l of a b s t r a c t i o n MINIZ can accomplish s i n g l e r e l a t i o n r e t r i e v a l s ( r e q u i r i n g no more than one pass) and t u p l e - a t - a - t i m e m o d i f i c a t i o n s . R e s p o n s i b i l i t i e s a l s o i n c l u d e a recovery f a c i l i t y and maintenance of f o u r system r e l a t i o n s . These system r e l a t i o n s s t o r e a l l i n f o r m a t i o n d e s c r i b i n g a l l r e l a t i o n s , domains, users and the p h y s i c a l extent and b l o c k i n g of r e l a t i o n s (the l a t t e r i s used o n l y by the f i l e system). MINIZ i s managed by the EXECUTOR. T h i s second l e v e l forms pa r t o f the r e l a t i o n a l language l e v e l o f ZETA and i s r e s p o n s i b l e f o r "snapshot" 2 r e l a t i o n s and m u l t i p l e r e l a t i o n queries* A "host programming language system" (HLS) and " s e l f - c o n t ained language system" (SLS) comprise, the language iAlmost a l l EDMSs we c o n s i d e r take t h i s approach* 2 T h i s should not be confused with d e r i v e d "views" or subschema. A snapshot r e l a t i o n i s independent of the r e l a t i o n s from which i t i s c r e a t e d * B. P r e l i m i n a r i e s 14 f a c i l i t i e s ( t h i r d l e v e l ) of ZETA. The l a t t e r i s i n t e r e s t i n g i n t h a t i t i s a c t u a l l y a "query language generator system" (QLS) which i s used t o generate a t a i l o r - m a d e SLS. The base r e l a t i o n s of ZETA must each be assigned a f i x e d s i z e d d i s k extent (which, of course, i m p l i e s a maximum number of t u p l e s allowed i n any such base r e l a t i o n ) . 1 Furthermore, a l l t u p l e s are added at the end of a r e l a t i o n ( t u p l e d e l e t i o n r e s u l t s i n a simple marking).. To b u i l d an a p p r o p r i a t e l y c l u s t e r e d r e l a t i o n , t h e r e f o r e , a user must f i r s t f i n d and a l l o c a t e a p h y s i c a l extent (he must a n t i c i p a t e r e l a t i o n s i z e ) then a p p r o p r i a t e l y order h i s t u p l e s p r i o r t o i n s e r t i o n . To r e c l a i m storage used by d e l e t e d t u p l e s the user must e s s e n t i a l l y r e b u i l d the r e l a t i o n . R e o r g a n i z a t i o n appears t o be p r e d i c t a b l e i n ZETA. However, v i r t u a l l y no automation of p h y s i c a l r e o r g a n i z a t i o n requirements i s provided. A l s o , the search f a c i l i t i e s used by MINIZ a r e not as s o p h i s t i c a t e d as other systems (most new q u e r i e s i n v o l v e s e q u e n c i a l scanning of base r e l a t i o n s ) . Another RDMS developed a t the U n i v e r s i t y of Toronto i s the OMEGA system [Schmid et a l . 76 ]. OMEGA was developed t o run under the c o n t r o l of the UNIX o p e r a t i n g system on a PDP 11/45 minicomputer. The l e v e l s which comprise: OMEGA (e x c l u d i n g the one r e p r e s e n t e d by the UNIX f i l e system) are e n t i r e l y analogous to the ZETA system. . *In f a c t , an OS JCL DD ca r d must be provided f o r each base r e l a t i o n f o r any job using ZETA. B» . P r e l i m i n a r i e s 15 O p e r a t i o n s implemented by MINI-OMEGA and by the "data s t r u c t u r e s system" comprise the f i r s t or a c c e s s path l e v e l . MINI-OMEGA p r o v i d e s the i n i t i a l r e l a t i o n a l a b s t r a c t i o n communicating d i r e c t l y with the UNIX f i l e system* The data s t r u c t u r e system maintains i n v e r t e d f i l e s and " b a s i c elements". The l a t t e r c o n s i s t s of two v a r i e t i e s c o r r e s p o n d i n g e s s e n t i a l l y t o i n t e r n a l temporary r e s u l t s and snapshot r e l a t i o n s . . The second l e v e l of OMEGA (termed the "a c c e s s s t r u c t u r e l e v e l " by the implementors), l i k e ZETA, forms p a r t o f what we understand as the r e l a t i o n a l language l e v e l * T h i s access s t r u c t u r e l e v e l i s r e s p o n s i b l e f o r the: o p t i m i z a t i o n and i n t e r p r e t a t i o n of a query language, the c o n s i s t e n c y o f e x i s t i n g access s t r u c t u r e s and t h e i r c r e a t i o n and d e s t r u c t i o n . The query language i s c a l l e d the " l i n k and s e l e c t o r language" (LSL) and must be the r e s u l t produced by a l l r e l a t i o n a l query languages comprising the user i n t e r f a c e s a t the t h i r d l e v e l * * As i n ZETA the l e v e l of automation of p h y s i c a l r e o r g a n i z a t i o n a l requirements i s r e l a t i v e l y low. There a r e , however, a number of advances over ZETA.. The automatic maintenance o f secondary i n d i c e s i s one example (although the c r e a t i o n and d e l e t i o n of i n d i c e s i s s t i l l the r e s p o n s i b i l i t y 4An e x p r e s s i o n i n LSL, as seen at the second l e v e l , i s t r e e s t r u c t u r e d . The c u r r e n t OMEGA implementation p r o v i d e s f o r a r e l a t i o n a l query language t h a t maps d i r e c t l y t o t h i s t r e e s t r u c t u r e . B. P r e l i m i n a r i e s 16 of a human 1). A l s o , the i n i t i a l t u p l e s of a r e l a t i o n are st o r e d i n a primary area s o r t e d by key value. I f access t o t u p l e s i s based p r i m a r i l y on the key domains of the r e l a t i o n t h i s r e s u l t s i n a more d e s i r a b l e (but not n e c e s s a r i l y optimal) c l u s t e r i n g of t u p l e s w i t h i n d i s k pages. Tuples t h a t are added f o l l o w i n g the i n i t i a l setup are l o c a t e d i n an overflow area which i n t u r n i m p l i e s a p e r i o d i c p h y s i c a l r e o r g a n i z a t i o n requirement. Another form of t u p l e c l u s t e r i n g i s a l s o r e a l i z e d by ONEGA. T h i s i n v o l v e s an a b i l i t y t o st o r e a l o g i c a l r e c o r d (tuple) s p l i t - u p i n t o s e v e r a l p h y s i c a l records i n se p a r a t e p h y s i c a l f i l e s ( i . e . the i - t h p h y s i c a l r e c o r d s of a s e t o f p h y s i c a l f i l e s comprise the i - t h l o g i c a l r e c o r d ) . The advantage of t h i s i s t h a t many more p a r t i a l - t u p l e s can now be st o r e d i n a s i n g l e block of a p h y s i c a l f i l e . Of course, such a " v e r t i c a l " p a r t i t i o n i n g of t u p l e s depends e n t i r e l y on the nature o f use of the r e l a t i o n * Again, the human i s r e s p o n s i b l e f o r d e termining such a p a r t i t i o n i n g . L a t e r on i n t h i s s e c t i o n we review a system (SODMS [ K o l l i a s e t a l * . 77], [ S t o c k e r and Dearnley 74]) which automates t h i s r e s p o n s i b i l i t y * INGEES [Stonebraker et a l . . 7 6 ] i s another EDMS developed to run under the c o n t r o l of the ONIX op e r a t i n g system* The s t r u c t u r i n g of the various components of INGEES, however, i s more i n v o l v e d t h a t e i t h e r ZETA or OMEGA. Our analogy o f these 4 I n f a c t , none of the systems we review provide f o r t h i s degree of automation. B. P r e l i m i n a r i e s 17 components to the "access path l e v e l " and " r e l a t i o n a l language" a b s t r a c t i o n s of [ T s i c h r i t z i s and Lochovsky 77] i s somewhat more s t r a i n e d . Due mainly to the l i m i t e d l o g i c a l address space.of PDP 11s (64k bytes) INGBES i s s t r u c t u r e d as a four l e v e l h i e r a r c h y of f o u r communicating processes.. The f i r s t of these (top l e v e l ) may be an i n t e r a c t i v e t e r m i n a l monitor o r an e x e c u t i n g user process. The former f a c i l i t a t e s d i r e c t use of INGRES by users and the l a t t e r access t o INGEES by user programs. The second l e v e l process (process 2) has r e s p o n s i b i l i t y f o r p a r s i n g q u e r i e s , concurrency and s e c u r i t y c o n t r o l and support f o r i n t e g r i t y c o n t r o l and d e r i v e d views* 1 S e c u r i t y and i n t e g r i t y c o n t r o l and d e r i v e d views are accomplished by a query m o d i f i c a t i o n procedure. Concurrency i s r e a l i z e d by use of a l o c k i n g p r o t o c o l r e s t r i c t i v e enough t o not r e q u i r e t r a n s a c t i o n deadlock d e t e c t i o n and re c o v e r y . L e v e l t h r e e of the process h i e r a r c h y i s r e s p o n s i b l e f o r query e v a l u a t i o n . T h i s i s accomplished by a query decomposition procedure and a o n e - v a r i a b l e query p r o c e s s o r (OVQP). Both are concerned t o a l a r g e extent with o p t i m i z a t i o n c o n s i d e r a t i o n s . A l l other INGBES commands are i n t e r p r e t e d by a s e t o f o v e r l a y i n g u t i l i t i e s a t process l e v e l f o u r * A s e t of f u n c t i o n s used i n common by processes two, three d e r i v e d views d i f f e r from snapshot r e l a t i o n s i n that they always r e f l e c t the c u r r e n t s t a t e o f the base : r e l a t i o n s from which they are d e r i v e d . B. P r e l i m i n a r i e s 18 and f o u r i s c a l l e d the "Access Methods I n t e r f a c e " (AMI)..This se t o f f u n c t i o n s communicates d i r e c t l y with the UNIX f i l e system t o implement the i n i t i a l r e l a t i o n a l a b s t r a c t i o n . I t i s the AMI tog e t h e r with process l e v e l s t h r e e (or at l e a s t OVQP) and f o u r t h a t corresponds t o our no t i o n of an access path l e v e l * The r e l a t i o n a l language l e v e l i s provided by processes one and two as a query language c a l l e d QUEL and an embedded 1 query language c a l l e d EQUEL.. The AMI supports f i v e d i f f e r e n t p h y s i c a l o r g a n i z a t i o n s o f base r e l a t i o n s . . These correspond t o a "heap" storage scheme and "hashed", "compressed hash", "ISAM" and "compressed ISAM" indexed storage schemes. . The f i r s t of these i s c o n s i d e r e d a p p r o p r i a t e f o r very s m a l l r e l a t i o n s * t r a n s i t i o n a l storage and temporary i n t e r n a l r e l a t i o n s . I f access to a r e l a t i o n i s p r i m a r i l y v i a a p a r t i a l match on i t s key a t t r i b u t e s then the hashed scheme i s a p p r o p r i a t e . In t h i s case the p h y s i c a l page l o c a t i o n of a t u p l e i s determined by a hash f u n c t i o n a p p l i e d to an ordered l i s t of va l u e s o f i t s key a t t r i b u t e s . The ISAM scheme i s t o be p r e f e r e d i n s i t u a t i o n s where access t o the r e l a t i o n has the form of a p a r t i a l range query on i t s key a t t r i b u t e s ( i . e . a search c o n d i t i o n s p e c i f y i n g a range.of v a l u e s f o r each key a t t r i b u t e o f the r e l a t i o n ) . T h i s i s because the p h y s i c a l page l o c a t i o n of a t u p l e i s determined by a key o r d e r i n g . As i n OMEGA, t h i s r e s u l t s i s a c l u s t e r i n g (on iEmbedded, t h a t i s , i n a general purpose programming language c a l l e d "C". B. P r e l i m i n a r i e s 19 the key a t t r i b u t e values) of t u p l e s w i t h i n d i s k pages.. INGRES, a l s o l i k e OMEGA, maintains overflow pages f o r i t s base r e l a t i o n s (an overflow page i s used whenever storage i s exhausted i n a primary page*). T h i s again r e q u i r e s p e r i o d i c p h y s i c a l r e o r g a n i z a t i o n . In INGRES, secondary i n d i c e s are supported and are maintained i n the same way as any of the base r e l a t i o n s . O v e r a l l INGRES i s analogous t o OMEGA with r e s p e c t t o i t s degree of p h y s i c a l r e o r g a n i z a t i o n requirements By f a r the most ambitious RDMS development t o date i s the System R p r o j e c t [Astrahan e t a l . 7 6 ] undertaken by IBM at t h e i r San Jose r e s e a r c h l a b o r a t o r y . . System R r e p r e s e n t s an attempt t o provide a complete r e l a t i o n a l database management system* In p a r t i c u l a r , support i s provided f o r a l a r g e degree of concurrency and system r e c o v e r y . A modified host o p e r a t i n g system (VM/370) d i r e c t l y supports m u l t i t h r e a d i n g (concurrency) at t he l e v e l of v i r t u a l machines which i n t u r n allows a pr o d u c t i v e use of a m u l t i p r o c e s s o r environment. E x t e n s i v e support f o r system recovery a t the l o g i c a l t r a n s a c t i o n l e v e l i s provided f o r both s o f t and hard system f a i l u r e * A system-1- wide monitor program has the r e s p o n s i b i l i t y f o r such re c o v e r y and f o r the d e t e c t i o n and recover y from t r a n s a c t i o n deadlock (the l e v e l of concurrency i n System R i s such t h a t deadlock can a r i s e ) . The System R p a r t of each v i r t u a l machine i s comprised o f lThe number of primary pages o f a base r e l a t i o n i s f i x e d at the time of i n i t i a l setup* B. P r e l i m i n a r i e s 20 two major components. The R e l a t i o n a l Storage System (RSS) together with i t s R e l a t i o n a l Storage I n t e r f a c e (RSI) corresponds i n f u n c t i o n to our n o t i o n of acc e s s path l e v e l * , The r e l a t i o n a l language l e v e l i s r e a l i z e d by the R e l a t i o n a l Data System (RDS) of System R together with i t s R e l a t i o n a l Data I n t e r f a c e (RDI; lowest l e v e l user i n t e r f a c e ) the main component of which i s the SEQUEL DML/DDL. M l a p p l i c a t i o n programs communicate with System R through the.RDI. F i g u r e I I - 1 below i l l u s t r a t e s the b a s i c a r c h i t e c t u r e of System R. A r c h i t e c t u r e of System R I I | MONITOR | I I I I- I I ~l | MASS | | STORAGE | » t F i g u r e I I-1 VM11 I I I | RDS | RSS | 1 I _ _ l VMn| I I I | RDS | RSS | I. 1, I A major component of the RDS i s the o p t i m i z e r . I t i s the r e s p o n s i b i l i t y o f t h i s component to f i n d the cheapest p o s s i b l e means o f executing SEQUEL statements given a v a i l a b l e l i n k s and i n d i c e s d e f i n e d on the base r e l a t i o n s . The o p t i m i z e r chooses from a v a r i e t y of methods to e v a l u a t e both r e s t r i c t i o n on r e l a t i o n s ( e q u i v a l e n t t o a one v a r i a b l e query i n QUEL under B. P r e l i m i n a r i e s 21 INGRES) and i n t e r / i n t r a - r e l a t i o n j o i n o p e r a t i o n s ( m u l t i p l e v a r i a b l e q u e r i e s i n QUEL).. T h i s i s accomplished p r i m a r i l y throuqh the use of a v a r i e t y o f s t a t i s t i c s maintained f o r each base r e l a t i o n . 1 The RDS a l s o has the r e s p o n s i b i l i t y f o r p r o v i d i n g f o r user d e f i n e d e v o l v i n g views, i n t e g r i t y a s s e r t i o n s , a u t h o r i z a t i o n and t r a n s a c t i o n " t r i g g e r s " . The l a t t e r f a c i l i t y i s analogous to the antecedent theorem of PLANNER [Hewitt 72]. In a d d i t i o n the RDS must p r o v i d e " a d v i c e " t o the RSS as t o where new t u p l e s should be l o c a t e d . T h i s i s accomplished by pas s i n g a t e n t a t i v e d i s k address f o r the new t u p l e s . A l l the system c a t a l o g u e r e l a t i o n s are a l s o maintained by the RDS. They provide the i n f o r m a t i o n about a l l other base r e l a t i o n s , views, images, l i n k s , i n t e g r i t y a s s e r t i o n s and t r i g g e r s . Images and l i n k s are s p e c i a l data s t r u c t u r e s t h a t permit e f f i c i e n t a s s o c i a t i v e access t o t u p l e s i n System R. Images are secondary i n d i c e s i n the form of B-trees [Bayer and McCreight 72] and are d e f i n e d on an e x p l i c i t ordered subset o f a t t r i b u t e s o f s i n g l e base r e l a t i o n s . L i n k s are a m u l t i r i n g data s t r u c t u r e [Wiederhold 77] p e r m i t t i n g a d i r e c t e f f i c i e n t means of r e p r e s e n t i n g one t o many r e l a t i o n s h i p s among a t t r i b u t e s w i t h i n and between base r e l a t i o n s . L i n k s d e f i n e d on a s i n g l e a t t r i b u t e i n a s i n g l e r e l a t i o n imply a p a r t i a l *As f a r as I know, these s t a t i s t i c s are not dynam i c a l l y maintained but r a t h e r are s p e c i f i e d by a user at i n i t i a l r e l a t i o n d e f i n i t i o n / s e t u p . B. P r e l i m i n a r i e s 22 o r d e r i n g of the t u p l e s on t h a t a t t r i b u t e . B i n a r y l i n k s ( i . e . l i n k s between a t t r i b u t e s i n separate r e l a t i o n s ) are analogous i n concept to a v a r i e t y of owner coupled set o f the DBTG model (see [CODASYL 71 J ) . Both images and l i n k s are maintained and used by the ESS component of System R. In a d d i t i o n t o base r e l a t i o n s 1 , users of System R a l s o have the r e s p o n s i b i l i t y f o r s p e c i f y i n g the RSS access paths ( i * e . images and l i n k s ) * . Research i s c u r r e n t l y underway, however, to automate t h i s f u n c t i o n . I t should be s t r e s s e d at t h i s p o i n t t h a t the DML component ( i n c l u d i n g the: query f a c i l i t y ) of the RDI of System R i n no way presumes or permits any knowledge of the e x i s t e n c e o f these access paths* Such a property of the DML i s p r e c i s e l y what d i s t i n g u i s h e s System R as a RDMS (verses a h i e r a r c h i c or network DMS) . The access path l e v e l of System R (the ESS) i s the most ambitious and s o p h i s t i c a t e d we s h a l l review. The.ESS p r o v i d e s the support f o r both p h y s i c a l (segment l e v e l ) and l o g i c a l ( t r a n s a c t i o n l e v e l ) recovery from t r a n s a c t i o n deadlock and system f a i l u r e . In a d d i t i o n the ESS supports a l l l o g i c a l and p h y s i c a l l o c k i n g requirements and the dynamic c r e a t i o n * maintenance and d e l e t i o n of base r e l a t i o n s , images and l i n k s . New f i e l d s may be added on the r i g h t of any e x i s t i n g base r e l a t i o n at any time without i n c u r r i n g the overhead of database r e l o a d or even the immediate update of the t u p l e s d e l a t i o n s , i n System B, are a c t u a l l y r e f e r e d t o as " t a b l e s " . B. P r e l i m i n a r i e s 23 w i t h i n t h e base r e l a t i o n . The s e a r c h i n g support of the ESS i s analogous i n c a p a b i l i t y t o the OVQP of the INGEES system. In f a c t , OVQP and ESS a re o f p a r t i c u l a r i n t e r e s t to us i n j u s t t h i s r e s p e c t s i n c e we s h a l l be adopting t h i s l e v e l of search c a p a b i l i t y as pa r t o f the design s p e c i f i c a t i o n of our own acc e s s path l e v e l . . A l e g a l search p r e d i c a t e t o the ESS i s a boolean expression o f p r i m i t i v e s i n d i s j u n c t i v e normal form (see [ S e l i n g e r e t a l . 79]),; Each p r i m i t i v e has the form of a comparison o p e r a t o r on an a t t r i b u t e (or f i e l d number) and a val u e . A l l p r i m i t i v e s i n a given search p r e d i c a t e must of course apply t o the.same r e l a t i o n ; A more f o r m a l i l l u s t r a t i o n of t h i s format i s given i n Table I I - 1 below i n terms of a BNF syntax c h a r t . The symbols "OB" and "AND" i n the c h a r t denote the l o g i c a l d i s j u n c t i o n and c o n j u n c t i o n o p e r a t o r s r e s p e c t i v e l y ; A l l i n f o r m a t i o n i n System B ( i n c l u d i n g t u p l e s , i n d i c e s , etc) are s t o r e d i n a set of segments (each of which has i t s own l o g i c a l address space). U n l i k e OMEGA a l l t u p l e s o f a given base r e l a t i o n are c o n s t r a i n e d t o occur i n a s i n g l e segment.. Un l i k e a l l other systems, however, a s i n g l e segment might c o n t a i n more than one r e l a t i o n ; Such a " v e r t i c a l " a g g r e g a t i o n 1 c a p a b i l i t y when combined with the l i n k access path can be used to g r e a t l y enhance the performance of r e l a t i o n a l j o i n o p e r a t i o n s * i B e c a l l i n our review of OMEGA that " v e r t i c a l " p a r t i t i o n i n g may a l s o be advantagous. B. P r e l i m i n a r i e s 24 R e c a l l t h a t the RDS provides t e n t a t i v e d i s k addresses f o r new t u p l e s to the RSS. T h i s f a c i l i t y e x i s t s f o r the purpose o f BNF Syntax of L e g a l Search P r e d i c a t e s f o r the RSS of System R < l e g a l predicate> ::= <conjunct> | (<legal predicate> OR <conjunct>) <conjunct> ::= <primitive> | (<conjunct> AND <primitive>) <primitive> ::= ( < f i e l d number> <operator> <value>) <operator> : := = I -.= I < I > I <= I >= Table II-1 c l u s t e r i n g t u p l e s o f base r e l a t i o n s i n d i s k pages. T h i s c l u s t e r i n g c a p a b i l i t y , however, i s l i m i t e d t o at most one image or l i n k per base r e l a t i o n ( c l u s t e r i n g on a l i n k makes sense i n System R because of the allowance f o r v e r t i c a l a g g r e g a t i o n ) . In a d d i t i o n , a l l such c l u s t e r i n g images or l i n k s must be e x p l i c i t l y d e c l a r e d through the RDI by the System R user. N e v e r t h e l e s s , the c l u s t e r i n g c a p a b i l i t y i s a fundamental and important aspect of System R. The p h y s i c a l r e o r g a n i z a t i o n requirements of System R are the l e a s t o f a l l RDMSs covered i n t h i s review* The access path l e v e l (SSS) : ".. . s u p p o r t s dynamic a d d i t i o n and d e l e t i o n of r e l a t i o n s , indexes, and l i n k s , with f u l l space r e c l a m a t i o n , and the a d d i t i o n of new f i e l d s to e x i s t i n g r e l a t i o n s - a l l without s p e c i a l B. P r e l i m i n a r i e s 25 u t i l i t i e s or database r e o r g a n i z a t i o n . " * As we have seen. System R a l s o provides f o r a c l u s t e r i n g c a p a b i l i t y ; There i s , however, much room f o r improvement. Users of System R must s t i l l a l l o c a t e d i s k segments and s u i t a b l y d i s t r i b u t e base r e l a t i o n s among them and they must s t i l l s p e c i f y which access paths are t o e x i s t . . Furthermore, they must understand w e l l enough the use of the base r e l a t i o n s to determine an optimal s e t of c l u s t e r i n g images and l i n k s . An i n t e r e s t i n g approach to a batch o r i e n t e d DMS i s the SODMS ( f o r S e l f - O r g a n i z i n g Data Management System) by [ S t o c k e r and Dearnley 74] (see a l s o [ S t o c k e r and Dearnley 72] and [ K o l l i a s et a l . .77]). As the name i m p l i e s , the design emphasis i n SODMS i s t o automate t h e : s e l e c t i o n of data s t r u c t u r e s and access methods. Such s e l e c t i o n s are based on an a n a l y s i s of a h i s t o r y of system usage (a query h i s t o r y ) . The. o p e r a t i o n a l environment of SODMS assumes the f o l l o w i n g : (a) user t o user i n t e r a c t i o n i s minimal, (b) the database f u n c t i o n s p r i m a r i l y i n responding to q u e r i e s and (c) query and update response or turnaround i s u n c r i t i c a l (thus p e r m i t t i n g use of batching t e c h n i q u e s ) . The breadth of a p p l i c a t i o n i s * t h e r e f o r e * somewhat more l i m i t e d than a g e n e r a l purpose RDMS such as System R. Never t h e l e s s , such r e s t r i c t i o n s have allowed the experimenters 1 T h i s passage i s taken from page 129 of [Astrahan e t a l . 76]. B. P r e l i m i n a r i e s 26 t o c o n c e n t r a t e on automating c o n t r o l over key p h y s i c a l o r g a n i z a t i o n d e t a i l s * The data model presented t o users by SODMS i s th a t of a set of t a b l e s or r e l a t i o n s . As yet, no subschema or p r o t e c t i o n f a c i l i t i e s are provided on these ( i * e . a l l us e r s see a l l t a b l e s ) . . In a n t i c i p a t i o n of c o n f u s i o n over terminology the des i g n e r s have l a b e l e d these t a b l e s as " f o l i o s " * A f o l i o i n SODMS may be r e a l i z e d by a v a r i e t y of d i f f e r e n t f i l e s r e p r e s e n t i n g r e c o r d s , sub-records, amendments, d i r e c t o r i e s , e t c . A l l d e c i s i o n s with regard t o the design and c r e a t i o n o f these f i l e s a r e made by the system i t s e l f and are based on an a n a l y s i s of past use. Co n s i d e r , f o r example, a f o l i o (a r e l a t i o n ) comprised o f the f i v e a t t r i b u t e s A1 t o A5. I f the l a s t e v a l u a t i o n of past use found t h a t the f o l l o w i n g t y p i f i e d use of the f o l i o : (a) given values f o r A1 and A2 get A4, (b) g i v e n a value f o r A1 get A5 and (c) get (order not important) a l l a t t r i b u t e s of a l l records* then we could expect the f o l i o as de p i c t e d i n Table II-2 ( a ). (Bracketed a t t r i b u t e s i n the t a b l e denote an o r d e r i n g on a f i l e thus p e r m i t t i n g a s s o c i a t i v e a c c e s s to the sub-records based on those a t t r i b u t e s . 1 ) In SODMS e v a l u a t i o n s are p e r i o d i c . I f the next e v a l u a t i o n of t h i s f o l i o were t o »The present implementation o f the system, however, a l l o w s o r d e r i n g s i n f i l e s t o be based on a t most one a t t r i b u t e . F i l e 2, t h e r e f o r e , cannot c u r r e n t l y be r e a l i z e d . . B. P r e l i m i n a r i e s 27 determine t h a t q u e r i e s of the form of (a) above no l o n g e r occurred and t h a t d i f f e r e n t q u e r i e s of the form: (d) given a value f o r A2 get A5. took t h e i r p l a c e then we would see the f o l i o a l t e r e d as i n Table I I - 2 (b). Notice t h a t f i l e 2 has not simply been destroyed but i n s t e a d has been a r c h i v e d t o tape. In g e n e r a l t h i s i s t r u e of a l l i n f o r m a t i o n i n SODMS. I t i s the o p i n i o n of the d e s i g n e r s , i n f a c t , t h a t no i n f o r m a t i o n , once added i n a DMS, should ever be destroyed.* T h i s i s not to say, however, t h a t amendments continue t o be a p p l i e d t o f i l e 2. In [ K o l l i a s et a l . 77J we see the a p p l i c a t i o n o f l i n e a r programming techniques to batched query e v a l u a t i o n and a type of f i l e a r c h i v i n g . An analogy i s made between developing an ex e c u t i o n s t r a t e g y f o r s m a l l batches of q u e r i e s and the " s e t c o v e r i n g problem"; a l s o f o r t h e a l l o c a t i o n of f i l e s t o o n - l i n e and o f f - l i n e d i s c packs and the "knapsack problem". C o n s i d e r a t i o n i s a l s o given to a r c h i v i n g p a r t s of f i l e s t o o f f l i n e d i s c packs*. With r e s p e c t t o System R, SODMS has automated a l l a l l o c a t i o n o f d i s k storage and the s e l e c t i o n of c l u s t e r i n g images ( l i n k i n g data s t r u c t u r e s as access methods are not supported i n SODMS). T h i s , i n i t s e l f , i s an important accomplishment of the system..SODMS i s , however, much more *There a r e a c t u a l l y many good reasons f o r t h i s o p i n i o n . Consider, i n the case of accounting a p p l i c a t i o n s f o r example, the u s u a l l y u n p r e d i c t a b l e a r r i v a l p a t t e r n s of government a u d i t o r s , law s u i t s * e t c . B. P r e l i m i n a r i e s 28 l i m i t e d i n i t s range of a p p l i c a t i o n s . Excluded from c o n s i d e r a t i o n , f o r example, are h i g h l y dynamic o n - l i n e environments r e q u i r i n g immediate update and f a s t query Example E f f e c t o f Change i n - Use of a F o l i o i n SODMS F i l e Number A t t r i b u t e ( s ) L o c a t i o n 1 A1,A2,A3,A4,A5 d i s k 2 (A1,A2) ,A4 d i s k 3 (A1),A5 d i s k (a) before r e o r g a n i z a t i o n F i l e Number A t t r i b u t e ( s ) L o c a t i o n 1 A1,A2,A3,A4,A5 d i s k 2 (A1,A2),A4 tape 3 (A1),A5 d i s k 4 (A2),A5 d i s k (b) a f t e r r e o r g a n i z a t i o n T a b l e I I -2 response. Secondary i n d i c e s (on f i l e s ) are a l s o not supported. Furthermore, experimental r e s u l t s i n [ S t o c k e r and Dearnley 74] suggest t h a t System B»s c a p a b a b i l i t y of a l l o w i n g the dynamic augmentation t o a f i l e of a d d i t i o n a l columns or a t t r i b u t e s i s h i g h l y d e s i r a b l e . U n l i k e System E, SODMS batches a l l amendments to i t s f o l i o s * These amendments are then a p p l i e d during p e r i o d i c f i l e (as opposed t o f o l i o ) r e o r g a n i z a t i o n * While some i n d e x i n g methods such as B-trees are u n a f f e c t e d i n t h e i r performance with on-going a l t e r a t i o n o t hers such as ISAM techniques B. P r e l i m i n a r i e s 29 degrade when such a l t e r a t i o n i s c o n t i n u a l l y a p p l i e d . . [Yao e t a l * 76] have developed a dynamic database r e o r g a n i z a t i o n a l g o r i t h m t h a t may be used f o r the l a t t e r case. Input to t h e i r a l g o r i t h m r e q u i r e s the f o l l o w i n g parameters f o r each p h y s i c a l o r g a n i z a t i o n o c c u r i n g i n the database: (a) an i n i t i a l s e a rch c o s t , (b) an i n i t i a l r e o r g a n i z a t i o n c o s t , (c) a r a t e o f d e t e r i o r a t i o n of the search cost without r e o r g a n i z a t i o n , (d) a r a t e of d e t e r i o r a t i o n of the search c o s t assuming continuous r e o r g a n i z a t i o n and (e) a r a t e of i n c r e a s e of r e o r g a n i z a t i o n c o s t * They f u r t h e r c o n s i d e r the more r e a l i s t i c s i t u a t i o n e n t a i l i n g n o n l i n e a r s e a r c h c o s t s and d e s c r i b e a system ( c a l l e d a F i l e Design Analyser) t h a t outputs search c o s t e s t i m a t e s f o r " r e a l world" p h y s i c a l o r g a n i z a t i o n s (such as s e q u e n t i a l and ISAM f i l e s ) . T h e i r system accepts t h r e e : b a s i c v a r i e t i e s of d e s c r i p t i v e parameters: (a) l o g i c a l r e c o r d o r g a n i z a t i o n (such as number of a t t r i b u t e s , r e c o r d l e n g t h , number of keywords, etc) , (b) u t i l i z a t i o n i n f o r m a t i o n (such as number of i n s e r t i o n s , d e l e t i o n s and m o d i f i c a t i o n s , number of q u e r i e s , etc) and (c) p h y s i c a l storage parameters (such as block s i z e , t r a n s f e r r a t e , e t c ) . The r e s u l t of t h e i r e f f o r t s i s an approach capable o f automating d e c i s i o n s r e g a r d i n g the sc h e d u l i n g of p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by r e c o r d amendment ( i n our case by the a d d i t i o n and d e l e t i o n of t u p l e s from an RDMS). T h i s i s c l e a r l y B. P r e l i m i n a r i e s 30 r e l e v e n t t o any access path l e v e l p e r m i t t i n g the e x i s t e n c e of non-dynamic data s t r u c t u r e s . C. D e t a i l e d Problem D e f i n i t i o n Two aspects of p h y s i c a l r e o r g a n i z a t i o n i m p l i e d by changing p a t t e r n s of usage o f e x i s t i n g r e l a t i o n s are co n s i d e r e d i n d e t a i l i n t h i s t h e s i s (see I n t r o d u c t i o n ) . I n f o r m a l l y , what are considered i s a v a r i e t y o f t u p l e (or record) c l u s t e r i n g which can be c h a r a c t e r i z e d as " h o r i z o n t a l c l u s t e r i n g " and t h e problem o f dynamic secondary index s e l e c t i o n * In terms of System B t h i s concerns the automation of the: s e l e c t i o n of images (both c l u s t e r i n g and non- c l u s t e r i n g ) .. T h i s s e c t i o n presents a d e t a i l e d i n t r o d u c t i o n and d e f i n i t i o n of these problems.. A "base r e l a t i o n " i n a EDMS corresponds t o what might i n t u i t i v e l y be c a l l e d a stored r e l a t i o n . The t u p l e s comprising a base r e l a t i o n p h y s i c a l l y e x i s t on some form of mass s t o r a g e media. In c o n s t r a s t a " d e r i v e d " r e l a t i o n i s d e f i n e d i n terms o f base r e l a t i o n s and r e q u i r e s some form of query e v a l u a t i o n i n order to " m a t e r i a l i z e " i t s t u p l e s . Throughout previous s e c t i o n s r e f e r e n c e has been made t o the n o t i o n o f c l u s t e r i n g of base r e l a t i o n s * In f a c t a v a r i e t y of types of c l u s t e r i n g have been mentioned. In the EDMS OMEGA [Schmid e t a l . . 76] and i n SODMS [ S t o c k e r and Dearnley 74 ] a base r e l a t i o n may have i t s a t t r i b u t e s or columns p a r t i t i o n e d C. P r e l i m i n a r i e s 31 i n t o (not n e c e s s a r i l y d i s t i n c t ) subsets of a t t r i b u t e s where each subset i s assigned i t s own f i l e . . Such " v e r t i c a l p a r t i t i o n i n g " r e q u i r e s each requested t u p l e of the base r e l a t i o n r e b u i l t from i t s component s u b - t u p l e s from each subset. In System E a " v e r t i c a l a g g r e g a t i o n " c a p a b i l i t y permits grouping s e t s o f a t t r i b u t e s from more than one base r e l a t i o n i n the same p h y s i c a l segment. In a d d i t i o n , grouping t u p l e s a c c o r d i n g t o t h e i r p r o b a b i l i t y of access i s the s u b j e c t of a d d i t i o n a l work on SODMS i n [ K o l l i a s et a l . 76]. C l e a r l y then, c l u s t e r i n g support a t the acc e s s path l e v e l of a EDMS i s o f fundamental importance (and up t o now we have r e l i e d on reader i n t u i t i o n i n understanding t h i s i s s u e ) . .A more f o r m a l o u t l i n e i s now presented. One e s s e n t i a l c h a r a c t e r i s t i c of a l l mass storage media i s th a t they are block o r i e n t e d . Communication with such media i s i n terms of u s u a l l y l a r g e ( r e l a t i v e to t u p l e s i z e ) p h y s i c a l b l o c k s . R e t r i e v a l of any p a r t i c u l a r t u p l e or p a r t i a l t u p l e always i m p l i e s r e t r i e v a l o f a much l a r g e r b l o c k . When one co n s i d e r s t h a t block r e t r i e v a l time from a mass storage media i s enormously slow i n comparison t o the time t o access main memory (on the order of f i v e t o s i x o r d e r s of magnitude i n the case of moving head disks) i t i s easy t o see t h a t the o r g a n i z a t i o n of tu p l e s w i t h i n b l o c k s (or the c l u s t e r i n g o f t u p l e s ) has great impact on the performance of the RDMS. We d e f i n e the problem of t u p l e : c l u s t e r i n g , t h e r e f o r e , as the f o l l o w i n g : D1. c l u s t e r i n g - C l u s t e r i n g concerns the placement of C. P r e l i m i n a r i e s 32 t u p l e s of a base r e l a t i o n on mass storage media i n such a way as to minimize block r e a d s / w r i t e s r e l a t i v e t o a " c h a r a c t e r i z a t i o n of usage'! o f the base r e l a t i o n . From the above d e f i n i t i o n i t i s c l e a r t h a t the c l u s t e r i n g o r g a n i z a t i o n of t u p l e s i s r e l a t i v e a t the l e a s t to each a p p l i c a t i o n of the.RDMS. The problem i s i n f a c t fundamentally r e l a t e d t o how each a p p l i c a t i o n i s used (eg. the v a r i e t y of q u e r i e s which a r i s e ) . I t i s important, t h e r e f o r e * t h a t a n o t i o n of " c h a r a c t e r i z a t i o n of usage" be more f u l l y understood. The number and v a r i e t y of q u e r i e s on a base r e l a t i o n i s c l e a r l y an important component o f such a c h a r a c t e r i z a t i o n o f use t h a t i s p o s s i b l e t o be a u t o m a t i c a l l y maintained. Some p o s s i b i l i t i e s of how such query use may be c h a r a c t e r i z e d a r e : (a) an enumeration of a l l permitted q u e r i e s * (b) a sample of (a) above and (c) query s t a t i s t i c s . Both (a) and (b) above seem at the l e a s t t o be somewhat unwieldy* With r e s p e c t t o c o n t r o l l i n g our own c l u s t e r i n g we w i l l , i n f a c t , be using query s t a t i s t i c s . , Furthermore, r e q u i r i n g a p r i o r i knowledge of the f u t u r e use of a base r e l a t i o n i s not p o s s i b l e to a u t o m a t i c a l l y p r e d i c t . A c h a r a c t e r i z a t i o n of query h i s t o r y i s more a p p r o p r i a t e f o r our purposes. A c h a r a c t e r i z a t i o n of use of a base r e l a t i o n a l s o determines what v a r i e t y of i n d e x i n g should e x i s t on the base r e l a t i o n . An index, i n our case, i s a performance C. P r e l i m i n a r i e s 33 c o n s i d e r a t i o n . Furthermore, a l l such i n d i c e s a t the acc e s s path l e v e l may be c h a r a c t e r i z e d as e i t h e r a "primary index" o r a "secondary index". T h i s d i s t i n c t i o n i s important enough t o warrant a more complete e x p l a n a t i o n of the purpose and r o l e o f these two b a s i c v a r i e t i e s of i n d e x i n g . In a c e r t a i n sense t h e i r " l o g i c a l " purpose has been equivalent!. In a proced u r a l " t u p l e - a t - a - t i m e " data management language (DML) o p e r a t i n g on a network model, f o r example, no d i s t i n c t i o n i s made : between a primary or secondary index e i t h e r on i n i t i a t i o n o r durin g execution o f a n a v i g a t i o n through the network. Knowledge of a d i s t i n c t i o n a r i s e s i n terms of performance only.. Concerning the r e l a t i o n a l model scanning a base r e l a t i o n i n System B v i a a c l u s t e r i n g image i s much more e f f i c i e n t than v i a an u n c l u s t e r e d image. The d i f f e r e n c e between a primary and secondary index, t h e r e f o r e , i s t h a t the former has the a d d i t i o n a l p r o p e r t y o f i n f l u e n c i n g the p h y s i c a l l a y o u t (more u s u a l l y l o c a t i o n ) of a data e n t i t y . In terms of a EDMS the primary index of a base r e l a t i o n d e f i n e s the p h y s i c a l l o c a t i o n of i t s t u p l e s * S i n c e any o t h e r purpose of a primary index may be subsumed by f u r t h e r secondary i n d e x i n g t h i s r o l e of a primary index i s a l l important. With r e s p e c t t o the proposed design of an access path l e v e l of a EDMS the f o l l o w i n g d e f i n i t i o n s t h e r e f o r e C. P r e l i m i n a r i e s 34 apply:* °2. secondary index - as an access path t o data a secondary index i s a data s t r u c t u r e p e r m i t t i n g a v a r i e t y of more e f f i c i e n t search c a p a b i l i t y on a base r e l a t i o n but with no r e l a t i o n s h i p or bearing whatever on the c l u s t e r i n g nature of the t u p l e s w i t h i n t h a t base r e l a t i o n * D3. primary index A primary index i s a l s o an access path to data* I t has the a d d i t i o n a l purpose, however, of f a c i l i t a t i n g the d e s i r e d c l u s t e r i n g of t u p l e s w i t h i n a base r e l a t i o n . Consider what i s i m p l i e d by the above d e f i n i t i o n of a primary index* By v i r t u e of f a c i l i t a t i n g c l u s t e r i n g , the:primary index p r o v i d e s f o r the mapping between the t u p l e s of a base r e l a t i o n and t h e i r l o c a t i o n on mass s t o r a g e media. C l u s t e r i n g , i n t u r n , i s determined by a c h a r a c t e r i z a t i o n of use ( i n our case a query h i s t o r y s t a t i s t i c ) . . In an i n d i r e c t sense, then, a primary index maps t u p l e s of a base r e l a t i o n t o p h y s i c a l s t o r e i n a way dependent on how the base r e l a t i o n i s used* There are a number of ways i n which the i n f o r m a t i o n i n a base r e l a t i o n can be c l u s t e r e d . The v e r t i c a l p a r t i t i o n i n g (OMEGA, SODMS) and v e r t i c a l a g gregation (System R) i s one such approach* In v e r t i c a l p a r t i t i o n i n g a base r e l a t i o n i s d i v i d e d v e r t i c a l l y i n t o a s e t of s m a l l e r t a b l e s each s t o r i n g a subset ( p o s s i b l y n o n - d i s j o i n t ) of the a t t r i b u t e v a l u e s f o r each t u p l e (see F i g u r e II-2 ( a ) ) . T h i s approach presupposes t h a t each such subset of the a t t r i b u t e v a l u e s f o r the s e t of t u p l e s i s *Keep i n mind, however, t h a t r e l a t i o n a l language l e v e l of a component) has no notion of secondary) . . the DML/DDL presented at t h e RDMS (or at l e a s t the DML an index (whether primary o r C. P r e l i m i n a r i e s 35 a l l t h a t i s r e q u i r e d f o r a s i g n i f i c a n t f r a c t i o n o f a l l q u e r i e s (a " c h a r a c t e r i z a t i o n of use"). C l e a r l y s m a l l e r subsets o f a t t r i b u t e values imply s m a l l e r p a r t i a l - t u p l e s * T h i s i n t u r n f a v o u r a b l y impacts mass storage I/O because of the l a r g e r b l o c k i n g f a c t o r of the p a r t i a l - t u p l e s w i t h i n the disk pages comprising the t a b l e s . The above suggests the f o l l o w i n g d e f i n i t i o n : D4. v e r t i c a l c l u s t e r i n g - V e r t i c a l c l u s t e r i n g of a set of base r e l a t i o n s (conceptual) i s a p a r t i t i o n i n g of t h e i r combined set of columns or a t t r i b u t e values i n t o a s e t of t a b l e s ( p h y s i c a l ) i n such a way as to reduce d i s k r e a d s / w r i t e s r e l a t i v e t o a c h a r a c t e r i z a t i o n of use. In [ S c h k o l n i c h 76] the problem of v e r t i c a l c l u s t e r i n g i s a p p l i e d t o a set of base r e l a t i o n s l o g i c a l l y r e l a t e d i n terms of a h i e r a r c h y * An e f f i c i e n t a l g o r i t h m i s presented which generates an op t i m a l v e r t i c a l aggregation of the r e l a t i o n s i n t o a s e t of t a b l e s c a l l e d l i n e a r address spaces or LASs i n the paper. The a l g o r i t h m i s d r i v e n by a f o r m u l a t i o n o f c h a r a c t e r i z a t i o n of use of the base r e l a t i o n s * The author mentions that the model f o r the c h a r a c t e r i z a t i o n of use can e a s i l y be expanded t o account f o r oth e r forms of use o f the base r e l a t i o n s * As we have seen. System B supports an a b i l i t y t o s t o r e (or aggregate) more than one base r e l a t i o n i n the same f i l e (thus r e s u l t i n g i n l o c a t i n g t u p l e s from d i f f e r e n t base r e l a t i o n s i n the same b l o c k ) . Such aggregation i s c o n t r o l l e d by the c l u s t e r i n g o p t i o n of l i n k s d e c l a r e d by the system's users. With the p o s s i b l e e x c e p t i o n of a p p l i c a b i l i t y of work C. P r e l i m i n a r i e s 36 (by Schkolnich) i n t r o d u c e d above t o an RDMS no c o n s i d e r a t i o n i n the l i t e r a t u r e , as f a r as the author i s aware, has been giv e n to automating the ch o i c e of such c l u s t e r i n g images.. The problem remains open..No f u r t h e r c o n s i d e r a t i o n by t h i s t h e s i s i s g i v e n , i n f a c t , t o the s u b j e c t of v e r t i c a l c l u s t e r i n g * . H o r i z o n t a l vs V e r t i c a l C l u s t e r i n g R (conceptual) A~|~A2~|~A3~| ---I 1 I 1 1 c | (p h y s i c a l ) T1| A1 | A2 | T2 a I a I b I 1 I d 2 I c | I. a I a I b I 1 | 1 | 2 I A3 | -"I c I d I c I T3 A2 | A3 1 1 I c 1 I d 2 I c (a) V e r t i c a l P a r t i t i o n i n g I n t o T a b l e s T1, T2 S T3. (conceptual) R A1 I A2 | A3 | — I I I a | 1 | c | a | 1 | d | -> b | 2 | c | B1 B2 (phys i c a l ) " A l " | ~ A2~ | "IT 1 1 a J 1 I c a I 1 I d a I 1 | c b | 2 | c (b) H o r i z o n t a l C l u s t e r i n g I n t o Blocks B1 & B2* F i g u r e II-2 H o r i z o n t a l c l u s t e r i n g i s another way i n which i n f o r m a t i o n i n a base r e l a t i o n can be c l u s t e r e d . Concern here i s with p o s s i b l e groupings of the t u p l e s (or p a r t i a l - t u p l e s i f v e r t i c a l p a r t i t i o n i n g c l u s t e r i n g has been applied) themselves C. P r e l i m i n a r i e s 37 w i t h i n the p h y s i c a l pages comprising the t a b l e (s) (see F i g u r e II-2 ( b ) ) . In g e n e r a l the s e t s of elements d e f i n e d by these groupings need not be d i s j o i n t . In F i g u r e II-2 (b), f o r example, two c o p i e s of one of the t u p l e s i s a c t u a l l y s t o r e d . I t i s t h i s form of c l u s t e r i n g t h a t w i l l be d e a l t with i n d e t a i l by t l i i s t h e s i s . a l l o w i n g r e p l i c a t i o n of a t u p l e , however, w i l l not be considered any f u r t h e r * , T h i s second dimension of c l u s t e r i n g has the f o l l o w i n g d e f i n i t i o n : D5* h o r i z o n t a l c l u s t e r i n g - H o r i z o n t a l c l u s t e r i n g of a base r e l a t i o n i s a p a r t i t i o n i n g of i t s t u p l e s i n t o f i x e d s i z e d b l o c k s or d i s k pages i n such a way as to reduce d i s k r e a d s / w r i t e s r e l a t i v e t o a c h a r a c t e r i z a t i o n of use. F i g u r e II-3 and Table II-3 i l l u s t r a t e how a c h a r a c t e r i z a t i o n of guery use impacts the performance of sample h o r i z o n t a l c l u s t e r i n g s o f a r e l a t i o n i n block reads..As expected, a random a l l o c a t i o n of the t u p l e s t o b l o c k s y i e l d s the p o o r e s t performance. More i n t e r e s t i n g i s the o b s e r v a t i o n t h a t s o r t i n g does not y i e l d the b e s t . I n f a c t , our approach to h o r i z o n t a l c l u s t e r i n g i n t h i s t h e s i s would r e s u l t i n the t h i r d h o r i z o n t a l p a r t i t i o n i n g of the t u p l e s ( i . e . c l u s t e r 3 ) which, as s h a l l be seen, i s o p t i m a l . . T h i s l e a d s us to our l a s t d e f i n i t i o n concerning c l u s t e r i n g : D6. o p t i m a l h o r i z o n t a l c l u s t e r i n g - The optimal h o r i z o n t a l c l u s t e r i n g of a base r e l a t i o n i s the h o r i z o n t a l c l u s t e r i n g r e s u l t i n g i n the minimal r e t r i e v a l cost r e l a t i v e t o a c h a r a c t e r i z a t i o n of query h i s t o r y where r e t r i e v a l c o s t i s d e f i n e d i n terms of d i s k r e a d s / w r i t e s . Some i n t e r e s t i n g work r e g a r d i n g automation of h o r i z o n t a l C. P r e l i m i n a r i e s 38 Example H o r i z o n t a l C l u s t e r i n g c l u s t e r l (random) | | . . . . _ _ .. • - • -A1 |A2 | B1 f 1 1 B2| | | B3| 1 1 B4 r I 1 1 2 | I 2 | 4 J I 1 1 3 | I 1 1 1 1 1 I I 3 1 1 1 I 3 | 3 | I 3 1 4 | I 3 2 1 I 2 | I 2 I 2 | 1 1 I 4 I I 4 1 4 | I 4 1 1 I 3 | I 4 | 2 | I 2 | 1 [ I 2 1 3 | I 4 3 1 I 4 | I. ___| ;_| l _ I __\ l_ |~ 2 I 1 I 2 I 2 | c l u s t e r 2 (sorted) 2 I 3 | . . . ... . . .. . ___ 2 I 4 | B1|' 1 1 B2| | | B3 | 1 1 B4|~ 3 I 1 I I 1 I 1 I I 2 | 1 | I 3 1 1 I | 4 1 3 I 2 | I 1 I 2 | I 2 | 2 | I 3 I 2 | | 4 2 3 I 3 | I 1 | 3 | I 2 | 3 | I 3 I 3 | j 4 3 3 I 4 I I 1 | 4 | I 2 | 4 | I 3 I 4 | | 4 4 4 I 1 I I | | l _ |_ 4 | 2 | 4 I 3 | c l u s t e r 3 ( m u l t i d i m e n s i o n a l ^ sorted) 4 I 4 | . . . _ . . | :_ | Bl | I I B2| | | B3| I I B4 | 1 1 I 1 \ I 1 I 3 | I 3 I 1 I I 3 3 1 1 I 2 | | 1 | 4 | I 3 I 2 | I 3 4 ! 2 1 1 I I 2 | 3 | I 4 I 1 I I 4 I 3 1 2 I 2 | I 2 | 4 | 1 4 I 2 | I 4 4 1 l _ I. F i g u r e I I - 3 c l u s t e r i n g concerns document r e t r i e v a l systems. How such systems are used d i f f e r s from "normal" a p p l i c a t i o n s of a DMS. Users of document r e t r i e v a l systems are u s u a l l y never c e r t a i n of what they wish r e t r i e v e d . That i s , r e c o r d s t h a t may s a t i s f y the query a c c o r d i n g to the user (and should t h e r e f o r e be r e t r i e v e d ) may not e n t i r e l y f u l f i l the user's a t t r i b u t e / v a l u e c o n s t r a i n t s . Furthermore, users of such a system place g r e a t e r weight on the system s a t i s f y i n g a "get about n of the most r e l e v a n t r e c o r d s " c o n s t r a i n t than on the more usual "get a l l and only r e l e v a n t r e c o r d s " . C. P r e l i m i n a r i e s 39 In [ S a l t o n e t a l . 77] (see a l s o [ S a l t o n 79a] and [ S a l t o n 79b]) a system i s d e s c r i b e d t h a t f a c i l i t a t e s t h i s environment. Although the h o r i z o n t a l c l u s t e r i n g 1 accommodated by the system C l u s t e r Performance R e l a t i v e t o a - Que r y Group c l u s t e r l c l u s t e r 2 c l u s t e r 3 Query c o s t | c o s t c o s t A 1=1 4 1 1 2 A 1=2 | 3 1 2 A 1=3 | 4 I 1 I 2 A 1=4 3 I 1 2 A 2=1 | 3 I 4 2 A 2= 2 2 I 4 I 2 A2=3 3 I 4 2 A2=4 | 2 I 4 2 t o t a l s 24 I 20 16 Table II-3 i s mora:of a by-product than a design g o a l i t s r e a l i z a t i o n i s d i r e c t l y implemented as the set of lowest l e v e l c l u s t e r groups* Two aspects of search i n t h e i r system are r e t r i e v a l " p r e c i s i o n " and " r e c a l l " . In t h e i r words: " P r e c i s i o n i s d e f i n e d as the p r o p o r t i o n of r e t r i e v a l items a c t u a l l y found r e l e v a n t , whereas r e c a l l i s the p r o p o r t i o n of r e l e v a n t m a t e r i a l s t T h e i r use of the word " c l u s t e r " has a d i f f e r e n t meaning from our own* z T h i s passage i s taken from page 13 of [ S a l t o n et a l . 77]. C. P r e l i m i n a r i e s 40 a c t u a l l y r e t r i e v e d * " 2 Such c h a r a c t e r i s t i c s of search w i l l not be c o n s i d e r e d , however, i n the proposed access path l e v e l . The assumption i s made t h a t "those asking" know e x a c t l y what they want.. Furthermore, the p a s s i v e environment of document r e t r i e v a l i s c o n s i d e r e d o v e r l y c o n s t a i n i n g . In c o n t r a s t t o S a l t o n 1 s s t a t i c data s t r u c t u r e s dynamic support f o r optimal h o r i z o n t a l c l u s t e r i n g i s provided by the access path l e v e l * The dynamic nature r e l a t e s t o accommodating changing c h a r a c t e r i z a t i o n s of usage* another important c o n s i d e r a t i o n i n determining how to group the t u p l e s of a base r e l a t i o n w i t h i n d i s k b l o c k s concerns each t u p l e ' s p r o b a b i l i t y of access..Consider, f o r example, a r e l a t i o n R with the a t t r i b u t e s SEX and NAME. Assume f u r t h e r t h a t there are 5 t u p l e s i n E with SEX= MALE and 95 with SEX=FEMALE. I f each d i s k b l o c k can h o l d 10 t u p l e s from E and i f h a l f the g u e r i e s s p e c i f y SEX=MALE and the other h a l f SEX=FEHALE then i t i s o b v i o u s l y d e s i r a b l e t o l o c a t e the 5 t u p l e s i n E with SEX=MALE i n the same d i s k block. Such a c l u s t e r i n g c o n s i d e r a t i o n i s j u s t another aspect o f our n o t i o n s of h o r i z o n t a l and v e r t i c a l c l u s t e r i n g s i n c e a c h a r a c t e r i z a t i o n o f use of B can o f course convey i n f o r m a t i o n (such as r e t r i e v a l p r o b a b i l i t y ) on i n d i v i d u a l t u p l e s and a t t r i b u t e s of B. I t i s n e v e r t h e l e s s u s e f u l to d i s t i n g u i s h t u p l e r e t r i e v a l p r o b a b i l i t y or any other i n f o r m a t i o n r e g a r d i n g i n d i v i d u a l t u p l e s from h o r i z o n t a l c l u s t e r i n g c o n s i d e r a t i o n s . There are s e v e r a l reasons: C* P r e l i m i n a r i e s 41 (a) Dynamic maintenance of a c h a r a c t e r i z a t i o n of use f o r the purposes o f h o r i z o n t a l c l u s t e r i n g i s g r e a t l y s i m p l i f i e d , (b) the problem of determining optimal h o r i z o n t a l c l u s t e r i n g becomes much more managable when i n f o r m a t i o n regarding i n d i v i d u a l t u p l e s does not have to be taken i n t o account and (c) i n e x c i s i n g i n d i v i d u a l t u p l e r e t r i e v a l p r o b a b i l i t y from h o r i z o n t a l c l u s t e r i n g c o n s i d e r a t i o n s we d i v o r c e the important s u b j e c t of i n f o r m a t i o n a r c h i v a l * One p o s s i b l e accommodation f o r t u p l e r e t r i e v a l p r o b a b i l i t y might i n v o l v e c o n s i d e r i n g i t a more g e n e r a l c l u s t e r i n g problem than h o r i z o n t a l or v e r t i c a l c l u s t e r i n g . . A base r e l a t i o n may then be d i s t r i b u t e d among s e v e r a l l e v e l s (ii.es. d i f f e r e n t areas/types of storage) ; each of which i s then d e d i c a t e d to s t o r i n g i n f o r m a t i o n with the same: l i k e l y h o o d of r e t r i e v a l ; each of which may then have h o r i z o n t a l and v e r t i c a l c l u s t e r i n g o c c u r r i n g w i t h i n . As a r e s u l t , r e l a t i o n R, above, would have t u p l e s o c c u r r i n g i n at l e a s t two l e v e l s s i n c e those with SEX=MALE have g r e a t e r p r o b a b i l i t y of r e t r i e v a l . . Such c o n s i d e r a t i o n s , however, remain i n the area of f u t u r e r e s e a r c h p o s s i b i l i t i e s . 1 The h o r i z o n t a l c l u s t e r i n g support d i s c u s s e d i n the t h e s i s assumes a uniform p r o b a b i l i t y of r e t r i e v a l o f the t u p l e s of a base r e l a t i o n . Our d e f i n i t i o n of a secondary index has so f a r been very general* Automating the s e l e c t i o n of i n d i c e s r e q u i r e s a much 40ne a p p l i c a b l e r e f e r e n c e i s work i n [ K o l l i a s e t al*,77] r e g a r d i n g the automatic a l l o c a t i o n of d i s k f i l e s to o n - l i n e and o f f - l i n e d i s k packs* C. P r e l i m i n a r i e s 42 more e x p l i c i t understanding of t h e i r format and of how they are used i n query evaluation;. D e t a i l s concerning both these i s s u e s with regard t o secondary i n d e x i n g c o n s i d e r e d by t h i s t h e s i s are given i n Chapter I I I . There a r e , however, some ge n e r a l design c h a r a c t e r i s t i c s t h a t are common to almost a l l secondary i n d e x i n g considered i n the l i t e r a t u r e * Secondary i n d i c e s are u s u a l l y implemented by i n v e r s i o n s . T h i s i s i n c o n t r a s t t o a m u l t i l i s t [Prywes and Gray 63] or m u l t i r i n g [Wiederhold 77] approach* Consider t h a t each value of an indexed a t t r i b u t e has a s e t of t u p l e s a s s o c i a t e d with i t . Each t u p l e can be re p r e s e n t e d by a t u p l e p o i n t e r t h a t i s the p h y s i c a l address of the t u p l e i n <page,displacement form. An i n v e r s i o n approach p h y s i c a l l y s t o r e s each t u p l e p o i n t e r s e t with i t s r e s p e c t i v e value i n the i n d e x * 1 In such a scheme i t can be b e n e f i c i a l t o compute t u p l e p o i n t e r s e t i n t e r s e c t i o n s and unions when e v a l u a t i n g q u e r i e s t h a t provide more than one indexed (inverted) value. Performing such o p e r a t i o n s i s much e a s i e r and guicker i f each t u p l e p o i n t e r s e t i n the i n d i c e s i s maintained i n s o r t e d order* The problem of secondary index s e l e c t i o n f o r i n v e r s i o n approaches has been the s u b j e c t of some res e a r c h . In [ S c h k o l n i c k 74] and [Anderson and Berra 77] f u n c t i o n s are d e r i v e d t h a t produce a c o s t value f o r a given permutation o f *See [ S a i t o h e t a l . 77 ] f o r an e x c e l l e n t b r i e f o u t l i n e of " i n v e r t e d f i l e " and " m u l t i l i s t " approaches t o secondary i n d e x i n g . C. P r e l i m i n a r i e s 43 indexing on the a t t r i b u t e s of a base r e l a t i o n . S c h k o l n i c k demonstrates a " r e g u l a r i t y c o n d i t i o n " on h i s c o s t f u n c t i o n p e r m i t t i n g a more e f f i c i e n t a l g o r i t h m t o f i n d the o p t i m a l i n d e x i n g s e t whereas Anderson and Berra r e l y on a brute f o r c e approach. The l a t t e r i n c l u d e much more implementation d e t a i l i n t o t h e i r c o s t f u n c t i o n however. S c h k c l n i c k assumes t h a t computing i n t e r s e c t i o n s and unions of t u p l e p o i n t e r s e t s r e t r i e v e d from secondary i n d i c e s can always be accomplished i n main memory and t h e r e f o r e c o n t r i b u t e s n e g l i g i b l y t o o v e r a l l c o s t . . H i s f u n c t i o n i n c o r p o r a t e s r e t r i e v a l and maintenance ( i . e . update, i n s e r t i o n and d e l e t i o n ) c o s t s . In a d d i t i o n to r e t r i e v a l and maintenance Anderson and Berra c o n s i d e r the c o s t of storage..They a l s o d e a l i n much more d e t a i l with v a r i o u s forms of i n v e r s i o n s * . I n p a r t i c u l a r they c o n s i d e r the merits of m a i n t a i n i n g sorted t u p l e p o i n t e r s e t s over unsorted s e t s . A much more pragmatic approach to secondary index s e l e c t i o n i s d e s c r i b e d i n [ F a r l e y and Schuster 75].„ The emphasis i n t h e i r paper i s on d e t a i l s concerning query e v a l u a t i o n on s i n g l e r e l a t i o n s with secondary i n d i c e s on s i n g l e a t t r i b u t e s implemented as i n v e r s i o n s with s o r t e d t u p l e p o i n t e r s e t s . The s t r a t e g y adopted i n query e v a l u a t i o n i n c o r p o r a t e s the use of a c o s t f u n c t i o n . 1 T h i s f u n c t i o n can be i T h i s i s t r u e of a l l query e v a l u a t i o n schemes of a l l RDHSs d i s c u s s e d i n t h i s t h e s i s . C. P r e l i m i n a r i e s 44 used to d e r i v e an o v e r a l l c o s t of e v a l u a t i o n of any query given a p a r t i c u l a r i n d e x i n g s e t . Using t h i s cost f u n c t i o n along with a r e p r e s e n t a t i v e sample of past q u e r i e s allows them t o measure the performance of any proposed i n d e x i n g s e t . As i n [Anderson and Berra 77] they then f o l l o w a brute f o r c e approach i n d e r i v i n g t h i s measurement f o r each p o s s i b l e i n d e x i n g s e t . T h e i r approach, however, f a i l s t o take i n t o c o n s i d e r a t i o n both maintenance and storage c o s t s (these c o n s i d e r a t i o n s along with the u l t i m a t e i n d e x i n g c h o i c e are l e f t t o the: database a d m i n i s t r a t o r ) . Another s i m p l i f y i n g assumption adopted by F a r l e y and Schuster as w e l l as Anderson, Berra and S c h k o l n i c k i s t h a t secondary in d e x i n g on s i n g l e a t t r i b u t e s only i s permitted* R e c a l l t h a t an image i n System R can be based on more than one a t t r i b u t e . The author i s not aware of any work r e g a r d i n g secondary index s e l e c t i o n d e a l i n g with t h i s g e n e r a l i z a t i o n * The s u b j e c t of dynamic secondary index s e l e c t i o n i s c l e a r l y n o n - t r i v i a l (see f o r example [Comer 7 8 ] ) . As i n the case of c l u s t e r i n g , such s e l e c t i o n i s determined p r i m a r i l y by how base r e l a t i o n s are used. A s u i t a b l e c h a r a c t e r i z a t i o n of use* t h e r e f o r e , i s as important to secondary index s e l e c t i o n as i t i s t o c l u s t e r i n g c o n s i d e r a t i o n s . One l a s t i s s u e concerning problem d e f i n i t i o n s ends t h i s s e c t i o n * . T h i s concerns terminology used i n d e s c r i b i n g v a r i o u s query types. A l l d e f i n i t i o n s given are r e l e v a n t t o an access path l e v e l of an RDMS. Furthermore, any i n s t a n c e of any o f the C. P r e l i m i n a r i e s H5 f o l l o w i n g guery types (the semantics of these i s obvious) i s assumed t o apply t o a r e l a t i o n S with k columns: D7. p a r t i a l match - A p a r t i a l match query provides a value f o r a proper subset of the a t t r i b u t e s of E. I t has the form: {C0Ln1=VALl} 5 {COLn2=VAL2} & ... .& {COLnj=VALj} where j<k and the COLni are d i s t i n c t ( i * e . i-*=m i m p l i e s ni-»=nm) . D8. exact match - An exact match query p r o v i d e s a value f o r each column of R. I t s form i s as above save t h a t j must now equal k. D9. p a r t i a l range query - A p a r t i a l range query pro v i d e s a range of v a l u e s f o r a proper subset o f the a t t r i b u t e s of B. I t has the form: (COLni between VAL11 and VAL123 & (C0Ln2 between VAL21 and VAL22) & ... ... & {COLnj between VALJ1 and VALJ2] where j<k and the COLni are d i s t i n c t . D10. range query - A range query provides a range of val u e s f o r each a t t r i b u t e o f E. I t s form i s as above save t h a t j must now e q u a l k. D11. i n t e r s e c t i o n query - I n f o r m a l l y , an i n t e r s e c t i o n query i s d i s j u n c t i o n s of the above. I t has the form: {Q1 or Q2 or .... or Qm} where m>0 and each Qi i s a query of any v a r i e t y of those above* D* S i m p l i f y i n g Assumptions There are a number of s i m p l i f y i n q assumptions r e g a r d i n g the hardware environment f o r a EDMS. The most important concerns the mass storage media i t s e l f . In p a r t i c u l a r , tape storage (along with the s e q u e n t i a l mode of use i t i m p l i e s ) i s no l o n g e r permitted f o r a p p l i c a t i o n s other than i n f o r m a t i o n a r c h i v a l and backup, s t a t i s t i c s a c q u i s i t i o n and t r a n s a c t i o n l o g g i n g * C u r r e n t l y a c t i v e base r e l a t i o n s are assumed to r e s i d e D. P r e l i m i n a r i e s 46 on storage media of at l e a s t the performance of moving head hard d i s k s , . Another important assumption concerns the amount of main memory and c e n t r o l p r o c e s s i n g u n i t (CPU) overhead r e q u i r e d t o support a EDMS. With phenominal t e c h n o l o q i c a l advances i n l o g i c d e n s i t i e s i t i s now f e a s i b l e t o assume an a b i l i t y t o a l l o c a t e l a r g e amounts of main memory to b u f f e r i n g a p p l i c a t i o n s and even to d e d i c a t e CPUs to the EDMS software. T r a d i t i o n a l b a l a n c i n g c o n s t r a i n t s f o r i n f o r m a t i o n c o n t r o l systems (and even o p e r a t i n g systems) such as main memory c o s t and CPU overhead are no l o n g e r considered to apply. Taking t h e i r p l a c e i s a requirement f o r g e n e r a l i t y , homogeneity and ease of use. Eesponse time i s considered a more important measurement o f system performance than e i t h e r throughput or m i n i m i z a t i o n of hardware c o s t s . S e v e r a l assumptions concerning the a p p l i c a t i o n s environment are made. Dedicated r e a l time c o n t r o l a p p l i c a t i o n s , f o r example, are not supported by the access path l e v e l . A requirement to ensure a constant upper bound f o r any o p e r a t i o n provided at the access path l e v e l would add c o n s i d e r a b l y to the complexity of the i s s u e s c o n s i d e r e d . Furthermore, the very important t o p i c of d i s t r i b u t e d support f o r a RDMS r e l a t i o n a l language l e v e l i s not c o n s i d e r e d . Because of the emphasis on response time no attempt i s made to apply b a t c h i n g t e c h n i q u e s to query e v a l u a t i o n or base r e l a t i o n D. P r e l i m i n a r i e s 47 maintenance. \ The access time f o r a block on a moving head hard d i s k i s comprised of three major components.. The f i r s t o f t h e s e , c a l l e d seek time, i s the time r e g u i r e d to p h y s i c a l l y move a r e a d / w r i t e head t o the d e s i r e d t r a c k . R o t a t i o n a l delay i s the time r e q u i r e d f o r the read/write head to l o c a t e the s t a r t o f the d e s i r e d b l o c k on the t r a c k * The t r a n s f e r time taken to read the d e s i r e d block i s the l a s t of these access time components. The problem of l o c a t i n g the b l o c k s c o n t a i n i n g the t u p l e s of base r e l a t i o n s on the d i s k i n a way t h a t reduces the o v e r a l l seek and r o t a t i o n a l delay times i n a c c e s s i n g these b l o c k s i s a l s o not considered by t h i s t h e s i s . Both t h i s and the s c h e d u l i n g of d i s k I/O o p e r a t i o n s are: c o n s i d e r e d the r e s p o n s i b i l i t y of lower l e v e l software or hardware* l R e c a l l t h a t the d e s i g n e r s of SODHS assumed j u s t the o p p o s i t e . A* The System 48 Chapter I I I : The System A. O v e r a l l Design The f i r s t p a r t o f t h i s s e c t i o n concerns t h e : s p e c i f i c a t i o n f o r a proposed access path l e v e l of a EDMS. The o v e r a l l d e s ign f o r an implementation c a l l e d SOEAAM (for S e l f - O r g a n i z i n g E e l a t i o n a l A s s o c i a t i v e Access Method) t h a t attempts t o s a t i s f y t h i s s p e c i f i c a t i o n completes the s e c t i o n . G e n e r a l l y , the access path l e v e l provides the i n i t i a l r e l a t i o n a l a b s t r a c t i o n . To accomplish t h i s , c o n t r o l over a l l p h y s i c a l o r g a n i z a t i o n d e t a i l s concerning a l l base r e l a t i o n s r e s i d e s completely wi t h i n i t . T h i s i n c l u d e s the f o l l o w i n g c l u s t e r i n g and i n d e x i n g c o n s i d e r a t i o n s : (a) dynamic support f o r the h o r i z o n t a l c l u s t e r i n g of a l l base r e l a t i o n s and (b) automatic s e l e c t i o n of s i n g l e - a t t r i b u t e i n v e r s i o n s . The d e c i s i o n s a t the ac c e s s path l e v e l with regard t o h o r i z o n t a l c l u s t e r i n g and secondary index s e l e c t i o n f o r a base r e l a t i o n are governed by a v a r i e t y of s t a t i s t i c s t h a t determine a c h a r a c t e r i z a t i o n of use and secondary index u s e f u l n e s s f o r the base r e l a t i o n . A l l s t a t i s t i c s used by the access path l e v e l are maintained by i t * . There i s good reason, however, t o endow the access path l e v e l with an a b i l i t y to take " a d v i c e " i n regard to p h y s i c a l o r g a n i z a t i o n d e t a i l s * Such s i t u a t i o n s r e q u i r i n g t h i s are an i n i t i a l base r e l a t i o n c r e a t i o n (when the system has had no ft. The System 49 o p p o r t u n i t y t o a c q u i r e a c h a r a c t e r i z a t i o n of use i t s e l f ) and the a c q u i s i t i o n of a p r i o r i knowledge by a user of a dramatic change i n the c h a r a c t e r i z a t i o n of use of an e x i s t i n g base r e l a t i o n (not p r e d i c t a b l e by the system). T h e : l a t t e r case may imply an immediate a l t e r a t i o n on the c u r r e n t l y e x i s t i n g secondary i n d e x i n g s e t . The a d v i c e t a k i n g a b i l i t y i s accomplished by p e r m i t t i n g the user t o d i r e c t l y o v e r r i d e the c o n t r o l i n f o r m a t i o n used by the access path l e v e l to govern p h y s i c a l o r g a n i z a t i o n * Support f o r a s s o c i a t i v e a c c e s s to i n f o r m a t i o n i s p r o v i d e d f o r s i n g l e base r e l a t i o n s only. T h i s i s c o n s i s t e n t with the c a p a b i l i t i e s of access path l e v e l s of most e x i s t i n g EDMSs. Furthermore* f i n d i n g a l l t u p l e s of a base r e l a t i o n s a t i s f y i n g any g i v e n search p r e d i c a t e never r e q u i r e s c o n s i d e r a t i o n o f a t u p l e w i t h i n the base r e l a t i o n more than once.. That i s , a t u p l e may be accepted or r e j e c t e d f o r r e t r i e v a l on f i r s t examination. In the worst case* t h e r e f o r e * responding to a query may r e q u i r e a t most one complete scan o f a l l the t u p l e s within a base r e l a t i o n . N e v e r t h e l e s s , a l l the f o l l o w i n g v a r i e t y of q u e r i e s are d i r e c t l y supported . (see s e c t i o n C of Chapter I I f o r the d e f i n i t i o n s ) : (a) exact match (b) p a r t i a l match (c) range query (d) p a r t i a l range query (e) i n t e r s e c t i o n query i n s e c t i o n D of Chapter II a number o f s i m p l i f y i n g A. The System 50 assumptions c o n c e r n i n g the proposed environment were o u t l i n e d . Assuming a l l of these, the access path l e v e l s t i l l r e t a i n s a l a r g e measure of s u i t a b i l i t y f o r i t s proposed environment ( i . e . the access path l e v e l of a RDMS which r e p l a c e s the f i l e system of an o p e r a t i n g system). Such an environment* however, n e c e s s i t a t e s support f o r a l a r g e measure of concurrency of access and maintenance on a l l base r e l a t i o n s . . T h i s i n c l u d e s concurrency at both the l o g i c a l l e v e l (more than one ongoing a c c e s s / m o d i f i c a t i o n on the same base r e l a t i o n ) and p h y s i c a l l e v e l (more than one ongoing access to the same p h y s i c a l block or disk page)• In a d d i t i o n , a type of concurrency p e r m i t t i n g normal use of a l l base r e l a t i o n s during any p h y s i c a l r e o r g a n i z a t i o n i s e s s e n t i a l (see s e c t i o n A of Chapter I I ) . We d i s t i n g u i s h t h i s l a t t e r as " r e o r g a n i z a t i o n concurrency" from the preceeding r e f e r e n c e to "user concurrency". In f a c i l i t a t i n g r e o r g a n i z a t i o n concurrency every e f f o r t i s made to minimize the degeneration of r e t r i e v a l performance of those base r e l a t i o n s undergoing p h y s i c a l r e o r g a n i z a t i o n . User concurrency may be understood as a type o f multiprogramming* In t h i s case the i n d i v i d u a l " j o b s " are t r a n s a c t i o n s * A t r a n s a c t i o n i n t h i s sense i s a sequence o f (u s u a l l y more than one) r e t r i e v a l and maintenance o p e r a t i o n s on a set of base r e l a t i o n s . The "shared r e s o u r c e s " , t h e r e f o r e , are the base r e l a t i o n s themselves. As a r e s u l t * l o c k i n g requirements a r i s e i n order t o ensure the l o g i c a l i n t e g r i t y of a l l base r e l a t i o n s t c each t r a n s a c t i o n . A l l such l o c k i n g requirements (both the l o g i c a l and A. The System 51 p h y s i c a l l e v e l s ) are the s o l e r e s p o n s i b i l i t y of the a c c e s s path l e v e l . Because of t h i s the l e v e l of user concurrency t h a t may a r i s e can r e s u l t i n t r a n s a c t i o n deadlocking* . The f a c i l i t i e s necessary t o support the l o c k i n g requirements must t h e r e f o r e i n c l u d e c a p a b i l i t i e s f o r t r a n s a c t i o n deadlock d e t e c t i o n * . Furthermore* recovery f a c i l i t i e s necessary f o r deadlock recovery (and, of course, system f a i l u r e ) are c a l l e d f o r . A l l such recovery f a c i l i t i e s are a l s o the s o l e r e s p o n s i b i l i t y of the access path l e v e l . We begin our d e s c r i p t i o n of the proposed system by an i n t r o d u c t i o n of the l a y o u t and data s t r u c t u r e s r e s i d e n t on mass storage t h a t are used by SOEAAM to access and maintain each base r e l a t i o n . F i g u r e I I I - 1 below i l l u s t r a t e s the s t o r a g e s t r u c t u r e s i n v o l v e d (the base r e l a t i o n i s assumed to have a t l e a s t j a t t r i b u t e s ) . The primary index of a base r e l a t i o n used to c o n t r o l the h o r i z o n t a l c l u s t e r i n g of t h e base r e l a t i o n i s the k-d t r e e data s t r u c t u r e (see [ B e n t l e y 75], [ B e n t l e y 78a] and [ B e n t l e y 78b]). T h i s data s t r u c t u r e i s extremely robust i n the v a r i e t y o f q u e r i e s t h a t i t supports. In f a c t , any i n s t a n c e o f an i n t e r s e c t i o n query can always be evaluated with the e x c l u s i v e use of t h e k-d t r e e primary index. J o t e t h e r e f o r e t h a t , given the d e f i n i t i o n above of the v a r i e t y of v a l i d q u e r i e s to our access path l e v e l (see the user i n t e r f a c e ) , the use of the primary index i s always p o s s i b l e . T h i s f a c t i s i n c o r p o r a t e d i n t o the search s t r a t e g i e s used (see S e c t i o n E of t h i s c h a p t e r ) . A..The System 52 Storage S t r u c t u r e of a Base R e l a t i o n i n SORAAM Primary Index « 1 I I I | k-d t r e e |- 1 - - • -, ,-„-• I I c l u s t e r e d t u p l e s J comprising the j — base r e l a t i o n | _ l _ l B - t r e e on COLn 1 B - t r e e on COLn 2 • • B - t r e e : on COLn j - I l_ The i n d e x i n g s e t i s : {n1, n2 r n j ) Fi g u r e I I I - 1 Another very important p r o p e r t y of k-d t r e e s with r e s p e c t t o h o r i z o n t a l c l u s t e r i n g i s t h a t , with a s u i t a b l e c h a r a c t e r i z a t i o n o f use of the base r e l a t i o n s and c o n t r o l o f the k-d t r e e s , o p t i m a l h o r i z o n t a l c l u s t e r i n g of t h e i r t u p l e s r e s u l t s * . Both t h i s property and the c o n t r o l i s s u e s are d i s c u s s e d i n d e t a i l i n S e c t i o n s B and C of t h i s chapter. The secondary i n d e x i n g i l l u s t r a t e d by F i g u r e I I I - 1 A* The System 53 i n v o l v e s the use of the B-tree data s t r u c t u r e . 1 A B-tree index may e x i s t f o r each p o s s i b l e " r e l a t i o n / a t t r i b u t e " p a i r of the base r e l a t i o n s * In the example i l l u s t r a t e d j a t t r i b u t e s o f the base r e l a t i o n are each indexed by a B-tree. _In SOEAAM a base r e l a t i o n need not be f u l l y i n v e r t e d ( i . e . a B-tree f o r each p o s s i b l e a t t r i b u t e ) . We denote the s e t of a t t r i b u t e s f o r a base r e l a t i o n t h a t a re indexed as the " i n d e x i n g s e t " . There are t h r e e . u s e r s u p p l i e d parameters f o r each base r e l a t i o n t h a t c o n t r o l i t s in d e x i n g s e t . These parameters (Plow, Phigh and S) are in t r o d u c e d below i n Table I I I - 1 . . The parameters along with v a r i o u s other s t a t i s t i c s ( i n c l u d i n g the important " g l o b a l performance measure" or g.p.m.) t h a t are maintained f o r each base r e l a t i o n a re used by our system t o c o n t r o l the s e l e c t i o n of the s i n g l e a t t r i b u t e i n v e r s i o n s * S e c t i o n D o f t h i s chapter p r e s e n t s more f u l l y the d e t a i l s o f these aspects of SOEAAM. There are th r e e base r e l a t i o n s t h a t must e x i s t i n a l l implementations of SOEAAM ( i * e . t h a t are r e q u i r e d and used by SOEAAM i t s e l f ) c a l l e d EELATIONS, ATTBIBOTES and PBOCESSES.„The e x i s t e n c e of a l l t h r e e i s necessary f o r the proper o p e r a t i o n o f the system. The t u p l e s i n RELATIONS and ATTBIBOTES d e f i n e l F 6 r an e x c e l l e n t i n t r o d u c t i o n t o B-tr e e s see. [Comer 79] or [Knuth 73], [Bayer and McCreight 72] f i r s t i n t r o d u c e d these s t r u c t u r e s . There has been much work on B-trees s i n c e . [Bayer and S c h o l n i c k 76], f o r example, d i s c u s s an i n t e r e s t i n g v a r i a n t c a l l e d t h e p r e f i x B-tree and [Bayer and Unterauer 76] c o n s i d e r concurrency of o p e r a t i o n s on them. [Lomet 79] presents an i n t e r e s t i n g method of o r g a n i z i n g i n t e r n a l nodes of a p r e f i x B- t r e e . A. The System 54 the e x i s t e n c e , l o c a t i o n and format of a l l base r e l a t i o n s and t h e i r a t t r i b u t e s respectively,* Furthermore, a l l s t a t i s t i c s used by the access path l e v e l are a l s o contained i n these r e l a t i o n s * PROCESSES e x i s t s f o r the purpose of those a s p e c t s of SORAAM concerning t r a n s a c t i o n maintenance. Tuples i n PROCESSES convey the e x i s t e n c e of processes with ongoing t r a n s a c t i o n s and t h e i r r e s p e c t i v e recovery p o i n t s (we e l a b o r a t e on t h i s i n the o u t l i n e o f the user i n t e r f a c e ) . . Table I I I - 1 i l l u s t r a t e s the format o f these base r e l a t i o n s * The a t t r i b u t e (s) enclosed i n p a r e n t h e s i s f o r a S p e c i a l R e l a t i o n s Used SORAAM R e l a t i o n Name RELATIONS ATTRIBUTES PROCESSES A t t r i b u t e L i s t (Rname), B s i z e , N, D, PIB, PIBloc, IX, STPB, f f 1 , f f 2 , P s i z e , g.p*m., Plow, Phigh, S, qno, qnoold, r a d i x l , RDQreads, DQreads, r a d i x 2 , Pcost, r a d i x 3 , buf (Rname, Aname), COL#, format, n* d, I f l a g , I s i z e , SIB, S I B l o c , f , f o l d , min, max, b (process*) , q u i e t * Table I I I - 1 r e l a t i o n comprise the key t o that r e l a t i o n * A l l users (or processes) are permitted access and maintenance o p e r a t i o n s on them (no s e c u r i t y c o n s i d e r a t i o n s are assumed by SORAAM). A l l o p e r a t i o n s on other base r e l a t i o n s r e q u i r e t h e i r d e f i n i t i o n by a p p r o p r i a t e e n t r i e s i n RELATIONS and ATTRIBUTES. A d e s c r i p t i o n A* The System 55 o f each of the a t t r i b u t e s i n Table I I I - 1 i s given i n Appendix A. One o f the d e c i s i o n s r e g a r d i n g the user i n t e r f a c e t o SOEAAM was to r e q u i r e no e x p r e s s i o n parsing..Because o f t h i s the user i n t e r f a c e are v a r i e t i e s of t r e e - l i k e s t r u c t u r e s with the nodes themselves denoting the o p e r a t i o n s r e q u i r e d and the c o n d i t i o n s f o r search as we l l as s e r v i n g simply as b u f f e r s f o r t u p l e s and l o g r e c o r d s . In a sense, " o p e r a t i o n t r e e s " submitted to SOEAAM are analogous to a parse t r e e c r e a t e d by the parse phase of a compiler. T h i s approach t o i n t e r f a c i n g i s not, however, unigue to SOEAAM (see f o r example the LSL i n t e r f a c e of OMEGA i n [Schmid e t a l * 7 6 ] ) . Communication, t h e r e f o r e , i n v o l v e s a v a r i e t y of s i x t e e n node types . the f i r s t t e n of which serve as c h o i c e s f o r the r o o t node o f an " o p e r a t i o n t r e e " . Each of the s i x non-root node types i s a s s o c i a t e d a l s o with a " s y n t a c t i c i d e n t i f i e r " enclosed i n b r a c k e t s of the form "<.;,>" which, i n t u r n , denote p o s s i b l e p o i n t e r values i n the p o i n t e r f i e l d s of node types having them. The f i r s t f i e l d of a l l node types i d e n t i f i e s the node type t o SOEAAM. A l l but one of the node types (type 14; or <values>) have a f i x e d format with a f i x e d number o f f i e l d s . A l l o p e r a t i o n s on SOEAAM may be c a t e g o r i z e d as one o f th r e e t y p e s : base r e l a t i o n r e t r i e v a l , base r e l a t i o n maintenance and t r a n s a c t i o n maintenance/recovery/audit o p e r a t i o n s . In the f o l l o w i n g , node types have the form "{n,.*,)" (the f i r s t f i e l d being a numeric value i d e n t i f y i n g A. The System 56 the node t y p e ) . Other f i e l d s have one of three other forms: a value of an a t t r i b u t e as d e s c r i b e d i n Appendix A* a s y n t a c t i c i d e n t i f i e r denoting a p o i n t e r f i e l d t o other non-root nodes o r a f i x e d s e l e c t i o n of values. Brackets of the form " [ . , , ] " e n c l o s e s e l e c t i o n s i n the l a t t e r case. Also* f i e l d s tagged with "*" must be s u p p l i e d by t h e user. We f i r s t i n t r o d u c e the v a r i e t y of root node types p o s s i b l e then f o l l o w t h i s by a d e s c r i p t i o n o f the s y n t a c t i c i d e n t i f i e r s along with t h e i r a s s o c i a t e d non-root node types. The reader i s encouraged t o bear i n mind when re a d i n g the f o l l o w i n g t h a t the design of the i n t e r f a c e t o SORAAM i s not intended as an end user interface..Knowledge and use of t h i s i n t e r f a c e i s p r o p e r l y the r e s p o n s i b i l i t y of a r e l a t i o n a l language l e v e l t h a t would comprise the top end of a f u l l RDMS. R e t r i e v a l Operations There are two root node types a v a i l a b l e f o r r e t r i e v a l o p e r a t i o n s ( f o r e x p r e s s i n g q u e r i e s ) . These have the form: (a) {0* rprocess#*,Path# #Rnarae*,<predicate>* r<values>} (b) {1*,Path#* r<values>} We note f i r s t t h a t "Process*" i d e n t i f i e s the process i s s u i n g the r e t r i e v a l o p e r a t i o n t o SORAAM. Both of these conform to a no t i o n of a "RETRIEVE" and "NEXT" o p e r a t i o n r e s p e c t i v e l y t h a t e s s e n t i a l l y " l i n e a r i z e s " the r e t u r n from a base r e l a t i o n o f the t u p l e s s a t i s f y i n g a p r e d i c a t e of a query. For a RETRIEVE o p e r a t i o n SORAAM f i r s t a s s i g n s a unique "Path*" t h a t i s used A. The System 57 by SORAAM t o i d e n t i f y the sequence o f t u p l e s t o be r e t u r n e d . The f i r s t t u p l e i n "Rname" s a t i s f y i n g "<predicate>" 1 i s then returned i n "<values>". Subsequent t u p l e s are returned v i a subsequent user s u p p l i e d NEXT o p e r a t i o n s u s i n g the provided "Path*" value. Such a l i n e a r i z a t i o n of r e t r i e v a l response i s not, o f c o u r s e , c l e a r l y d e s i r a b l e . As a consequence g r e a t e r r e s p o n s i b i l i t y f o r r e t r i e v a l c o n t r o l r e s t s on higher l e v e l users ( i i . e . the r e l a t i o n a l language l e v e l ) : We f e e l , however, t h a t the b e n e f i t s r e s u l t i n g from g r e a t l y s i m p l i f i e d b u f f e r management f a r outweigh any such disadvantages. The format of these o p e r a t i o n s f a c i l i t a t e more than one ongoing r e t r i e v a l by the same process..Furthermore, no opening or c l o s i n g of any form i s c a l l e d f o r * C l e a r l y there i s , however, an implementation i s s u e concerning v a l u e s of "Path*". There i s a question of under what c o n d i t i o n s may a path number and i t s a s s o c i a t e d r e s o u r c e s be f r e e d f o r use by o t h e r r e t r i e v a l o p e r a t i o n s . Exhaustinq the s e t of t u p l e s s a t i s f y i n g "<predicate>" i s one such c o n d i t i o n but the q u e s t i o n deserves g r e a t e r thought. We l e a v e , f o r now however, t h i s implementation i s s u e . Maintenance O p e r a t i o n s i l f hd <predicate> i s s u p p l i e d i n the RETRIEVE node then a l l t u p l e s i n "Rname" are assumed t o q u a l i f y f o r r e t r i e v a l . A scan o f a l l t u p l e s i n a base r e l a t i o n i s t h e r e f o r e f a c i l i t a t e d . A. The System 58 There are three root node types a v a i l a b l e f o r maintenance o p e r a t i o n s on base r e l a t i o n s . T h e i r form i s : (a) {2*,process**,Rname*,<values>*} (b) { 3*, p r o c e s s * * , Rname*, < p r e d i c a t e d ) (c) {4*,process**,Rname*,<predicate>*,<values>*} These correspond t o "INSERTION", "DELETION" and "UPDATE" o p e r a t i o n s r e s p e c t i v e l y * A l l f i e l d s i n the above are user s u p p l i e d . Besides i d e n t i f y i n g the r e q u e s t i n g process to SORAAM the " p r o c e s s * " f i e l d i s a l s o necessary f o r purposes of t r a n s a c t i o n recovery* D e s t r u c t i v e o p e r a t i o n s on base r e l a t i o n s are logged along with the value of "process*" t o f a c i l i t a t e t h i s . The r o l e of the other f i e l d s i n the performance by SORAAM o f these maintenance o p e r a t i o n s should be.obvious.. T r a n s a c t i o n Maintenance/Recpyery/Audit Operations The f i r s t f o u r of the r o o t node types d e s c r i b e d below provide f o r t r a n s a c t i o n maintenance and recovery. A u d i t t r a i l requirements are f a c i l i t a t e d by a l o g g i n g r o o t node t y p e . j These formats are: (a) {5*,process**,guiet*} (b) {6*, process**] (c) {7*,process**,quiet*} (d) {8*,process**,quiet**} (e) {9*,process**,<buffer>*) These correspond t o "begin t r a n s a c t i o n " (BEGINTRANS), "end t r a n s a c t i o n " (ENDTRANS), " c h e c k p o i n t " (QUIET), "checkpoint A. . The System 59 r e c o v e r y " (RECOVER) and "LOG" o p e r a t i o n s r e s p e c t i v e l y . The f i r s t f o u r are e n t i r e l y analogous to the System R RDI ( R e l a t i o n a l Data I n t e r f a c e ) operators 11 BEGIN_TRAN S" r "END_TRANS", "SAVE" and "RESTORE" (see [ A s t r a h a n et a l . . 7 6 ] ) . A "LOG" o p e r a t i o n i n v o l v e s w r i t i n g the contents of "<buffer>" t o a system l o g * R e c a l l t h a t maintenance o p e r a t i o n s use t h i s same system l o g (see above). A l l maintenance o p e r a t i o n s on base r e l a t i o n s must occur i n s i d e "BEGINTRANS"/"ENDTRANS" p a i r s . . In t h i s way a l l processes not c u r r e n t l y i n v o l v e d i n a t r a n s a c t i o n can be assured ( f o r t h e i r r e t r i e v a l operations) of both the l o g i c a l and p h y s i c a l i n t e g r i t y of the base r e l a t i o n s . , Furthermore, a l l processes with ongoing t r a n s a c t i o n s are " i n s u l a t e d " from f e e l i n g the e f f e c t s of other ongoing ( i . e . . uncompleted) t r a n s a c t i o n s . The "QDIET" and "RECOVER" o p e r a t i o n s are u s e f u l f o r l a r g e t r a n s a c t i o n s i n v o l v i n g a great d e a l of maintenance on the base r e l a t i o n s * Recovery from a deadlock may then i n volve.backup t o a more rec e n t "quiet#" of the t r a n s a c t i o n . S y n t a c t i c I d e n t i f i e r s and Non-root-Nodes Types We now i n t r o d u c e the v a r i e t y of non-root node types t h a t can occur i n an o p e r a t i o n t r e e communicated to SORAAM. Each i s a s s o c i a t e d with a s y n t a c t i c i d e n t i f i e r some o f which were used t o denote p o i n t e r f i e l d s i n t h e above* A s y n t a c t i c i d e n t i f i e r , however, may sometimes i n v o l v e more than one form.. We t h e r e f o r e d e a l with each such i d e n t i f i e r i n t u r n . A. The System 60 R e t r i e v a l and maintenance o p e r a t i o n s d e s c r i b e d above allow the use of a <predicate>. T h i s has one o f three forms (the f i r s t two of which are ot h e r s y n t a c t i c i d e n t i f i e r s ) : (a) <primitive> (b) <conjunct> (c) {10*,<conjunct>*,<predicate>*} Note t h a t the t h i r d form of <predicate> i s a non-root node type t h a t permits a p o i n t e r f i e l d t o another <predicate>. T h i s t h i r d form i s i n t e r p r e t e d by SORAAM as a d i s j u n c t i o n (OR) o f the form's second and t h i r d argument. A <conjunct> s y n t a c t i c i d e n t i f i e r has the f o l l o w i n g two p o s s i b l e forms: (a) <primitive> (b) {11*, <primitive>*,<conjunct>*) Note again t h a t the second form of <conjunct> i s a non-root node type t h a t permits a p o i n t e r f i e l d t o another <conjunct>. Analogously, t h i s second form i s i n t e r p r e t e d as a c o n j u n c t i o n (AND) o f the form's l a s t two arguments. A <primitive> s y n t a c t i c i d e n t i f i e r a l s o has two p o s s i b l e forms* Both of these, however* are non-root node types: r - i * l< ( K=l (a) {12*,Aname*,|= |,value*} I -=l l>=l l> I L J where " v a l u e " i s a value of a t t r i b u t e "Aname". A*.The System 61 r T * r l "* (b) {13*, low*, | < | , Aname*, | < |,high*} l<=l K=l L J L J where "low" and " h i g h " are values o f the a t t r i b u t e "Aname". The f i r s t o f these non-root node t y p e s (note that both are le a v e s i n an o p e r a t i o n tree) i s i n t e r p e t e d by SORAAM as s p e c i f y i n g an e x p l i c i t value or an upper or lower bound f o r an a t t r i b u t e . The second i s i n t e r p r e t e d as s p e c i f y i n g a range o f values f o r an a t t r i b u t e . Consider now t h a t the d e f i n i t i o n of the <predicate> s y n t a c t i c i d e n t i f i e r r e s u l t i n g from the above i s nothin g more than a t r e e s t r u c t u r e d d i s j u n c t i v e normal form e x p r e s s i o n . .The <primitive> forms allowed, t h e r e f o r e , f a c i l i t a t e the ex p r e s s i o n of any i n s t a n c e of an i n t e r s e c t i o n query as a <predicate> t o SORAAM. Furthermore, the t r e e s t r u c t u r e corresponds t o an e a s i l y used parse of the query* We s h a l l use the term "<predicate> i n s t a n c e " , t h e r e f o r e , to denote a non- n u l l <predicate> f i e l d o c c u r r i n g i n RETRIEVE, DELETION or UPDATE r o o t node types* The two remaining s y n t a c t i c i d e n t i f i e r s used by root node types a r e <values> and <buffer>..Both have only one form t h a t are non-root node types. The forms f o r <values> and <buffer> r e s p e c t i v e l y a re: (a) {14*,Aname1*,v1*,Aname2*,v2*,...,Anamei*,vi*} where v 1 * , v 2 * , . . . , v i * a re values of a t t r i b u t e s A n a m e i , A n a m e 2 , A n a m e i r e s p e c t i v e l y . . A. .The System 62 (b) {15*,log} where " l o g " i s the t e x t t o be w r i t t e n t o the l o g f i l e . The i n t e r p r e t a t i o n of the above non-root node types by SORAAM should be obvious. An Example R e t r i e v a l We conclude t h i s f i r s t s e c t i o n of Chapter I I I with a much needed example use of SORAAM. The o p e r a t i o n t r e e : f o r a query on an example base r e l a t i o n i s qiven. Consider t h a t the base r e l a t i o n "PARTS" on which our query a p p l i e s has the f o l l o w i n g a t t r i b u t e s (given i n b r a c k e t s ) : PARTS[P#,PNAME,PCOLOR,PWEIGHT,QTY] The key of the r e l a t i o n i s the "P#" numeric a t t r i b u t e * Other numeric a t t r i b u t e s are "PWEIGHT" and "QTY". Each t u p l e i n "PARTS" d e s c r i b e s a s p e c i f i c p a r t by i t s name, c o l o r , weight and q u a n t i t y . Note t h a t t u p l e s d e s c r i b i n g "PARTS" must a l r e a d y r e s i d e i n "RELATIONS" and "ATTRIBUTES" before any o p e r a t i o n i s p o s s i b l e on i t . There are of course s i x such t u p l e s n ecessary (one f o r the r e l a t i o n e n t r y ; f i v e f o r the a t t r i b u t e e n t r i e s ) . A l l s i x would have been added by p r i o r maintenance o p e r a t i o n s on SORAAM. The query we c o n s i d e r may be expressed i n E n g l i s h as: A. The System 63 "Get a l l b o l t s more than 10 ounces and nuts of which we have l e s s than 40." T h i s immediately decomposes i n t o two d i f f e r e n t o p e r a t i o n t r e e s c orresponding t o : (a) Get the f i r s t nut or b o l t then (b) get the r e s t . F i g u r e I I I - 2 i l l u s t r a t e s the o p e r a t i o n t r e e s f o r both p a r t s . Example Operation Trees On SOEAAM {EETEIEVE , process 1 , ?path#, PATtT, + , + } I I (a) "get the f i r s t nut o r b o l t " {NEXT,path#,+3 I V {14,?tuple} (b) "get the r e s t " I V | {14,?tuple} I V {OB,+,+} I V {AND, + ,+} I I I V | {12,QTY,<,40} I V {12,PNAME, = ,nuts3 {ANDr+,+} I I I V | £1 2,PWEIGHT,>, 10} I V {12,PNAME, = ,bolt} F i g u r e I I I - 2 For r e a d a b i l i t y , node type f i e l d s have been r e p l a c e d by more meaningful mnemonics* The f i r s t d e f i n e s the r e l a t i o n and p r e d i c a t e to SOEAAM..The f i r s t t u p l e i s re t u r n e d i n " ? t u p l e " A. The System 64 and SORAAM a s s i g n s a "path#" value. T h i s value i s then used i n the second o p e r a t i o n t r e e by the r o o t "NEXT" node to a c q u i r e , v i a " ? t u p l e " again, the next nut or b o l t . S u c c e s s i v e use of the second o p e r a t i o n t r e e accomplishes " g e t t i n g the r e s t " . B» The Primary Index In preceeding s e c t i o n s we have discussed the r o l e of the primary index of a base r e l a t i o n * In gene r a l * the primary index determines the c l u s t e r i n g of the t u p l e s i n f i x e d s i z e d b l o c k s * With SORAAM, however, we have l i m i t e d o u r s e l v e s t o a c o n s i d e r a t i o n of h o r i z o n t a l c l u s t e r i n g c o n s i d e r a t i o n s only. R e c a l l t h a t such c o n s i d e r a t i o n s themselves are de f i n e d i n terms of a c h a r a c t e r i z a t i o n o f use of the base r e l a t i o n * In t h i s s e c t i o n we i n t r o d u c e our chosen c h a r a c t e r i z a t i o n o f use. C o n s i d e r a t i o n i s then given of a number of access methods used as primary i n d i c e s t o r e a l i z e d e s i r a b l e h o r i z o n t a l c l u s t e r i n g determined by the c h a r a c t e r i z a t i o n . The f i r s t s e c t i o n of t h i s t h e s i s i n t r o d u c e d the p a r t i c u l a r data s t r u c t u r e t h a t we have adopted f o r t h i s r o l e as the k-d t r e e * An i n t r o d u c t i o n to k-d t r e e s and an i n i t i a l demonstration o f t h e i r s u i t a b i l i t y are presented followed by a d e t a i l e d examination o f the problems t h a t a r i s e . The next s e c t i o n c o n s i d e r s i n d e t a i l the s p e c i f i c problem of " d i s c r i m i n a t o r s e l e c t i o n " . The c h a r a c t e r i z a t i o n of use of a base r e l a t i o n used by B. The System 65 SOEAAM i s a set of s t a t i s t i c s based on the h i s t o r y of q u e r i e s on the base r e l a t i o n * A l l such s t a t i s t i c s occur i n the s p e c i a l r e l a t i o n s "RELATIONS" and "ATTRIBUTES" as values o f the a t t r i b u t e s "qno" and " f " * Each base r e l a t i o n has a "qno" value and a value o f " f " a s s o c i a t e d with each of i t s a t t r i b u t e s . . T h e "qno" value r e p r e s e n t s a count of the t o t a l number of q u e r i e s t h a t have been a p p l i e d t o a base r e l a t i o n and the " f " f o r each a t t r i b u t e r e p r e s e n t a count of the number of times a value f o r each was s u p p l i e d i n past q u e r i e s . The d e s c r i p t i o n of the user i n t e r f a c e i n the pre c e e d i n g s e c t i o n i n t r o d u c e d and d e f i n e d the s y n t a c t i c i d e n t i f i e r "<predicate>". Each occurrence of a <predicate> i n a "RETRIEVE", "DELETION" or "UPDATE" r o o t node type o f an o p e r a t i o n t r e e (each <predicate> i n s t a n c e ) r e c e i v e d by SORAAM i s i n t e r p r e t e d as an occurrence of a query. The a r r i v a l o f one of these r o o t node types together with i t s <predicate>, t h e r e f o r e , s i g n a l s t o SORAAM t o increment the "qno" value of the a s s o c i a t e d "Rname" (given a l s o i n the roo t node) i n RELATIONS. For each "<primitive>" o c c u r r i n g i n the p r e d i c a t e t h e r e , i s an a s s o c i a t e d a t t r i b u t e name "Aname". On <predicate> a r r i v a l these a t t r i b u t e names together with "Rname" o f the ro o t node: are used by SORAAM to l o c a t e and increment the a s s o c i a t e d " f " v a l u e s . In t h i s way the f o l l o w i n g i n f o r m a t i o n r e g a r d i n g a c h a r a c t e r i z a t i o n of use of each base r e l a t i o n i s a v a i l a b l e t o SOEAAM: (a) a r e l a t i v e frequency of s p e c i f i c a t i o n count " f " f o r each "Rname/Aname" p a i r and B. The System 66 (b) with r e s p e c t t o the above a r e l a t i v e count "qno" of the t o t a l number of q u e r i e s on a base r e l a t i o n . I f the maximum value a numeric a t t r i b u t e such as "qno" may have i s a l i m i t i n g f a c t o r then "recovery" from r e a c h i n g t h i s maximum i s necessar y . C l e a r l y t h e r e i s a clanger t h a t the s t a t i s t i c s d e s c r i b e d above may become u s e l e s s i f the "qno" and " f " reach t h e i r maximum values and stay t h a t way.. A p a r t i c u l a r l y simple recovery approach, then, i s simply t o d i v i d e t h e "qno" and " f " v a l u e s of a base r e l a t i o n i n h a l f whenever "qno" exceeds i t s maximum. Note t h a t the " f " v a l u e s , as a r e s u l t , are ensured never to exceed the value o f "qno" and* more i m p o r t a n t l y , stay i n proper p r o p o r t i o n t o each o t h e r and to "qno" (thus p r e s e r v i n g the q u a l i t y o f the c h a r a c t e r i z a t i o n of use). There i s an i n t e r e s t i n g r e p e r c u s s i o n of t h i s r e c o v e r y approach t h a t r e s u l t s from a l o s s o f i n f o r m a t i o n whenever a " d i v i d e : b y 2" i s c a l l e d f o r . The maximum value then determines how much of the query h i s t o r y w i l l a c t u a l l y be remembered. Because these s t a t i s t i c s c o n t r o l h o r i z o n t a l c l u s t e r i n g (and because the support f o r h o r i z o n t a l c l u s t e r i n g i n SORAAM i s dynamic i n nature) c o n t r o l over the maximum value y i e l d s c o n t r o l over the r a t e of a d a p t a t i o n of the h o r i z o n t a l c l u s t e r i n g to changing p a t t e r n s of use* The a t t r i b u t e " r a d i x l " i n RELATIONS i s provided f o r j u s t t h i s purpose. Reducing the value of " r a d i x l " i n c r e a s e s the r a t e of ada p t a t i o n o f the h o r i z o n t a l c l u s t e r i n g (and a l s o the system overhead n e c e s s a r i l y involved) while i n c r e a s i n g " r a d i x l " r e s u l t s i n B. The System 67 g r e a t e r immunity to s h o r t and temporary b u r s t s of unusual use but l e s s w i l l i n g n e s s t o change, i There are a number of e x i s t i n g access methods that might be used as the c h o i c e f o r the primary index of base r e l a t i o n s i n SOEAAM. The access methods we c o n s i d e r take p r o g r e s s i v e l y more advantage of the above freguency of access s t a t i s t i c s c o mprising the c h a r a c t e r i z a t i o n of use t o accomplish p r o g r e s s i v e l y b e t t e r degrees of h o r i z o n t a l clustering.„ P o s s i b l y the l e a s t l i k e l y c a n d i d a t e s f o r the primary index are methods i n v o l v i n g hashing techniques* T h e i r most condemning c h a r a c t e r i s t i c i s c l e a r l y the r e s u l t i n g "randomization" of the r e l a t i v e p h y s i c a l l o c a t i o n s of t u p l e s i n a base r e l a t i o n . 2 Regardless of the choice of columns t h a t comprise the domain of a hash f u n c t i o n no s i g n i f i c a n t r e l a t i o n s h i p can r e s u l t f o r t u p l e s l o c a t e d p h y s i c a l l y near to each other* No c h a r a c t e r i z a t i o n of use, t h e r e f o r e , can have any c o n s t r u c t i v e bearing on a primary index based on a hashing method* Another major problem with a t r a d i t i o n a l hash f i l e i s the requirement of knowing i n advance the s i z e of the f i l e expected ( i . e . the number of t u p l e s comprising a base r e l a t i o n ) . T h i s i s necessary i n order to determine an a p p r o p r i a t e range f o r the hash f u n c t i o n i t s e l f * T h i s problem. lA new i n s i g h t i n c o g n i t i v e s c i e n c e perhaps. 2 N o t e t h a t g e n e r a l purpose order p r e s e r v i n g hashing f u n c t i o n s do not e x i s t . B. The System 68 however, has been r e c e n t l y overcome with the advent of an "extendable hashing" access method f i r s t i n t r o d u c e d i n [ F a g i n 78]. The expected case performance of r e t r i e v a l on the hash key u s i n g t h i s method i s two block reads f o r any f i l e s i z e . . Other c a n d i d a t e s f o r the primary index are methods i n v o l v i n g t r e e s t r u c t u r e s . Of these the most predominant c u r r e n t l y used by e x i s t i n g RDMSs f o r both primary and secondary i n d e x i n g i s the B-tree* There are s e v e r a l reasons f o r t h i s * For one, a B-tree always remains balanced r e g a r d l e s s of the v a r i e t y and number of i n s e r t i o n s and d e l e t i o n s performed on i t . S i n c e no degeneration of search performance occurs no p h y s i c a l r e o r g a n i z a t i o n i s ever necessary. Furthermore, by i t s nature, a B-tree can be used t o maintain a p h y s i c a l sequence of t u p l e s i n a s o r t order determined by an ordered subset of the a t t r i b u t e s of a base r e l a t i o n . T h i s second property of B-trees endows them with an a b i l i t y t o make c o n s t r u c t i v e use of the c h a r a c t e r i z a t i o n o f use s t a t i s t i c s i n t r o d u c e d above. The s o r t order of the t u p l e s of a base r e l a t i o n would be chosen as the order i m p l i e d by the s e q u e n c e : r e s u l t i n g from a r r a n g i n g the a t t r i b u t e s of the base r e l a t i o n i n descending order of t h e i r " f " v a l u e s . The primary s o r t column would t h e r e f o r e correspond to the. a t t r i b u t e most f r e q u e n t l y s p e c i f i e d i n q u e r i e s . Tuples would then be p h y s i c a l l y s t o r e d nearer those with s i m i l a r values f o r t h i s primary s o r t a t t r i b u t e . Thus we can see t h a t a b e n e f i c i a l h o r i z o n t a l c l u s t e r i n g r e s u l t s . B e c a l l from F i g u r e II-3 and Table I I - 3 , however, t h a t the B. .The System 69 h o r i z o n t a l c l u s t e r i n g r e s u l t i n g from the maintenance i n s o r t order o f the t u p l e s of a base r e l a t i o n can be improved upon. In f a c t a t h i r d c l u s t e r i n g , " c l u s t e r 3 " , y i e l d e d an a b s o l u t e improvement i n b l o c k reads over a sorted c l u s t e r i n g (that would r e s u l t from the use of a B-tree as the. primary index) eq u a l i n g the improvement gained by the s o r t e d c l u s t e r i n g over a random c l u s t e r i n g (that would r e s u l t from the use of extendable hashing f o r example). F i g u r e I I I - 3 i l l u s t r a t e s the r e a l i z a t i o n of " c l u s t e r 3 " i n F i g u r e I I - 3 with a "k-d t r e e " index (another t r e e s t r u c t u r e ) . C l e a r l y , then, a k-d t r e e h olds the p o t e n t i a l f o r improved h o r i z o n t a l c l u s t e r i n g . For t h i s reason a k-d t r e e data s t r u c t u r e was chosen as the main contender f o r the r o l e of primary index of a l l base r e l a t i o n s i n SOEAAM. We s h a l l see i n the next s e c t i o n t h a t i t s s u i t a b i l i t y f o r t h i s r o l e i s enhanced by the f a c t t h a t , with v a r i o u s simple i n t e r p r e t a t i o n s of our c h a r a c t e r i z a t i o n of use, an o p t i m a l h o r i z o n t a l c l u s t e r i n g r e s u l t s * . A simple i n t r o d u c t i o n to k-d t r e e s now f o l l o w s * Each i n t e r n a l or non-leaf node i n a k-d t r e e has the f i e l d s d e f i n e d by the l a y o u t diagrammed i n F i g u r e I I I - 3 . The l a t t e r t h r e e f i e l d s ( i . e . "key v a l u e " , " l e f t s u b t r e e " and " r i g h t subtree") are e n t i r e l y analogous i n purpose and f u n c t i o n to those of a r e g u l a r b i n a r y t r e e . The a d d i t i o n a l " d i s c r i m i n a t o r " f i e l d simply determines the column or a t t r i b u t e to which the "key v a l u e " a p p l i e s , " l e v e l 1" of the k-d t r e e diagrammed (the root node) d i s c r i m i n a t e s on the f i r s t column w h i l e " l e v e l 2" d i s c r i m i n a t e s on the second. T h e r e f o r e , B. The System 70 A k-d Tree Index t o "cluster3|' (from F i g u r e II-3) I n t e r n a l Node Layout I I I I I I + I + I • I + I i i i i. i i i i i i _ __>right subtree > l e f t subtree ___>key value ___>discriminator (column*) l e v e l l e v e l F i g u r e I I I - 3 only t u p l e s with values f o r t h e i r f i r s t (second) a t t r i b u t e g r e a t e r than or equal t o 3, the "key value"s, occur i n the r i g h t subtree of the r o o t node ( l e v e l 2 nodes)* Searching on a k-d t r e e i s simple* Consider an exact match query that would a r i s e , f o r example, from t u p l e B. The System 71 i n s e r t i o n . S t a r t i n g a t the r o o t node we f o l l o w the l e f t or r i g h t p o i n t e r s as i m p l i e d by the comparison of the a p p r o p r i a t e a t t r i b u t e value determined by a node's d i s c r i m i n a t o r with the node's key value* In t h i s way we a r r i v e a t a s i n g l e l e a f b l ock i n which we must then search. In a p a r t i a l match query, however, a node may d i s c r i m i n a t e on an a t t r i b u t e f o r which no value i s p r o v i d e d . In t h i s case the search must r e c u r s e down both the l e f t and r i g h t subtrees* As an example c o n s i d e r the p a r t i a l match query {column*1=2} a p p l i e d to the base r e l a t i o n given i n Figure I I I - 3. He would f o l l o w the l e f t subtree of the r o o t node s i n c e the f i r s t column value provided by the guery compares l e s s than the r o o t node key value of 3. However, s i n c e no value i s provided f o r the second column i n our guery both the l e f t and r i g h t s u b t r e e of the l e v e l 2 node must be searched. As a r e s u l t the f i r s t two l e a f b l o c k s must be searched f o r t u p l e s s a t i s f y i n g our query {column#1=2}. Note t h a t both range q u e r i e s and p a r t i a l range g u e r i e s can s i m i l a r l y be f a c i l i t a t e d . There are, however, a number of problems with k-d t r e e s : (a) With r e s p e c t to our c h a r a c t e r i z a t i o n of use how should d i s c r i m i n a t o r values be s e l e c t e d i n order t o a r r i v e at a best p o s s i b l e h o r i z o n t a l c l u s t e r i n g of the t u p l e s of a base r e l a t i o n ? (b) How do we ensure b a l a n c i n g of the i n t e r n a l nodes of a k-d t r e e ? (c) How do we c l u s t e r the i n t e r n a l nodes of a k-d t r e e themselves w i t h i n f i x e d s i z e b l o c k s i n order to ensure minimal disk accesses? (d) How do we f a c i l i t a t e dynamic p h y s i c a l B. The System 72 r e o r g a n i z a t i o n of the h o r i z o n t a l c l u s t e r i n g n e c e s s i t a t e d by a changing c h a r a c t e r i z a t i o n o f use? He d e f e r d i s c u s s i o n o f the f i r s t of these problems u n t i l the next s e c t i o n . The l a t t e r t h r e e of the problems, however, are co n s i d e r e d i n the remainder o f t h i s s e c t i o n * Our f i r s t problem, then, concerns b a l a n c i n g of the i n t e r n a l nodes. Imbalance i n a k-d t r e e can occur when the t r e e i t s e l f must be dynamically generated a c c o r d i n g to the t u p l e i n s e r t i o n sequence ( i . e . the sequence of INSERTION o p e r a t i o n s on a base r e l a t i o n submitted to SORAAM) . To b e t t e r understand how t h i s problem a r i s e s i t i s necessary to have a c l e a r e r i d e a of the i n s e r t i o n process through the k-d t r e e primary index. F i g u r e III-4 below i l l u s t r a t e s both t h i s i n s e r t i o n process and a r e s u l t i n g imbalance o f the i n t e r n a l nodes (more s p e c i f i c a l l y the r o o t node) a r i s i n g when the l a s t t u p l e s are added. The l e a f b l o c k s i n the f i g u r e c o n t a i n the t u p l e s and are l a b e l l e d a c c o r d i n g t o t h e i r order of c r e a t i o n * Each l e a f block i s assumed t o have a c a p a c i t y o f a t most f o u r t u p l e s . A l a b e l of the form "*n" denotes the order of t u p l e a r r i v a l . . The phrase " l e a f page s p l i t t i n g " best d e s c r i b e s the event r e s u l t i n g i n the c r e a t i o n of a new l e a f block and i n t e r n a l node. Such page s p l i t s occur when a new t u p l e must be added t o a l e a f t u p l e block a l r e a d y f u l l . The a r r i v a l , f o r example, o f the f i f t h t u p l e f o r c e s both t u p l e block "B1" i n Fig u r e I I I - 4(a) to be s p l i t i n t o two t u p l e b l o c k s "B1" and "B2" and an i n t e r n a l node c r e a t e d t o d i s t i n g u i s h between t h e i r t u p l e s i n B..The System 73 Sorted Order I n s e r t i o n i n a k-d Tree - B1 1 I 1 | 1 | 1 | 1 2 3 4 (a) I | *1 | *2 | *3 | *4 t B1 I I I I I 1 I 2 | + | I _ _ _ _ _ _ _ _ I I v B2| 1 1 I I 2 | 1 |*5 1 2 I I 2 | 2 |*6 1 3 I I 2 | 3 |*7 1 4 I I 2 | 4 |*8 _ _ l l _ l .1 _ l (b) B1 I I I I I I 1 I 2 | • | + | l _ l _ _ _ l _ l _ 1 _ l_l I I I _V__ I 1 I 1 1 I 2 1 I 3 1 I <» B2| I I I I I I 2 | 2 2 I 3 I I 3 | I J I V_ I ~ l I 1 I 2 I 1 2 B3| | *9 |*10 I I 3 I + I + • _ i i i 2 2 3 3 I 3 3 4 I I f*11 |*12 (c) F i g u r e I I I - 4 F i g u r e I I I - 4 (b). In a s i m i l a r f a s h i o n t u p l e block "B2" i s s p l i t twice more on a r r i v a l of the n i n t h and t h i r t e e n t h t u p l e s . B. The System 74 I I I I I I 1 I 2 | + | + | 9 f f _ • i _ i _ i F i g u r e I I I - 4 (cont'd) B- The System 75 The f i r s t t h i n g t o remember about k-d t r e e s i s t h a t they behave e x a c t l y as 1-d (or binary) t r e e s i n expected case performance f o r random t u p l e i n s e r t i o n s : "Thus we know t h a t t y p i c a l i n s e r t i o n s and record lookups i n a k-d t r e e w i l l examine approximately 1*386 l o g n nodes." 1 2 Indeed, should the t u p l e s of F i g u r e I I I - 4 have been i n s e r t e d i n a random order i l l u s t r a t e d by " c l u s t e r l " of F i g u r e I I - 3 i n s t e a d of the s o r t e d order i l l u s t r a t e d by " c l u s t e r 2 " of F i g u r e I I - 3 the p e r f e c t l y balanced k-d t r e e : i l l u s t r a t e d i n F i g u r e I I I - 3 would have r e s u l t e d (the reader i s encouraged to v e r i f y t h i s f o r h i m s e l f ) . One must r e a l i z e * however, t h a t k-d t r e e s are b a s i c a l l y a s t a t i c data s t r u c t u r e * 2 Problems a r i s e when i n s e r t i o n s are not random. In such cases a k-d t r e e can become unacceptably unbalanced ( p o s s i b l y when i n s e r t i o n s are made i n some s o r t e d order as i n F i g u r e I I I - 4 ) . B a l a n c i n g " t r i c k s " used by 1-d t r e e s such " r o t a t i o n s " i n the case of AVL t r e e s and " i n t e r n a l node s p l i t t i n g " i n the case of B-trees cannot g e n e r a l l y be a p p l i e d to k-d t r e e s . The next b e s t approach, then, i s to design a r e b u i l d i n g phase that accomplishes b a l a n c i n g but t h a t ' T h i s passage i s taken from page 512 of [ B e n t l e y 75]. A d e r i v a t i o n of t h i s r e s u l t f o r b i n a r y t r e e s may be found on pages 426-427 of [Knuth 73]. 2A "dynamic m u l t i d i m e n s i o n a l s e a r c h i n g data s t r u c t u r e " a p p r o p r i a t e f o r exact or p a r t i a l match and range or p a r t i a l range s e a r c h i n g and e f f i c i e n t i n both space and time (no worse than 0[n l o g n] f o r e i t h e r ) does not y e t e x i s t . . B..The System 76 permits concurrent normal use made of the k-d t r e e primary index and base r e l a t i o n * The phase would be i n i t i a t e d upon d e t e c t i o n of a s u i t a b l e imbalance c o n d i t i o n . There are many ways that imbalance can be dete c t e d . A p a r t i c u l a r l y simple c o n d i t i o n suggested by Knuth (see page 451 of (Knuth 73]) t h a t we adopt has the form: D > c . l o g (IX+1) (1) 2 In the above e x p r e s s i o n "D" r e p r e s e n t s the maximum depth o f a k-d t r e e index and "IX" the number of i n t e r n a l nodes; Both va l u e s are a u t o m a t i c a l l y maintained by SOEAAH and are l o c a t e d i n the "RELATIONS" s p e c i a l base r e l a t i o n (see Appendix A). Note t h a t t h e number of l e a f t u p l e blocks equals IX+1..A minimum value of 1 f o r the c o n s t a n t " c " i m p l i e s i n t o l e r a n c e f o r any but p e r f e c t l y balanced k-d t r e e s * Higher values o f c allo w g r e a t e r degrees of imbalance to e x i s t i n the primary index. For example, c=1 would imply t h a t the k-d t r e e i n F i g u r e I I I - 4 ( e ) was imbalanced whereas c=1.5 would not*. The c o n d i t i o n (1) would be checked f o r a base r e l a t i o n whenever a p a g e - s p l i t r e s u l t s from an i n s e r t i o n o p e r a t i o n (thus i ncrementing the value of IX f o r t h a t base r e l a t i o n ) * . The r e b u i l d i n g phase adopted by SORAAM i n v o l v e s the random d e l e t i o n and r e - i n s e r t i o n o f the t u p l e s of the imbalanced k-d t r e e . In t h i s case the value of c should be chosen t o be no l e s s than approximately 1.386 (see above q u o t a t i o n from [ Bentley 75]). The f i r s t mechanism used to f a c i l i t a t e t h i s i n v o l v e s the B. The System 77 use of a s p e c i a l i n t e r n a l node. Such a node, which we denote as "N*", d i f f e r s from a r e g u l a r i n t e r n a l node i n the number of f i e l d s present and i t s e f f e c t on s e a r c h i n g and i n s e r t i o n . C o n s i d e r , f o r example, t h a t we have a k-d t r e e : t h a t has become unbalanced i n the sense t h a t the above c o n d i t i o n (1) has become t r u e : N__ I f I I L _N | | where "N" are i n t e r n a l nodes and | I "L" are l e a f t u p l e blocks. . L _N r ~~i i i L L In t h i s circumstance the s p e c i a l i n t e r n a l node N* i s added t o the f r o n t of the r o o t node of the roo t page of the k-d t r e e as f o l l o w s : N* I I I .N_ - i . I I I I L .N__ I I where "N" and "L" are d e f i n e d I | as above. L N_ I ~ l I I I L No d e s c r i m i n a t o r o r key value f i e l d s a re de f i n e d f o r the N* node;, Furthermore, searc h i n g and i n s e r t i o n are a f f e c t e d by N* as f o l l o w s : (a) A l l s e a r c h i n g r e c u r s e s u n c o n d i t i o n a l l y down both the l e f t and r i g h t subtree of N* and (b) a l l i n s e r t i o n s are made to the r i g h t subtree o f B. The System 78 N* only. . Note t h a t the o p e r a t i o n of aaaing N* t o a k-d t r e e primary index can be accomplished q u i c k l y and, once done, a l l o w s c o n t i n u i n g use made of the l e f t and/or r i g h t subtrees* . Note a l s o t h a t N* could be i n s e r t e d p r i o r t o any unbalanced subtree (a property t h a t might p o s s i b l y be used t o advantage should we have b a l a n c i n g i n f o r m a t i o n p e r t a i n i n g to each i n d i v i d u a l i n t e r n a l node of a k-d t r e e ) . . The p o i n t of t h i s o p e r a t i o n i s t h a t i t now allo w s an " i d l e p r o c e s s " ( i * e. a c e r t a i n u t i l i t y program e x e c u t i n g when the CPU i s i d l e ) t o d e l e t e and r e - i n s e r t the t u p l e s of the l e f t subtree of N* (the o l d unbalanced k-d t r e e ) . In t h i s way the l e f t subtree e v e n t u a l l y d i s a p p e a r s ( i n c l u d i n g N*) and a new balanced k-d t r e e i s the r e s u l t . Such a b a l a n c i n g process must ensure t h a t randomness i s in t r o d u c e d i n the s e l e c t i o n of t u p l e s t o d e l e t e and r e - i n s e r t . I d e a l l y the randomness should apply t o the : s e t of t u p l e s c o m p r i s i n g the base r e l a t i o n . U n f o r t u n a t e l y , random d e l e t i o n of t u p l e s from a k-d tre e r e s u l t s i n an unacceptable space u t i l i z a t i o n of the l e a f t u p l e b l o c k s . There a r i s e s the p o s s i b i l i t y o f an a r b i t r a r i l y l a r g e percentage of the l e a f b l o c k s c o n t a i n i n g only a few t u p l e s each.* He t h e r e f o r e i n t r o d u c e randomness i n the s e l e c t i o n of l e a f t u p l e b l o c k s *Note t h a t " r o t a t i o n s " used to ensure good space u t i l i z a t i o n i n B-trees when d e l e t i n g r e c o r d s cannot be used i n the case of k- d t r e e s * B. The System 79 from which t u p l e s are d e l e t e d and r e - i n s e r t e d * There are many ways i n which t h i s may be accomplished. One p o s s i b i l i t y i s t o i n t r o d u c e a hash f i l e : ( p o s s i b l y u s i n g F a g i n * s extendable hash f i l e scheme) i n which to s t o r e l e a f t u p l e block addresses. That i s , the address of each new l e a f block a r i s i n g from a l e a f page s p l i t would be added to the hash f i l e . A scan of the hash f i l e i n the p h y s i c a l s t o r a g e seguence would then i n t r o d u c e the d e s i r e d randomness i n the s e l e c t i o n of l e a f t u p l e b l o c k s . Now when N* i s added (due t o imbalance) t h i s hash f i l e i s f i x e d (blocks under the r i g h t subtree o f N* are added t o another "new" hash f i l e ) and the s e q u e n t i a l scan of the f i x e d hash f i l e i s made. i T h i s scan proceeds as f o l l o w s : (a) A f i x e d number m f i r s t b l o c k s (independent of the s i z e of the base r e l a t i o n ) are c o n s i d e r e d . (b) D e l e t i o n and r e - i n s e r t i o n of t u p l e s from each of these m blocks i s made i n t u r n . (c) Whenever one of the m b l o c k s i s exhausted the next block i s r e t r i e v e d from the hash f i l e * Some o b s e r v a t i o n s apparent with t h i s approach are: (a) The expected d i s t r i b u t i o n of the range o f values f o r a l l a t t r i b u t e s of the t u p l e s i n the m b l o c k s i s randomly d i s t r i b u t e d (m pa r c e l s ) throughout the value range of a l l a t t r i b u t e s * (b) At most m (a f i x e d s m a l l number) of b l o c k s i n the l e f t subtree of N* can be p o o r l y u t i l i z e d . The occurrence of imbalance i n a k-d t r e e primary index does i n t r o d u c e a s i g n i f i c a n t r e b u i l d i n g phase. T h i s r e b a l a n c i n g o p e r a t i o n does, however, permit c o n c u r r e n t normal use of the k-d t r e e data structure.„ Furthmore, such a B..The System 80 r e b a l a n c i n g o p e r a t i o n i s never a b s o l u t e l y necessary* Consider now the t h i r d problem c o n c e r n i n g the c l u s t e r i n g of the i n t e r n a l nodes of a k-d t r e e themselves w i t h i n f i x e d s i z e d d i s k b l o c k s . An approach i s d e s i r e d t h a t ensures both good performance i n terms of page f a u l t s t h a t occur while c h a s i n g i n t e r n a l nodes and good d i s k block u t i l i z a t i o n . . Furthermore, the approach should f a c i l i t a t e c o n c u r r e n t use o f the primary index d u r i n g i t s i n v o c a t i o n and should be dynamic i n the sense t h a t no r e f o r m a t t i n g of an e n t i r e k-d t r e e i s ever necessary.. The approach taken i n SOEAAM i s very simple. Consider the f o l l o w i n g s i t u a t i o n i n which a primary index block with a maximum c a p a c i t y o f s i x i n t e r n a l nodes of a k-d t r e e i s f i l l e d p r i o r to the a r r i v a l of another i n t e r n a l node from below: In t h i s case a page s p l i t occurs c r e a t i n g another primary index block* As a r e s u l t the i n t e r n a l node " N 1 " i s sent one l e v e l above: Bi The System 81 A I N1 • I I B1| |~ | B2| | I I I I I I N2_ | | _N3_ i l l I I I I I I I I I I I I N4 N5 | I N6 N7 Note, t h e r e f o r e , t h a t an i n t e r n a l node can a r r i v e from below as the r e s u l t of a l e a f t u p l e b l o c k s p l i t or an i n t e r n a l node block s p l i t . . In the above, the i n t e r n a l node "N1" may cause f u r t h e r i n t e r n a l node block s p l i t s above "B1" and "B2".. Because of the p r i o r concern with b a l a n c i n g we are assured of a reasonable i n t e r n a l node d i s k b l o c k u t i l i z a t i o n (approximately 50 percent worst c a s e ) . The . obvious analogy with the B*-tree data s t r u c t u r e a l s o ensures good performance i n terms of page f a u l t rates*.An exact match search on the k-d t r e e y i e l d s an expected number of d i s k reads g i v e n by: | 1 + l o g r i (IX + 1) | (2) I 1 1 X | | I I— + 1 1 I |PIB | L J where [IX/PIB ]+1 i s the expected branching f a c t o r o f each i n t e r n a l node block and (IX+1) i s the number of l e a f t u p l e b l o c k s . 1 Note t h a t a l l o w i n g o n l y one i n t e r n a l node per tSee Appendix A f o r a d e f i n i t i o n of "IX" and "PIB" B..The System 82 i n t e r n a l block ( i . e. IX=PIB) y i e l d s an expected number of d i s k reads given by: r T | 1 + l o g (IX + 1) | I 2 | Since we can expect a t l e a s t 50 percent u t i l i z a t i o n o f i n t e r n a l node b l o c k s the f o l l o w i n g h o l d s : r i (2) > | 1 + l o g (IX+1) | I _ | 2 where "B" i s the i n t e r n a l node b l o c k i n g f a c t o r * The l a s t problem concerning a k-d t r e e primary index t h a t we deal with i n t h i s s e c t i o n concerns how a dynamic p h y s i c a l r e o r g a n i z a t i o n of the h o r i z o n t a l c l u s t e r i n g n e c e s s i t a t e d by a changing c h a r a c t e r i z a t i o n of use i s f a c i l i t a t e d . The problem i s comprised of two p a r t s : (a) determining when a c h a r a c t e r i z a t i o n of use has changed s u f f i c i e n t l y to warrant p h y s i c a l r e o r g a n i z a t i o n and (b) a c complishing the d e s i r e d r e o r g a n i z a t i o n . In SORAAM the c h a r a c t e r i z a t i o n of use t h a t determines the s e l e c t i o n o f d i s c r i m i n a t o r s during l e a f t u p l e block s p l i t s are the "gnocld" and " f o l d " v a l u e s found i n the s p e c i a l base r e l a t i o n s RELATIONS and ATTRIBUTES r e s p e c t i v e l y (see Appendix A). These values t h e r e f o r e determine.the c u r r e n t l y e x i s t i n g h o r i z o n t a l c l u s t e r i n g . Now r e c a l l from the s t a r t o f t h i s s e c t i o n t h a t a c h a r a c t e r i z a t i o n of use i s dynam i c a l l y maintained v i a the "gno" and " f " values found a l s o i n RELATIONS and ATTRIBUTES. B..The System 83 Determining a s u f f i c i e n t change between the " f " and " f o l d " (and s i m i l a r l y the "qno" and "qnoold") values t o warrant p h y s i c a l r e o r g a n i z a t i o n ( i . e . r e - h o r i z o n t a l l y c l u s t e r ; (a) above) i s accomplished i n SORAAM i n a simple f a s h i o n . In the next s e c t i o n we d e r i v e a c o s t f u n c t i o n based on weights d e r i v e d from the above and on the "d" value s ( r e p r e s e n t i n g the number of l e v e l s i n the k-d t r e e d i s c r i m i n a t i n g on each a t t r i b u t e * ) . L e t t i n g t h i s f u n c t i o n be denoted as "C" a s u f f i c i e n t change i s deemed to have occurred by SORAAM when the f o l l o w i n g c o n d i t i o n h o l d s : C (fold.qnoold) > c (3) C (f ,qno) where " c " i s an e x p e r i m e n t a l l y determined c o n s t a n t g r e a t e r than 1. The t e s t ; then* i s simple and overcomes the f i r s t p a r t of our problem ((a) above). We note i n passing t h a t a more e l a b o r a t e t e s t (weighing the b e n e f i t s o f r e o r g a n i z a t i o n v e r s e s the r e o r g a n i z a t i o n cost i t s e l f ; as i n [Yao e t a l . 76]) n e c e s s a r i l y i n v o l v e s the i n t r o d u c t i o n of a " c h a r a c t e r i z a t i o n of change o f use". Such i n f o r m a t i o n i s not c u r r e n t l y maintained or used by SORAAM. Accomplishing the d e s i r e d r e o r g a n i z a t i o n ((b) above) i s now simple. SORAAM simply reuses the mechanism used t o maintain b a l a n c i n g i n a k-d t r e e . That i s , when c o n d i t i o n (3) becomes t r u e f o r a p a r t i c u l a r base r e l a t i o n : *Note t h a t the "d" v a l u e s , themselves, are a f u n c t i o n of the "qnoold" and " f o l d " values s i n c e d i s c r i m i n a t o r s e l e c t i o n determines the former and i s determined by the l a t t e r . B. The System 84 (a) the s p e c i a l i n t e r n a l node "N*" i s added, (b) the "qnoold" and " f o l d " values are r e s e t from the "qno" and " f " v a l u e s r e s p e c t i v e l y ( f o r t h a t base r e l a t i o n ) and (c) the r e b a l a n c i n g process i s i n i t i a t e d . Note from the above t h a t d i s c r i m i n a t o r s e l e c t i o n f o r the r i g h t subtree of N* i s now based on the c u r r e n t c h a r a c t e r i z a t i o n o f use* Furthermore, as with r e - b a l a n c i n g , normal c o n c u r r e n t use o f a base r e l a t i o n and the primary index i s a l s o f a c i l i t a t e d . We have seen i n t h i s s e c t i o n how SOEAAM has automated one aspect of a EDMS's p h y s i c a l r e o r g a n i z a t i o n a l requirements. S p e c i f i c a l l y , a l l management of h o r i z o n t a l c l u s t e r i n g c o n s i d e r a t i o n s i n c l u d i n g a c h a r a c t e r i z a t i o n of use have been d e s c r i b e d * In the next s e c t i o n we d e a l with the s p e c i f i c problem of d i s c r i m i n a t o r s e l e c t i o n a r i s i n g from l e a f t u p l e block s p l i t t i n g . C. D i s c r i m i n a t o r S e l e c t i o n E e c a l l t h a t the primary i n d i c e s i n SOEAAM grow dyn a m i c a l l y i n the number of i n t e r n a l nodes ac c o r d i n g to the s i z e of the base r e l a t i o n s . In p a r t i c u l a r , a new i n t e r n a l node i s generated whenever an i n s e r t i o n o p e r a t i o n f o r c e s a l e a f t u p l e b l o c k s p l i t * One of the problems o u t l i n e d i n the previous s e c t i o n concerned the problem of s e l e c t i n g the a t t r i b u t e or column number on which t o d i s c r i m i n a t e between the two l e a f t u p l e b l o c k s t h a t a r i s e when t h i s event oc c u r s . C. The System 85 Such d i s c r i m i n a t o r s e l e c t i o n should c l e a r l y be a f u n c t i o n of our chosen c h a r a c t e r i z a t i o n o f use s t a t i s t i c s ( i n p a r t i c u l a r : " f o l d " and "qnoold" v a l u e s ) . In SORAAH a l l d i s c r i m i n a t o r s o c c u r i n g i n i n t e r n a l nodes t h a t are at the same l e v e l i n the k-d t r e e are equal* We can t h e r e f o r e c h a r a c t e r i z e the problem of d i s c r i m i n a t o r s e l e c t i o n as the problem of determining a sequence of d i s c r i m i n a t o r values h such t h a t cost f u n c t i o n s d e s c r i b i n g : P1. average e x t e r n a l t u p l e page v i s i t s per query (most importantly) and P2. average i n t e r n a l node v i s i t s per query ( l e s s importantly) are* i n some sense, minimized. In F i g u r e I I I - 3 i n the preceeding s e c t i o n , f o r example, h = {h1,h2} = {column* 1 ,column#2] = {1,2}.. In the remainder o f t h i s s e c t i o n we o u t l i n e two c o s t f u n c t i o n s t h a t c h a r a c t e r i z e the f i r s t o f these problems based on our chosen c h a r a c t e r i z a t i o n of use..An i d e n t i c a l c l o s e d form s o l u t i o n t o problem P1 above i s d e r i v e d f o r both these c o s t f u n c t i o n s and a method f o r choosing between the two i s gi v e n . F o l l o w i n g t h i s we demonstrate t h a t the two problems above are i n t i m a t e l y r e l a t e d and d e r i v e a dynamic a l g o r i t h m s o l v i n g both problems t h a t d i r e c t l y generates h. SORAAM's implementation of t h i s a l g o r i t h m completes the s e c t i o n * Consider t h a t the c h a r a c t e r i z a t i o n of use s t a t i s t i c s " f o l d " and "qnoold" of a p a r t i c u l a r base r e l a t i o n R d i r e c t l y r e f l e c t i t s query h i s t o r y . That i s , a l l q u e r i e s t h a t have a r r i v e d a g a i n s t R have provided one and only one a t t r i b u t e C. The System 86 value. The p r o b a b i l i t y of a p a r t i c u l a r a t t r i b u t e i being s p e c i f i e d i n a query i s then given by: f o l d p - i - i qnoold Note t h a t the d i r e c t n e s s assumption above i m p l i e s : k k Zf o l d c** - - i - = y p = 1 where k i s the number qnoold i of a t t r i b u t e s i n E. i=1 i=1 L e t t i n g D = | {h) | and d i = I (h j | i = j and 1<j<D) | where {.,,} denotes a s e t and |'{...} I denotes i t s s i z e we may c h a r a c t e r i z e P1 as: k 1 D-di D p 2 where ^ix=T) and 2 i s the number (4) i i=1 of l e a f t u p l e b l o c k s i n E.. Taking out c o n s t a n t s (assuming a f i x e d s i z e d E) and l e t t i n g wi=pi we d e r i v e a c o s t f u n c t i o n c h a r a c t e r i z i n g e x t e r n a l node v i s i t s : k F(d ) = 2W (5) i=1 C i - d i (d ) = / w i ^ i su b j e c t t o the c o n s t r a i n t ^di=D In [Aho and Oilman 79] we a r e given an account o f a d e r i v a t i o n of a c l o s e d form e x p r e s s i o n y i e l d i n g optimum d i s t r i b u t i o n of d i s c r i m i n a t i n g b i t s (determining "bucket" addresses) among a base r e l a t i o n ' s a t t r i b u t e s f o r p a r t i a l C. The System 87 match r e t r i e v a l . T h e i r c h a r a c t e r i z a t i o n of use, however* i s based on the assumption t h a t a t t r i b u t e s are independently s p e c i f i e d . The p r o b a b i l i t y of a query s p e c i f y i n g the f i r s t m of k a t t r i b u t e s of E, t h e r e f o r e , would be: p • p . . . . • p • (1-p ) . . . . . ( 1 - p) 1 2 m m + 1 k or m k 7 T p . TT (1-p ) i=1 i i=m+1 i S i m i l a r l y , t h e i r e x p r e s s i o n c h a r a c t e r i z i n g average bucket accesses (in our case e x t e r n a l node v i s i t s ) c o r r e s p o n d i n g t o (4) above i s given by: k r n D T T J p i | - d i 2 P / (1 + | 1 2 ) (6) " M - p i l k i = 1 «- -• where P = 77 (1-pi) and i=1 D, d i and p i are d e f i n e d as above. Again t a k i n g out c o n s t a n t s but now l e t t i n g wi=pi/(1-pi) we have a cost f u n c t i o n c h a r a c t e r i z i n g e x t e r n a l node v i s i t s assuming independently s p e c i f i e d a t t r i b u t e s : k - d i G(d ) = / / (1 + w 2 ) (7) (d ) = JJ (1 + w 2 X) i=1 s u b j e c t to the c o n s t r a i n t ^di=D. Using the method of Lagrangian m u l t i p l i e r s t o s o l v e f o r the d i i n (5) above we have: C. The System 88 d dd i=1 - d i 2 + I ^ i |i=1 L - D = 0 f o r i=1,2,...,k or - d i - w (ln 2) 2 + ~}\ = 0 i or d = l o g w - c f o r some co n s t a n t c, i 2 i (8) The reader may check that (8) i s a l s o d e r i v e d when we apply the same method above to solve f o r the d i i n (7) . . From [ Aho and Oilman 79], then, (8) l e a d s to the c l o s e d form: k D 1 - r 7 d = - + l o g w • - / l o g w i k 2 i k ̂  2 i i=1 (9) An o p t i m a l i n t e g e r valued s o l u t i o n can now be d e r i v e d from the above by a simple rounding process* . The same r e f e r e n c e demonstrates t h a t the f o l l o w i n g proceedure y i e l d s such an opt i m a l i n t e g e r s o l u t i o n : "Add the f r a c t i o n a l p a r t s (which must sum to an in t e g e r ) t o determine how many d i ' s must be rounded up. S e l e c t t h i s number of d i ' s having the l a r g e s t f r a c t i o n a l p a r t s and round them up; round the o t h e r s down."* 4From page 174 of [Aho and Oilman 79], C. The System 89 C o n s t r a i n i n g the d i i n the v a l u e s they may assume by a lower and/or upper bound i s a l s o e a s i l y accommodated. In (4) and (6) above, f o r example, the d i are c o n s t r a i n e d to be non-negative. [Aho and Ullman 79] give us the f o l l o w i n g procedure to d e a l with such c o n s t r a i n t s : (a) From the a p p l i c a t i o n of (9) determine a l l i f o r which d i i s o u t s i d e a bound. (b) Set each such d i t o i t s n e a r e s t bound and then r e s o l v e the system with these d i d e l e t e d . (c) Should other d i , as a r e s u l t , "overstep a l i m i t " then repeat from (b) above. Note t h a t accomplishing step (b) i s simple s i n c e : (9) i m p l i e s t h a t u n c o n s t r a i n e d d i vary with a constant amount i n a change of D. We have now d e r i v e d a s o l u t i o n to P1 i n the s t a t i c case of f i x e d D y i e l d i n g a d i s t r i b u t i o n of the d i s c r i m i n a t o r values d i among the a t t r i b u t e s of a base r e l a t i o n . The d i s c r i m i n a t o r v alue sequence h can now be generated from these d i v a l u e s . Furthermore, the average i n t e r n a l node v i s i t c o s t (P2 above) can now be expressed as a f u n c t i o n of the permutation of the d i w i t h i n h. In the f o l l o w i n g we d e r i v e a dynamic a l g o r i t h m f o r k-d t r e e g e n e r a t i o n t o o p t i m a l l y h o r i z o n t a l l y c l u s t e r t u p l e s of a base r e l a t i o n a c c o r d i n g t o e i t h e r of the two i n t e r p r e t a t i o n s of our c h a r a c t e r i z a t i o n of use. T h i s a l g o r i t h m w i l l be seen to minimize both P1 and P2 above* To r e i t e r a t e * the two i n t e r p r e t a t i o n s of our c h a r a c t e r i z a t i o n of use can be d e s c r i b e d as f o l l o w s : (a) i f we map f o l d i / g n o o l d p i then p i -> wi we are assuming a d i r e c t r e p r e s e n t a t i o n of query h i s t o r y C. The System 90 and (b) i f we map -> p i as i n (a) and p i / ( 1 - p i ) •-> wi we are assuming an independent r e p r e s e n t a t i o n of guery h i s t o r y . . C l e a r l y the c h o i c e of mapping depends on which more c l o s e l y r e l a t e s the c h a r a c t e r i z a t i o n of use with query h i s t o r y . The approach used i n SORAAM to decide among these.mappings i s the f o l l o w i n g : k i f 21 f o l d > c . qnoold ( f o r some e x p e r i m e n t a l l y i=1 i determined c o n s t a n t c) P i then choose -> w (independent model) (1-Pi) i otherwise choose p -> w ( d i r e c t model) i i Regardless of the choice o f mappings the f u n c t i o n s given i n (4) , (5) , (6) and (7) above are a l l minimized a t l i k e d i s t r i b u t i o n s d i of the d i s c r i m i n a t o r values given by h. Consider now an a l t e r n a t i v e f o r m u l a t i o n of (4). Given a k-d t r e e o f depth D with the d i s c r i m i n a t o r value sequence (from r o o t to leaves) h=h1,h2,.,,,hD we d e f i n e an i - t r e e of the k-d t r e e (1<i<k) as the subtree with the c h a r a c t e r i s t i c t h a t a l l nodes with d i s c r i m i n a t o r value i have one son only* Given the k-d t r e e i n F i g u r e I I I - 3 , f o r example, with D=2 and h={1,2} the 1-tree would be: C..The System 91 . N I I .N__ | | where "N" are i n t e r n a l nodes and I j "L" are l e a f t u p l e b l o c k s . L L and the 2-tree would be: N . I I I I N N | | where "N" and "L" are d e f i n e d j | as above. I L Now we can ref o r m u l a t e (4) a s : k H V 1 (h) = / w L where h, D f w are d e f i n e d as (10) ^ i D i i i=1 above and L are the number D of l e a v e s i n t h e . i - t r e e s (a f u n c t i o n of h). Likewise problem P2 can now be c h a r a c t e r i z e d as minimizing the c o s t f u n c t i o n : k D-1 J (h) = > w M where M = ? L ( i * e ; the number D — * i D D j of i n t e r n a l nodes i=1 j=1 of the i - t r e e ) i and M = 0 (11) 1 He can r e l a t e (10) and (11) as f o l l o w s : Ci The System 92 k | D-1 | k D-1 f (h) = / w | / L | = / V w L D ^— 1 ± | ^— 1 j | ^ ^ i j i=1 |j=1 | i=1 j=1 L J D-2 k = ) w L + ) w L i-—i L a i j L.—» i D-1 j=1 i=1 i=1 D-1 = J (h) + H (h) = 5JH (h) (12) D-1 D-1 i=1 i From (9) and i t s i n t e g e r s o l u t i o n i t i s c l e a r t h a t a minimal Hm (h) and Hm-1(h) ( f o r m<D) have the property t h a t t h e r e e x i s t s a unigue i such t h a t : (a) d = d» f o r i-»=j and j j m m-1 (b) d = d« +1 f o r i = j j j m m-1 where d = | {h | i = j and 1<j<n}| i j n T h i s now suggests how we can generate h t o minimize both H and i J above* R e c a l l i n g t h a t I i s the number o f l e a v e s i n the D i - t r e e g i v e n by h (with depth D). T h i s may be r e w r i t t e n as: C. The System 93 1 D-N (h) i D n L (h) = 2 where N (h) i s d e f i n e d as D m the number o f h (i<m) i such t h a t h =n. i The approach i n v o l v e s a j u d i c i o u s choice o f a summand i i n (10) whenever the depth of the k-d t r e e i n c r e a s e s by one., We simply choose the l a r g e s t summand i . T h i s approach ensures t h a t f o r a l l 1<m<D the f o l l o w i n g h o l d s : j hm m-N m-N m m w 2 < w 2 f o r 1<j<k j " hm or j hm log w - N < l o g w - N (13) 2 j m 2 hm m Note t h a t an o p t i m a l h minimizing H and J may not be unique s i n c e , f o r example, there may be more than one c h o i c e f o r hD (i i , e . a t the lowest l e v e l ) s a t i s f y i n g (13). By convention SORAAM always chooses the minimal column number f o r hm. S p e c i f i c a l l y , when a l e a f t u p l e block at depth m of base r e l a t i o n R s p l i t s SORAAM behaves as f o l l o w s : C. The System 94 (a) The s e t S of c o n s t r a i n e d d i s c r i m i n a t o r values j (1<j<k) i s determined as those j such t h a t : j N +1 m 2 < c . n ( f o r an e x p e r i m e n t a l l y determined constant c) where n, found i n ATTRIBUTES, i s the number of d i s t i n c t values of the: a t t r i b u t e with {Aname=R S COL#=j}.. (b) The d i s c r i m i n a t o r value i i s chosen as the minimal i such t h a t : j i log w - N < l o g w - N 2 j m ~* 2 i m f o r a l l i , j not i n S. j j (c) N i s assigned N f o r (1<j<k) and (j-«=i) and m + 1 m i i N i s assigned N + 1. m+1 m (d) I f m=D then both D found i n RELATIONS with {Rname=R3 and d found i n ATTRIBUTES with {Rname=R & COI#=i) are incremented. . The a t t r i b u t e s n, d and D used above are l o c a t e d i n the s p e c i a l base r e l a t i o n s RELATIONS and ATTRIBUTES (see Appendix A). The N values are represented as a s i n g l e k - v e c t o r K i n each l e a f t u p l e block as f o l l o w s : i k K = N where 2 K = m i m i=1 i Step (c) above, t h e r e f o r e , i s accomplished by simply i n c r e m e n t i n g K i . L a s t l y , the weights w used i n ste p (b) above are determined by the a p p r o p r i a t e mapping from the " f o l d " and C. The System 95 "qnoold" values found a l s o i n the s p e c i a l base r e l a t i o n s . , According to two fundamentally d i f f e r e n t i n t e r p r e t a t i o n s of our c h a r a c t e r i z a t i o n of use, t h e r e f o r e * we have d e r i v e d a dynamic a l g o r i t h m f o r d i s c r i m i n a t o r s e l e c t i o n i n primary i n d i c e s t h a t r e a l i z e s an op t i m a l h o r i z o n t a l c l u s t e r i n g o f the t u p l e s w i t h i n base r e l a t i o n s . . D. Secondary Indexing T h i s s e c t i o n d e a l s p r i m a r i l y with the automatic s e l e c t i o n o f s i n g l e a t t r i b u t e i n v e r s i o n s on base r e l a t i o n s i n SORAAM. Such i n v e r s i o n s are c a l l e d secondary i n d i c e s and are d e s i r a b l e f o r t h e i r p o t e n t i a l of f a c i l i t a t i n g more e f f i c i e n t search w i t h i n base r e l a t i o n s * 1 We s h a l l see t h a t automating s e l e c t i o n of secondary i n d i c e s , however, r e q u i r e s t h a t SORAAM have an i n t i m a t e knowledge of both the p h y s i c a l and l o g i c a l a s p e c t s o f such i n d i c e s * For t h i s reason we begin the s e c t i o n with a d e t a i l e d p r e s e n t a t i o n of the v a r i e t y o f secondary i n d e x i n g c u r r e n t l y c o n s i d e r e d by SORAAM. C o n s i d e r a t i o n i s then g i v e n of a v a r i e t y of b a l a n c i n g c o n s t r a i n t s used to c o n t r o l t h e : i n d e x i n g . The c o n t r o l parameters adopted by SORAAM f o r t h i s purpose are then presented f o l l o w e d by the index s e l e c t i o n d e t a i l s themselves. tSee i n Chapter I I , Sec t i o n C our d e f i n i t i o n of secondary i n d e x i n g . D. The System 96 The s e c t i o n c o n c l u d e s with a more g e n e r a l c o n s i d e r a t i o n of the no t i o n o f secondary i n d e x i n g . F i g u r e I I I - 5 below prese n t s the example base r e l a t i o n " P J" represented i n terms of i t s primary index, l e a f t u p l e b l o c k s and s i n g l e secondary index on the. second column a t t r i b u t e "J#". 1 The two i n t e r n a l nodes of the primary index have the form " d i s c r i m i n a t o r * : k e y value". The t u p l e s i n the l e a f t u p l e b l o c k s of Fi g u r e I I I - 5 (b) are addressed by " t u p l e i d e n t i f i e r s " or " t i d s " of the form t i j where i and j index t u p l e b l o c k s and t u p l e s within b l o c k s r e s p e c t i v e l y . In SORAAM a secondary index may e x i s t on any of the t h r e e a t t r i b u t e s P#, J * and QTY of PJ but not, f o r example, on a con c a t e n a t i o n of P#+QTY. Furthermore, any such index i s an i n v e r s i o n (not a m u l t i l i n k or m u l t i r i n g ) . I n v e r s i o n s are a c t u a l l y comprised of two components.. The f i r s t i s a B-tree index with e n t r i e s of the form {v,p} where v i s a v a l u e of the indexed a t t r i b u t e o c c u r r i n g i n the base r e l a t i o n and p i s a p o i n t e r t o a p h y s i c a l l y s e q u e n t i a l l i s t of t i d s o f t u p l e s having t h a t value. The second component o f an i n v e r s i o n i s the s e t of secondary t u p l e p o i n t e r b l o c k s (STPBs) i n which the t i d l i s t s themselves are stored* In F i g u r e I I I - 5 (c) f o u r such STPBs are used to c o n t a i n the f o u r t i d l i s t s . Furthermore, a l l t i d s i n each t i d l i s t are:maintained i n 1 We may c o n s i d e r t h a t PJ i s i n t e r p r e t e d by a c o n t r a c t i n g f i r m as r e p r e s e n t i n g the q u a n t i t y ("QTY") of each p a r t ("P#") s u p p l i e d by the f i r m t o each job ("J#"). D. The System 97 Example Base R e l a t i o n With Primary and Secondary Indexing ( l o g i c a l view PJ p# | J# | QTY P1 | J1 I 4 P1 | J2 | 1 P1 I J3 I 3 P1 J4 I 5 P2 I J1 I 4 P2 J2 I 2 P3 J2 I <* P3 J3 I 4 P3 J4 I 3 P4 I J3 I 5 (a) I I. t 1 1 | P l J2 1| t 2 1 | P l J1 4| t31|P3 J2 4| t12|P1 J3 3| t22|P1 J4 5| t32|P3 J3 4| t13|P2 J2 2| t23|P2 J1 4| t33|P3 J4 3| I I I I I I I. I t34|P4 J3 5| !_______I ( l e a f t u p l e blocks) (b) (E-tree index on J#) I I |J1+,J2+,J3+,J4+| I—I. _ _ l I I _ l I IV I IV I IV || V I I t 2 1 * , t 2 3 , | j t 1 1 * , t 1 3 , t 3 1 * , | | t 1 2 * , t 3 2 * r | | t 3 4 , t 2 2 * , t 3 3 * | I I I - •• - • I I _ _ _ _ _ _ _ _ II. (secondary t u p l e p o i n t e r blocks) (c) F i g u r e I I I - 5 s o r t e d order. A scan on PJ v i a the J# i n v e r s i o n would produce the sequence of t i d s given by t h e i r l e f t to r i g h t sequence i n D. The System 98 F i g u r e I I I - 5 ( c ) . Notice that such a scan v i s i t s each STPB only once* When r e t r i e v i n g the t u p l e a s s o c i a t e d with a t i d i t may be necessary t o r e t r i e v e the t u p l e b l o c k , i n which i t r e s i d e s , from mass storage* Such an event i s a f u n c t i o n of the r e l e v e n c e t o the scanned a t t r i b u t e of the h o r i z o n t a l c l u s t e r i n g , of the s i z e of the t i d l i s t s themselves and of the number o f main memory b u f f e r s a v a i l a b l e f o r the o p e r a t i o n . Assuming t h a t one b u f f e r only i s a v a i l a b l e t o c o n t a i n a l e a f t u p l e block then scanning PJ v i a the J# i n v e r s i o n would produce seven t u p l e block reads. Those t i d s i n F i g u r e I I I - 5 (c) causing such reads are p o s t f i x e d with "*"„ In the next s e c t i o n on search s t r a t e g y we e l a b o r a t e on e s t i m a t i n g such c o s t s . I f an i n s e r t i o n or d e l e t i o n occurs i n a base r e l a t i o n then a d d i t i o n a l e f f o r t i s r e g u i r e d t o s u i t a b l y a l t e r any secondary i n d i c e s on the base r e l a t i o n ' s a t t r i b u t e s . C o n s i d e r , f o r example, an i n s e r t i o n i n t o PJ of the t u p l e T={P2,J3,6}. According to the primary index T would be i n s e r t e d i n t o the second l e a f t u p l e block and a s s i g n e d the t i d t24.. Since the value "J3" a l r e a d y e x i s t s i n the B-tree then updating the secondary index simply i n v o l v e s adding t24 to J3's t i d l i s t . Another e n t r y i n the t h i r d STPB would then ensue. The r e s u l t i n g STPB would appear as: I _________ IV I It12,t24,t32,| I __________ I New STPBs are c r e a t e d whenever t i d i n s e r t i o n s cause ov e r f l o w s . D. The System 99 Should the maximum c a p a c i t y of an STPB be three t i d s and should T's assigned t i d have been t35 then the f o u r t h STPB would s p l i t c r e a t i n g a new STPB i n order to accommodate t35 as f o l l o w s : I I IV I |t31,t35,| |t22,t33| I » I I Note from the above t h a t J4*s t i d l i s t p o i n t e r must a l s o be updated* F u r t h e r d e t a i l s concerning our v a r i e t y of secondary i n d e x i n g can now be simply deduced. I t should be c l e a r , f o r example, t h a t the STPBs f o r a p a r t i c u l a r index be doubly l i n k e d t o one another i n the obvious manner. Furthermore, the s p l i t t i n g approach t o STPB c r e a t i o n i m p l i e s a worst case of 50 percent space u t i l i z a t i o n . However, the expected case space u t i l i z a t i o n i s much b e t t e r and i s very l i k e l y uniform f o r a l l secondary i n d i c e s of any base r e l a t i o n . For each base r e l a t i o n t h i s expected case u t i l i z a t i o n i s represented as a f r a c t i o n a l value of the a t t r i b u t e " f f 2 " i n RELATIONS (see Appendix A).. The reason f o r our concern with making so e x p l i c i t the a r c h i t e c t u r e of permitted secondary i n d e x i n g i s t h a t d e c i d i n g which secondary i n d e x i n g t o c r e a t e and d e l e t e i s i n t i m a t e l y r e l a t e d to search s t r a t e g y (next section) which i n turn i s i n t i m a t e l y concerned with cost e s t i m a t i o n of query e v a l u a t i o n . T h i s c y c l e i s i l l u s t r a t e d by F i g u r e I I I - 6 below with arrows denoting the dependency. The next t h i n g t o c o n s i d e r i n automating the s e l e c t i o n o f D. The System 100 A C y c l i c Dependency In Secondary Index S e l e c t i o n Cost I E s t i m a t i o n \ / Secondary Indexing / S t r a t e g y v Search F i g u r e I I I - 6 the v a r i e t y of i n d e x i n g d i s c u s s e d above i s the s p e c i f i c c o n t r o l mechanism with which d e c i s i o n s about the c h o i c e o f ind e x i n g are made. Suppose, f o r example, t h a t we adopt a design c o n s t r a i n t t h a t r e q u i r e s complete i s o l a t i o n of a l l index s e l e c t i o n c o n s i d e r a t i o n s from SOEAAM* s users. T h i s might be accommodated by a method based on any of the f o l l o w i n g : (a) A l l emphasis i s placed on r e t r i e v a l performance only. That i s * no c o n s t r a i n t e x i s t s i n s e l e c t i n g i n d i c e s . In t h i s case a l l base r e l a t i o n s would e x i s t f u l l y i n v e r t e d ( i * e . with an index on each a t t r i b u t e ) . (b) A l l emphasis i s placed on minimizing t o t a l I/O t r a f f i c t o mass storage* In t h i s case the b a l a n c i n g c o n s t r a i n t t o index s e l e c t i o n i s the i n c r e a s e d I/O im p l i e d by maintenance o p e r a t i o n s . A secondary index reduces t o t a l block reads f o r r e t r i e v a l e v a l u a t i o n and i n c r e a s e s block reads and w r i t e s f o r i n s e r t i o n s , updates and d e l e t i o n s . (c) A g l o b a l space/time t r a d e o f f i s assumed by SORAAM. In t h i s case a s e l e c t i o n of the most advantagous subset of the p o s s i b l e i n v e r s i o n s f i t t i n g w i t h i n the permitted space overhead of a base r e l a t i o n i s made. Should a l l base r e l a t i o n s be f u l l y i n v e r t e d then the problem of index s e l e c t i o n i s t r i v i a l . . Both the space and Di. The System 101 p o t e n t i a l l y l a r g e maintenance overhead of t h i s approach, however* are c l e a r l y unacceptable i n the environment assumed by SOEAAM. Consi d e r , f o r example, a base r e l a t i o n implementing a s p o o l i n g or a u d i t i n g f u n c t i o n . The o p e r a t i o n s on such a r e l a t i o n may almost e n t i r e l y i n v o l v e i n s e r t i o n s and d e l e t i o n s . A l t e r n a t i v e l y , i n using maintenance I/O as a b a l a n c i n g c o n s t r a i n t a t r a d e o f f i s assumed between query and maintenance performance.. SOEAAM would u n c o n d i t i o n a l l y weight a block read or write caused by an i n s e r t i o n equal t o a block read caused by a r e t r i e v a l . There are c l e a r l y a p p l i c a t i o n s f o r which such a t r a d e o f f would be e n t i r e l y i n a p p r o p r i a t e . . c o n s i d e r , f o r example, an o n - l i n e c r i t i c a l p a t i e n t i n f o r m a t i o n system f o r a h o s p i t a l * One can understand t h a t r e t r i e v a l performance would rank f a r above maintenance performance f o r base r e l a t i o n s i n v o l v e d i n such an a p p l i c a t i o n . T h i s same c r i t i c i s m a l s o a p p l i e s i n a more obvious manner to an approach i n v o l v i n g a g l o b a l space/time t r a d e o f f assumption*. The r e t r i e v a l performance requirement f o r one base r e l a t i o n can e n t i r e l y d i f f e r from another. For these reasons a design c o n s t r a i n t of complete i s o l a t i o n of a l l index s e l e c t i o n c o n s i d e r a t i o n s w i t h i n SOEAAM has not been adopted. Instead an attempt i s made t o render the necessary user communication concerning index s e l e c t i o n i n as simple a form as p o s s i b l e . Towards t h i s end SOEAAM maintains f o r each base r e l a t i o n a measure o f the i n d e x i n g e f f e c t i v e n e s s * The g l o b a l performance measure (g.p*m.) i s simply a r a t i o of the number D..The System 102 of r e l e v a n t block reads ( i . e . a d i s k read of a l e a f t u p l e block c o n t a i n i n g a d e s i r e d tuple) to t o t a l block reads ( i n c l u d i n g primary index b l o c k s , B-tree b l o c k s and STPBs) f o r past r e t r i e v a l o p e r a t i o n s . The a t t r i b u t e s " g .p.m."RDQreads" and "DQreads" i n RELATIONS e x i s t f o r t h i s purpose (see Appendix A). In the same way as " r a d i x l " can be used t o c o n t r o l the r a t e of ada p t a t i o n of the c h a r a c t e r i z a t i o n o f use s t a t i s t i c s * t he a t t r i b u t e " r a d i x 2 " can be used t o s i m i l a r l y c o n t r o l the g. p. m. The c o n t r o l over secondary in d e x i n g f o r each base r e l a t i o n i s now accommodated by the use of the f o l l o w i n g t h r e e user s u p p l i e d parameters: (a) Plow, Phigh - These are a lower and upper bound r e s p e c t i v e l y on g.p.m. (the r e t r i e v a l performance). I n c r e a s i n g Plow would t h e r e f o r e tend t o c r e a t e f u r t h e r i n d e x i n g while d e c r e a s i n g Phigh would tend to d e l e t e c u r r e n t i n d e x i n g . For each base r e l a t i o n SORAAM attempts t o ensure t h a t the f o l l o w i n g c o n d i t i o n holds: Plow < q.p.m. = RDQreads < Phigh DQreads (b) S - T h i s parameter conveys the maximum percentage o f i n d e x i n g storage overhead permitted f o r the base r e l a t i o n * A l l b l o c k s used by the i n t e r n a l nodes of the primary index and by secondary i n d i c e s (both the B-tree nodes and STPBs) are regarded as storage overhead. A l l t h r e e o f the Plow, Phigh and S are a l s o l o c a t e d i n RELATIONS. For each base r e l a t i o n t he l a t t e r parameter S i s assumed *See beginning of Chapter I I I Section- B. D. The System to have precedence over Plow. That i s , should the user i m p l i e d minimum r e t r i e v a l performance imply an indexing s e t ( i . e . s e t of s i n g l e a t t r i b u t e . i n v e r s i o n s f o r a base r e l a t i o n ) exceeding the permitted storage overhead i m p l i e d by S then SORAAM d e r i v e s an a l t e r n a t i v e i n d e x i n g s et best u t i l i z i n g the permitted storage;. A l s o , should Plow exceed a c e r t a i n t h r e s h o l d value then a f u l l i n v e r s i o n of the base r e l a t i o n may ensue. S e t t i n g Plow t o a high v a l u e : then a l l o w s a d i r e c t c o n t r o l of a space/time t r a d e o f f v i a S by a user.. In a d d i t i o n t o g.p.m. SORAAM a u t o m a t i c a l l y maintains a v e c t o r b on each base r e l a t i o n with elements b i (i>0) o f the r e l a t i v e number of times each p o s s i b l e secondary index would have been used should i t have always e x i s t e d . Note t h a t i denotes a secondary index cn the a t t r i b u t e with (COL#=i). With r e s p e c t to t h i s v e c t o r another s t a t i s t i c " Pcost" i s a l s o maintained. P c o s t , however, denotes a value comparable to t h a t of DQreads with the assumption t h a t no i n d e x i n g has ever e x i s t e d . . The value of Pcost i s i n c r e a s e d by an estimate i n block reads o f the c o s t f o r each <predicate> i n s t a n c e e v a l u a t i o n v i a the primary index. Furthermore, should the maximum depth o f the primary index i n c r e a s e by one then P c o s t i s a l s o incremented* In t h i s case the d i s c r i m i n a t o r COL# f o r the new l e v e l determines the increment. In p a r t i c u l a r * the i n c r e a s e i n primary index and l e a f t u p l e b lock v i s i t s doubles f o r the f r a c t i o n of q u e r i e s not s u p p l y i n g values f o r the new d i s c r i m i n a t i n g COL#. That i s : D. The System 104 qnoold - f o l d P cost <- Pcost + " Pcost qnoold These s t a t i s t i c are p r i m a r i l y maintained by a component of the search s t r a t e g i e s (see S e c t i o n E of t h i s chapter) which i s invoked f o r each such <predicate> i n s t a n c e submitted to SORAAM. Vector b i s s t o r e d as values of the a t t r i b u t e "b" i n ATTRIBUTES; Pcost i s found i n RELATIONS.. The s p e c i f i c search s t r a t e g y f o r e v a l u a t i n g any i n d i v i d u a l o p e r a t i o n r e q u i r e d o f SORAAM always i n v o l v e s the use of a s i n g l e index only (be i t primary or secondary). As a r e s u l t o f t h i s the b i are independent i n t h e i r i m p l i c a t i o n o f index a p p l i c a b i l i t y t o guery e v a l u a t i o n ; t h a t i s , the sum bi+bj ( i - , = j ) makes sense. . SORAAM•s approach t o index s e l e c t i o n , t h e r e f o r e , i s t o apply approximate a l g o r i t h m s f o r an a p p r o p r i a t e l y formulated knapsack problem.» In l i g h t o f t h i s we d e f i n e the f o l l o w i n g b e n e f i t v e c t o r : r i r i b' = b . max{0,| block reads i f |-| block reads i f |} i i li n d e x i doesn't e x i s t | |index i does e x i s t | L J L J (A) (B) D-di where (A) = 2 . {(IX + 1) + PIB} = (IX+PIB + 1) D d i 2 2 ( i . e . a f r a c t i o n of t u p l e blocks + index pages of k-d tree) *A f a m i l y of such a l g o r i t h m s may be found i n [ S a h n i 75]. Di The System 105 D-di and (B) = l o g r n ^ + STPB - + min {N •, 2 } I i | i n n | +1| i i ISIB | (14) ** i ( i . e . index block reads + STPB reads + t u p l e : b l o c k reads) In the above the values of Dr d i and IX have already been i n t r o d u c e d i n p r e v i o u s s e c t i o n s . The other v a l u e s PIB, n i , S I B i , STPB and N are maintained i n GELATIONS and ATTRIBUTES. The reader i s r e f e r e d to Appendix A f o r a d e s c r i p t i o n of each. The b e n e f i t v e c t o r given by (14) i l l u s t r a t e s the requirement f o r i n t i m a t e knowledge of the a r c h i t e c t u r e of the i n d e x i n g by index s e l e c t i o n components. Cost e s t i m a t i o n i n block reads o f u s i n g the v a r i o u s p o s s i b l e access paths c l e a r l y r e q u i r e s such knowledge. The t h i r d component of the c o s t e s t i m a t i o n f o r (B) above, f o r example, i n c o r p o r a t e s the p o s s i b l e e f f e c t of h o r i z o n t a l c l u s t e r i n g by the primary index. A c o s t v e c t o r c i s d e f i n e d i n terms of the number o f b l o c k s necessary f o r each index. More s p e c i f i c a l l y : r T c = |# of index b l o c k s and STPBs| i | necessary f o r index i | L J = SIB + STPB i Note t h a t c i s d e f i n e d i n terms of storage c o s t only. Should one assume t h a t maintenance I/O overhead i s uniform over any i n d e x i n g s e t then such a cost f u n c t i o n i n d i r e c t l y i n c o r p o r a t e s maintenance c o s t * D. The System 106 We now wish t o c a l c u l a t e a t h r e s h o l d performance requirement P f o r a new i n d e x i n g s e t . Denoting the c u r r e n t i n d e x i n g s et as a vector x* where: r x' = | 1 i f C0L#=i c u r r e n t l y i n d e x e d 1 i | 0 otherwise L f o r i v a r y i n g over a l l p o s s i b l e C0L# i n the base r e l a t i o n o f concern then the f o l l o w i n g r e p r e s e n t s another estimate of a value f o r DQreads ( i . e * b l o c k reads f o r past r e t r i e v a l o p e r a tions) r e l a t i v e to an i n d e x i n g s e t : Pcost - (b«. x') Allowing Plow = RDQreads/y we have: Pcost - (b*. x') = Pcost - P DQreads y Now s o l v i n g f o r P we have: P = Pcost - RDQreads . Pcost - fb'. xM Plow DQreads = Pcost - q.p.mi. . f Pcost - (b'.x ' ) } Plow We can now f o r m a l l y s p e c i f y the approach used by SORAAM to determine an i n d e x i n g s et x f o r a base r e l a t i o n * Whenever e i t h e r o f the f o l l o w i n g two c o n d i t i o n s occur: (a) g*p.m. < Plow ( i . e . not enough indexing) 4 I f l a g i s tr u e i n t h i s case (see Appendix A). D. The System 107 (b) g.p.m. > Phigh ( i . e . too much indexing) then SORAAM s o l v e s the f o l l o w i n g knapsack problem: minimize c . x (15) s u b j e c t t o b'. x > P where x i n {0,1} i and, s i m i l a r l y , whenever e i t h e r of the f o l l o w i n g occur:*• (c) (c . x) + PIB > (S . [IX + 1]) ( i . e . above s o l u t i o n (15) r e q u i r e s too much storage overhead) (d) (c . x') + PIB > (S . [IX + 1]) ( i * e . c u r r e n t i n d e x i n g s e t r e q u i r e s too much storage overhead) then the problem becomes: maximize b*. x (16) s u b j e c t t o c . x < (S . [IX+1]) - PIB The chosen i n d e x i n g set corresponds to a l l xi=1 i n x. Thus we see how SORAAM automates the s e l e c t i o n of s i n g l e - a t t r i b u t e i n v e r s i o n s thereby accomplishing automation o f another major p h y s i c a l o r g a n i z a t i o n d e t a i l . There are a number of ways i n which these r e s u l t s may be extended. The most obvious i s to add to the c h o i c e of i n d e x i n g data s t r u c t u r e s a v a i l a b l e t o SORAAM. Perhaps minimally one should c o n s i d e r hashing and TRIE techniques and even v a r i o u s other s t a t i c s e a r c h i n g s t r u c t u r e s . L i n k i n g data s t r u c t u r e s such as the l i n k access path i n System R are a l s o c l e a r l y d e s i r a b l e . 1Note t h a t i n both c o n d i t i o n s the c o s t of the primary index i s i n c l u d e d * r D..The System 108 Consider a s i t u a t i o n i n which the c o n s t r a i n t s of an aspect o f a world model d i c t a t e a very r e g u l a r d i s t r i b u t i o n o f values on a s p e c i f i c a t t r i b u t e of a r e l a t i o n . . C o n s i d e r a l s o t h a t the model d i c t a t e s t h a t the a t t r i b u t e i s a key of the r e l a t i o n . The obvious r e p r e s e n t a t i o n of an index f o r t h i s " n i c e " a t t r i b u t e would simply be a f i x e d s i z e : FORTHAN-like a r r a y . Furthermore, such a s i t u a t i o n i s not so u n l i k e l y . The " e n t i t y domains" f i r s t presented i n [ H a l l e t a l . . 76] and adopted i n [Codd 79] are a p o s s i b l e example. , Values i n e n t i t y domains are system generated and unseen and unknown t o end user s ; One may even consi d e r a design d e c i s i o n t o not commit SORAAM t o a f i x e d s m a l l number of se a r c h i n g data s t r u c t u r e s and search a l g o r i t h m s . SORAAM would f a c i l i t a t e the easy e x t e n s i o n of both i t s secondary i n d e x i n g c a p a b i l i t i e s and i t s user i n t e r f a c e ( i . e . augmenting the v a r i e t y of non-root node t y p e s ) . J u s t i f i c a t i o n f o r such a d e c i s i o n seems a l l the more c l e a r when one c o n s i d e r s the e x i s t e n c e o f se a r c h i n g data s t r u c t u r e t r a n s f o r m a t i o n s as i n [ B e n t l e y 78c]. „ Accommodating a d d i t i o n a l i n d e x i n g data s t r u c t u r e s i s r e l a t i v e l y simple as long as we maintain a c o n s t r a i n t t h a t no more than one be used t o e v a l u a t e any given i n s t a n c e of a <predicate>; as long as problems (15) and (16) are w e l l d e f i n e d . . Should n d i f f e r e n t v a r i e t i e s of i n d e x i n g data s t r u c t u r e s be a v a i l a b l e then f o r a base r e l a t i o n of k columns the b e n e f i t v e c t o r would be o f s i z e nk. The f o r m u l a t i o n of the problem of index s e l e c t i o n given D. The System 109 by (15) and (16), however, seems to f a i l when a r b i t r a r y combinations of i n d i c e s are permitted i n <predicate> e v a l u a t i o n * The r e s u l t s a l s o f a i l to extend n i c e l y when i n v e r s i o n s based on a c o n c a t e n a t i o n of more than one a t t r i b u t e are permitted t o e x i s t . In such s i t u a t i o n s t he:values c . x i n (15) and b*. x i n (16) no l o n g e r make sense. . We note i n c o n c l u d i n g t h i s s e c t i o n t h a t a s p e c i f i c m u l t i - a t t r i b u t e i n v e r s i o n may indeed be implied by s t a t i s t i c s a l r e a d y maintained by SORAAM. We c o n j e c t u r e .the i m p l i c a t i o n o f such an i n v e r s i o n i n s i t u a t i o n s where the "qno" value of a s p e c i f i c column i g r e a t l y exceeds i t s b value* T h i s s i g n a l s t h a t q u e r i e s s u p p l y i n g values f o r column i almost always do so i n combination with a t l e a s t one o t h e r a t t r i b u t e j whose index i s p r e f e r e d by t h e search s t r a t e g i e s over an index on i i n e v a l u a t i n g such q u e r i e s * E. Search S t r a t e g i e s T h i s f i n a l s e c t i o n of Chapter I I I concerns the : d e c i s i o n process f o r the s e l e c t i o n of access paths t o be used i n the s e a r c h i n g components of o p e r a t i o n s on SORAAM.. Fundamental to t h i s s e l e c t i o n process i s a c o s t estimate of the number o f block reads t h a t would be i n c u r r e d f o r each index a v a i l a b l e . The f i r s t p a r t of the s e c t i o n presents the method used by SORAAM to determine such c o s t s and the i m p l i e d search s t r a t e g y . The s e c t i o n ends with a d i s c u s s i o n of r e l a t e d work E. The System 110 concerning the BDMSs System E and INGEES. One of the s i m p l i f y i n g assumptions adopted by SOEAAM i s t h a t the use of one index o n l y i s permitted i n the e v a l u a t i o n of any <predicate> i n s t a n c e . * Such an assumption g r e a t l y s i m p l i f i e s the s e l e c t i o n of access path and, as we have seen i n the pre v i o u s s e c t i o n , the problem of automating index s e l e c t i o n . Given a query, t h e r e f o r e , (defined as a <predicate> inst a n c e ) the problem i s to determine which of the c u r r e n t l y e x i s t i n g secondary i n d i c e s or the k-d t r e e primary index t o use f o r the search* l e t t i n g k be the number of columns i n the base r e l a t i o n and I subset {0,1;...,k} rep r e s e n t the i n d e x i n g s e t (where i>0 i n I i m p l i e s the a t t r i b u t e with C0L#=i has a secondary index and where i=0 denotes the primary index) then the problem i s to determine the j i n I y i e l d i n g the l e a s t block reads i n the search i m p l i e d by the query. We now o u t l i n e the procedure used by SOEAAM t o handle t h i s problem* E e c a l l i n q t h a t a l l q u e r i e s are i n d i s j u n c t i v e normal form we c o n s i d e r t h i s procedure i n two p a r t s . The f i r s t , P1, concerns each <conjunct> (a p a r t i a l match or p a r t i a l range query; see Table II-1 f o r i l l u s t r a t i o n ) and the second, P2, the query as a whole. For each <conjunct> P1 r e t u r n s to P2 a l i s t of the form: ^Throughout t h i s s e c t i o n we w i l l be assuming f a m i l i a r i t y with the design of the user i n t e r f a c e o u t l i n e d i n S e c t i o n A o f t h i s chapter*. E. The System 111 L = {cost0, c o s t 1, . , . ,costk] (17) denoting estimated c o s t i n block reads to e v a l u a t e the query u s i n g the primary index (costO), a secondary index on the f i r s t column ( c o s t l ) , on the second ( c o s t 2 ) , e t c . . I f i i s not i n I then c o s t i i s assigned a l a r g e number which we denote as LARGENUHBEE. otherwise f o r each i i n I P i does the f o l l o w i n g : 1. c o s t <- LABGENUMBER i 2. f o r each <primitive> i n the <conjunct> do 2. 1 i f COL# -.= i {where COL# i s im p l i e d by the "Aname" f i e l d of <primitive>] then repeat from 2 f o r next <primitive>. 2.2 determine the s e l e c t i v i t y f a c t o r SF 2* 3 temp <- SF . (*) 2. 4 i f temp < c o s t then c o s t <- temp i i 2.5 repeat from 2 f o r next <primitive> 3. r e t u r n . An i n t r o d u c t i o n to and d e f i n i t i o n of SF (a s e l e c t i v i t y f a c t o r ) c l o s e l y q u a l i f y i n g f o r our own purposes i s g i v e n i n [ S e l i n g e r e t a l . 79].. B a s i c a l l y , SF i s an attempt a t e s t i m a t i n g the f r a c t i o n of t u p l e s or b l o c k s t h a t would be accessed should an index i be used with an a p p r o p r i a t e < p r i m i t i v e > . T h i s estimate, a f u n c t i o n of the:<primitive> and of i n f o r m a t i o n r e g a r d i n g the indexed a t t r i b u t e and maintained by SORAAM, i s c a l c u l a t e d as f o l l o w s : (a) For a non-root node type 13 <primitive>: E. The System 112 high - low SF <- 1 — max - min i i where max and min are boundary val u e s of the i i a t t i b u t e with COL#=i and are l o c a t e d i n ATTRIBUTES. (b) For a non-root node type 12 <primitive> i f the comparison operator i s one of {<,<=} then: value - min SF <- — i max - min i i or i f i t i s one of {>,>=} then: max - value SF <- i max - min i i e l s e i f the comparison o p e r a t o r i s "-=" or "=" then: 1 SF <- 1 or SF <- - n i r e s p e c t i v e l y . N o t i c e from the above t h a t we are assuming the v a l i d i t y o f the minus operator f o r range <primitive>s. There a r e , of course, some a t t r i b u t e s f o r which only the "=" and "-=" comparison o p e r a t o r s would be v a l i d . Notice a l s o t h a t the esti m a t e s assume a uniform d i s t r i b u t i o n of both index v a l u e s and t u p l e s on v a l u e s . Step 2.3 s e t s the estimate "temp" of bl o c k reads i n using index i with <primitive> as the product of SF and (*) . (*) , t h e r e f o r e , i s an estimate of the c o s t i n block reads o f a scan E. The System 113 o f the e n t i r e base r e l a t i o n v i a index i . In SORAAM t h i s e s t i mate i s a f u n c t i o n of: (a) the number o f b u f f e r s a v a i l a b l e t o hold index and t u p l e blocks read from mass storage ( t h i s i s given by the value of the a t t r i b u t e ; "buf 1 1 i n ATTRIBUTES), (b) the nature of the h o r i z o n t a l c l u s t e r i n g on the base r e l a t i o n and (c) the a v a i l a b i l i t y of v a r i o u s scanning a l g o r i t h m s on the STPBs of the secondary index. There are f o u r STPB scanning a l g o r i t h m s c o n s i d e r e d by SORAAM. A l l vary i n t h e i r i m p l i e d CPU overhead and minimum number o f r e q u i r e d b u f f e r s * These a l g o r i t h m s can be d e s c r i b e d as f o l l o w s : SCAN 1. A s e q u e n t i a l scan of the STPBs. SCAN2. An a q u i s i t i o n and s o r t of a l l r e l e v a n t t i d s . SCAN 3. An a q u i s i t i o n and s o r t of n STPB b l o c k s of t i d s at a time. SCAN4. Merge of m t i d l i s t s . . Assuming we have j b u f f e r s a v a i l a b l e then devoting a l l but two of them t o b u f f e r i n g l e a f t u p l e blocks suggests the f i r s t scan a l g o r i t h m SCAN 1 above. T h i s assumes t h a t the number of l e v e l s i s the primary index d i s c r i m i n a t i n g on the r e l e v a n t index (di) i s l a r g e enough. That i s , d i must p a r t i t i o n the s e t of l e a f t u p l e b l o c k s i n t o u n i t s s m a l l enough such t h a t each may be e n t i r e l y c o ntained w i t h i n the a v a i l a b l e main memory b u f f e r s . Should t h i s not be the case then we may s t i l l c o n s i d e r i n t e l l i g e n t use of the a v a i l a b l e b u f f e r s f o r b u f f e r i n g index and STPB b l o c k s themselves. Since t i d s are almost always much E. The System 114 s m a l l e r i n s i z e than t u p l e s a much s m a l l e r number of b u f f e r s may only be r e q u i r e d . SCAN2, t h e r e f o r e , i s a simple f i r s t s t e p i n t h i s d i r e c t i o n ; In t h i s case we r e q u i r e enough b u f f e r s t o r a g e t o c o n t a i n a l l STPB b l o c k s c o n t a i n i n g a l l r e l e v a n t t i d s ( i . e ; t i d s p o i n t i n g t o t u p l e s s a t i s f y i n g the query <predicate>). , SCAN 1 took advantage of the p a r t i t i o n i n g of the s e t o f l e a f t u p l e : b l o c k s by d i * . Consider now t h a t t h i s same p a r t i t i o n i n g must n e c e s s a r i l y apply a l s o to the s e t of index and STPB blo c k s . Each subset of index or STPB b l o c k s r e s u l t i n g i n t h i s p a r t i t i o n i n g i s a s s o c i a t e d with i t s own d i s j o i n t subset o f l e a f t u p l e b l o c k s . Both SCAN3 and SCAN4 may t a k e advantage of t h i s phenominon. We may now concern o u r s e l v e s with a c q u i r i n g and s o r t i n g the t i d s f o r each element of the STPB p a r t i t i o n i n the case of SCAN3 or merging the t i d l i s t s f o r each element of the index block p a r t i t i o n i n the case o f SCAN4. Because o f the l a c k of knowledge of the exact l o c a t i o n of these p a r t i t i o n s i n the case of secondary i n d i c e s we can expect t h a t boundary o v e r l a p w i l l roughly double the scan c o s t s of these approaches over t h a t i m p l i e d by the use o f SCAN1. * F i n a l l y we note that should the range of index values be s u f f i c i e n t l y s m a l l then SCAN4 may a l s o apply; In t h i s case an 4 I f we had adopted the overhead of marking these index and STPB p a r t i t i o n s i n the design of our secondary i n d i c e s ( i . e . marked index values o c c u r r i n g as v a l u e s i n the primary index) then t h i s d o u b l i n g of scan c o s t over b e s t p o s s i b l e would not o c c u r . E. The System 115 m-way merge o f the t i d l i s t s i s performed where m i s the number of index v a l u e s o c c u r r i n g i n the range. T a b l e I I I - 2 below summarizes t h i s v a r i e t y of scan a l g o r i t h m i n terms of the minimum number of b u f f e r s r e g u i r e d and the r e s u l t i n g c o s t e s t i m a t e (*) i n block reads f o r a scan of the t u p l e s i n a base r e l a t i o n . . T h e use of t h i s t a b l e by SORAAM i n c a l c u l a t i n g (*) proceeds as f o l l o w s : (a) For each approach f o r which enough b u f f e r s are a v a i l a b l e the estimated scan c o s t i s computed. (b) From these the approach i n v o l v i n g the l e a s t disk I/O and then the l e a s t CPO overhead i s s e l e c t e d and (*) i s assigned i t s c a l c u l a t e d scan c o s t . The approach s e l e c t i o n i m p l i e d by step (b) above i s c o n s i s t e n t with the design emphasis i n SORAAM on r e t r i e v a l response. With r e s p e c t t o CPO overhead, i f we assume s e q u e n t i a l scans are cheaper t h a t merges which i n t u r n are cheaper than s o r t s , then the o r d e r i n g of approach s e l e c t i o n i s i m p l i e d by the top t o bottom o r d e r i n g of Table I I I - 2 . We have now seen how c o s t i (i>0) i s c a l c u l a t e d by P1 (the f i r s t p a r t of our search s t r a t e g i e s procedure) » I n e s t i m a t i n g a c o s t f o r the primary index (costO) SORAAM a b b r e v i a t e s the r e p r e s e n t a t i o n o f a <conjunct> as a p a r t i a l match query Q subset {1#..,k} where i i n Q i m p l i e s t h a t an "Aname" i n a <primitive> i n the <conjunct> e x i s t s such t h a t the COL# a s s o c i a t e d with Aname i s i . Using t h i s r e p r e s e n t a t i o n f o r the <conjunct> costO i s c a l c u l a t e d by P1 as f o l l o w s : E. The System 116 Scan Cost of a Base R e l a t i o n V i a a Secondary Index min. necessary Scan Cost i n Approach # of " b u f , , f e r s Algorithm Block Reads IX + 1 d i 2 + 2 SCAN 1 Best* (B) r i ISFin | 1 i l + 2 SCAN4 r T (m=|SF.n |) I i l Best* (C) r i (SF.STPBI + 2 SCAN2 Best* (D) r T 1 n | 1 1 d i-1| |2 | + 2 SCAN4 r l i n - | (m=l__i_ 1) 1 d i-1| .12 I 2.Best * (E) r i ISTPB | 1 1 I a i - 1 1 |2 | + 2 SCAN3 r T j STPB | (n=| 1) 1 d i-1| |2 | 2* Best* (F) 3 SCAN 1 D e f a u l t 2 i " B e s t " i s d e f i n e d as (SIB + STPB + IX + 1); i . e * no index, i STPB or t u p l e block accessed more than once;, r i |N D-di| 2 , , D e f a u l t " i s d e f i n e d as SIB + STPB + n . .min|-,2 |. The i i |n | L i 1 worst case here occurs when each t u p l e r e f e r e n c e by i t s t i d r e q u i r e s a block read. The t h i r d term takes i n t o account the e f f e c t of s o r t e d t u p l e l i s t s . T h i s e f f e c t i s d i f e l t as 2 approaches n . i Table I I I - 2 E. The System 117 1 costO <- _ d i {PIB + IX + 1} (18) i i n Q 2 T h i s a b b r e v i a t i o n of <conjunct> format i n e s t i m a t i n g costO i s j u s t i f i e d when one c o n s i d e r s t h a t range <primitive>s can o n l y improve the performance of the primary index.. That i s , the r a t i o o f r e l e v a n t l e a f t u p l e block reads t o t o t a l block reads (a t r u e measure of index performance) can only improve* Note t h a t costO need only be evaluated once f o r any p r e d i c a t e i n s t a n c e . Now c o n s i d e r t h a t the second p a r t of our index s e l e c t i o n procedure, P2, has f o r each <conjunct> of the query <predicate> a l i s t L given by (17) denoting estimated c o s t s t o e v a l u a t e the <conjunct> f o r each p o s s i b l e index* The f i n a l index s e l e c t i o n by P2 i s determined as f o l l o w s : (a) Any index i f o r which c o s t i f o r any <conjunct>=LARGENUMBER i s d i s q u a l i f i e d from c o n s i d e r a t i o n . Note t h a t , by t h i s c r i t e r i o n , the primary index i s never d i s q u a l i f i e d * (b) Of those i n d i c e s remaining the index i chosen has the minimum of the v a l u e s : c o s t or / cost f o r i>0. 0 — J i <conjunct> T h i s completes the o u t l i n e of c o s t e s t i m a t i o n and index s e l e c t i o n used by SORAAM. N o t i c e t h a t the query e v a l u a t i o n f a c i l i t a t e d has (admittedly not g r a c e f u l l y ) f i n e s s e d any c o n s i d e r a t i o n or n e c e s s i t y of performing i n t e r s e c t i o n s and E..The System 118 unions of t i d l i s t s * T h i s immediately f o l l o w s from the r e s t r i c t i o n s t a t e d at the beginning of t h i s s e c t i o n o f p e r m i t t i n g one index o n l y i n the e v a l u a t i o n of any <predicate> i n s t a n c e . F o l l o w i n g the det e r m i n a t i o n of the a c t u a l index t o use f o r the given p r e d i c a t e i n s t a n c e SORAAM reuses the p r e c e e d i n g procedures to determine the most d e s i r a b l e index (whether or not i t c u r r e n t l y e x i s t s ) f o r t h i s purpose. Should a secondary index be chosen i n t h i s case then the a s s o c i a t e d b i value i n ATTRIBUTES i s incremented. Furthermore, the "Pcost" value a s s o c i a t e d with the i m p l i e d base r e l a t i o n i s incremented by the e s t i m a t e d value of costO; by the value given i n (18) above. In t h i s way the search s t r a t e g i e s maintains the necessary i n f o r m a t i o n f o r use by the secondary index s e l e c t i o n proceedures o u t l i n e d i n the preceeding s e c t i o n . To i l l u s t r a t e the e f f e c t of SORAAM*s h o r i z o n t a l c l u s t e r i n g t o g e t h e r with i t s v a r i o u s base r e l a t i o n scan a l g o r i t h m s c o n s i d e r the case o f a p a r t i c u l a r base r e l a t i o n schema R (a1,a2,...,ak) where R and a i r e p r e s e n t the r e l a t i o n and a t t r i b u t e names r e s p e c t i v e l y . Assume t h a t R c o n t a i n s 120,000 t u p l e s i n 4,000 l e a f t u p l e b l o c k s at 30 t u p l e s per block. Assume a l s o t h a t secondary i n d i c e s e x i s t on the f i r s t two a t t r i b u t e s only and t h a t : (a) D=12; d1=6; d2=6 - i m p l y i n g t h a t access t o R i s e x c l u s i v e l y v i a a1 and a2 and t h a t <predicate>s supply values f o r a1 as o f t e n as f o r a2.. (b) Psize=1000; ff2=2/3 - imp l y i n g t h a t the number of STPBs per index i s 180. E..The System 119 Approach (E) using SCAN3 i n Table I I I - 2 now t e l l s us t h a t with j u s t 8 b u f f e r s a scan o f B v i a the secondary index on a1 or a2 y i e l d s an expected 8,000 l e a f t u p l e block reads! A mean c o s t i n l e a f t u p l e block reads f o r the complete scan o f E v i a any secondary index i s t h e r e f o r e 8,000 (again assuming j u s t 8 b u f f e r s a v a i l a b l e ) . T h i s compares very f a v o u r a b l y with scan c o s t s f o r secondary i n d i c e s o f a l l BDMSs reviewed i n t h i s t h e s i s t h a t support them* Con s i d e r t h a t [ S e l i n g e r et a l ; . 7 9 ] p r e d i c t a comparable c o s t f o r a base r e l a t i o n scan v i a u n c l u s t e r e d images i n System E as the c a r d i n a l i t y of the base r e l a t i o n . A mean c o s t o f 120,000 t u p l e block reads t h e r e f o r e a r i s e s even with the a v a i l a b i l i t y of hundreds of b u f f e r s . In t h i s case SOEAAM e x h i b i t s an order of magnitude decrease i n mass storage I/O. The search s t r a t e g i e s component of SOEAAM accomplishes only a simple p a r t of the more gen e r a l task of c o m p i l i n g best p o s s i b l e d e t e r m i n i s t i c e x e c u t i o n s t r a t e g i e s from non- p r o c e d u r a l DHLs. That i s , the kind of se a r c h i n g i m p l i e d by a non-procedural DML of an EDMS i s much more complex than t h a t supported by SOEAAM. Aggregation o p e r a t o r s such as MAX or AVG and data r e l a t a b i l i t y o p e r a t o r s such the the JOIN operator o f the r e l a t i o n a l a l g e b r a c o n t r i b u t e enormously t o the d i f f i c u l t y o f the problem* L i t e r a t u r e d e a l i n g with the s u b j e c t of t h i s s e c t i o n , t h e r e f o r e , u s u a l l y concerns the more g e n e r a l s e t t i n g . In System B the " o p t i m i z e r " , a component of the EDS (the r e l a t i o n a l language l e v e l of System E ) , has f u l l E. The System 120 r e s p o n s i b i l i t y f o r a l l s e a r c h i n g s t r a t e g y c o n s i d e r a t i o n s . T h i s i n c l u d e s s e a r c h i n g w i t h i n and between base r e l a t i o n s . . [ Blasgen and Eswaran 77] and more r e c e n t l y [ S e l i n g e r e t a l . 79] p r e s e n t the d e t a i l s o f System R's o p t i m i z e r . In SORAAM i t i s necessary to s p l i t the r e s p o n s i b i l i t y o f search s t r a t e g y s e l e c t i o n between the access path and r e l a t i o n a l language l e v e l s . SORAAM's commitment t o c o n t r o l l i n g a l l c l u s t e r i n g and index c r e a t i o n d e t a i l s i m p l i e s t h a t i t must a l s o be r e s p o n s i b l e f o r search s t r a t e g y w i t h i n base r e l a t i o n s . That i s , s i n c e SORAAM can only know of the e x i s t e n c e o f a secondary index at a given p o i n t i n time.then SORAAM must decide when such an index i s to be used. SORAAM, however, does not r e p r e s e n t a precedent i n t h i s regard. The DML of INGRES c a l l e d QUEL i s based on the r e l a t i o n a l c a l c u l u s . In QUEL the: module r e s p o n s i b l e f o r o p t i m i z i n g o n e - v a r i a b l e q u e r i e s i s d i s t i n c t from the module r e s p o n s i b l e f o r o p t i m i z i n g m u l t i - v a r i a b l e q u e r i e s . The s e a r c h i n g c a p a b i l i t y of the former ( c a l l e d OVQP1) i s analogous to t h a t provided by SORAAM. For an i n t e r e s t i n g o u t l i n e of the approach used by INGRES i n h a n d l i n g m u l t i - v a r i a b l e q u e r i e s see [Wong and Y o u s s e f i 76]. Query p r o c e s s i n g i n SORAAM i s d i s t i n c t from OVQP of INGRES and the o p t i m i z e r of the RDS of System R with r e s p e c t t o the more complex degree of c l u s t e r i n g supported. We have *See the review of INGRES i n S e c t i o n B of Chapter I I . . E. .The System 121 seen i n a preceeding example t h a t enormous r e d u c t i o n s i n I/O t r a f f i c can ensue from support f o r m u l t i d i m e n s i o n a l h o r i z o n t a l c l u s t e r i n g i n performing search w i t h i n base r e l a t i o n s . A. Future D i r e c t i o n s and C o n c l u s i o n s 122 Chapter IV: Future D i r e c t i o n s and C o n c l u s i o n s A» The Next Step In t h i s f i r s t s e c t i o n of the f i n a l chapter we d i s c u s s what may be c o n s i d e r e d the next l o g i c a l s t e p i n the study o f a s e l f - o r g a n i z i n g access path l e v e l : a demonstration of f e a s i b i l i t y of the concept of SOEAAM.. T h i s n e c e s s a r i l y r e g u i r e s f u r t h e r e m p i r i c a l evidence.. The nature o f t h i s evidence and the t e s t i n g environment necessary t o a c q u i r e i t are d i s c u s s e d * . A number of " e x p e r i m e n t a l l y determined c o n s t a n t s " (or t u n i n g parameters) were introduced throughout Chapter I I I . . A t e s t system i n which t o e m p i r i c a l l y d e t e r m i n e . s u i t a b l e v a l u e s f o r these c o n s t a n t s i s c l e a r l y necessary. To r e i t t e r a t e , the problems f o r which s o l u t i o n s i n v o l v e d the use of these c o n s t a n t s can be o u t l i n e d as f o l l o w s : (a) the d e t e c t i o n of a s u i t a b l e imbalance c o n d i t i o n i n the primary index o f a base r e l a t i o n . In S e c t i o n B of Chapter I I I imbalance was determined by the value of the c o n d i t i o n : D > c1 . l o g (IX+1) 2 Furthermore, we saw t h a t the r e a l valued constant c1 was c o n s t r a i n e d to be g r e a t e r than 1,386. (b) the i n t r o d u c t i o n of randomness i n the s e l e c t i o n o f l e a f t u p l e b l o c k s from which t u p l e s are d e l e t e d and r e - i n s e r t e d a c c o r d i n g to the b a l a n c i n g or r e - c l u s t e r i n g phase of a primary index. A l s o i n S e c t i o n B of Chapter I I I an i n t e g e r constant number c2 o f l e a f t u p l e blocks are s e l e c t e d a t a time. (c) The d e t e r m i n a t i o n of a s u f f i c i e n t change i n the A. Future D i r e c t i o n s and C o n c l u s i o n s 123 c h a r a c t e r i z a t i o n of use t o warrant r e - c l u s t e r i n g of a base r e l a t i o n * Such r e - c l u s t e r i n g (Section B; Chapter III) was determined by the value of the c o n d i t i o n : C (fpld,qnoold) > c3 C(f,qno) where C i s d e f i n e d as the value of the f u n c t i o n s F or G (see S e c t i o n C; Chapter III) and c3 i s a r e a l valued constant g r e a t e r than 1. (d) the i n t e r p r e t a t i o n of the c h a r a c t e r i z a t i o n of use s t a t i s t i c s f o r the purpose of d i s c r i m i n a t o r s e l e c t i o n i n the primary index. In S e c t i o n C of Chapter I I I the c h o i c e f o r the d i r e c t verses indepentent models was based on the f o l l o w i n g c o n d i t i o n : k ^ f o l d > c4 . qnoold i=1 i where cU i s again a r e a l valued constant g r e a t e r than 1. (e) the acceptance of a minimal number of unique e x i s t i n g a t t r i b u t e values f o r q u a l i f i c a t i o n of d i s c r i m i n a t i o n i n the primary index. In S e c t i o n C of Chapter I I I we permitted d i s c r i m i n a t i o n on an a t t r i b u t e i n a l e a f t u p l e block s p l i t o n l y i f the f o l l o w i n g c o n d i t i o n was s a t i s f i e d : k j + 1 2 < c5 . n where c5 i s a r e a l v alued constant i n the range (0,1). In a d d i t i o n t o determining s u i t a b l e v a l u e s f o r the c o n s t a n t s c1,.,,c5 above a number of other i s s u e s r e q u i r i n g the use of a t e s t system must be r e s o l v e d . These i s s u e s are o u t s t a n d i n g i n regard to an u l t i m a t e demonstration o f f e a s i b i l i t y of p r a c t i c a l s e l f - o r g a n i z i n g or s e l f - a d a p t i n g sytems. In order to make d e c i s i o n s about i t s p h y s i c a l A. Future D i r e c t i o n s and C o n c l u s i o n s o r g a n i z a t i o n SOEAAM r e q u i r e s the dynamic maintenance o f v a r i o u s i n f o r m a t i o n * T h i s i n f o r m a t i o n can be c a t e g o r i z e d as f o l l o w s : (a) i n f o r m a t i o n comprising a c h a r a c t e r i z a t i o n of use and (b) i n f o r m a t i o n r e g a r d i n g a n t i c i p a t e d index e f f e c t i v e n e s s and i n e f f e c t i v e n e s s . I t i s c e r t a i n l y not yet c l e a r t h a t dynamic u n c o n d i t i o n a l maintenance o f such i n f o r m a t i o n i s f e a s i b l e * .. With our hig h regard f o r t r a n s a c t i o n response such " p r o c e d u r a l " overhead must be e x p e r i m e n t a l l y demonstrated a c c e p t a b l e . . There are a number of asp e c t s of the kind of h o r i z o n t a l c l u s t e r i n g supported i n SOEAAM (of the c h o i c e of data s t r u c t u r e f o r the primary index) t h a t a r e , as y e t , u n c l e a r . The most obvious q u e s t i o n s one may ask concern comparisons o f our m u l t i - d i m e n s i o n a l c l u s t e r i n g with t r a d i t i o n a l 1- dimensional c l u s t e r i n g i n regard to r e t r i e v a l response..More s u c c i n c t l y , i s i t a l l worth the e x t r a t r o u b l e ; i s mass s t o r a g e I/O per query s i g n i f i c a n t l y reduced? 1 SOEAAM has a l s o i n c u r r e d a r e b u i l d i n g c o s t a r i s i n g from i t s s e l e c t i o n of a s t a t i c data s t r u c t u r e f o r the primary index. I t i s t h e r e f o r e d e s i r a b l e to a c q u i r e a more c o n c r e t e measure of t h i s overhead i n r e l a t i o n t o v a r i o u s t u p l e i n s e r t i o n sequences, c h a r a c t e r i z a t i o n s of use. and, indeed, values of c1 above. E e c a l l , f o r example, t h a t s o r t e d i n p u t may *The simple example given i n S e c t i o n E of Chapter I I I , however, must c e r t a i n l y c o n t r i b u t e to one's enthusiasm f o r k-d t r e e s . . A. Future D i r e c t i o n s and c o n c l u s i o n s 1 2 5 imply f r e q u e n t l y o c c u r r i n g imbalance i n the growth of a primary index. In SORAAM the c o n t r o l of secondary index s e l e c t i o n i s not e n t i r e l y t r a n s p a r e n t . In p a r t i c u l a r there were three user s u p p l i e d parameters "Plow", "Phigh" and "S" t h a t had g r e a t i n f l u e n c e i n t h i s r e g ard. Index c r e a t i o n i s c o n t i n g e n t not only on these parameters but on the nature of use of the base r e l a t i o n ; . C o n s i d e r , f o r example* t h a t the more.complex a g i v e n <predicate> i n s t a n c e becomes the more: probable the u n c o n d i t i o n a l s e l e c t i o n of the primary index f o r the query e v a l u a t i o n by the search s t r a t e g i e s ; 1 A much c l e a r e r p i c t u r e o f the nature of i n t e r a c t i o n of these user parameters and o f query complexity i s s t i l l necessary as w e l l as the experimental determination of s u i t a b l e d e f a u l t v a l u e s f o r the parameters themselves. Although harder t o e v a l u a t e , some e s t i m a t i o n of the o v e r a l l e f f e c t i v e n e s s of the search s t r a t e g i e s component o f SORAAM i s c l e a r l y c a l l e d f o r . In p a r t i c u l a r , the accuracy o f the cost e s t i m a t i o n r o u t i n e s i n p r e d i c t i n g "best p o s s i b l e " c h o i c e of access path i s , as yet, unknown; We have seen* t h e r e f o r e , a number of o u t s t a n d i n g i s s u e s r e q u i r i n g the design and implementation of a t e s t environment f o r t h e i r r e s o l u t i o n . Such a t e s t system would d i f f e r markedly S u r p r i s i n g l y , should a l l <predicate> i n s t a n c e s be complex enough f o r a base r e l a t i o n then, no matter how h e a v i l y used, no b e n e f i t may accrue from the c r e a t i o n of any secondary index. A. Future D i r e c t i o n s and C o n c l u s i o n s 1 2 6 from a " f u l l - s c a l e " p r o duction system i n a number of important ways: (a) Support f o r a c t u a l s t o r a g e on mass storage media i s not necessary. R e a l i s t i c turnaround f o r s i m u l a t i o n s i n the t e s t system imply complete main memory residences. . Large numbers of l e a f t u p l e b l o c k s would then imply s m a l l " t o y " b u f f e r s i z e s . (b) The t e s t system would i n v o l v e the use of an e n t i r e l y d i f f e r e n t user i n t e r f a c e . I t must be p o s s i b l e f o r the user to s p e c i f y whole s e t s of t r a n s a c t i o n s and t i m i n g of t r a n s a c t i o n a r r i v a l , determine the accumulation and r e p o r t i n g of much more e x t e n s i v e s t a t i s t i c a l i n f o r m a t i o n and have more e f f e c t i v e c o n t r o l over a l l aspects of the system (e.g. the values of c1 t o c5 above).. (c) A p r o d u c t i o n system would almost c e r t a i n l y imply e x t e n s i v e m o d i f i c a t i o n to the f i l e system of the a n t i c i p a t e d o p e r a t i n g system environment; the t e s t system would not. The t e s t system would be comprised of f o u r major p a r t s : (a) a t r a n s a c t i o n stream generator ( i n c l u d i n g maintenance t r a n s a c t i o n s ) * , (b) a t e s t data g e n e r a t o r 1 , (c) an access methods s i m u l a t o r and (d) a s t a t i s t i c a l a c q u i s i t i o n and r e p o r t generator. Fundamental t o the design of the t e s t system would be the nature o f the user i n t e r f a c e f o r the above. Note t h a t the simple design o u t l i n e of the components given here of a t e s t system would have l i t t l e correspondence to those of a p r o d u c t i o n system. Again we s t r e s s t h a t the m o t i v a t i o n s f o r a t e s t i n g environment d i s c u s s e d here are to f a c i l i t a t e iSee [ F a r l e y and Schuster 75] f o r a d i s c u s s i o n of these components f o r an analogous t e s t i n g environment* „ A, Future D i r e c t i o n s and C o n c l u s i o n s 127 a c q u i s i t i o n o f e m p i r i c a l evidence of f e a s i b i l i t y of access path l e v e l design ( i n p a r t i c u l a r t h a t o f SORAAM) . B. Co n c l u s i o n s In t h i s t h e s i s we have begun an i n v e s t i g a t i o n o f automating p h y s i c a l r e o r g a n i z a t i o n a l reguirements of an RDMS. A review of " s t a t e - o f - t h e - a r t " experimental systems a p p r o p r i a t e f o r o n - l i n e a p p l i c a t i o n s r e v e a l e d a l a c k of regard f o r t h i s problem. The f i r s t s e c t i o n of Chapter I I , t h e r e f o r e , presented arguments s t r o n g l y motivating a d d i t i o n a l r e s e a r c h * We observed the growing tendancy of o p e r a t i n g systems to assume more d i r e c t r e s p o n s i b i l i t y f o r support of data management needs. In p a r t i c u l a r , an o p e r a t i n g system environment i n which an RDMS has supplanted the r o l e o f a f i l e system f i r m l y i m p l i e d the need f o r such automation. A design s p e c i f i c a t i o n and o u t l i n e f o r the access path l e v e l of an RDMS a p p r o p r i a t e f o r such an environment (SORAAM) was g i v e n . I m p l i c i t i n t h i s design was the automation of two major p h y s i c a l o r g a n i z a t i o n d e t a i l s concerning a type of re c o r d or t u p l e c l u s t e r i n g and index s e l e c t i o n . A n a l y s i s o f these d e t a i l s has y i e l d e d an access path l e v e l with s i g n i f i c a n t advantages gained over other systems* We f e e l the r e s u l t s of t h i s a n a l y s i s have c o n t r i b u t e d s i g n i f i c a n t l y t o our o v e r a l l optimism f o r both the d e s i r a b i l i t y and f e a s i b i l i t y of such automation. B. Future D i r e c t i o n s and C o n c l u s i o n s 128 a complete demonstration of f e a s i b i l i t y f o r our proposed access path l e v e l and f o r i t s design p r i n c i p l e s r e s t s h e a v i l y , however, on f u t u r e experimental r e s e a r c h * We: have o u t l i n e d what we b e l i e v e t o be the next such step i n the p r e c e e d i n g s e c t i o n * . W i t h regard t o the l e g i t i m a c y of the m o t i v a t i n g n o t i o n of an "RDMS i n s t e a d of a f i l e system" i n a g e n e r a l purpose computing environment perhaps the only a c c e p t a b l e evidence r e s t s with the user feedback from a f u l l - s c a l e p r o d u c t i o n system* In f o r m u l a t i n g the problems of c l u s t e r i n g and index s e l e c t i o n c a r e f u l a t t e n t i o n to the meaning of e x i s t i n g t e rminology was necessary. In p a r t i c u l a r a strong advantage was r e a l i z e d i n d i s t i n g u i s h i n g between a primary index, i n which we d e a l t with h o r i z o n t a l c l u s t e r i n g , and secondary i n d i c e s , i n which we improved i n d e x i n g performance (reduced, t h a t i s , the l e n g t h of access path)• We saw, however, t h a t these two notions are f a r from independent. A primary index, by v i r t u e of i t s e f f e c t on t u p l e c l u s t e r i n g , c l e a r l y i n f l u e n c e d the s e l e c t i o n of a secondary index f o r the purpose of guery e v a l u a t i o n which i n t u r n i n f l u e n c e d the s e l e c t i o n of the secondary i n d e x i n g set i t s e l f , a more s u b t l e r e l a t i o n s h i p (not p r e v i o u s l y discussed) e x i s t s i n the o p p o s i t e d i r e c t i o n . The c h o i c e of secondary i n d e x i n g s e t can a f f e c t the a c t i o n of the primary index towards t u p l e c l u s t e r i n g . Should, f o r example, an a t t r i b u t e a of a base r e l a t i o n R s a t i s f y the f o l l o w i n g : (a) a secondary index always e x i s t s on a. B. Future D i r e c t i o n s and C o n c l u s i o n s 129 (b) no q u e r i e s s p e c i f y value ranges f o r A, (c) the query sequence i m p l i e s a random d i s t r i b u t i o n of A-values and (d) the number of unique values i n B f o r A i s c l o s e t o the number o f t u p l e s i n B then l i t t l e j u s t i f i c a t i o n e x i s t s f o r ever d i s c r i m i n a t i n g on A i n the primary index. The c h a r a c t e r i z a t i o n of A given above i s an example of the s o r t of a t t r i b u t e f o r which t u p l e b l o c k i n g can have no b e n e f i c i a l e f f e c t . Accounting f o r such cases i n SORAAM, however, i s an open problem. The a r c h i t e c t u r e of the s p e c i f i c v a r i e t y of secondary i n d e x i n g c o n s i d e r e d by SOBAAM and the problem of automating t h e i r s e l e c t i o n l e d , i n an almost obvious manner, to a f o r m u l a t i o n of t h i s problem i n terms of the w e l l known (and w e l l studied) "knapsack problem". . The l a t t e r i s a l s o w e l l known t o be a c o m p u t a t i o n a l l y d i f f i c u l t problem. 1 In l i g h t o f the very b a s i c secondary i n d e x i n g considered such a r e s u l t lends credence to the use of a p p r o p r i a t e h e u r i s t i c approaches to the problem of secondary index s e l e c t i o n i n g e n e r a l . R e c a l l t h a t such a h e u r i s t i c approach was h i n t e d at the end of S e c t i o n D of Chapter I I I i n the e x t e n s i o n o f SOBAAM t o support f o r m u l t i - a t t r i b u t e i n v e r s i o n s as secondary i n d i c e s . Such an e x t e n s i o n i s cne of many p o s s i b l e avenues f o r f u t u r e r e s e a r c h * .We conclude with some sugg e s t i o n s i n t h i s r e g a r d : (a) the e x t e n s i o n of SOBAAM t o support f o r other * I t i s a member of the c l a s s o f such d i f f i c u l t problems denoted as "NP-complete". B. Future D i r e c t i o n s and C o n c l u s i o n s 130 forms of secondary i n d e x i n g and index s e l e c t i o n methodology* (b) c o n s i d e r a t i o n f o r the problem of i n f o r m a t i o n a r c h i v i n g . (c) the ext e n s i o n of SOEAAM to support f o r l a r g e numbers of s m a l l s i z e d base r e l a t i o n s * V e r t i c a l c l u s t e r i n g of base r e l a t i o n s becomes very s i g n i f i c a n t i f base r e l a t i o n s i z e i s small compared t o the s i z e of a d i s k b l o c k . We have c l e a r l y concentrated i n t h i s t h e s i s on support f o r base r e l a t i o n s where the opposite i s t r u e . (d) the design of the r e l a t i o n a l language l e v e l (SOEAAM's u s e r ) . (e) c o n s i d e r a t i o n of the remaining p h y s i c a l r e o r g a n i z a t i o n a l requirements (see the I n t r o d u c t i o n ) . (f) the implementation of a complete o p e r a t i n g system environment i n c l u d i n g the " i n t e g r a t e d " EDMS t h a t i s completely s e l f - o r g a n i z i n g with regard t o the p h y s i c a l o r g a n i z a t i o n o f i t s e l f . References 131 References [Aho ana Oilman 79J Aho, A l f r e d V. .ana Ullman, J e f f r e y D. , "Optimal P a r t i a l - Match R e t r i e v a l When F i e l d s Are Independently S p e c i f i e d " , ACM T r a n s a c t i o n s on Database Systems, Volume 4 Number 2, pp. 168- 179, June(1979). [Anderson and Berra 77] Anderson, Henry D. and B e r r a , Bruce P., "Minimum Cost S e l e c t i o n of Secondary Indexes f o r Formatted F i l e s " , ACM T r a n s a c t i o n s on Database Systems, Volume 2 Number 1, pp* 68-90, March (1977) . . [Astrahan e t a l . 76] Astrahan, M. M., Blasgen, D. D., Chamberlin, D.D., Eswaran, K. P., Gray, J . N., G r i f f i t h s , Pi,. Pi,, King, W. F. , L o r i e , R. A., McJones, P. R., Mehl, J..W., P u t z o l u , G. R., T r a i g e r , I.. L., Wade, B. W. and Watson, V. , "System R: R e l a t i o n a l Approach to Database Management", ACM T r a n s a c t i o n s on Database Systems, Volume 1 Number 2, pp*.97-137, June(1976). [Bayer and McCreight 72] Bayer, R. . and McCreight, E., " O r g a n i z a t i o n and Maintenance o f Large Ordered I n d i c e s " , Acta I n f o r m a t i c a , Volume 1, S p r i n g e r - V e r l a g p u b l i s h e r , pp. 173-189, (1972). [Bayer and Schkolnick 76] Bayer, R. and S c h k o l n i c k , M., "Concurrency of Operations On B - t r e e s " , Research Report RJ 1791, IBM Research L a b o r a t o r y , San Jose, May(1976). [Bayer and Onterauer 76] Bayer, R. and Onterauer, K., " P r e f i x B - t r e e s " , Research Report RJ 1796, IBM Research Laboratory, San Jose, June (1976) . [ B e n t l e y 75] Bentley, Jon L o u i s , " M u l t i d i m e n t i o n a l Binary Search Trees Osed For A s s o c i a t i v e Searching", CACM Volume 18 Number 9, pp. 509-517, September (1975) . [ B e n t l e y 78a] References 132 , "A Survey of Algorithms and Data S t r u c t u r e s f o r Range Searching", T e c h n i c a l Report CMU-CS-78-136, Carnegie-Mellon U n i v e r s i t y , August (1978) . [ B e n t l e y 78b] , " M u l t i d i m e n s i o n a l Binary Search Trees In Database A p p l i c a t i o n s " , T e c h n i c a l Report CMU-CS-78-139, Ca r n e g i e - Mellon U n i v e r s i t y , September(1978). [ B e n t l e y 78c] — - — , "Decomposable Searching Problems", T e c h n i c a l Report CMU-CS-78-145, Carnegie-Mellon U n i v e r s i t y , August (1 978) . [Berryman and Fowler 79] Berryman, J . and Fowler, A;, " B u i l d i n g a Database Management System Into MTS", I n t e r n a l Memo* The Computing Center* U n i v e r s i t y of B r i t i s h Columbia, May(1979).. [ B e r s t e i n 79] B e r s t e i n , P h i l i p A* ( e d i t o r ) , "Proceedings of the ACM- SIGMOD I n t e r n a t i o n a l Conference on Management o f Data", May (1979). [Blasgen and Eswaran 77] Blasgen, M. W. and Eswaran, K. P., "Storage and Access i n E e l a t i o n a l Data Bases", IBM Systems J o u r n a l , Volume 16 Number 4, pp. 363^377, November (1977) . [ B r o d i e et a l . 75] Br o d i e , M. L. ( e d i t o r ) , Chan, S., Czarnik* B., Leong, E. Sch u s t e r , S..and T s i c h r i t z i s , D., "ZETA: A Prototype R e l a t i o n a l Data Base Management System", T e c h n i c a l Report CSRG-51, Computer S c i e n c e Research Group* U n i v e r s i t y o f Toronto, February(1975)* [ C a r g i l l 79] C a r g i l l , T. A., "A View of Source Text f o r D i v e r s e l y C o n f i g u r a b l e Software", proposed Ph.D. t h e s i s . U n i v e r s i t y of Waterloo, (1979). [Chamberlin e t a l . 76] Chamberlin, D. D., Astrahan, M. M., Eswaran, K. P., G r i f f i t h s * P. P., l o r i e , R. A., Mehl, J. ,W. , Reisner, P. and Wade, B.. W., "SEQUEL 2: A U n i f i e d Approach to Data References 133 D e f i n i t i o n , M a n i p u l a t i o n , and C o n t r o l " , IBM J o u r n a l o f Research and Development, November(1976) ., [Chamberlin 76] Chamberlin, D. D., " R e l a t i o n a l Data Base: Management Systems", Research Report RJ 1729* IBM Research Laboratory, San Jose, February(1976). [Codd 70 ] Codd, E. F., "A R e l a t i o n a l Model of Data For Large Shared Data Banks", CACM Volume 13 Number 6* pp..377-387, June (1970). [Codd 79 ] , "Extending the Database R e l a t i o n a l Model To Capture More Meaning", Supplement to Proceedings of ACM- SIGMOD I n t e r n a t i o n a l Conference on Management of Data, pp. . 29-52, see [ B e r s t e i n 79]. [CODASYL 71 ] CODASYL* "Data Base Task Group Report", CODASYL DBTG, ACM, New York, A p r i l ( 1 9 7 1 ) . [CODASYL 73] , "Data D e f i n i t i o n Language Committee J o u r n a l o f Development", CODASYL DDLC, IFIP A d m i n i s t r a t i v e Data P r o c e s s i n g Group, Amsterdam, June(1973). [Comer 78 ] Comer, Douglas, "The D i f f i c u l t y of Optimum Index S e l e c t i o n " , ACM T r a n s a c t i o n s on Database Systems, Volume 3 Number 4, pp. 440-445, December(1978). [Ccmer 79 ] , "The U b i q u i t o u s B-Tree", ACM Computing Surveys, Volume 11 Number 2, pp. 121-137, June (1979) . [Datapro 79 ] Datapro Research Corp*, "TOTAL, Cincom Systems, I n c . " , databook 70 - the EDP buyer's b i b l e . Volume: 3, software s e c t i o n , (1 978) . [Date 77 ] Date, C. J . , "An I n t r o d u c t i o n to Database Systems", References 134 Second E d i t i o n , Addison-Wesley P u b l i s h i n g Company, ISBN 0-201-14456-5, (1977) . [ F a g i n 78] F a g i n , Ronald, "Extendable Hashing - A Fast Access Method f o r Dynamic F i l e s " , Research Report RJ 2305, IBM Research l a b o r a t o r y , San Jose, J u l y (1978).. [ F a r l e y and Schuster 75] F a r l e y , G i l l e s J . H. and Schuster, Stewart A., "Query Execution and Index S e l e c t i o n f o r R e l a t i o n a l Data Bases", T e c h n i c a l Report CSRG-53, Computer Systems Research Group, U n i v e r s i t y of Toronto, March(1975) . . [Gray 78] Gray, J. , "Notes On Data Base Operating Systems", Research Report RJ 2188, IBM Research Laboratory, San Jose, February(1978) . [ H a l l et a l . , 76 ] H a l l , P., Owlett, J . and Todd, S., " R e l a t i o n s and E n t i t i e s " , Proceedings I F I P TC-2 Working Conference on M o d e l l i n g i n Database Management Systems, pp. 201-220, s e e [ N i j s s e n 76b]. [Henry 78 J Henry, G. . G., " I n t r o d u c t i o n to IBM System/38 A r c h i t e c t u r e " , IBM S/38 TECH. DEV., pp. . 3-6, see [IBM 78], [ H e w i t t 72] Hewitt* C a r l , " D e s c r i p t i o n s and T h e o r e t i c a l A n a l y s i s (using Schemata) of PLANNER: A Language f o r Proving Theorems and Manipulating Models i n a Robot", Ph.D. T h e s i s , AI-TR-258, The A r t i f i c i a l I n t e l l i g e n c e L a b o r a t o r y , Massachusetts I n s t i t u t e of Technology, Cambridge, Massachusetts, (1972). [IBM 78] IBM, "IBM System/38 T e c h n i c a l Developments", IBM Product Design and Development, General Systems D i v i s i o n , ISBN 0- 9333186-00-2, (1S78).. [ K l i m i i e and Koffeman 74] Klimbie, J . W. and Koffeman, K. L. ( e d i t o r s ) , "Data Base References 135 Management", North-Holland P u b l i s h i n g Company, North- Holland ISBN: 0 7204 2809 2, (1974). [Knuth 73] Knuth, Donald E., "The Art of Computer Programming S o r t i n g and Searching", Volume 3, Addison-Wesley P u b l i s h i n g Company, ISBN 0-201-03803-X, (1973).. [ K o l l i a s et a l . . 77 ] K o l l i a s , J . G. , Stocker, P. M., and Dearnley, P. , A., "Improving the Performance of an I n t e l l i g e n t Data Management System", The Computer J o u r n a l , The B r i t i s h Computer S o c i e t y , Volume 20 Number 4, ISSN 0010-4620, November (1977). [Lomet 79] Lomet, David B., " M u l t i - T a b l e Search f o r B-tree F i l e s " , Proceedings of ACM-SIGMOD I n t e r n a t i o n a l Conference on Management of Data, pp. 35-42, see [ B e r s t e i n 79].. [ L o r i e 74] L o r i e , R. A., "XRM - An Extended (n-ary) R e l a t i o n a l Memory", T e c h n i c a l Report No. 320-2096, IBM Cambridge S c i e n t i f i c Center, January (1974) . [McDonell 77] McDonell, Ken J , , "A U n i f i e d Approach To Secondary Storage Input-Output Operations", Ph. D* .Thesis, T e c h n i c a l Report TR77-6, U n i v e r s i t y of A l b e r t a , September(1977). [McGee 76 ] McGee, Wil l i a m C , "On User C r i t e r i a f o r Data Model E v a l u a t i o n " , ACM T r a n s a c t i o n s on Database Systems, Volume 1 Number 4, pp. 370-387, December (1976) . [ N i j s s e n 76a] N i j s s e n , G.. M., "A Gross A r c h i t e c t u r e , f o r the Next Generation Database Management Systems", Proceedings IFIP TC-2 Working Conference on Modelling i n Database Management Systems, pp* 1-24, see [ N i j s s e n 76b]. [ N i j s s e n 76b] N i j s s e n , G.. M. ( e d i t o r ) , " M o d e l l i n g i n Database Management Systems", North-Holland P u b l i s h i n g Company, ISBN 0 7204 0459 2, (1976). References 136 [Prywes and Gray 63] Prywes, N. S. and Gray, H..J. , "The O r g a n i z a t i o n of a M u l t i l i s t - T y p e A s s o c i a t i v e Memory", IEEE T r a n s a c t i o n s on Communication and E l e c t r o n i c s , pp* 488-492, September (1963) . [Rodriguez and Echhouse 77] Rodriguez, R o s e l l J . and Echhouse, R., "Management o f Data by Future Operating Systems", New D i r e c t i o n s f o r Operating Systems* A Workshop Report, Browne, J . . C. ( e d i t o r ) , ACM SIGOPS Operating Systems Review, Volume 11 Number 1, pp*23-25, (1977). [ S a h n i 75] Sahni, S a r t a j , "Approximate Algorithms f o r the 0/1 Knapsack Problem", J o u r n a l of the ACM, Volume 22 Number 1, pp. 115-124, January(1975) . [ S a l t o n e t a l . 77 ] S a l t o n , Gerard, Bergmark, D. and Wong, A., "Generation and Search of C l u s t e r e d F i l e s " , T e c h n i c a l Report TR 77- 299, Dept. of Comp. S c i , C o r n e l l U n i v e r s i t y , (1977). [ S a l t o n 7 9a] S a l t o n , Gerard, "Suggestions f o r a Uniform Representation of Query and Record Content i n Data Base and Document R e t r i e v a l " , T e c h n i c a l Report TR79-363, C o r n e l l U n i v e r s i t y , (1979). [ S a l t o n 79b] , "A Progress Report On Automatic Information R e t r i e v a l " , T e c h n i c a l Report TR79-368, C o r n e l l U n i v e r s i t y , (1979). [ S c h k o l n i c k 74] Sc h k o l n i c k , M., "The Optimal S e l e c t i o n of Secondary I n d i c e s f o r F i l e s " , T e c h n i c a l Report, Carnegie-Mellon U n i v e r s i t y * November(1974). [ S c h k o l n i c k 76] * "A C l u s t e r i n g A l g o rithm f o r H i e r a r c h i c a l S t r u c t u r e s " , Research Report RJ 1806, IBM Research Laboratory, San Jose, J u l y ( 1 9 7 6 ) . [Schmid et a l . 75 ] References 137 Schmid, H. A. (editor) , B e r n s t e i n * P. A. (editor) , Arlow, B., Baker, R. and Pozgaj, S. r "The R e l a t i o n a l Data Base System OMEGA (Progress R e p o r t ) " , T e c h n i c a l Report CSRG- 72, Computer Systems Research Group, U n i v e r s i t y o f Toronto, J u l y ( 1 9 7 6 ) . [ S e l i n g e r e t a l . 79 J S e l i n g e r , P. G., Astrahan, M. M. , Chamberlin, D. D., L o r i e , R. A. and P r i c e , T. G., "Access Path S e l e c t i o n i n a R e l a t i o n a l Database Management System", Proceedings of ACM-SIGMOD I n t e r n a t i o n a l Conference on Management o f Data, pp. 23-34, see [ B e r s t e i n 79].. [ S t o c k e r and Dearnley 72] Stocher, P. . M. and Dearnley, P. A*, " S e l f - O r g a n i z i n g Data Management Systems", B r i t i s h Computer J o u r n a l , Volume 16 Number 2, pp. 100-105, (1972). [ S t o c k e r and Dearnley 74 ] , "A S e l f - O r g a n i z i n g Data Base Management System", Proceedings IFIP Working Conference on Data Base Management, pp. 337-349, see [ K l i m b i e and Koffeman 74]. [Stonebraker et a l . 76] Stonebraker, M., Wong, E., Kreps* P. and Held, G., "The Design and Implementation of INGRES" ACM T r a n a c t i o n s on Database Systems, Volume 1 Number 3, pp. 189-222, September (1976). [Todd 76 ] Todd, S. J . P., "The Pe t e r l e e R e l a t i o n a l T e s t V e h i c l e - a system overview", IBM Systems J o u r n a l , Volume 15 Number 4, pp. 285-308, (1976). [ T s i c h r i t z i s and Lochovsky 77] T s i c h r i t z i s , D i o n y s i o s C. and Lochovsky, F r e d e r i c k H., "Data Ease Management Systems", Academic Press, I n c., ISBN 0-12-701740-2, (1977). [Watson and A b e r l e 78] Watson, C. T. and Aberle, G. F., "System/38 Machine Data Base Support", IBM S/38 TECH* DEV., pp 59-62, see [IBM 78]. [Wiederhold 77] References 138 Wiederhold, Gio, "Database Design", McGraw-Hill p u b l i s h e r , ISBN 0-07-070130-X, (1977). [Hong and Y o u s s e f i 76] Wong, Eugene and Y o u s s e f i , K a r e l , "Decomposition - A Strategy f o r Query P r o c e s s i n g " , ACM T r a n s a c t i o n s on Database Systems, Volume 1 Number 3, pp. 223-241, September (1976). [Yao et a l . 76] Yao, S. B., Das, K. S. and Teorey* T. J . , "A Dynamic Database R e o r g a n i z a t i o n A l g o r i t h m " , ACM T r a n s a c t i o n s on Database Systems, Volume 1 Number 2, pp. . 159-174, June (1976) . Appendix A. 139 Appendix A. Base R e l a t i o n s Used by. SORAAM This appendix presents a more d e t a i l e d i n t r o d u c t i o n to the t h r e e base r e l a t i o n s and t h e i r a t t r i b u t e s used by SORAAM itself,,. Table I I I - 1 i n the f i r s t s e c t i o n of Chapter I I I o u t l i n e d the schema f o r these r e l a t i o n s . A d e s c r i p t i o n of each o f the a t t r i b u t e s given i n Table I I I - 1 f o l l o w s . The d e s c r i p t i o n s themselves are i n t e n t i o n a l l y c u r s o r y s i n c e t h e i r motivation and use are a c e n t r a l t o p i c i n Chapter I I I o f the t h e s i s . The reader i s encouraged to r e f e r t o t h i s appendix duri n g the course of readi n g Chapter I I I . In the f o l l o w i n g d e f i n i t i o n s those a t t r i b u t e s whose value s a r e a u t o m a t i c a l l y maintained by SORAAM are p o s t f i x e d with "*": RELATIONS Rname = the c h a r a c t e r s t r i n g i d e n t i f i e r of a base r e l a t i o n . B s i z e * = b l o c k s i z e i n terms of re c o r d s per block of a base r e l a t i o n . N* = the number of t u p l e s i n a base r e l a t i o n . D* = the maximum depth of the primary index t r e e s t r u c t u r e of a base r e l a t i o n (eguals the sum of the a s s o c i a t e d a t t r i b u t e ' s "d" v a l u e s ) . PIB* = the number of d i s k b l o c k s used f o r the primary index of a base r e l a t i o n * P I B l o c * = the l o c a t i o n o f the ro o t block f o r the primary index o f a base r e l a t i o n * IX* = the number o f i n t e r n a l nodes used i n the primary index of a base r e l a t i o n * STPB* = the number o f secondary t u p l e p o i n t e r b u f f e r s per index on a base r e l a t i o n (equals N/[ f f 2*Psize J) * f f 1 * = the f i l l f a c t o r of a secondary index. f f 2* = the f i l l f a c t o r of a t u p l e p o i n t e r b u f f e r . P s i z e * = the t u p l e p o i n t e r b l o c k s i z e i n terms of p o i n t e r s per b l o c k . g* p.m.* = the g l o b a l performance measure of a base r e l a t i o n (equals RDQreads/DQreads) . Appendix A 140 Plow = the user s u p p l i e d lower bound on the g.p.m. of a base r e l a t i o n . Phigh = the user s u p p l i e d upper bound on the g.p.m. of a base r e l a t i o n . S = the f r a c t i o n of storage overhead permitted f o r secondary i n d i c e s on the a s s o c i a t e d base r e l a t i o n s . qno* = the r e l a t i v e number to the " f " o f the a s s o c i a t e d a t t r i b u t e s of q u e r i e s on the base r e l a t i o n . qnoold* = as above but r e l a t i v e i n s t e a d to the " f o l d " of the a s s o c i a t e d a t t r i b u t e s . . r a d i x l = the maximum permitted "qno" v a l u e . RDQreads* = the r e l a t i v e number t o "DQreads" of r e l e v e n t block reads f o r query response of a base r e l a t i o n . A r e l e v e n t block was one t h a t c o n t a i n e d a t u p l e t h a t was r e t r i e v e d . DQreads* = the r e l a t i v e number to "RDQreads" of t o t a l block reads f o r query response o f a base r e l a t i o n * T h i s i n c l u d e s primary and secondary index b l o c k s * r a d i x 2 = the maximum permitted "DQreads" value. Pcost* = the r e l a t i v e number of the t o t a l block reads f o r query response, assuming no i n d e x i n g , t o the "b" of the a s s o c i a t e d a t t r i b u t e s * , r a d i x 3 = the maximum permitted "Pcost" value. buf = the number of main memory b u f f e r s a v a i l a b l e f o r o p e r a t i o n s on a base r e l a t i o n . ATTRIBUTES Aname = the c h a r a c t e r s t r i n g i d e n t i f i e r of an a t t r i b u t e name* C0I# = the column number i n which v a l u e s of an a t t r i b u t e occur i n t u p l e s of a base r e l a t i o n * format = the storage a t t r i b u t e . n* = the number a t t r i b u t e o c c u r r i n g i format f o r values of an of d i s t i n c t v a l u e s of an n a base r e l a t i o n (assumed Appendix A 141 not g r e a t e r than N . . . t h e number o f t u p l e s i n the base r e l a t i o n ) . . d* = "d" conveys a t t r i b u t e dependent i n f o r m a t i o n about the primary index of a base r e l a t i o n . S p e c i f i c a l l y i t r e p r e s e n t s the number of l e v e l s i n the k-d t r e e d i s c r i m i n a t i n g on the a s s o c i a t e d a t t r i b u t e . I f l a g * = i f a secondary index c u r r e n t l y e x i s t s on the a s s o c i a t e d a t t r i b u t e of a base r e l a t i o n then I f l a g i s a t r u e value; otherwise f a l s e * I s i z e * = the index b l o c k s i z e i n terms o f a t t r i b u t e v a l u e s per block. SIB* = the number of secondary index b l o c k s used i f a secondary index e x i s t s on the a s s o c i a t e d a t t r i b u t e (equals n/[ f f 1 * I s i z e ]) . S I B l o c * = the l o c a t i o n of the root block of a secondary index (should i t e x i s t ) on the a s s o c i a t e d a t t r i b u t e . f * = the r e l a t i v e access frequency t o the " f " of the other a t t r i b u t e s and "qno" of the a s s o c i a t e d base r e l a t i o n . f o l d * = as above but r e l a t i v e i n s t e a d t o the " f o l d " and "gnoold" v a l u e s . min* - f o r numeric a t t r i b u t e s the c u r r e n t minimum value o c c u r r i n g i n the base r e l a t i o n * max* = f o r numeric a t t r i b u t e s the c u r r e n t maximum value o c c u r r i n g i n the base r e l a t i o n . b* = the " b e n e f i t " value of a secondary index on the a s s o c i a t e d a t t r i b u t e o f a base r e l a t i o n . T h i s value i s r e l a t i v e to the "b" of the other a t t r i b u t e s and to "Pcost" of the a s s o c i a t e d base r e l a t i o n * PROCESSES pro c e s s * = the i d e n t i f i c a t i o n of a process using SORAAM (assigned u s u a l l y by the o p e r a t i n g system) . q u i e t * = a " q u i e t " l og re c o r d i d e n t i f i e r . .

Cite

Citation Scheme:

    

Usage Statistics

Country Views Downloads
United States 5 0
China 4 0
France 2 0
City Views Downloads
Beijing 4 0
Unknown 3 1
Mountain View 2 0
Wilmington 1 0
Ashburn 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}

Share

Share to:

Comment

Related Items