Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

PVIT: A task-based approach for design and evaluation of interactive visualizations for preferential… Bautista, Jeanette Lyn 2008

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2008_spring_bautista_jeanette.pdf [ 1.44MB ]
Metadata
JSON: 24-1.0051291.json
JSON-LD: 24-1.0051291-ld.json
RDF/XML (Pretty): 24-1.0051291-rdf.xml
RDF/JSON: 24-1.0051291-rdf.json
Turtle: 24-1.0051291-turtle.txt
N-Triples: 24-1.0051291-rdf-ntriples.txt
Original Record: 24-1.0051291-source.json
Full Text
24-1.0051291-fulltext.txt
Citation
24-1.0051291.ris

Full Text

PVIT: A Task-Based Approach for Design and Evaluation of Interactive Visualizations for Preferential Choice by Jeanette Lyn Bautista  B.Sc., The University of Manitoba, 1997 B.Sc., The University of Winnipeg, 2003 B.A., The University of Winnipeg, 2003  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science in The Faculty of Graduate Studies (Computer Science)  The University Of British Columbia (Vancouver) April, 2008 c Jeanette Lyn Bautista 2008  Abstract In decision theory the process of selecting the best option is called preferential choice. Many personal, business, and professional preferential choice decisions are made every day. In these situations, a decision maker must select the optimal option among multiple alternatives. In order to do this, she must be able to analyze a model of her preferences with respect to the objectives that are important to her. Prescriptive decision theory suggests several ways to effectively develop a decision model. However, these methods often end up too tedious and complicated to apply to complex decisions that involve many objectives and alternatives. In order to help people make better decisions, an easier, more intuitive way to develop interactive models for analysis of decision contexts is needed. The application of interactive visualization techniques to this problem is an opportune solution. A visualization tool to help in preferential choice must take into account important aspects from both fields of Information Visualization and Decision Theory. There exists some proposals that claim to aid preferential choice, but some key tasks and steps from at least one of these areas are often overlooked. An added missing element in these proposals is an adequate user evaluation. In fact, the concept of a good evaluation in the field of information visualization is a topic of debate, since the goals of such systems stretch beyond what can be concluded from traditional usability testing. In our research we investigate ways to overcome some of the challenges faced in the design and evaluation of visualization systems for preferential choice. In previous work, Carenini and Lloyd proposed ValueCharts, a set of visualizations and interactive techniques to support the inspection of linear models ii  Abstract of preferences. We now identify the need to consider the decision process in its entirety, and to redesign ValueCharts in order to support all phases of preferential choice. We present our task-based approach to the redesign of ValueCharts grounded in recent findings from both Decision Analysis and Information Visualization. We propose a set of domain-independent tasks for the design and evaluation of interactive visualizations for preferential choice. We then use the resulting framework as a basis for an analytical evaluation of our tool and alternative approaches. Finally, we use an application of the task model in conjunction with a new blend of evaluation methods to assess the utility of ValueCharts.  iii  Table of Contents Abstract  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  ii  Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  iv  List of Tables  vii  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dedication  xii  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii  1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1  2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  4  2.1  2.2  Preferential Choice and Decision Theory . . . . . . . . . . . . . .  4  2.1.1  Multiattribute Utility Theory . . . . . . . . . . . . . . . .  5  2.1.2  Additive Multiattribute Value Function . . . . . . . . . .  6  ValueCharts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  8  3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  11  3.1  Evaluation of Information Visualization . . . . . . . . . . . . . .  11  3.2  Task Analysis of Information Visualizations . . . . . . . . . . . .  13  3.3  Visualization Techniques Supporting Preferential Choice . . . . .  21  3.3.1  Tools for Exploring Alternatives and Domain Values . . .  22  3.3.2  Systems with Limited Support for Preference Model Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  23 iv  Table of Contents 3.3.3  Tools for Decision Support . . . . . . . . . . . . . . . . .  27  3.3.4  Visualizations that Support Preference Model Inspection  28  3.4  Evaluation of Preferential Choice Decision Aids . . . . . . . . . .  32  3.5  Previous Empirical Studies of ValueCharts . . . . . . . . . . . . .  36  4 Task Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  40  4.1  Higher-level Visualization Tasks . . . . . . . . . . . . . . . . . . .  41  4.2  Applying Tasks from Decision Theory . . . . . . . . . . . . . . .  44  4.3  Application of the Task Taxonomy . . . . . . . . . . . . . . . . .  50  5 Analytic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . .  51  5.1  Survey of Related Tools . . . . . . . . . . . . . . . . . . . . . . .  51  5.2  Comparison Scoring . . . . . . . . . . . . . . . . . . . . . . . . .  52  5.3  Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . .  54  5.3.1  Construction . . . . . . . . . . . . . . . . . . . . . . . . .  54  5.3.2  Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . .  61  5.3.3  Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . .  66  6 Redesign Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . .  71  6.1  Rotation of Display . . . . . . . . . . . . . . . . . . . . . . . . . .  71  6.2  Construction Interface . . . . . . . . . . . . . . . . . . . . . . . .  75  6.3  Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . .  81  7 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . .  84  7.1  Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  84  7.2  Applying PVIT to Empirical Evaluation . . . . . . . . . . . . . .  85  7.3  Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . .  88  7.4  Controlled Study . . . . . . . . . . . . . . . . . . . . . . . . . . .  90  7.4.1  Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . .  90  7.4.2  Data/Domain . . . . . . . . . . . . . . . . . . . . . . . . .  90  7.4.3  Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  91  7.4.4  Results . . . . . . . . . . . . . . . . . . . . . . . . . . . .  96  v  Table of Contents 7.4.5 7.5  7.6  Part A Sign Test . . . . . . . . . . . . . . . . . . . . . . .  99  Exploratory Study . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7.5.1  Domain Data Sets . . . . . . . . . . . . . . . . . . . . . . 102  7.5.2  Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . 104  7.5.3  Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105  Evaluation Summary . . . . . . . . . . . . . . . . . . . . . . . . . 109  8 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . 112 8.1  Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112  8.2  Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112  Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 A Additional Material . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 A.1 Pre-study questionnaire . . . . . . . . . . . . . . . . . . . . . . . 123 A.2 Post-study questionnaire . . . . . . . . . . . . . . . . . . . . . . . 124 A.3 Consent Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 A.4 BREB Approval . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 A.5 Training tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 A.6 Testing tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130  vi  List of Tables 3.1  Visual tasks taxonomy . . . . . . . . . . . . . . . . . . . . . . . .  19  3.2  The visual accomplishment dimension . . . . . . . . . . . . . . .  20  3.3  The visual implication dimension . . . . . . . . . . . . . . . . . .  20  5.1  Task numbering scheme for PVIT . . . . . . . . . . . . . . . . . .  53  6.1  SMARTER questions for weight elicitation . . . . . . . . . . . . .  80  7.1  Sample inspection tasks mapped to the House domain . . . . . .  93  vii  List of Figures 2.1  A decision analysis flowchart with steps grouped by higher-level task phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5  2.2  An example of AMVF: hotels in Vancouver . . . . . . . . . . . .  6  2.3  Applying AMVF to 3 alternatives . . . . . . . . . . . . . . . . . .  8  2.4  The original ValueCharts and a decision model based on the hotel domain  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  9  3.1  Shneiderman’s TTT: Task by data type taxonomy . . . . . . . .  14  3.2  Bridging the analytical gap . . . . . . . . . . . . . . . . . . . . .  15  3.3  Spotfire: a data visualization technique technique based on starfield displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3.4  Attribute Explorer: a data visualization technique based on attribute bar charts . . . . . . . . . . . . . . . . . . . . . . . . . . .  3.5  23  SmartClient’s data visualizations: a) starfield display b) parallel coordinates view . . . . . . . . . . . . . . . . . . . . . . . . . . .  3.6  22  24  SmartClient a) preference refinement by posting critiques b) simple bar visualization for preference inspection . . . . . . . . . . .  25  3.7  VEIL: Visual Exploration and Incremental Utility Elicitation . .  26  3.8  Visual Interactive Sensitivity Analysis . . . . . . . . . . . . . . .  27  3.9  Treemaps used to visualize preference models based with AHP .  28  3.10 A proposed treemaps visualization for MAUT. . . . . . . . . . .  29  3.11 CommonGIS: a) Utility signs b) Parallel coordinates c) Sliders for setting weights . . . . . . . . . . . . . . . . . . . . . . . . . .  30  viii  List of Figures 3.12 Possible ways to represent hierarchy of objective values in CommonGIS visualizations . . . . . . . . . . . . . . . . . . . . . . . .  32  3.13 The alternative by attribute matrix visualization. Cells contain alternative scores of the corresponding objective. . . . . . . . . .  33  3.14 Ufinder interface with ValueCharts view displaying the selection of the highest valued alternative . . . . . . . . . . . . . . . . . .  37  3.15 Ufinder evaluation results: Subjects found it more useful a) with ValueCharts than b) without, and c) generally preferred UFinder with ValueCharts . . . . . . . . . . . . . . . . . . . . . . . . . . .  38  3.16 ESA implementation a) user right-clicks on the primitive label of the objective hierarchy b) the pop-up utility function interface . 4.1  39  A decision analysis flowchart with steps grouped by higher-level task phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  40  4.2  Decomposition of goal to 3 decision making tasks or phases . . .  41  4.3  Addition of TTT tasks to PVIT  41  4.4  Integrating data and model visualization: expansion of TTT’s  . . . . . . . . . . . . . . . . . .  “relate” task with Knowledge tasks . . . . . . . . . . . . . . . . .  42  4.5  Integration of TTT tasks into PVIT . . . . . . . . . . . . . . . .  43  4.6  Integration of Knowledge Tasks into PVIT . . . . . . . . . . . . .  43  4.7  Integration of VC tasks into PVIT . . . . . . . . . . . . . . . . .  45  4.8  Integration of supplementary tasks from Decision Theory . . . .  47  4.9  The PVIT model: Preferential choice Visualization Integration of Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  49  5.1  Analytical evaluation results - construction phase . . . . . . . . .  54  5.2  [CommonGIS] Wizard screen to select criteria of importance . . .  56  5.3  [VISA] Objective construction of the House example . . . . . . .  56  5.4  [CommonGIS] Filtering based on total scores. . . . . . . . . . . .  57  5.5  [VISA] Creating an alternative list. . . . . . . . . . . . . . . . . .  58  5.6  [CommonGIS] Limited control of value function . . . . . . . . . .  59  ix  List of Figures 5.7  [VISA] a) Value function setting options for both continuous (linear and nonlinear) and discrete types b) graphically setting nonlinear value functions . . . . . . . . . . . . . . . . . . . . . . . . .  60  5.8  [CommonGIS] Selection of alternatives . . . . . . . . . . . . . . .  61  5.9  Analytical evaluation results - inspection phase . . . . . . . . . .  61  5.10 [CommonGIS] Visualization techniques to support inspection tasks 62 5.11 [ValueCharts] Alternatives sorted by total score with stacked bars representing the results  . . . . . . . . . . . . . . . . . . . . . . .  64  5.12 [Treemaps] Increasing objectives and varying proximity can hinder comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . .  65  5.13 [VISA] Many windows to view weights by segmented score bars by family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  65  5.14 Analytical evaluation results - sensitivity analysis phase . . . . .  66  5.15 [Treemaps] Sensitivity analysis methods . . . . . . . . . . . . . .  67  5.16 [CommonGIS] Automatic Variation of Weights . . . . . . . . . .  68  5.17 [VISA] Sensitivity graphs . . . . . . . . . . . . . . . . . . . . . .  69  6.1  Original VC proposal: a) mismatching orientation b) limited space 72  6.2  The new vertical design: ValueCharts Plus  . . . . . . . . . . . .  73  6.3  A zoom-in view of the value function . . . . . . . . . . . . . . . .  74  6.4  The construction interface: objective modelling . . . . . . . . . .  76  6.5  The construction interface: alternative domain values  . . . . . .  77  6.6  The construction interface: specifying objective value function . .  78  6.7  The construction interface: using SMARTER for initial weighting 79  6.8  Rearranging objectives for flexible tradeoff . . . . . . . . . . . . .  82  6.9  VC+ and the final evaluation summary . . . . . . . . . . . . . .  83  7.1  Original orientation: new domain value view  . . . . . . . . . . .  86  7.2  Pruning PVIT . . . . . . . . . . . . . . . . . . . . . . . . . . . .  88  7.3  Demographic breakdown of subjects . . . . . . . . . . . . . . . .  90  7.4  Value trees: a) Training b) Testing . . . . . . . . . . . . . . . . .  91  x  List of Figures 7.5  Evaluating the decision-making process . . . . . . . . . . . . . .  91  7.6  Interaction logging window . . . . . . . . . . . . . . . . . . . . .  95  7.7  Lower-weighted objectives . . . . . . . . . . . . . . . . . . . . . .  97  7.8  With VC+H (a), alternative labeling is on the score display, thus identifying the scores along the total display is quicker. . . . . . .  7.9  98  Insight Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106  7.10 Results of Part B Post-Study questions  . . . . . . . . . . . . . . 108  7.11 Results of general post-study questions . . . . . . . . . . . . . . . 110  xi  Acknowledgements First and foremost, I thank my wonderful supervisor Giuseppe Carenini for his tremendous support and encouragement throughout my graduate studies. It would not have been possible to complete this work without his guidance. I would also like to express my appreciation to David Poole for taking on the role of my second reader. I am extremely grateful to my family, especially my sister Gina, for pushing me, believing in me, and always being there for me. I also thank all my dear friends for their support, in particular Jeremy Mesana, Norie Figueroa, Steve Wilson and Roman Holenstein. Finally, I thank Blair Tennessy for his understanding and reassurance throughout this chapter of my life.  xii  Dedication My work is dedicated to my best friend, dear cousin, and true inspiration: Arnold Galman December 3, 1975 - June 3, 2000  xiii  Chapter 1  Introduction In decision theory the process of selecting the best option out of a set of alternatives is called preferential choice. Many personal, business, and professional preferential choice decisions are made everyday. Often these are complex decisions that require consideration of multiple objectives. We seek the perfect win-win situation, but in most cases, this solution does not exist and we are forced to consider tradeoffs among our objectives. For instance, consider the decisions that may have to be made when planning a vacation. When selecting a hotel within a specified price range, you may find one that is situated at the ideal location but does not have all the amenities you seek. In this case you will have to consider the tradeoffs. People are generally not very effective at considering tradeoffs among objectives, and require support to make this process easier [19]. Decision analysis in the last several decades has investigated methods to support decision-making with conflicting objectives [38], but the use of tools to facilitate the learning and use of these methods are not widespread. According to prescriptive decision theory, effective preferential choice should include all steps of the decision-analysis process. Three distinct interwoven phases are usually considered. First, in the model construction phase, the decision maker builds her decision model based on her objectives: what objectives are important to her, the degree of importance of each objective, and her preferences for each objective outcome. Secondly, in the inspection phase, the DM analyzes her preference model as applied to a set of alternatives. Finally, in sensitivity analysis, the DM has the ability to answer “what if” questions, such as “if we make a slight change in one or more aspects of the model, does it effect the optimal decision?” [19]. The first goal of this thesis is the development of 1  Chapter 1. Introduction a tool for preferential choice that provides full support for - and fluid interaction between - all three phases of model construction, inspection and sensitivity analysis, which are essential in making good decisions. In [16], the authors presented ValueCharts (VC), a set of interactive visualization techniques to support preferential choice. VC in its original form was designed by mainly focusing on supporting the model inspection phase. Preliminary user studies were conducted, demonstrating the need to consider the other phases of the decision process. Furthermore, the design of the interface relied on a rather simple task analysis exclusively based on decision theory. In this thesis we present the second major iteration in the development of VC, in which we have redesigned the system by taking into account the decision process in its entirety. The second key goal of this thesis is to conduct an informative and intuitive assessment of our design by addressing some major challenges of Information Visualization (InfoVis) evaluations. In our attempt to achieve these two goals we applied a common task-based solution: we propose a set of domain-independent tasks for the design and evaluation of interactive visualizations for preferential choice. As we conduct the task analysis we begin with the inclusion of the decision-making phases while offering a taxonomy of tasks that can be used to standardize the evaluation of such tools. The outcome of the task analysis is the Preferential choice Visualization Integrated Task model (PVIT): a much more sophisticated compilation of domain-independent tasks than our previous set as it considers all aspects of preferential choice, some new ideas from Decision Theory, and more importantly, an integration of task frameworks from the area of Information Visualization. To summarize, in this thesis we propose the PVIT taxonomy as a guide for the design and evaluation of InfoVis systems intended to aid in preferential choice. To first illustrate the usefulness of our task model, we performed an analytical evaluation of the original ValueCharts and of other tools for preferential choice previously presented in the literature. This analytical process identified both the strengths and weaknesses of the original VC as well as other tools, driving the redesign of our new system that aims to provide support for all the 2  Chapter 1. Introduction PVIT tasks. We then applied our task model to the empirical evaluation of the new design as a triangulation of methods to determine if users find it a good tool for decision-making. As a preview of the thesis, in Chapter 2 we provide a brief background of preferential choice and our original proposal of ValueCharts. In chapter 3 we discuss the related work including proposed tools, taxonomies, and empirical evaluations. We describe the task analysis and how the PVIT was constructed in Chapter 4, then apply it in the analytical evaluation of VC and competing tools in Chapter 5. In Chapter 6 we present the new design of ValueCharts, and explain the rationale behind it. Finally, we describe the design and results of the empirical evaluation of our newly designed preferential choice tool in Chapter 7, then conclude and discuss future work in the final chapter.  3  Chapter 2  Background 2.1  Preferential Choice and Decision Theory  Preferential choice is the situation where a decision maker is faced with a choice to make among several options (e.g. selecting a house, deciding on a digital camera, etc.). The key step in making the decision is to rank the alternatives according to how valuable they are to the DM. Decades of research in behavioral decision theory have shown that people are not very effective at ranking complex entities according to their value. As a result, Decision Theory has developed a methodology to support such complex decisions based on preferences. The decision maker should develop a quantitative model of her preferences which includes several key considerations. First, the DM should identify the aspects of the entity she cares about. Second, the DM should determine how the achievement of each objective outcome should be measured. Finally, tradeoffs among objectives are quantified in the model by weighting each objective depending on how important it is for the decisionmaker. Any proposed solution can then be applied the resulting quantitative model to compute its value for the decision maker. After completing this process for each alternative, a solution should not be sought by simply ranking alternatives. According to Decision Theory, the fundamental goal of building models of preferences is instead to help the decision maker organize all the information relevant to generate the ranking into a structure that can be effectively analyzed. The decision maker should at least be able to analyze (i) the model itself (ii) the final (and partial) results of applying the model to the alternatives and (iii) how all these results are sensitive to the model 4  Chapter 2. Background parameters. According to prescriptive decision theory, effective preferential choice should include all steps of the decision-making process. We identify this iterative process as three distinct interwoven phases, as depicted in Figure 4.1 [19]. b. a. construction  Identify the decision situation and understand objectives Identify alternatives Decompose and model the problem  inspection  Choose the best alternative Sensitivity analysis  sensitivity analysis  Is further analysis needed? Implement chosen alternative  Figure 2.1: A decision analysis flowchart with steps grouped by higher-level task phases  First, in the model construction phase, the DM builds her decision model based on her objectives: what objectives are important to her, the degree of importance of each objective, and her preferences for each objective outcome. Secondly, in the inspection phase, the DM analyzes her preference model as applied to a set of alternatives. Finally, in sensitivity analysis, the DM has the ability to answer “what if” questions, and determine how all these results are sensitive to changes in the model parameters [19].  2.1.1  Multiattribute Utility Theory  The field of decision analysis has developed a few different methods and models to support preferential choice. The purpose of these models is to enable the DM to organize all the information relevant to a complex decision into a structure that can be effectively analyzed. Several linear models of preferences exist, such 5  Chapter 2. Background as Multiattribute Utility Theory (MAUT) [38], a well-respected and widely used methodology of systematically exploring preference models [37].  2.1.2  Additive Multiattribute Value Function  An additive multiattribute value function (AMVF) is a model derived from the popular MAUT. An AMVF consists of a value tree and a set of component value functions. A value tree is a decomposition of an entity value into a hierarchy of objectives (criteria, or decision-making attributes). Upon facing a decision situation, the objective hierarchy should be created first. Figure 2.2a represents a sample AMVF in the hotel domain. COMPONENT VALUE FUNCTION  VALUE TREE rate  .20  .15  hotel .25  room  .55  size  .85  Internetaccess  .40  area  location ..60  a.  skytraindistance  Rate - R $100<=x1<=$150  V1(R) 1/50*(x1-100)  Size - S 200<=x2<=300 m2  V2(S) 1/100*(x2-200)  Internet access - I none dialup high-speed  V3(I) 0 0.2 1.0  Area - A Uptown Downtown Eastside  V4(A) 0.5 1.0 0 V5(T) 1-(1/8*(x5-1))  Train-Distance - T 1<=x1<=9 blocks  b.  Figure 2.2: An example of AMVF: hotels in Vancouver  Each leaf-node of a value tree is referred to as a primitive objective (e.g., Rate, Size), while non-leaf-node is referred to as an abstract objective (e.g., Location). Assigning Preferences The next step is for the DM to associate each primitive objective with a component value function. This function expresses the preferability of each objective outcome as a number in the [0,1] interval, with the most preferable domain-  6  Chapter 2. Background value mapped to 1 and the least preferable mapped to 0. For instance, in Figure 2.2b the Downtown neighborhood is the most preferred by DM, and a skytrain-distance of 3 blocks has preferability (1 - (1/8 * (3 - 1)))=0.75. This serves as the DM’s measure of how an alternative measures with respect to the objective. Finally, the DM must set the importance weighting for each primitive objective: how valuable it would be for the decision maker to move from the worst to the best level of each objective in comparison to the others. The sum of the weights at each level is always equal to 1. Formally, an AMVF predicts the value v(e) of an entity e as a linear combination of the values of the primitive objectives: v(e) = v(x1 , . . . , xn ) = Σwi vi (xi ) where • (x1 , . . . , xn ) is the vector of primitive objective values for an entity e • ∀ primitive objective i, vi is the component value function and wi is its absolute weight, with 0 ≤ wi ≤ 1 and wi = 1;  wi is equal to the  product of all the weights on the path from the root of the value tree to the primitive objective i. The value v(e) of an entity e will be in [0,1], with 1 meaning the best possible e and 0 meaning the worst possible e [16]. Applying model to alternatives Figure 2.3 shows graphically the application of the DM’s AMVF to three alternatives. From right to left, first the component value functions are applied to the domain value. Next the computed values are multiplied by the corresponding absolute weights. Finally the value of the higher nodes is computed by summing up the values of their children, therefore as we roll up the tree until we finally reach the total scores at the root “Hotel”. 7  Chapter 2. Background  hotel1  COMPONENT VALUE FUNCTION  VALUE TREE .1  .2  .5  0  1  .04 .54 .81 .57  .25 .213 .612  .25  0  Rate - R $100<=x1<=$150  V1(R) 1/50*(x1-100)  0  .5  Size - S 200<=x2<=300 m2  V2(S) 1/100*(x2-200)  Internet access - I none dialup high-speed  V3(I) 0 0.2 1.0  rate  .20  hotel  1  room  size  .22  location .193 .399 .509  area .5  .33  .2  Internetaccess  .21  .55  1  1  .5  1  skytraindistance .25 .875  .875  Area - A Uptown Downtown Eastside Train-Distance - T 1<=x1<=9 blocks  V4(A) 0.5 1.0 0 V5(T) 1-(1/8*(x5-1))  125 300 highspeed uptown 7  hotel2 125 300 highspeed uptown 7  hotel3 125 300 highspeed uptown 7  Figure 2.3: Applying AMVF to 3 alternatives  This illustration is a very simple example of AMVF. As the number of objectives, value tree depth, and total alternatives increase, the model can become much more complex and tedious to analyze in this manner. This can be a cumbersome process, considerably difficult to compare component values across alternatives, and even more daunting to consider changing the model in order to perform sensitivity analysis. All these difficulties may likely deter the DM from using this powerful decision-making method of analysis. What decision makers need is a tool to help them organize this information and effectively visualize relevant details, and in turn, successfully analyze their model of preferences as applied to the alternatives. We argue that an intuitive interactive visualization that supports key tasks for preferential choice can lead to better decision-making.  2.2  ValueCharts  ValueCharts is a set of interactive visualization techniques for preferential choice that aid in the analysis of a DM’s preference model based on the Additive Multiattribute Value Function 1 . 1 For  a detailed description of ValueCharts and AMVF, see [16]  8  Chapter 2. Background The objectives in the AMVF are arranged hierarchically, and are represented at the column headings of the VC (See Figure 2.4). The horizontal length of each column indicates the relative weight assigned to each objective. Each row represents an alternative, thus each cell portrays an objective corresponding to an alternative. The amount of filled color relative to cell size depicts the alternative’s preference value of the particular objective, with a filled cell representing the best possible value and an empty one representing the worst possible value. The values are then accumulated and presented in a separate aligned display in the form of horizontal stacked bars, displaying the resulting score of each alternative.  Figure 2.4: The original ValueCharts and a decision model based on the hotel domain  Several interactive techniques are available in the current prototype to further enable the inspection of the preference model. For instance, sensitivity analysis of objective weights is enabled by sliding the column headings to the desired weight. Double-clicking on the column heading (primitive objective, abstract objective, or total) ranks the alternatives accordingly. Center-clicking on each cell and objective label displays the corresponding domain value and range of values, respectively. Clicking and dragging the left edge of any column (primitive objective) specifies a threshold value that an objective must meet in order to be considered. The original ValueCharts proposal enables all tasks that were intended to be supported. From the description it is apparent that VC provides only limited support for preferential choice when considering the decision process in its entirety [19]. More specifically, it does not support tasks for constructing the model, but only for inspection and some sensitivity analysis. 9  Chapter 2. Background Since the introduction of ValueCharts, two studies and preliminary evaluations were performed by various students. The first investigated how the preference model enhances a data exploration tool by integrating ValueCharts with a dynamic query interface [10]. The second was a case study intended to observe users working with VCs in a domain of their own interest [18]. Results of both studies indicated that although VC was very well-received by the subjects, the fact that VC does not effectively support all the tasks involved in preferential choice has a negative impact on the quality of the decision process and on the users’ perception of it. These results further suggest the need to take a more comprehensive look at all tasks that ValueCharts should support (See Chapter 3.5 for more detail).  10  Chapter 3  Related Work Here we discuss various work in the literature, including related evaluation studies, task analyses, and proposed systems.  3.1  Evaluation of Information Visualization  In the new and growing field of Information Visualization, convincing user evaluations are challenging and still relatively rare [50]. There seems to be very little in the way of agreement over what constitutes a good visualization hence the evaluation criteria differ. Traditional human-computer interaction (HCI) ideals, although critical, can be considered only a beginning to the complexity of empirical evaluation of information visualizations, and it is clear that there is more than usability when it comes to InfoVis. Kosara et al. present a short introduction to user studies of information visualization tools and techniques, together with practical tips from past experiences [41, 42]. These provide some basic principles and an initial understanding of some differences between HCI and InfoVis evaluations, but mainly from basic interaction and perception points of view. In Catherine Plaisant’s effort to understand how new visualization techniques can be adopted into commercial tools, she discusses several challenges of information visualization evaluation [50]. Several challenges are of particular interest to our task-based approach, and we attempt to address them in our evaluation. We will describe some of these challenges in light of recent related work (information visualization comparison studies) as well as how we attempt to address them in our study. 11  Chapter 3. Related Work 1. It is important that the empirical study matches tools with users, tasks, and real problems. Tasks should also be ecologically valid, that is, the tasks chosen represent actual tasks that users would perform with the tool. In a comparison of three different commercial visualization systems (Eureka, InfoZoom, and Spotfire), the experimenters selected tasks that were meaningful in each of the carefully chosen domains (i.e. interesting to their intended subjects) [40]. On the other hand, the SpaceTree evaluation (a comparison to Hyperbolic Browser and MS Windows Explorer) [51] presented visualizations to the subject pool of computer scientists in the likely unfamiliar domain of biology. Subjects were asked to perform some tasks that are of questionable ecological validity, or irrelevant to the purpose of the tool (e.g. “read the path up the tree, find this branch 3 nodes with more than 10 direct descendants, and which of the three branches of this node contain more nodes”). Although our tool is intended to support decision-making in different domains, in our evaluation we present decision situations that are of interest to the subjects and emphasize careful selection of low-level tasks that are helpful in the decision-making process. 2. User testing can be improved by focusing on a more comprehensive set of tasks. Most studies center on locate and identify tasks (e.g. [40] [51]) but those requiring users to compare, associate, distinguish, rank, cluster, and correlate (i.e. visual tasks outlined in [68] and [73]) are rarely covered. Morse et al [45] take a task-based approach that use the low-level visual task taxonomies of [68] and [73] as a guide and starting point of their study. We incorporate these visual tasks to our task model to address this issue. 3. Comparative studies often report overall performance for a combined set of tasks (e.g. [32], [40]) but reporting results per task (as in [51]) is preferable.  12  Chapter 3. Related Work We will deal with this aspect by reporting on tasks at the different levels of our hierarchical task analysis presented in Chapter 4. 4. Measures of discovery and insight should be included in user studies. Discovery is seldom an instantaneous event, but requires studying and manipulating the data repetitively. In [56] the authors describe a set of several distinct characteristics of insight, on which they based an empirical evaluation of five popular visualization tools. Although their study was on microarray vis for biological insight, these characteristics can be generalized and applied to any domain, as we did in our study. 5. Data and task selection remains an ad-hoc process that would be aided by the development of task taxonomies and benchmark repositories of datasets and tasks. Literature on InfoVis task taxonomies only provide general guidelines e.g. [3, 58, 68, 73] (See Chapter 3.2). A move toward more defined tasks is the InfoVis Contest [27] and proposed task taxonomies for specific visualization data types e.g. graph [65], multidimensional [44]. In our work we suggest a taxonomy of tasks for preferential choice visualizations. We begin our effort to overcome some of these challenges by looking closely at the tasks that must be supported for visualization systems that support preferential choice.  3.2  Task Analysis of Information Visualizations  Notwithstanding Plaisant’s emphasis on the importance of task selection in information visualization evaluations, few task taxonomies have been proposed. We discuss the works that are relevant to our study, ordered from generalized to more detailed, including those that focus on effective analysis to aid in decisionmaking through visualizations.  13  Chapter 3. Related Work  Grounded in the literature is Ben Shneiderman’s proposal of a task by data type taxonomy (TTT) [58] where the data types (1-, 2-, 3-dimensional, temporal, multidimensional, tree and network) are on the left side of the TTT, and are organized by the problems users are trying to solve (Figure 3.1). overview zoom filter details-on-demad relate history extract  TASK 1-dimensional 2-dimensional 3-dimensional temporal multi-dimensional tree network  DATA TYPE  Figure 3.1: Shneiderman’s TTT: Task by data type taxonomy  For the task domain (along the top of the TTT in the figure above) he describes a set of tasks that are at a high level of abstraction, noting that more refinements would be the next natural steps in expanding this table. This set of tasks that expands the information-seeking mantra of “Overview first, zoom and filter, then details on demand” can be described as follows: • Overview: a view of the total collection • Zoom: a view of an individual item • Filter: removing unwanted items from the display set • Details-on-demand: getting the details of a selected item • Relate: viewing the relationships between selected items • History: the actions of undoing, replaying, and refining using historic information 14  Chapter 3. Related Work • Extract: the extraction or focusing in on sub-collection and other parameters of a given set Shneiderman described the ideas in his paper as descriptive and explanatory. Although he does not claim for it to be prescriptive, TTT evidently acts as an inspiration and guideline for designers. It has been widely cited by researchers developing novel information visualization tools as a justification for their methodological approaches. Craft and Cairns’ [21] literature survey found 52 citations which they categorized as implementations (34), methods (7), evaluations (6), taxonomies (4) and other (1). Their analysis suggests that Shneiderman’s information-seeking mantra should merely be used as a guideline.  Most recent of the literature in evaluation is that of Amar and Stasko [3]. They note that frameworks like Shneiderman’s typically center on faithful correspondence and representation of data (coined by the authors as “representational primacy”), but fail to facilitate higher-level analytical tasks such as decisionmaking. They argue that there are three reasons for this. First, limited affordances and simple operations such as database queries are only useful for initial exploration of data sets. Secondly, visualization representations are predetermined and not particularly agile, forming simple and static cognitive models from historical data. Third, most visualization systems do not deal with uncertainty of data and interlinked causes and effect very well. These are the basis for what the authors call “analytical gaps”, which are the gap between representation and analysis (Figure 3.2).  representation of data  analyst's perceptual processes  Analytical Gap  Worldview gap do we show the right things to the users?  higher analytic activity  perceiving Rationale gap explaining useful will users understand relationshps relationships what they will see?  Figure 3.2: Bridging the analytical gap  15  Chapter 3. Related Work To bridge these analytical gaps, the authors propose a taxonomy of common subtasks to better support designers and evaluators of information visualization systems. They present higher-level knowledge tasks that visualization systems should support for complex decision-making and learning. The knowledge tasks are categorized into two major types of gaps that need to be bridged: Rationale (the gap between perceiving a relationship and actually being able to explain confidence in and usefulness of that relationship) and Worldview (the gap between what is being shown and what actually needs to be shown to draw a conclusion). We provide task descriptions accompanied by examples to reflect a decision-support visualization interface for preferential choice.  Worldview Gap • Determine domain parameters: provide facilities for creating, acquiring and transferring knowledge about important domain parameters within a data set - clarification of domain-specific measures, relative positive or negative connotations of parameters (e.g. is proximity to the train station a cost or benefit to the DM?) • Multivariate explanation: provide support for discovery of useful correlative models and constraints - visualization of correlation between more than three explanatory variables (e.g. the relationship between multiple objectives with respect to the total value of hotel alternatives) • Confirm hypothesis: provide support for the formation and verification of hypothesis - tools that help users define hypothesis, simulate possible outcomes, and verify the truth about hypotheses (e.g. does the DM’s initial preferred alternative - based on a holistic judgement - fare as well when applied to her preference model?)  16  Chapter 3. Related Work Rationale Gap • Expose uncertainty: expose and show possible effects of uncertainty in data measures and aggregations - dealing with inconsistencies, missing information, representing estimates and standard error (e.g. how to represent missing or estimated values of alternatives?) • Concretize relationships: clearly present what comprises the representation of a relationship, present concrete outcomes where appropriate - further details (i.e. domain values on-demand) and intuitive representation of the nature of how data is related (e.g. how does the 200-square-foot room of Hotel2 contribute to the total value of Hotel2?) • Formulate cause and effect: clarify possible sources of causation - facilities for sensitivity analysis (e.g. What if the DM weights the importance of the nightly rate more, compromising on Internet access? Would it change the ranking of alternatives?) Accordingly, the authors support the use of taxonomies for organizing tasks that a visualization should facilitate. They propose their knowledge task-based framework for systematic design and evaluation of information visualization techniques. In this thesis we have adopted Amar and Stasko’s framework, in conjunction with Shneiderman’s TTT, to organize a set of cognitive tasks for preferential choice previously developed by Carenini & Lloyd [16].  Taking a decision-theoretic point of view, Carenini and Lloyd outline a basic set of cognitive tasks that should be enabled by any interface that supports the analysis of linear models expressing preferences and evaluations [16] (e.g. AMVF, see the previous chapter). These tasks are a good starting point for our study, but were not meant to be exhaustive nor laid out in any meaningful taxonomy or order: 17  Chapter 3. Related Work 1. Comparison of alternatives with respect to total value (e.g., compare the value of Hotel1 with the value of Hotel2) 2. For each alternative, assessment of the contribution to its total value of: (a) value of each primitive objective (e.g., contribution of Skytrain-distance to hotel value of Hotel1) (b) value of each abstract objective (e.g., contribution of Location to hotel value of Hotel3) 3. Comparison of alternatives with respect to: (a) value of each primitive objective (e.g., compare the value of Hotel2 Room-size with the value of Hotel4 Room-size) (b) value of each abstract objective (e.g., which alternative has the best overall Location?) (c) value across objectives, so values for all objectives need to be expressed in same unit (e.g., compare the value of Hotel1 Room-size with the value of Hotel5 Area) 4. Inspection of the hierarchy of objectives (value tree, see Chapter 2) (e.g. identify an objective of Location) 5. Assessment of the extent to which each objective weight contributes to the total (1 when normalized) (e.g. which is the strongest objective of Hotel3?) 6. Sensitivity analysis of changing a weight: (a) how does it effect other weights? (e.g., if the weight of Internet-access is changed, how will the weight of Skytrain-distance change?) (b) how does it effect the value of the alternatives, both the total and for each objective? (e.g., if the weight of Rate is changed, how will the overall value of Hotel4 change? How will the value of Skytraindistance for Hotel6 change?) 18  Chapter 3. Related Work 7. Inspect component value functions 8. Inspect the range on which each primitive objective is defined (e.g., Roomrate varies between 100 to 150 CAD) 9. Maintain overview of all the relevant information Based on these proposed tasks, the initial prototype of ValueCharts was designed. The task set was also used by the authors as a reference for discussion of related work. Now shifting back to a visual perspective, we also look at the types of lowlevel tasks that Plaisant recommends for information visualization evaluation. Wehrand and Lewis [68] present a low-level, domain-independent taxonomy of tasks that users might perform in a visual environment. Zhou and Feiner [73] extend this set by defining and parameterizing additional tasks (Table 3.1). Relational visual tasks  Associate<?x, ?y > Collocate<?x, ?y >  Correlate<?x1, ..., ?xn >  Locate<?x, ?locator >  Plot<?x1, ..., ?xn >  Position<?x, ?locator >  MarkCompose<?x1, ..., ?xn >  Connect<?x, ?y >  Situate<?x, ?locator >  Distinguish<?x, ?y >  Unite<?x, ?x − part >  MarkDistribute<?x, ?y >  Attach<?x, ?x − part >  Isolate<?x?y >  Background<?x, ?backgr >  MarkDistribute<?x1, ..., ?xn > Cluster<?cluster, ?x1, ..., ?xn > Outline<?cluster >  Outline<?x, ?locator > Rank<?x1, ..., ?xn, ?attr >  Emphasize<?x − part?x > Focus<?x − part?x >  Categorize<?x1, ..., ?xn >  Pinpoint<?x, ?locator >  Time<?x1, ..., ?xn, ?t > Reveal<?x − part?x >  Isolate<?x − part?x >  Expose<?x − part?x >  Reinforce<?x − part?x >  Itemize<?x − part?x > Specify<?x − part?x >  Generalize<?x1, ..., ?xn >  Individualize<?cluster >  Separate<?x − part?x >  Merge<?x1, ..., ?xn >  Compare<?x, ?y >  Identify<?x, ?identif ier >  Differentiate<?x, ?y >  Name<?x, ?name >  Intersect<?x, ?y >  Portray<?x, ?image >  Switch<?x, ?y >  Individualize<?x, ?attr > Profile<?x, ?prof ile > Direct visual organizing and encoding tasks Encode<?x >  Iconify<?x >  Structure<?x >  Label<?x >  Portray<?x >  Trace<?x >  Symbolize<?x >  Tabulate<?x >  Map<?x >  Quantify<?x >  Plot<?x >  Table 3.1: Visual tasks taxonomy  This taxonomy interfaces high-level presentation intents to low-level visual techniques by grouping the tasks into two major dimensions: visual accomplishments (tasks that describe the type of presentation intents that a visualization  19  Chapter 3. Related Work might help to achieve) and visual implications (tasks that specify a particular type of visual action that a visual task may carry out). Inform  Enable  Elaborate  Summarize  Explore  Emphasize  Associate  Search  Verify  Sum  Compute Differentiate  Reveal  Background  Categorize  Categorize  Correlate  Correlate  Categorize  Cluster  Compare  Locate  Locate  Cluster  Compare  Correlate  Rank  Rank  Compare  Correlate  Distinguish  Correlate  Distinguish  Identify  Distinguish  Emphasize  Locate  Generalize  Identify  Rank  Identify  Locate  Reveal  Locate  Rank  Rank  Reveal  Table 3.2: The visual accomplishment dimension  The visual accomplishments breaks down into a hierarchical structure: the major branches describe tasks that “inform” and “enable”. The former are described as elaborate and summarize tasks, while the later are further decomposed into exploration tasks and compute tasks (See Table 3.2). Three main implication types make up the visual implication tasks: visual organization, visual signaling, and visual transformations. The overall structure of the implications dimension of the visual taxonomy is shown in Table 3.3. Implication  Type  Subtype  Elemental Tasks  Organization  Visual Grouping  Proximity  Associate, cluster, locate  Similarity  Categorize, cluster, distinguish  Continuity  Associate, locate, reveal  Closure  Cluster, locate, outline  Signaling  Transformation  Visual attention  Cluster, distinguish, emphasize, locate  Visual sequence  emphasize, identify, rank  Visual composition  associate, correlate, identify, reveal  Structuring  Tabulate, plot, structure, trace, map  Encoding  Label, symbolize, portray, quantify  Modification  Emphasize, generalize, reveal  Transition  Switch  Table 3.3: The visual implication dimension  20  Chapter 3. Related Work These taxonomies have been successfully applied in practice. For instance, Morse et al use it as a guideline for their evaluation design [45]. They argue that by defining tests by using a visual task taxonomy rather than customary tasks from the application domain are less restricting and more meaningful for visualizations. We use these taxonomies in our empirical evaluation (Chapter 7) in order to map tasks from PVIT to tasks of specific domains for the user studies.  3.3  Visualization Techniques Supporting Preferential Choice  Several visualization techniques have been proposed and applied to support preferential choice: the selection of a preferred alternative out of a set of entities each described by the values associated with a set of attributes. In order to perform evaluative comparison studies, we require competing interactive visualization tool(s) to compare against ValueCharts. We surveyed the related work in the literature in our search for candidate interfaces, as well as to gain additional insight for our task analysis and redesign. From our research we found that proposed visualization techniques for multiattribute decision-making fell within four categories: • Proposed visualizations that support exploration only according to domain values • Systems that do consider the user’s preference model, but only provide minimal consideration of it • Decision support tools that use traditional visualization methods • Novel visualization techniques for inspecting linear models of preferences  21  Chapter 3. Related Work  3.3.1  Tools for Exploring Alternatives and Domain Values  In contrast with the ValueCharts approach (See Section 2.2 for details), many proposals allow the user to explore the set of available alternatives according to their attributes’ domain values and not according to a model of user’s preferences. The Dynamic HomeFinder [69], and other such tools like FilmFinder[2] and the commercial tool Spotfire [1] (Figure 3.3) are systems based on starfield dynamic query displays and intended to support the exploration of data in selecting a preferred alternative. Attribute Explorer [61] displays a distribution of all objects’ domain values over each attribute and provides effective techniques for querying and filtering (Figure 3.4). Other similar approaches include MultiNav [43], Parallel Bargrams [71], and FOCUS [62].  Figure 3.3: Spotfire: a data visualization technique technique based on starfield displays  Although these approaches support the user in exploring and understanding the alternatives and their domain properties, they do not support users in following decision strategies that lead to optimal decisions [16]. They may be 22  Chapter 3. Related Work  Figure 3.4: Attribute Explorer: a data visualization technique based on attribute bar charts  fine for quick and dirty low-stake decisions (e.g. cost < $20), but for medium to high-stake decisions they should be combined with a system that supports the analysis of preference models.  3.3.2  Systems with Limited Support for Preference Model Analysis  In comparison to the data visualization techniques listed above, SmartClient incorporates a DM’s preferences and values in the process of decision analysis [54]. SmartClient is a personalized decision search tool for finding travel products (i.e. round-trip flights). It uses a starfield overview display which allows comparing the entire range of possibilities according to selected criteria (See Figure 3.5a), a parallel coordinates visualization to compare small sets of alternatives and their attributes (Figure 3.5b), a search engine, and a user interface supporting a user-system interaction model called example critiquing. The DM starts the search by specifying one or any number of preferences 23  Chapter 3. Related Work  a.  b.  Figure 3.5: SmartClient’s data visualizations: a) starfield display b) parallel coordinates view  24  Chapter 3. Related Work in the query area (e.g. destination, travel dates etc.), and based on this initial preference model, the search engine will find and display a set of matching results. The user then can refine her preferences by taking the presented results and posting critiques to it. For example, she might choose an alternative and indicate (by selection from a dropdown menu) that she wants something that is “less expensive” or “closer” to a specified location (Figure 3.6a.). In addition, she can modify the relative preference importance by setting bars to indicate how strongly she desires that preference to be respected: selecting to “compromise” the attribute sets the preference weighting of the selected attribute to zero (Figure 3.6b.). a.  b.  Figure 3.6: SmartClient a) preference refinement by posting critiques b) simple bar visualization for preference inspection  The only method of inspection of the DM’s preference model that she can perform is in this front-end interface. Thus, in this approach, the preferences are communicated in a different manner and the resulting visualizations (Figure 3.5) do not portray the DM’s preference model, but only of the resulting options and domain values of a set of attributes.  25  Chapter 3. Related Work Similar to SmartClient, VEIL (Visual Exploration and Incremental Utility Elicitation) [13] is a system intended to support the selection of round-trip flights. It combines visualizing alternatives according to both the attributes’ domain values and to a linear model of the user preferences. All alternative flights for a given trip are displayed to show objective domain values such as departure time, arrival time, duration etc. In addition, each graphical object corresponding to a flight is colored on a gray-scale based on how preferable it is for the user according to a linear model of her preferences. The DM can change the preference model in two ways: directly, through a window in which each attribute and its weights are shown and can be changed, and indirectly, by stating a preference on the flights displayed on the screen (e.g. by expressing preference for a flight departing at 1pm over the flight departing at 12PM, the utility function is updated). Although this visualization approach includes the preference model and its application to the alternatives, it only provides minimal inspection.  Figure 3.7: VEIL: Visual Exploration and Incremental Utility Elicitation  26  Chapter 3. Related Work  3.3.3  Tools for Decision Support  In addition to the visualization techniques described above, there are a number of commercial decision support tools for analyzing models of preferences. These tools do not offer a novel visualization method to represent preference models, but in fact offer a combined view of very traditional means including the decision tree, simple bar charts and dot plots. Visual Interactive Sensitivity Analysis (VISA) [11], which employs a visual interactive approach [12] is a typical example of such tools that allow the DM to create the objective hierarchy by assembling a value tree, setting value functions, weighting objectives, and creating alternatives (Figure 3.8).  Figure 3.8: Visual Interactive Sensitivity Analysis  The visualization used to assess the total values is a simple bar chart and domain values can be similarly viewed by a simple chart of values. A temp gauge displays the total value of alternatives for each objective (a different display for each one), weighting must be performed by pulling up a bar chart for each objective “family” i.e. an abstract objective and its children, and the 27  Chapter 3. Related Work total value for each alternative is displayed on yet another bar chart. From this description it is apparent that there are many different views and windows required to inspect the preference model and alternatives. Other examples of this type of support are Logical Decisions [33], HiView [31], HiPre3+ [28] and WebHiPre [46].  3.3.4  Visualizations that Support Preference Model Inspection  Asahi and Shneiderman propose a method for the analysis of a preference model that is slightly different from MAUT [7]. The model is based on the Analytic Hierarchy Process (AHP), which is a decision-making methodology for preferential choice alternative to MAUT that nicely maps into a treemap visualization method [57] (Figure 3.9). Although quite different from our method and criticized by some [23], AHP is nonetheless a linear model of preferences, making their proposal a comparable tool.  Figure 3.9: Treemaps used to visualize preference models based with AHP  28  Chapter 3. Related Work Figure 3.10 illustrates (using the Hotel example) how treemaps might be used to visualize a preference model based on MAUT. Here each primitive objective is assigned a space on the treemap based on its weighting, then divided evenly among the number of alternatives (6) and given a distinguishing color. The filled area represents the value of each alternative according to the DMs preferences. Moreover, sensitivity analysis of objective weights would be enabled by adjusting the respective area. hotel location area  room size  rate  Internet-access  skytrain-distance  Figure 3.10: A proposed treemaps visualization for MAUT.  Correspondingly, the treemaps approach as applied to MAUT poses weaknesses in examining certain aspects of the DM’s preference model. The perspective difficulties include assessing the contribution of each objective as well as comparing alternatives with respect to an objective. These problems arise because such tasks require summing and comparing along non-adjacent areas. For a thorough critique of this technique see further discussion in Chapter 5.  29  Chapter 3. Related Work  CommonGIS is a tool for interactive exploration and analysis of geo-referenced data which also supports multiple criteria decision making [5]. Furthermore, it supports the optional analysis of preferences for other decision-making methodologies. One such method is the ideal-point method that involves the comparison of alternatives to both the best objective the worst objective scores (See [35] for further details). Two interfaces are provided for visualizing preferences: utility signs and parallel coordinates views.  a.  b.  c.  Figure 3.11: CommonGIS: a) Utility signs b) Parallel coordinates c) Sliders for setting weights  Alternatives are represented by utility signs, and are spatially distributed and displayed on a map. A utility sign consists of several graphical elements: bars in utility bar charts and circle segments in utility wheels (Figure 3.11a). Each element corresponds to one of the objectives under consideration. One dimension of an element (height in a bar and radius in a circle segment) represents the value of the objective for the represented alternative. The other dimension (width in a bar and angle in a circle segment) represents the preferences of each objective. The user also has the option to frame the utility sign in order to simplify estimation of areas and, thereby, visual evaluation of options. This area corresponds to best possible values of all the attributes. When the user 30  Chapter 3. Related Work interactively changes the objective weights (with representing sliders, shown in Figure 3.11c.), the signs on the map are immediately redrawn. The parallel coordinates view (Figure 3.11b.) includes several horizontal axes, one axis per objective under consideration [4].  Difference in relative  weights of criteria are reflected by variation of lengths of the axes: the more important the criterion, the longer is the corresponding axis (where the highest weighted objective’s axis length is set to the maximum length and all others scaled accordingly). An alternative is represented by a line (“value path”) connecting the positions on the neighboring axes corresponding to the alternative’s objective values (where higher valued objectives of each alternative fall closer to the right pole). Two different result views can be added as additional axes: the summary evaluation scores of the options (given the current weights of the criteria) and the ranking (order) of the options according to the scores (the example in Figure 3.11b. shows both computation results as the bottom two axes of the plot). When the user alters any of the weights, the scores are immediately re-computed, and the results are reflected in the parallel coordinates plot. The method presented also supports sensitivity analysis. One major drawback of CommonGIS decision support design is that it does not provide support for an objective hierarchy. Can utility signs and parallel coordinates support a hierarchical visualization? A single-level example used in CommonGIS can be seen as a set of abstract objectives representing the sum of the values of the primitive objectives. Since utility signs are spatially displayed and represent values with a filled area, we can envision displays such as Figure 3.12a. The dimensional representation of value and weight remain as aforementioned, and each bar represents the abstract objective of the highest level. Lower levels are proportionally filled in, much like a treemap. This representation will likely encounter problems as the number of hierarchical levels and objectives increase, particularly because of the limited area of each sign. The parallel coordinate view can be arranged in the same manner. A parallel coordinate tree, combining the advantages of a familiar tree layout and multidimensional analysis capabilities of parallel coordinates, was proposed to visualize 31  Chapter 3. Related Work  a.  b.  Figure 3.12: Possible ways to represent hierarchy of objective values in CommonGIS visualizations  hierarchically organized survey data [14]. A balanced tree with uniform depth is depicted in (Figure 3.12b), but it may be difficult to lay out in an imbalanced situation with a varied number of levels. In general, we argue that this layout may require too much unnecessary cognitive overhead for the DM. Furthermore, the slider display will also need to incorporate a hierarchical arrangement. Despite the lack of support for hierarchical representation of the value tree, CommonGIS provides coordinated visualization techniques for the analysis of the DM’s preferences. We find their visual decision support features comparable to ours, and a more detailed assessment is provided in Chapter 5.  3.4  Evaluation of Preferential Choice Decision Aids  In spite of extensive previous work on visualizations to support preferential choice, very little evaluation has been reported in the literature. In our investigation we have identified only three empirical studies. 32  Chapter 3. Related Work The first study examined the influence of computer-based decision aids on decision strategy selection. Todd and Benbasat [63] presented an interface for preferential choice based on AMVF. They used a decision support system (DSS) with a very simple visualization of an attribute by alternative matrix, as shown in Figure 3.13. The matrix cells contain the score of the corresponding alternative for the corresponding objective, assuming that a value function has been applied. Attribute weights are specified in an additional column and alternative totals in an additional row. weights  house1  house2  house3  house4  house5  house6  neighborhood  0.28  0.280  0.224  0.280  0.140  0.140  0.280  park-distance  0.42  0.378  0.294  0.378  0.210  0.336  0.042  kitchen-size  0.21  0.210  0.147  0.210  0.084  0.084  0.210  storage  0.09  0.018  0.090  0.072  0.045  0.072  0.072  total:  0.886  0.755  0.940  0.479  0.632  0.604  Figure 3.13: The alternative by attribute matrix visualization. Cells contain alternative scores of the corresponding objective.  The authors propose that: all other things being equal, if a decision aid reduces the effort associated with employing a particular strategy relative to other strategies, a decision maker will be more inclined to employ that strategy. Furthermore, given two equally effortful strategies, decision makers are more likely to employ the one they perceive will provide a better solution. The experiment looked at two decision strategies: additive compensatory (AC) and elimination by aspects (EBA). The AC strategy is roughly equivalent to AMVF, whereas EBA is a noncompensatory strategy where excellent values on some attributes cannot compensate for poor values on other attributes. Basically, EBA is based on a comparison of attribute values to some threshold level 33  Chapter 3. Related Work with the elimination of any alternative that does not meet the threshold level for any one objective. Thus, AC comparatively requires more effort than EBA, but it is considered the superior strategy. In the experiment, the DSS provided a set of commands that support both AC and EBA. A different subset of these commands, varying in the degree of support provided for each strategy, were assigned to different treatment groups. The experimental results showed that subjects used the AC strategy when support for AC was high and EBA when support for AC was low. This demonstrated that effort is an important mediator between decision aid use and decision strategy selection. These findings encourage our work because we argue that the ValueCharts visualization requires even less effort than the attribute by alternative matrix display of the interface used in the study.  In the second study, Pu and Kumar performed an evaluation of the examplebased preference elicitation approach used in the SmartClient search tool [55]. The approach they used was to compare their example-critiquing interface (called “tweakingUI”) with a standard and widely used ranked list (a list of alternatives ranked on a single attribute i.e. price). Their evaluation was based on a simple search task and a set of tasks based on performing tradeoffs. Task completion was measured by how long it took to answer each question (e.g. “Find your most preferred apartment”, “Find an apartment that is 5 square meters bigger than [the answer to the previous question]”) and error rate was measured by the number of wrong answers over the total amount of questions. A post-study questionnaire measured subjective satisfaction and confidence in the answers they found. The result of this study showed tweakingUI provides a useful tool for making multiple tradeoff functions, especially as the complexity of tradeoff tasks increase. 22 out of the 32 total participants expressed more satisfaction with tweakingUI, so it was generally the most preferred interface. In terms of satisfaction, the authors found the more familiar users were with the interface (i.e. 34  Chapter 3. Related Work more time spent with them training), the more they preferred tweakingUI. Surprisingly, subjects rated RankedList with a higher confidence level even though they made fewer errors with tweakingUI. The authors took these findings and extrapolated the results in order to make some preliminary inferences about other example-critiquing interfaces. They reasoned that the experiment results suggested 4 criteria of a good examplecritiquing interface: ease of use, access to weights, ability to perform simple tradeoffs and ability to perform complex tradeoffs. 4 tools were compared in light of these 4 aspects. According to their analysis, SmartClient was the superior interface. In our study we take a slightly different approach. Rather than running the experiment to determine what criteria to consider when analytically evaluating other interfaces, we employ a method that determines the criteria (task list) first. We carry out an analytical evaluation of our prototype as well as selected competing tools, and then a final empirical evaluation of our redesigned system.  Zapatero et al. explored several commercial tools that support multiattribute decision-making. The study evaluates the benefits to decision making due to using these decision support software packages (Logical Decisions, Expert Choice, VIMDA, Criterium, and VISA) as compared to using only a spreadsheet package (Quattro Pro) [72]. Subjects were given a case study problem to solve, so there was only one high-level task performed: make a decision based on the given scenario. The experiment tested users on objective measures (time required to perform experiment) and as well as subjective satisfaction in user-friendliness, confidence in procedure, and confidence in solution. In summary, VISA ranked among the top for all the above categories: 1. VISA and Expert Choice were ranked highest out of the six tools 2. Expert Choice and VISA had the highest factor scores for user-friendliness 35  Chapter 3. Related Work 3. VISA and Expert Choice had the highest factor scores for confidence in the procedure 4. Criterium and VISA required the least time for decision makers to reach their solution. These two packages took significantly less time than the baseline tool Quattro Pro The authors note that VISA was considered to employ a “highly interactive visual approach”, which may indicate the reasons as to why it appears to be the more superior tool. We take a closer look at VISA in Chapter 5. In contrast to this approach, our comparison evaluation examines competing tools and their support for key lower-level cognitive and visual tasks that collectively determine how effective each tool is for preferential choice. Our empirical method tests objectively based on low-level tasks performance and subjectively based on the subject’s perception of the outcome of a high-level decision-making task. Furthermore, since we are testing visualizations for decision support, we also incorporate a measure of insight to our user study.  3.5  Previous Empirical Studies of ValueCharts  Since the introduction of ValueCharts in 2004, two unpublished studies were performed for submission as projects for Computer Science courses at the University of British Columbia. The UFinder project [10] was developed for a graduate course in Information Visualization (CPSC 533C) and the addition of support for step-wise linear utility sensitivity analysis [18] was selected as Directed Studies for the Cognitive Systems (COGSYS) Program (CPSC 448B). In both projects, the students extended the ValueCharts with additional visual means, and conducted informal assessments of the prototype with users. The development of UFinder was an opportunity to incorporate data exploration with preference model analysis. The interface, designed in the spirit of The Dynamic HomeFinder [69], used a starfield display and dynamic querying to explore a set of hierarchically arranged university data. At any point the 36  Chapter 3. Related Work  Figure 3.14: Ufinder interface with ValueCharts view displaying the selection of the highest valued alternative  user was able to pull up a ValueCharts view of resulting universities based on the filtered results. The UFinder interface was enhanced by the additional preference modeling offered by ValueCharts, and the ValueCharts gained a filtering tool as well as an integration to the full set of each alternative’s domain value through the preview pane of the UFinder (See Figure 3.14). Accompanying this work was an informal user evaluation to find out how ValueCharts enhances decision-making over standard dynamic queries. To test this, six subjects were presented the University data set with two different interfaces: The UFinder tool both with the ValueCharts enhancement and without it. Interestingly, it was observed that often subjects simply eliminated columns of unvalued objectives. This finding suggests that the users need to build their preference model, in addition to using UFinder to help construct their alternative set through filtering. Nonetheless, the study determined that subjects found the addition of ValueCharts more useful and preferable (Figure 3.15).  37  Chapter 3. Related Work We find these initial results to be a promising start for our empirical studies presented in this thesis.  a.  b.  c.  Figure 3.15: Ufinder evaluation results: Subjects found it more useful a) with ValueCharts than b) without, and c) generally preferred UFinder with ValueCharts  The second development project addressed future work noted in [16]. In the Extending Sensitivity Analysis (ESA) project, ValueCharts was redesigned to support the integration of step-wise utility sensitivity analysis. Several design ideas were proposed, and the final implementation employed the on-demand appearance of the interactive utility graphs upon right-clicking the primitive objectives’ titles (Figure 3.16). The informal case study provided valuable insights for our evaluation design. An interesting method was used: the subject selected a decision problem of their own to explore. Each subject discussed the problem with the experimenter, who then took the information and created the initial decision tree. The experimenter then plugged the information into a ValueChart for the subject to analyze. 38  Chapter 3. Related Work  a.  b.  Figure 3.16: ESA implementation a) user right-clicks on the primitive label of the objective hierarchy b) the pop-up utility function interface  In general, the subjects were pleased with the system. However, it was found that although we propose a rather simple interface, users still felt that ValueCharts is “pushing their learning capacity”. Users seemed confused at first, but became more comfortable as they were able to piece the components of the visualization and decision model together. In addition, some users wanted to make some changes to the objectives or alternatives, as they learned more about the decision model. We take these salient findings in consideration in our redesign.  39  Chapter 4  Task Analysis In order to identify a set of domain-independent tasks that should be supported by interactive visualizations for preferential choice, we incorporate frameworks and task taxonomies from the field of information visualization with concepts from decision theory. Our top-down, task-based approach begins with the goal to select the best alternative. According to prescriptive decision theory, effective preferential choice should include all steps of the decision-making process. We identify this iterative process as three distinct interwoven phases, as depicted in Figure 4.1 [19]. b. a. construction  Identify the decision situation and understand objectives Identify alternatives Decompose and model the problem  inspection  Choose the best alternative Sensitivity analysis  sensitivity analysis  Is further analysis needed? Implement chosen alternative  Figure 4.1: A decision analysis flowchart with steps grouped by higher-level task phases  First, in the model construction phase, the DM builds her decision model based on her objectives: what objectives are important to her, the degree of impor40  Chapter 4. Task Analysis tance of each objective, and her preferences for each objective outcome. Secondly, in the inspection phase, the DM analyzes her preference model as applied to a set of alternatives. Finally, in sensitivity analysis, the DM has the ability to answer “what if” questions, and determine how all these results are sensitive to changes in the model parameters [19]. The main steps in the decision-analysis process can be identified within 3 main phases (or higher level decision-making tasks): construction, inspection, and sensitivity analysis. We take these phases to represent the first decomposition in our task analysis (Figure 4.2).  Goal: Select the best alternative construction  sensitivity analysis  inspection  Figure 4.2: Decomposition of goal to 3 decision making tasks or phases  4.1  Higher-level Visualization Tasks  We take a shift in our perspective and focus on taxonomies of tasks, both old and new, proposed in the field of Information Visualization, beginning with Shneiderman’s TTT [58]. Since the findings of Craft and Cairn’s survey of 52 TTT citations suggests that it should merely be used as a guideline [21], we use the tasks of Shneiderman’s famous mantra in this initial stage of our analysis as the next abstract level (Figure 4.3).  Goal: Select the best alternative construction Overview  inspection Zoom/filter Relate  sensitivity analysis  Details-on-demand History  Extract  Figure 4.3: Addition of TTT tasks to PVIT  41  Chapter 4. Task Analysis At this point we take a closer look at the type of tasks that are described by Ben Shneiderman. These tasks allow the DM to explore the set of alternatives according to their attributes’ domain values. The TTT framework fares well for interactive visualizations supporting data visualizations such as systems including Attribute Explorer [61] and The Dynamic HomeFinder [69]. On the other hand, a different type of visualization for preferential choice is one that intends to help the DM to examine alternatives according to a model of her preferences. Amar and Stasko’s knowledge task-based framework supports this notion of model visualization by going beyond the mere representational primacy and bridging the analytical gap between the data and the representing model [3].  TASK  overview zoom filter details-on-demad relate history extract  KNOWLEDGE TASKS Worldview determine domain parameters multivariate explanation confrm hypothesis  Rationale expose uncertainty concretize relationships formulate cause and effect  Figure 4.4: Integrating data and model visualization: expansion of TTT’s “relate” task with Knowledge tasks  As Wesley Johnston [36] suggests, rather than thinking of data visualization, we should also be thinking about model visualization, as well as the linkages of the model to the data. He takes on the view that the best solution is to have both model and data visualization and to integrate them well. Based on this principle, we conceive that any interactive visualization that supports preferential choice should strive to accomplish this integration. In order to do so, we conjoin Shneiderman’s framework for data visualization with Amar and Stasko’s model visualization framework. Since the analytical gaps represent what is needed for effectively perceiving and explaining relationships (Chapter  42  Chapter 4. Task Analysis 3, Figure 3.2), the juncture between the two frameworks result from expanding the “relate” task from [58] with the set of Knowledge Tasks from [3] (Figure 4.4).  Goal: Select the best alternative construction  inspection  sensitivity analysis  Overview  History  Details-on-demand  Extract  Zoom/filter  Relate  Figure 4.5: Integration of TTT tasks into PVIT  The resulting tasks from this integration can then be designated as subtasks that support each of the decision-making phases. Although“relate” would have to span all 3 of construction, integration, and sensitivity analysis (Figure 4.5), the remaining TTT tasks and further refined Knowledge Tasks fittingly classify into each phase (Figure 4.6). These high-level visualization tasks constitute our general framework for the design of a visual interface that supports preferential choice.  Goal: Select the best alternative construction  inspection  Zoom/filter  Details-on-demand  sensitivity analysis  Overview History Extract Relate  confirm hypothesis determine domain parameters  multivariate explanation  expose uncertainty  formulate cause & effect  concretize relationships  Figure 4.6: Integration of Knowledge Tasks into PVIT  43  Chapter 4. Task Analysis  4.2  Applying Tasks from Decision Theory  We continue our top-down approach with the next step to apply concepts from decision theory to our high-level framework. In our compilation of basic tasks to support preferential choice visualizations, we integrate a decision-theoretic point of view at this level of the hierarchy. We first apply the basic set of cognitive tasks proposed by Carenini and Lloyd [16]. The proposed tasks suitably arrange as subtasks of the higher-level tasks of the InfoVis layer. • overview – Maintain overview of all the relevant information • multivariate explanation – For each alternative, assessment of the contribution to its total value of: a) value of each primitive objective b) value of each abstract objective – Comparison of alternatives with respect to: a) value of each primitive objective b) value of each abstract objective c) value across objectives – Inspection of the hierarchy of objectives (value tree) – Assessment of the extent to which each objective weight contributes to the total (1 when normalized) • concretize relationships – Comparison of alternatives with respect to total value – Inspection of component value functions – Inspection of the range on which each primitive objective is defined  44  Chapter 4. Task Analysis • formulate cause and effect – Sensitivity analysis of changing a weight: a) How does it affect other weights b) How does it affect alternative values This set of tasks as applied to PVIT is represented in Figure 4.7. It is apparent from this illustration that their initial ideas for ValueCharts focused on the inspection of the DM’s preference model with some support for sensitivity analysis. Goal: Select the best alternative construction  inspection  sensitivity analysis  Overview Maintan an overview of all relevent information  History  Zoom/filter Extract  Details-on-demand Relate  confirm hypothesis  multivariate explanation  determine domain parameters  For each alternative, asessment of the contribution to: a. total value of each primitive objective b. total value of each abstract objective Comparison of alternatives with respect to: a. value if each primitive objective b. value if each abstract objective c. value across objectives  expose uncertainty concretize relationships Comparison of alternatives with respect to total value  formulate cause & effect Sensitivity analysis of changng a weight: a. how it affect other weights b. how it affects alternative values  Inspection of component value functions Inspection of range on which each primitive object is defined  Assessment of the extent to whch each objective weight contributes to total Inspection of the hierarchy of objectives  Figure 4.7: Integration of VC tasks into PVIT  In order to support the tasks that ultimately lead to selecting the best alternative, we must consider the task framework (Figure 4.6) in its entirety. We present this supplementary list of lower-level tasks to complete our set:  45  Chapter 4. Task Analysis Construction • zoom and filter – Filter out uninteresting alternatives – Create list of alternatives • confirm hypothesis – Selection/marking of an alternative (assumption of the best alternative) • determine domain parameters – Selection of objectives (construction of objective hierarchy) – Definition of value function of each primitive objective – Definition of initial objective weighting Inspection • details-on-demand – Inspection of domain values of each alternative • expose uncertainty – Represent and display missing data - How does the uncertainty affect alternative scores? Sensitivity Analysis • history & extract – Comparison of results among different evaluations - Save - Undo - Print  46  Chapter 4. Task Analysis • formulate cause and effect – Sensitivity analysis of changing a component value function: - How does it affect the total value of alternatives?  Goal: Select the best alternative construction Filter out uninteresting alternatives (construction of alternative list)  sensitivity analysis  inspection  Zoom/filter  Overview  Details-on-demand  Maintan an overview of all relevent information  Inspection of domain values of each alternative  History  Extract  Comparison of results among different evaluations  Relate  confirm hypothesis Selection/marking of an alternative  determine domain parameters Definition of value function of each primitive objective Selection of objectives (construction of objective hierarchy) Determine initial objective weighting  multivariate explanation For each alternative, asessment of the contribution to total value of each objective Comparison of alternatives with respect to objective values Assessment of the extent to which each objective weight contributes to total Inspection of the hierarchy of objectives  expose uncertainty Represent and display missing data  concretize relationships  formulate cause & effect Sensitivity analysis of changng a weight:  Sensitivity analysis of changing a component value function  Comparison of alternatives with respect to total value Inspection of component value functions Inspection of range on which each primitive object is defined  Figure 4.8: Integration of supplementary tasks from Decision Theory  New Conceptual Shifts in Decision Theory At first glance, it appears that this consequential list is sufficient for our analysis. On the contrary, it fares well for traditional decision theory methods, but is incomplete due to the necessary consideration of the changes in decision theory over the last two decades. In behavioural decision theory, a large number of studies have shown that human decision making is inherently adaptive and constructive [48, 49]. When people are deciding on how to decide, they are adaptive to both the decision task and the decision environment. They have several decision strategies at 47  Chapter 4. Task Analysis their disposal and when faced with a decision they select a strategy depending on a variety of factors related to the task, context and individual differences. Furthermore, additional studies investigating the contingent nature of decision making indicate that individuals often do not possess well-defined preferences on many objects and situations, but construct them in a highly context-dependent fashion during the decision process. In prescriptive decision theory, we have witnessed a move from an alternativefocused approach to a value-focused thinking (VFT) approach [39]. In the alternative approach the DM, given a decision problem, should follow three basic steps: (i) identify a set of plausible alternatives (ii) specify the values relevant to evaluate the alternatives and (iii) apply these values to choose the best alternative for her. Value-focused thinking reverses the first 2 steps of the process: once a decision problem is recognized, full specification of the relevant values follows, then these values are used to creatively identify alternative possibilities and to carefully assess their desirability. There are several implications of these two relatively recent changes in decision theory on preference elicitation [17] but here we will only focus on the ones that affect our goal of task design for a model and alternative construction interface. One key property of the adaptive and constructive nature of decision-making is that stating preferences is a process rather than a one-time permanent listing of preferences. This view is further supported by a key aspect of Value-Focused Thinking (VFT) that distinguishes it from traditional decision theory: its emphasis on how the iterative process of refinement and of objective quantification can reveal further hidden objectives. Another aspect of VFT that should be considered in our task analysis is that focusing on values first stimulates the DM to search for more desirable alternatives or possibly creatively devise new alternatives that better achieve her objectives. The first perspective is already partly addressed by the sensitivity analysis tasks in our proposed task list. However, two additional construction tasks should be enabled to reflect the new conceptual shifts in decision theory: 48  Chapter 4. Task Analysis • Addition or modification of objectives at any point. • Addition or modification of alternatives at any point. The complete list is outlined in Figure 4.9 as subtasks of the three main tasks of construction, inspection, and sensitivity analysis. It depicts the integration of Shneiderman’s TTT and Amar and Stasko’s Knowledge Task framework, and further illustrates how they fit tasks that deal with visualization of data (alternatives), the preference model, and a combination of both. Goal: Select the best alternative A L T E R N A T I V E S  construction  sensitivity analysis  inspection  Zoom/filter  Details-on-demand  1  Filter out uninteresting alternatives (construction of alternative list)  2  Addition/modification of alternatives at any point  8  Inspection of domain values of each alternative  9  Maintan an overview of all relevent information  Overview M O D E L  History  Extract  of results 18 Comparison among different evaluations  Relate  and  A L T E R N A T I V E S  multivariate explanation  confirm hypothesis  3  Selection/marking of an alternative  determine domain parameters  M O D E L  4  Definition of value function of each primitive objective  5  Selection of objectives (construction of objective hierarchy)  6  Determine initial objective weighting  7  Addition/modification of objectives at any point  10  For each alternative, asessment of the contribution to total value of each objective  11  Comparison of alternatives with respect to objective values  12  Assessment of the extent to which each objective weight contributes to total  13  Inspection of the hierarchy of objectives  expose uncertainty  14  formulate cause & effect  Represent and display missing data  concretize relationships  15  Comparison of alternatives with respect to total value  16  Inspection of component value functions  17  Inspection of range on which each primitive object is defined  19  Sensitivity analysis of changng a weight  20  Sensitivity analysis of changing a component value function  legend:  decision support tasks  Shneiderman's Amar & Stasko's TTT Knowledge Tasks  Carenini & Lloyd's basic cognitive tasks  New tasks from our task analysis  New tasks from AFT and VFT  Figure 4.9: The PVIT model: Preferential choice Visualization Integration of Tasks  49  Chapter 4. Task Analysis  4.3  Application of the Task Taxonomy  The result of our task analysis is a set of 20 basic tasks that we argue is beneficial to the design and evaluation of visualization interfaces to support preferential choice. The task set can be used to guide the design of such decision support systems, as well as act as a set of heuristics for the analytic evaluation of prototypes. In addition, the PVIT can also serve as a basis of user tasks for empirical evaluations. We first use the PVIT set to compare alternative designs for tools intended to support preferential choice. Meanwhile we take a closer look at the current design of ValueCharts to identify aspects that may need some redesign. Moreover, this set of tasks may be further broken down and described based on lower-level visual tasks [68, 73] and detailed actions. Interactive tasks will describe the construction phase, visual tasks will make up the inspection phase, and a combination of both will describe sensitivity analysis tasks. In our evaluation design (see Chapter 7) we will map these low-level tasks to the selected domain(s).  50  Chapter 5  Analytic Evaluation The tasks of the PVIT model provide a good basis for an analytical evaluation. We investigated the extent of support of each task by ValueCharts, as well as 3 competing interfaces. In contrast to approaches such as [16, 55], we intend to evaluate each interface with constructive intent rather than just critically. With the evaluation of ValueCharts, our objective is to address the tasks that are not currently enabled. We also intend to enhance components that have room for improvement, and those that may be implemented better by the other systems. With the assessment of the competing tools, not only do we point out their shortcomings, but we recognize the particular strengths of their methods as well.  5.1  Survey of Related Tools  In Chapter 3, we discussed many different proposals of visual interactive techniques for preferential choice. Among these we discovered three types of proposals that fall short of the requirements that we seek: (i) those that only allow the user to explore the set of available alternatives according to their attributes domain model and not according to a model of a user’s preference (e.g. [61, 69]) (ii) systems that do consider the users’ preference model, but only provides a minimal inspection of it (e.g. [13, 54]) and (iii) proposals and commercial tools inspired by decision theory that we find only incorporate simple charts and traditional visual means that we claim were poorly integrated (e.g. [33, 63]). After this preliminary screening, we decided to conduct the analytical evaluation of ValueCharts (VC) and three systems that support visualizations of 51  Chapter 5. Analytic Evaluation alternative domain values when applied to preference models: AHP Treemaps (TM) [7] and CommonGIS (CGIS) [6] were research tools we found most comparable, and Visual Interactive Sensitivity Analysis (VISA) [11] fared as the commercial tool that had the most focus on complex and effective visualization techniques.  5.2  Comparison Scoring  Our method of evaluation compares each interface task by task (please refer to the numbering scheme provided in Table 5.1), with further refinements of the basic tasks into subtasks where appropriate, whether they are subtasks of [16] (see sections 3.2 and 4.2 for more detail) or broken down further when more detail was necessary. We first looked at whether or not the task was supported, and if there was an obvious difference in the extent of support we ranked them appropriately, according to known concepts from Information Visualization and Human-Computer Interaction. Note that a missing or zero-value for a task could mean either the task was a) not supported or b) supported but ranked last overall. The ranked scores were then applied to the SMARTER [24] weighting technique and then normalized to a value between 0 and 1 (See 6.2 for more details about this technique). The result of our analysis was then plugged into a ValueChart2 .  2 In  addition to preferences, ValueCharts was also developed for displaying evaluations, as  we demonstrate in this section. See [16] for details.  52  Chapter 5. Analytic Evaluation  #  Description  Construction 1.  Alternative definition a) filter out uninteresting alternatives b) creation of alternative list  2.  Addition or modification of alternatives at any point  3.  Selection/marking of an Alternative  4.  Establish Objectives a) selection of objectives b) creation of objective hierarchy  5.  Definition of value function of each primitive objective  6.  Determine initial objective weighting  7.  Addition or modification of objectives at any point  Inspection 8.  Inspection of domain values of each alternative  9.  Maintain an overview of all relevant information  10.  For each alternative, assessment of the contribution to total value of a) value of each primitive objective b) value of each abstract objective  11.  Comparison of alternatives with respect to a) value of each primitive objective b) value of each abstract objective c) value across objectives  12.  Assessment of the extent to which the objective weight contributes to total  13.  Inspection of hierarchy of objectives  14.  Represent/display missing data  15.  Comparison of alternatives with respect to total value  16.  Inspection of component value function  17.  Inspection of range on which each primitive objective is defined  Sensitivity Analysis 18.  Comparison of results among different evaluations a) save b) undo c) print  19.  Sensitivity analysis of changing a weight a) how it affects other objectives i) changes all other objective weights proportionally ii) tradeoff between two objectives b) how it affects alternative values i) dynamic ii) computational representation  20.  Sensitivity analysis of changing a component value function  Table 5.1: Task numbering scheme for PVIT  53  Chapter 5. Analytic Evaluation  5.3  Evaluation Results  In order to report the results of the analytical evaluation, we present a summary of each phase in a ValueCharts view followed by a detailed discussion by phase.  5.3.1  Construction  Figure 5.1: Analytical evaluation results - construction phase  From the summary in Figure 5.1 we can ascertain that CommonGIS and VISA are the only tools that provide a method of constructing the preference model. Their total scores indicate that they are missing support for some construction tasks, and a closer look at the individual objective scores indicate a number of reasons for this. At this point it is important to make the distinction between the two different types of preferential choice situations that we are dealing with. On the one hand, we have the decision situations which are presented to us with a provided data set that includes the list of objectives and an alternative set with domain values provided. An example of these situations is the information found in a catalog of products. Most of the information visualization techniques that have been discussed in Chapter 3.3.1 deal with this type of situation. On the other hand, there are decision situations where the objectives and alternatives must be created by the DM. For example, in the ECA study [18] 54  Chapter 5. Analytic Evaluation the subjects communicated a decision situation, built the objective hierarchy and described the alternatives with respect to the objectives. The commercial decision support tools discussed in Chapter 3.3.3 provide support for this. Most decisions that we face every day require us to define our objectives and identify alternatives. Suitably, the two systems that will be discussed in this section are examples of each type. CommonGIS requires the user to open a data set in order to use the tool, while VISA provides the ability to create new objectives and alternatives from scratch. Creating and selecting alternatives and objectives (Tasks 1 and 4) The interfaces support the two different types of preferential choice problems that we discussed above. First, CGIS supports only selection from given information, whereas VISA only supports construction of new models: for tasks 1 and 4, CGIS only supports a) filtering alternatives and selecting objectives, and VISA only supports b) alternative and objective creation. We further decomposed these tasks into the above subtasks because we argue that an ideal interface should support both methods. To describe CommonGIS in this section we will use the Wallis data set (ski resorts in Switzerland) provided with the tool. To create an objective list (task 4), the DM is presented with a wizard-like interface that initiates the decision-support feature (this is an additional support for preferential choice: CommonGIS is a tool primarily used to explore geo-referenced data). As shown in Figure 5.2, the DM decides on which objectives are important to her and selects it from the left-hand listing, building an an objective list (recall that CGIS does not support an objective hierarchy). The resulting objective list is then the basis for the decision support features of CGIS: parallel coordinates and utility signs view (for simplicity we will only discuss utility bars in this chapter).  55  Chapter 5. Analytic Evaluation  Figure 5.2: [CommonGIS] Wizard screen to select criteria of importance  VISA provides extensive and flexible support, allowing the DM to rearrange the value tree as she sees fit: to add an objective the DM right-clicks on the parent objective and selects the “Set as Parent for New Criteria” option, then by right-clicking on whitespace and selecting “Add New Criterion”, a new primitive objective is added.  a.  b.  c.  d.  Figure 5.3: [VISA] Objective construction of the House example  In order to rearrange the criteria, the DM must right-click and drag - from the objective to be rearranged to the new parent objective - an awkward action since 56  Chapter 5. Analytic Evaluation click-and-drag is usually performed with the left mouse button. Nonetheless, adding and rearranging the tree can be accomplished in order to set the DM’s hierarchy of objectives. Figure 5.3 illustrates an example of how a DM might construct the objective hierarchy for the House example, adding and rearranging information in a-c to arrive at the resulting model in d. VISA allows for loading a data file, and criteria can be removed from the tree and saved with the model. In terms of alternatives (task 1), The CommonGIS approach displays all the alternatives first and then allows the user to filter, but only on the utility signs view (the parallel coordinates view always displays the entire set of alternatives). It only allows for filtering on the total value, instead of each objective, and there is no method of filtering based on domain values. Figure 5.4 illustrates how filtering is performed: the user clicks and drags the arrows that represent the minimum and maximum values (a & b) to create a range (c).  a.  b.  c.  Figure 5.4: [CommonGIS] Filtering based on total scores.  VISA provides good support for alternative creation where name and domain value information is added through a simple table display. By selecting the option under the Alternative menu, default zero-value (or lowest value) alternatives are created and appear in the alternative window. To set each primitive objective domain value, the DM clicks on the default value and either types in the continuous value or selects the discrete value from a list (Figure 5.5). 57  Chapter 5. Analytic Evaluation  Figure 5.5: [VISA] Creating an alternative list.  ValueCharts provides a bit of support for filtering on each primitive objective, but by utility and not domain value. Furthermore, it is not meant for creating an alternative list, but for visualizing the effect of specifying a threshold value. Addition or modification of alternatives and objectives at any point (Tasks 2 and 7) With the incorporation of tasks that arose from the study of value-focused thinking [39] and and adaptive decision-making [48], a visual interface for preferential choice should allow flexible changes of the alternative list, the objective hierarchy, and details of individual alternatives and objectives (tasks 2 and 7, respectively). CommonGIS and VISA provide support for each of these in comparable manner as in tasks 1 and 4 above. CommonGIS supports task 2 only in reverse of the filtering process (broadening the range). It does not support modification of alternatives through the interface (the DM would have to alter the data file). VISA provides consistency by allowing the user to add alternatives in the same way as in task 1. Suitably, both CGIS and VISA provide the same abilities as in task 2 for task 7. CommonGIS allows for the same wizard to appear when adding an objective, but when added it automatically sets the new objective(s) to the average weight (i.e. when adding a fifth objective, it is defaulted to a weight of 20 (100/5) and the others adjusted proportionally; when adding a fifth and sixth, they are both defaulted to weight 17 and the others adjust proportionally, etc.). VISA, on the 58  Chapter 5. Analytic Evaluation other hand, defaults the new objective’s weight to 0 (and the alternative scores to 0 or lowest value). Definition of value function and initial weighting (Tasks 5 and 6) In addition to specifying the initial alternative list and objective hierarchy, the DM must also establish an understanding of how she values each alternative outcome and finds how important each objective is. CommonGIS provides only minimal capability for specifying value function. First of all, since only continuous objectives can be added to the decision model, obviously only continuous objective value functions can be set. Furthermore, CGIS only allows for the user to select between two possibilities: positive linear and negative linear (“cost” or “benefit” criteria) (See Figure 5.6).  Figure 5.6: [CommonGIS] Limited control of value function  VISA, on the other hand, has extensive ability to define value functions for objectives of both continuous (including non-linear) and discrete values. Figure 5.7a shows the options possible. For objectives of continuous domain, if the curve is linear the DM inputs the best and worst values, and if the curve is non-linear she specifies the range (max and min values). An optional unit can be added, and the curve can be viewed and modified by clicking the Show Curve 59  Chapter 5. Analytic Evaluation button, bringing a new window with the plotted value function that the user can change (Figure 5.7b). For those of discrete domain, VISA provides predefined mappings (i.e. an H/M/L scale where High=100, Medium=50, and Low=0) as well as an option to customize her own qualitative scale by specifying the list of domain value and score pairs.  a.  b.  Figure 5.7: [VISA] a) Value function setting options for both continuous (linear and nonlinear) and discrete types b) graphically setting non-linear value functions  Neither tool supports the ability to define the initial weighting of objectives (task 6): definition is only performed in the same manner as they would set weights in sensitivity analysis (see section SA). Instead, both tools provide a default value for the weights: VISA defaults all new weights to 0 (not affecting the current weight distribution), whereas the default weight in CGIS is 1/n where n is the total number of objectives in the list (all other objective weights decrease proportionally). Selection and marking of an alternative (Task 3) From the knowledge task confirm hypothesis springs task 3. CommonGIS provides support for selecting and marking alternatives, not only for use to compare the presupposed best option to the others, but also as a means to compare only selected alternatives. Simply clicking on the utility sign of the option to be 60  Chapter 5. Analytic Evaluation selected gives it a distinct marking. A simple toggle allows the user to view only the selected objects, as Figure 5.8 illustrates. VISA provides no support for this task.  Figure 5.8: [CommonGIS] Selection of alternatives  5.3.2  Inspection  Figure 5.9: Analytical evaluation results - inspection phase  ValueCharts outscores the other interfaces in the inspection phase, though the analysis indicates room for improvement (Figure 5.9).  61  Chapter 5. Analytic Evaluation Viewing details (Tasks 8, 13, 16 and 17) The only interface that does not support a hierarchy of objectives (task 13) is CommonGIS. On the contrary, CGIS is superior for its details-on-demand domain value feature (task 8), since there are several other means of viewing this information: the DM can toggle an information panel to view listed information, or select other views to compare details such as table lens, scatterplot, etc. VC provides only limited capability for comparison of domain values: only one alternative’s value of only one objective is available. VISA has a simple table format view, and AHP Treemaps showed no indication of this feature.  Figure 5.10: [CommonGIS] Visualization techniques to support inspection tasks  62  Chapter 5. Analytic Evaluation In order to help concretize the relationship between the alternative domain value and scoring, inspection of the component value function is important (task 16). CGIS has a persistent display that represents the cost or benefit criteria (as we saw in Figure 5.6), VISA has a graphical view of the continuous functions, and VC provides a textual representation. A description or representation of the rankings required in AHP would be an equivalent for TM in this task, but is not provided by the interface. The range of values (task 17) is important to the DM’s ability to set the weighting of each objective (refer to Chapter 6.2 for a detailed explanation). CommonGIS provides a consistent view of the minimum and maximum values, whereas ValueCharts only provides an on-demand textual view for one objective at a time. Treemaps and VISA provide no support for this. Assessment and comparison (Tasks 10, 11, 12 and 15) Comparison of alternatives’ values should be done on both the overall and objective levels. All interfaces provide support for comparison with respect to total value (task 15): VC displays the results with stacked bars and provides the ability to sort by score, CGIS’ parallel coordinates views provides both an evaluation score and ranking, TM provides a separate gauge view of the resulting scores, and VISA displays scores on the value tree. On the objective level (task 11), CGIS is limited because it does not support a hierarchy so we consider support for only primitive objectives (a), where comparison can be reasonably done by using selection methods in the parallel coordinates view. ValueCharts ranks high for both (a) and (b) because results can be ordered by any objective (on any level) by double clicking on the objective name. VISA and Treemaps do not rank as well: although VISA has a barchart and thermometer view for comparison, they can only be seen withinfamily which must be pulled up with different screens; comparison between different objectives in Treemaps becomes more difficult as the number of objectives and levels increase (See Figure 5.12). Comparison across objectives (c)  63  Chapter 5. Analytic Evaluation  Figure 5.11: [ValueCharts] Alternatives sorted by total score with stacked bars representing the results  is best supported by VC because the columns can be moved in order to make a side-by-side comparison for both abstract and primitive objectives. Comparison with Treemaps can be accomplished but depending on the proximity of the objectives and alternatives, the task can be difficult. In a similar manner, objectives in CommonGIS may be difficult to compare if the utility bars are too far away from each other on the map (See Figure 5.10). VISA provides little support since different windows require opening, and must be sized the same in order to be compared. The assessment of the contribution to total value of each objective for each alternative (task 10) is supported similarly by all tools, where the proportion of the filled area of the objective to total area determines the contribution. CommonGIS is again limited due to lack of hierarchical support, thus ranking last in task 10b. VISA’s representation falls short because the “Segment Score Bars” view must be selected (it is not the default), and the segmented bars only represent the weighting for the highest levels of the family (See Figure 5.13). ValueChart and Treemaps both support task 12 (the assessment of the contribution of each objective weight to total), where the length of bars in the exploded divided bar chart and space size of the treemap represent it well. 64  Chapter 5. Analytic Evaluation  Figure 5.12: [Treemaps] Increasing objectives and varying proximity can hinder comparison  Figure 5.13: [VISA] Many windows to view weights by segmented score bars by family  65  Chapter 5. Analytic Evaluation CommonGIS’ i) small utility signs’ width ii) parallel coordinates’ length and iii) positioning of the weight slider all combine to help support this task. VISA is ranked low because the weight proportion is only presented textually on the objective tree for an overall view, and in separate windows for each family. Overview and missing information (Tasks 9 and 14) Relevant information for the overview of the model (task 9) include the objectives, the alternatives, the alternative values (of both objectives and total) when applied to the model, and the weighting. Treemaps does a good job of this since all this information is readily seen. CommonGIS, despite not supporting the hierarchy, provides all this information on the visualizations and the panels. The main problem with ValueCharts is that objective name readability is hindered by low weighting, long objective names, or increased number of objectives. Finally, since VISA requires the alternative and objective family information to be viewed in different windows, it ranks last. No tool provides support for task 14: representing missing data.  5.3.3  Sensitivity Analysis  Figure 5.14: Analytical evaluation results - sensitivity analysis phase  66  Chapter 5. Analytic Evaluation When looking at the sensitivity analysis results (Figure 5.14), VC’s low ranking strongly suggests that there are clear opportunities for improvement in our redesign. An important consideration would be the incorporation of value function sensitivity analysis, since no tool provides support for task 20. The subtasks of task 19 from [16] are broken down even further in our analysis. ValueCharts provides support for task 19a-i, where the weights can only be changed between two objectives: increasing one decreases the other. VISA and CGIS provide support for task 19a-ii: increasing one objective weight affects all other weights proportionally e.g. increasing one decreases the rest. Treemaps supports both methods with a hook tool for 19a-i and a pump feature for 19a-ii. a.  b.  Figure 5.15: [Treemaps] Sensitivity analysis methods  To tradeoff between two objectives, the hook is selected, changing the cursor, and the DM grabs the edges, sliding it accordingly (Figure 5.15a). To increase 67  Chapter 5. Analytic Evaluation the weighting proportionally the pump feature is selected and clicking on the selected objective increases it by a specified increment while decreasing the others (Figure 5.15b). The sensitivity analysis techniques of Treemaps and ValueCharts update the result display dynamically (task 19b-i). CGIS only displays changes dynamically for the decision support method on which the weight is being manipulated (e.g. if both parallel coordinates and utility signs view are open, the weight change is only reflected on the one view being manipulated and does not update the other). VISA dynamically displays the weight information on the bar charts if the windows are open. VISA and CommonGIS are the only tools that provide a computational method of viewing the decision model (task 19b-ii). CGIS has a procedure of automatic variation of the weights. On the parallel coordinates view is a button labeled “Run Sensitivity Analysis with Current Weights” which opens the window in Figure 5.16a.  a.  b.  Figure 5.16: [CommonGIS] Automatic Variation of Weights  For each criterion, the user specifies the variation range and the number of intermediate steps between the minimum and the maximum. In response, the system assigns different weights from the specified interval to the criterion, proportion68  Chapter 5. Analytic Evaluation ally adjust the weights of the other criteria, and computes the aggregated scores and the ranks for the resulting set of weights. The same procedure is repeated for each criterion. The computation results are summarized into four new attributes specifying for each option the minimum rank received in the course of weight variation, the maximum rank, the mean rank, and the variance. These attributes can be visualized with any of the CGIS visualization methods, for example, using utility bars (Figure 5.16b, with partial view of the resulting map). VISA provides the ability to create and analyze sensitivity graphs (Figure 5.17) . By specifying two objectives VISA plots a graph with the specified score of one objective plotted on the Y-axis and weight of another on the X-axis. This feature illustrates which alternatives are most sensitive to different weightings of specific objectives. If this window is kept open, it is dynamically updated with any weight changes.  Figure 5.17: [VISA] Sensitivity graphs  Finally, for task 18, VISA and CGIS provide the most support. Both systems provide a save function (a) as well as a snapshot function that saves an image of the current view. Printing the display is available for both CGIS and VISA (c), and VISA offers an extensive undo function (b).  69  Chapter 5. Analytic Evaluation  With the assessment of the three competing tools, not only did we point out their shortcomings, but we recognized the particular strengths of their methods as well. In contrast to approaches such as [16, 55], we evaluated each interface with constructive intent rather than just critically. With the redesign of ValueCharts, our objective is to address the tasks that are not currently enabled, as well as enhance components that have room for improvement and may be implemented better by the other tools. Several key findings from this formative evaluation are now considered in the redesign.  70  Chapter 6  Redesign Rationale We now take the information gathered from our task analysis and analytical evaluation to redesign the new interface: ValueCharts Plus (VC+).  6.1  Rotation of Display  One key result of our task analysis was that in the sensitivity analysis phase the user should also be allowed to effectively manipulate the value function (task 20). The Extending Sensitivity Analysis study (See Section 3.5) incorporated this feature into ValueCharts, but was inconsistent with the VC design as it showed the value function only on demand. Accordingly, we explored other possibilities that would allow the value functions to be visible all the time. We will explain the rationale behind the new design to support this task and then describe its effects to rectify various details noted in our analytical evaluation. To keep our compact design consistent, we concluded that since there is a value function for each primitive objective, the most natural way to add the value function view to VC is to have it correspond with the objectives by placing each value function at the bottom of each corresponding objective column. However, such a solution presents a serious problem. We would have a mismatch between how value is displayed for alternatives (as horizontal bars) and how value is represented in the value function views (i.e. vertically). This would not only be visually misleading but the associated sensitivity analysis interaction would also be confusing. The user would be changing the function points up and down while the resulting chart view dynamically updates the view left and right (Figure 6.1a). In addition, the placement of the value function 71  Chapter 6. Redesign Rationale below the objective column may be problematic because the VC column width would be too small to accurately represent the graph (e.g. the Size objective in Figure 6.1b: how to present the value function effectively in the limited and changing space?).  a.  b.  Figure 6.1: Original VC proposal: a) mismatching orientation b) limited space  So, instead of trying to incorporate the value function to our design, we decided to adjust our design to fit the value function. We propose a new version, the rotation of ValueCharts (Figure 6.2). This vertical design addresses the problems described above by providing a consistent orientation between value bars and function axis. The value function has been added to a rotated version of the ValueCharts display, beside the corresponding labels of the objective tree. Each primitive objective’s value function is displayed, with moveable points found as either circles for continuous objectives or squares for discrete. As the points are moved, the chart display is updated automatically to reflect the new function. As the objective weight is changed, the value function view changes in size as well. Because the movement is up/down, the shape of the function remains consistent and the value function height (y-axis) remains scaled to the height of the row or maximum fill of the corresponding cells. To examine the implications of this rotation on our analytical evaluation, we 72  Chapter 6. Redesign Rationale  Figure 6.2: The new vertical design: ValueCharts Plus  will first look at how it affects the inspection of the component value function (task 16). CGIS, VISA and VC received the same score on this task but for different reasons. Value functions in CGIS are permanently on the display, but the view is limited to showing only the value direction (positive/negative). VISA provides a more complete graphical view (at least for continuous values), but it requires several clicks to access it. VC provides a textual specification of the value function but it can be accessed with only one click. In contrast, we argue that the rotated version of VC ranks higher because it provides a graphical view, is persistent, and is available for all types of objectives. The rotated version also addresses two of VC’s shortcomings indicated in the analytical evaluation. First, task 17 (inspection of the range on which each primitive objective is defined) is improved, since the range of domain values for each objective is always readily visible. This is important for the DM’s awareness of actual tradeoff between two objectives. Because of the importance of  73  Chapter 6. Redesign Rationale this during weight sensitivity analysis, we decided to place the value function beside the objective labels. This way, as the DM is changing the weights, the best and worst values for the objectives are in view (more on this concept is described in the following section). Secondly, objective names are more readable (which affects task 9 - maintain overview of relevant information), because the label width is now only affected by the depth of the tree instead of the number of objectives. Also notice that readability of alternatives is not compromised by the rotation, since we can now take advantage of text slanting (this was not an option for objectives because of the hierarchical structure). In addition, the new orientation further strengthens support for concretizing relationships: aligning domain values (from the utility graphs) with cell value (height of cell fill) enhances the visualization of the relationship of the data and model. Lastly, most decision analysis trees, including value trees, expand left to right; therefore the rotation of the presentation places the hierarchy of objectives in this orientation, correctly portraying the direction of expansion. We do recognize potential problems with this design. With increasing objectives and decreasing row height, value function sensitivity analysis is hindered due to the small graph size. For this reason, an on-demand feature is enabled, presenting a bigger graph on a separate view (Figure 6.3).  Figure 6.3: A zoom-in view of the value function  74  Chapter 6. Redesign Rationale In addition, we designed the graphs so that if the objective weighting is too small, the moveable points and x-axis details are removed thus the general shape of the function is retained (see Figure 6.2, “size”). Also, the entire view is optional and can be removed with a menu item selection. Another issue was the possible waste of screen real estate at the top-left corner. We use it to our advantage to improve our capability of domain value-view (task 8), in which we now provide a listing of the selected alternative’s domain values.  6.2  Construction Interface  The goal of the development of the construction interface is to provide an intuitive method of preference model-building that relies as much as possible on the representations used in VC. From our analysis we deduce that there are four main tasks (steps) required to build an initial VC: definition of objectives, alternatives, value function, and initial weighting (tasks 1, 4, 5 and 6). We considered a separate wizard-like interface so that users focus on only one step at a time, but instead we chose a tabbed interface - for the sake of flexibility - in which each page encompasses the information required to complete these initial steps. According to value focused thinking (See Section 4.2), the objectives of the decision problem should be considered first. We present the construction interface with the objective view at the first tab (Figure 6.4). We adopted a concept similar to - yet simpler than - that of VISA, in which the decision maker builds a value tree graphically by adding to the objective that will represent the new objective’s parent. In addition, we included a separate panel or list of objectives that the DM can use to help arrange the tree. For example, the DM can load objective data into the application, and the available objectives will be listed on the panel. The user can then select objectives from this list to add to the tree, or remove objectives from the tree while retaining the objective information in the data file. The hierarchy of objectives is built by either adding new objectives by right-clicking on the parent objective and selecting Add from the context menu 75  Chapter 6. Redesign Rationale (eliminating VISA’s additional step of setting the parent), or selecting objectives from the list on the right-hand side by drag-and-drop. The DM can also remove or rearrange objectives on the tree in the same manner. This intuitive display closely resembles the exploded-divided bar chart, so the DM seemingly builds the VC view directly.  Figure 6.4: The construction interface: objective modelling  Once there are at least two objectives in the hierarchy, the user can input information about the alternative by clicking on the Alternative tab (Figure 6.5). In the case where the data is available in a stored file, this information is already available and populates into the fields. However, when the situation calls for newly created objectives, the domain values need to be entered. For consistency with the VC representation, the alternative view displays a table with objectives listed along the left-hand side, the columns represent the alternatives, and the data can be entered directly into the corresponding cells. After all the data is input and the DM clicks the Value Function tab, the application gathers required information in order to prepare for the next step. Each unique domain value for each primitive objective is an object of evaluation: the range (minimum and maximum) is determined for continuous objectives and the full list of possible outcomes is required for each of the discrete objectives.  76  Chapter 6. Redesign Rationale  Figure 6.5: The construction interface: alternative domain values  The DM must now set each value function according to her initial preferences. In order to do this, the DM first identifies what she feels are the best and worst outcomes for each objective, and assign them a value (utility) of 1 and 0, respectively. For objectives of continuous domain, any outcome value that falls within the maximum and minimum (inclusive) can be selected, and the DM sets all others as she sees fit. In other words, the DM is setting each of the outcomes to a normalized value (0 to 1) according to how desirable it is to her. In our interface, assigning values to outcomes is performed in the same manner as the sensitivity analysis graphs described in the previous section. Each objective, when selected from a list, will present a graph according to the alternative values: for objectives of discrete domain, each possible value will be presented along the bottom (Figure 6.6a), and for objectives of continuous domain, the x-axis ranges from the lowest possible value to the highest possible value according to the alternative set (Figure 6.6b). For these objectives, the DM can right-click and select from a list of pre-set functions (i.e. positive, neg-  77  Chapter 6. Redesign Rationale  a.  b.  Figure 6.6: The construction interface: specifying objective value function  ative linear, etc), as well as specify the steps (points) for step-wise sensitivity analysis. The DM cannot move on to the next step until all value functions are set. There is no default function: all points are set at 0.5 utility, so it is necessary for the DM to think about their preferences and set the individual functions. If a best or worst value is not set for a particular objective, this step is incomplete: the objective label in the list is highlighted and the Weighting tab is disabled. The importance of setting the value function according to alternative outcomes is apparent when we move to the next step. Often decision-makers assign weights to decision criteria incorrectly, as they tend to consider only “importance” of the objectives relative to the others. What must also be considered is the range of each objective’s alternative domain values based on the user’s preferences, or the best or worst outcomes. For example, when making the decision to purchase a new car, usually price is considered one of the most important factors. But would it still be as important if the prices of all cars being considered fall within a range of $10? The SMARTER [24] preference elicitation technique has been developed in decision theory to help decision makers express weights effectively. We incor78  Chapter 6. Redesign Rationale porate SMARTER in our interface. This technique elicits from the DM a rank importance ordering. It uses a series of questions (the Swing Weighting [66] procedure) that gets the decision-maker thinking about how important each of the objectives is to her by considering an improvement of each objective from the worst to the bast values. After the DM ranks the objectives, weights are assigned to each objective with the Barron-Barrett solution. If K is the number of attributes, then the weight of the kth attribute is: K  (1/i).  wk = (1/K) i=k  The theory behind SMARTER weighting is comparable to other techniques [52], and has proven to be a more a effective method for weighting than others [9]. The SMARTER weighting method was selected over other techniques for its simplicity of elicitation and easier judgements from the DM [24].  a.  b.  Figure 6.7: The construction interface: using SMARTER for initial weighting  Upon clicking the weighting tab of the construction interface, the application lists all the objectives and its best (utility = 1) and worst (utility = 0) outcomes. When the SMARTER weighting method is chosen, a wizard-like interface leads the DM through the weighting process (Figure 6.7). Through the wizard, the 79  Chapter 6. Redesign Rationale DM selects the objectives in the order according to importance - considering the range according to best and worst values - until all the objectives are ranked (Table 6.1 lists the simple questions of this elicitation method used). The wizard disappears and the initial view of the weighting page lists the objectives in ranked order, and includes the weighting assigned by SMARTER. This list represents the user’s initial weighting, and can be changed with the inspection view and sensitivity analysis. Question #  Dialog  1  Imagine the worst possible alternative (i.e. scoring 0 on all objectives), and for some reason you were required to choose it. If you can choose one objective to improve from its WORST value to its BEST, which would you choose to improve?  2 to n-1  Next, imagine that you are stuck with the worst possible alternative and allowed to improve any objective EXCEPT [list of objectives already selected] from its worst value to its best. Which would it be?  n  [last remaining objective] is the last objective that you would choose to improve. Click OK to complete the SMARTER weighting technique.  Table 6.1: SMARTER questions for weight elicitation  Upon completion of the weighting, the OK button is enabled, signalling that all steps of the construction phase are complete. At this point the DM can go back to any of the pages of the tabbed construction interface to make changes, but certain changes may affect the information in the other tabs and require the DM to add to or edit other pages. For example, if a new objective is created and added, the alternative values must be input, a corresponding value function set, and the SMARTER weighting process redone. Once the DM is satisfied with her constructed model, she clicks the OK button and the VC inspection view is presented (Figure 6.2). The DM can then select an initial preferred alternative to be marked (satisfying task 3) by right-clicking on the alternative label. Our construction interface allows the DM to import a data file, as well as construct the decision model from scratch. Our analysis of CommonGIS and  80  Chapter 6. Redesign Rationale VISA are key contributors to this combined design. In addition, this construction interface is not only used for the initial construction of the decision model, but also to modify, add or delete objectives and/or alternatives at any point (tasks 2 and 7).  6.3  Sensitivity Analysis  As we researched the proper weighting methods, it was evident that the original design of ValueCharts for sensitivity analysis of weights was very limited. Users could only perform a tradeoff between two objectives within a family (according to the hierarchy). For example, in this case, the DM can achieve a tradeoff weighting between Internet-access and Size within Room, but not between Size and Area (Figure 6.8a). Since the initial weighting methods that we employed ask the DM to rank all primitive objectives, the DM should be able to tradeoff between any two primitive objectives. We now allow users to do so, by rearranging the objectives by position then performing the slide weighting manipulation. Now the DM can perform a tradeoff between Size and Area, and other combinations can be accomplished (e.g. Size and Skytrain-distance) by repositioning the objectives (via drag and drop) in the tree (as in Figure 6.8b). We also enhanced the sensitivity analysis of changing weights by adding support for task 19a-i: the ability for users to see how the changes in weight of one objective affects the decision process if there is no specific objective to perform the tradeoff against. To support this task, we introduced a pump tool, similar to that of the Treemaps system [7]. When the pump option is turned on, the user clicks on an objective to change it by a certain increment, and all other objectives will change accordingly. We added several other capabilities to compare results of different sensitivity analysis (task 18). In addition to undo and print features, the save function not only captures the ValueChart information, but also the additional objective/alternative information that may be considered in future analysis of the same decision model. The ValueChart file (.vc) is loaded with all the construc81  Chapter 6. Redesign Rationale  a.  b.  Figure 6.8: Rearranging objectives for flexible tradeoff  tion information including objectives that are not included in the hierarchy. We also provided two other functions. Like CGIS and VISA, we added a snapshot function that takes a screenshot of the current ValueChart area for later viewing. We also incorporated the ability to open a second window of the current preference model for comparison with sensitivity analysis results.  Figure 6.9 presents a summary of a new PVIT based evaluation in which VC+ is compared with VC and all other alternatives. Notice the great improvement of VC+ from VC, as well as scoring 30% or better than the other tools. To visualize the results of this analysis we use VC+ itself.  3 For  3  this summary, the three phases were weighed equally, and the basic tasks are of equal  weight within each phase.  82  Chapter 6. Redesign Rationale  Figure 6.9: VC+ and the final evaluation summary  83  Chapter 7  Empirical Evaluation 7.1  Overview  After an analysis of the tools with respect to tasks of the PVIT model, we conducted an empirical evaluation. Since the analytical evaluation indicates that the other tools fare considerably worse than ValueCharts Plus in supporting the tasks of the PVIT model, we gave low priority to a comparison study. Instead, we performed a user study to assess the usefulness and effectiveness of VC+. With our assessment of VC+, we intend to accomplish the following: 1. gain insight on our new design 2. determine if VC+ is perceived a good tool for decision-making 3. apply our integrated model of tasks 4. address some challenges in evaluating visualizations We seek results from both a controlled experimental study as well as those of an observational or contextual inquiry. Although we are interested in key changes of our design, seeking the effectiveness as a decision-making tool would call for a more exploratory study. We propose a triangulation of evaluation methods that intersect at our PVIT model. We intend to accomplish goals 1 and 2 by having a split, 2-part experiment, which ties into our integrated task model (goal 3), and, in turn, addresses some challenges (goal 4) that we discussed in Chapter 3.  84  Chapter 7. Empirical Evaluation  7.2  Applying PVIT to Empirical Evaluation  In our PVIT model the main task - or goal - is to select the best alternative. The DM must be able to effectively analyze her preference model in order to help her select an alternative based on these preferences. Our hierarchical PVIT model outlines several levels of different tasks. The lowest level can be further decomposed and these domain-independent tasks must be mapped to specific domains in order to define tasks for empirical evaluation. Our claim is that if there is good support for the tasks of PVIT, then the DMs should be able to effectively analyze the decision and lead them to selecting a preferred alternative. By pruning the tree we can look more closely at the different higher level tasks to see how well they are supported. As a simple example we can look at how a tool fares in each of the three phases, as we discussed in Chapter 5. In order to narrow the scope of our evaluation, some pruning was done. First we looked at what we considered was important to evaluate, and after deciding on which aspects of the tool required some investigation, we then determined which corresponding area of the tree to focus on. Vertical Versus Horizontal The rotation of display is a key design change that we incorporated, and was proposed to improve the visualization as well as strengthen support for the tasks in the inspection phase (see Chapter 6). One aspect of our empirical evaluation was to test this design change. Both the horizontal and vertical interfaces incorporated the enhancements noted in Chapter 6 (e.g. support for construction), but in the cases where the new vertical orientation provides solutions to problems that the horizontal can not integrate, we modified the horizontal design to suit the limitations of the original view. Two notable design aspects were affected. Since the horizontal orientation of ValueCharts can not effectively include a persistent display of the value function (see Chapter 6), we concluded that the 85  Chapter 7. Empirical Evaluation value function sensitivity analysis feature best fits the horizontal design as an on-demand popup, where the user double clicks on the objective label to bring up the view (as was implemented for the ESA study [18], see Chapter 3.5). We also added to the horizontal interface a domain value view where the information for selected or all alternatives are displayed directly on the cells (Figure 7.1). The spatial positioning of the cell relative to the alternative and objective clearly identifies what the value represents. This provides a persistent view for comparison, instead of the originally implemented on-demand popup (which is still available in case the weighting is too small and cuts off the text). No further screen real estate is required.  Figure 7.1: Original orientation: new domain value view  Other major differences in the views did not require a different design, but may affect the effectiveness in the display. For example, the readability of labels, persistent view of the range of objective outcomes via the value function display, etc. These key differences in the displays highlight the areas of the PVIT model that we decided to focus on. As a final note, although we described the vertical interface as our new design, we have no intention of simply dismissing the horizontal view. We plan to offer both views in the final product, as factors in different situations may call for one over the other. For example, VC might be used as a supplementary tool to information provided on a website (whereas available screen space may incorporate the horizontal view better) or it may be a stand-alone application that a user can maximize use of all visuals (thus the user gains more from the 86  Chapter 7. Empirical Evaluation use of the vertical view). In addition, the nature of the data and decision model - including number of objectives, tree depth, alternative count, etc. - may be better used in specific orientations. The flexibility of ValueCharts in this manner is important. Since users have the natural inclination to use the default view in applications [40], the results of the small comparison study gives us further insight on which orientation should serve as the default view in our tool. Experimental PVIT We now refer back to the PVIT model to see how we decided to prune the tree. The driving factor behind the rotation of display was the value function sensitivity analysis feature. In addition to supporting the task Sensitivity analysis of changing a component value function, the persistent feature also lends support to the tasks Inspection of component value function and Inspection of range of values, and indirectly helps support Sensitivity analysis of changing a weight by providing more readable objective names and a proximal view of best and worst values for effective tradeoffs. Referring back to Chapter 4: by focusing on these tasks, rolling up the task hierarchy and taking a closer look at the PVIT model’s higher-level Knowledge Task-Based Framework [3], we reveal that these tasks all help to bridge the Worldview Gap (Figure 7.2). According to [3], the Worldview gap is the gap between what is being shown and what actually needs to be shown to draw a straightforward representational conclusion for making a decision. Thus, according to our PVIT model, if these tasks are well-supported and used properly by the DMs, the visualization tool should effectively help the user in decision-making. Our intent is to evaluate how users perform the lower level tasks of the PVIT, focusing on the applicable tasks of the Worldview gap (as illustrated in Figure 7.2). Although we pruned the tree, we still intend for users to be exposed to all the functions and enhancements of ValueCharts to help in the decision-making  87  Chapter 7. Empirical Evaluation  analyst's perceptual processes  Worldview gap expose uncertainty Represent and display missing data  concretize relationships  formulate cause & effect  perceiving useful relationships  Sensitivity analysis of changng a weight  Sensitivity analysis of changing a component value function  Comparison of alternatives with respect to total value Inspection of range on which each primitive object is defined  Figure 7.2: Pruning PVIT  process. Since the construction interface is very similar for both orientations, we did not include this in our evaluation, but introduce it in a tutorial (see Chapter 7.4.3). We tried to cover all the inspection tasks, keying in on the tasks above. In addition, we set out to determine how well our tool aids users to accomplish the higher-level task of decision-making. We believe that if these lowerlevel tasks are well performed by the user, the DM should be able to effectively analyze the preference model of the decision at hand, thus having the ability to select the best alternative. We introduce an experimental design with two parts: a quantitative study that measures the performance of users at the low level tasks, and a qualitative study that determines how effective the tool is for decision-making.  7.3  Evaluation Methodology  We used several distinct methodological approaches in order to achieve our goals of the evaluation. First, we took a quantitative approach by performing a controlled usability study to see how users performed the low-level tasks of the PVIT. Second, in our qualitative approach we observed the subject using the tool in a real decision-making context. The subjects then answered a number of questions regarding their experience with ValueCharts in the decision-making  88  Chapter 7. Empirical Evaluation process. In addition, we attempted to measure the users’ insight in the decision problem. And finally, we used interaction logging throughout the experiment for further study. By triangulation of methods we were able to more fully understand the DM’s experience of analyzing their preference model while gaining insights on our new design and determining how effective ValueCharts is in supporting decision-making. Since we were comparing two versions of the same tool, we decided on a between-subjects experimental design to avoid the obvious learning effect that would come with a within-subjects design. Each subject was assigned to either the vertical (VC+V) or horizontal (VC+H) ValueCharts Plus interface. Subjects Subjects were recruited through the Reservax  4  online experiment reservation  system. Prior to beginning the experiment, each subject read, signed and dated a consent form and filled out a pre-study questionnaire (see Appendix A.4 and A.1). 20 subjects, all students at UBC, agreed to spend 60 minutes with our experiment and receive $10 in compensation. Of the sample of 20 subjects, 8 were male, and ranged from late teens to 50+ in age. All subjects were fairly computer proficient, ranging from 10 - 50+ hours per week. Each subject worked with only one ValueCharts interface: 10 subjects worked with VC+V and the other 10 worked with VC+H. Figure 7.3 illustrates the demographic breakdown in detail. We found a good match in the grouping of the subjects in each treatment. Both groups had the same breakdown in sex and English proficiency, and the average computer use was very close. There was a slight difference in average age group: VC+V subjects were a younger group overall, half of them being less than 20 years old, and in the VC+H group, most subjects were in the 20-29 age group. All subjects had no previous exposure to formal decision analysis methods. 4 HCI@UBC  Subject Sign-up System http://www.reservax.com/hciatubc/index.php  89  Chapter 7. Empirical Evaluation  Sex  Age  6 female male  4 2 0  10 8 50+ 40-49  6  30-39 20-29  4  < 20  2 VC-V  VC-H  0  participants  8 participants  10  8 participants  10  Computer use (hours/week)  50+ 30-49  6  20-29 10-19  4  <10  2 VC-V  VC-H  0  VC-V  interface  interface  VC-H interface  Figure 7.3: Demographic breakdown of subjects  7.4 7.4.1  Controlled Study Hypothesis  H0 : There is no difference between the vertical and horizontal interfaces. In this section we attempted to measure subject performance on a set of tasks to determine whether or not there is a difference between the two orientations. In addition to the total time to complete and correctness of tasks, we looked at each task individually.  7.4.2  Data/Domain  For this controlled study, we used data sets accompanied by scenarios: for training, we used the scenario of shopping for a used television set, and for testing we put the user in the situation of deciding on a hotel to stay at in Vancouver. For the training phase we used the TV domain to describe the VC+ interface with 4 primitive objectives grouped into 2 abstract objectives (quality and cost). Of the primitive objectives, 2 were of discrete domain (condition and shipping) and 2 were of continuous (size and price). Other objectives such as stereo, age, and brand were shown to illustrate how objectives can be added and rearranged in the construction phase.  90  Chapter 7. Empirical Evaluation The hotel example that we have presented thus far was used in our testing phase. In this scenario there were 5 primitive objectives, 2 of which were grouped into 2 abstract objectives (location, room) with price as a leaf on its own. Price, distance from skytrain, and room size were continuous, and area and Internetaccess were discrete objectives. The value trees of the decision models are displayed in Figure 7.4.  rate size quality  size  condition  used television  room  hotel  Internetaccess  shipping  area  location  cost price  a.  b.  skytraindistance  Figure 7.4: Value trees: a) Training b) Testing  7.4.3  Tasks  Since the construction view is essentially identical for both interfaces, it was not included in the testing. It was, however, demonstrated in both training and testing in order to help the subject understand the decision situation. Hence, the first part of the study assumed that the construction phase was already completed, and participants performed inspection tasks interspersed with sensitivity analysis tasks. demonstrated  construction  inspection  evaluated  sensitivity analysis  Figure 7.5: Evaluating the decision-making process  91  Chapter 7. Empirical Evaluation We considered the following four basic types of sensitivity analysis tasks (instances of the formulate cause and effect in the PVIT model) that are important for the subject to perform: 1. What if [objx]’s weight is increased of k, and consequently [obj y]’s weight decreased of the same amount? 2. What if [objx]’s weight is increased of k, and [all other n objectives] decreased of k/n? 3. What if a component value function is changed in a numerical domain (e.g money)? 4. What if it is changed in categorical domain (e.g., neighborhood)? After each of the sensitivity analysis tasks, subjects completed a round of inspection tasks. In total, there were five rounds of inspection tasks (including one as the interface is first presented). The inspection tasks were derived based on the tasks outlined in our task model. We applied the PVIT inspection tasks to the taxonomy that Zhou & Feiner proposed to determine visual techniques [73]. We utilized their classifications in a different manner, in which we determined the experimental evaluation tasks for this part of the study. From the visual task taxonomy outlined in tables 3.3 and 3.2 (See Section 3.2), we started with the PVIT tasks and deduced visual tasks that we then mapped to the scenarios of our experiment. We considered all nine primitive inspection tasks from the PVIT model. However, since four of the inspection tasks are implied by sensitivity analysis tasks (e.g. Inspection of component value function is implied when the user is asked to perform value function sensitivity analysis), we explicitly tested only the remaining five (see Table 7.1 for an example of these five tasks mapped to the house domain).  92  Chapter 7. Empirical Evaluation What are the top 3 alternatives accord-  List the 3 highest valued houses  ing to total value? rank, identify For a specified alternative, which ob-  For HouseX, which is its strongest fac-  jective contributes to its total value the  tor according to your preferences?  most? compare, identify What is the domain value of objective  How many bathrooms are there in  x for alternative y? identify  House1?  What is the best alternative when con-  Which is the least expensive house?  sidering only objective x? rank, compare What is the best outcome for a objec-  What is the best bus-distance?  tive x? quantify, emphasize  Table 7.1: Sample inspection tasks mapped to the House domain  Tutorial and Training A very valuable insight gained from the ESA study (See Section 3.5) is that although ValueCharts appears to be a rather simple visualization technique, users still felt that VC is “pushing their learning capacity” [18]. One possible solution for this reaction is that in the ESA study, users may have had troubles more with understanding the decision process than the visualization. The subjects were presented with the visualization only after the model had been built by the experimenter. Our redesign helps to rectify this, as the steps to create the ValueCharts are presented to the user in the construction interface. According to our PVIT, the construction tasks are important for helping the user bridge the Rationale gap, or learning [3]: with respect to ValueCharts, it should help the DM learn the decision problem at hand, the model, and the decision analysis technique in general. Although we did not evaluate this part of our system, it was important to include the construction interface in order to help the user learn and understand the various aspects of the decision problem. In effect, a preliminary scripted tutorial carefully explained the decision process and AMVF while walking the user through construction with ValueCharts.  93  Chapter 7. Empirical Evaluation Participants were briefly introduced to the inspection interface with an introduction to what preferential choice decisions entail. Then the construction view was pulled up to “take apart” the ValueChart in detail. Participants were told that they would not be observed using this interface, and it was just a way to help describe the visualization. With the construction interface, the experimenter explained the objective hierarchy (factors), the given alternative data (options), specifying value function (preference) and SMARTER weighting (importance). After constructing the chart, the experimenter described the inspection interface in detail, including all the relevant interactions that could be performed. The training session was performed on the TV domain. After the experimenter demonstrated the construction phase with the TV model, the features of ValueCharts were then demonstrated, covering all the types of tasks that the subject was to complete. The subject was then given a set of tasks to perform on the given model, which were task examples of the testing phase. The experimenter explained that this was a practise session, and that time and correctness would be measured. The subject had the ability to ask the experimenter any questions at any point during the training session if needed. Part A - Procedure After the training phase, each participant had an opportunity to ask questions for clarification before the testing phase began. Subjects were reminded that time and correctness were being measured, and that this time they did not have the opportunity to ask questions. Once again, the experimenter walked the subject through the construction of the model (this time using the Hotel data set), but the testing did not start until after the ValueChart view was in place. The subject was then given a set of tasks much like in the tutorial. Each subject performed each task, writing down the answer to applicable tasks that asked a question about the data. For a complete list of training and testing tasks, see Appendices A.5 and A.6.  94  Chapter 7. Empirical Evaluation Interaction Logging A log of interactions was built in the prototype for the evaluation. A panel was added to the experiment interface as a separate floating window that the subject was able to move wherever on the screen they felt most comfortable (Figure 7.6). For the vertical interface, the panel was originally placed to the right of the ValueChart, and for the horizontal, just below. a.  c.  b.  d.  Figure 7.6: Interaction logging window  The subject was provided with a blank numbered answer sheet where he wrote down the answers to applicable tasks. When ready, the subject clicked the “begin” button (a). Timing for the first task started at this point. Inspection tasks that required a written answer were displayed in black font (b) and tasks that only required making changes (sensitivity analysis) were displayed in blue font (c). At the end of the task set, the subject clicked the button and the timing stopped (d). All interaction data and times for the total session and individual tasks were logged.  95  Chapter 7. Empirical Evaluation  7.4.4  Results  All subjects completed Part A successfully. In terms of correctness of tasks performed, there was no significant difference between the two interfaces. In fact, there were very few mistakes made during testing and the average overall mean score for ValueCharts was 18.5 (or 97.4% correct). The high percentage of correctness does give us a good indication that subjects did well. However, we could not determine if the subjects performed well overall for time to complete tasks, since there is no benchmark to compare against in this measure. Instead, we looked closely at these results to find an indication that there is a better interface for performing the tasks. The results showed that there was more variance found across subjects in time to complete tasks. However, the resulting differences were still nonsignificant. The mean time to complete all tasks was only slightly better for VC+V than VC+H for the training phase. Although still insignificant, there was more of a prominent difference seen in the testing phase, in which subjects performed better with the rotated interface. When we broke down the evaluation by task, there were similarly no significant differences found. Here we note tasks that resulted in differences closer to significance: • Test question: What is the actual SIZE of the room at Hotel3? Subjects answered this question faster in VC+V than VC+H because the weighting of SIZE at any point that this task was given was relatively small (.04), therefore the objective name was very hard to see in VC+H. Hence, subjects likely needed more time to figure out which column corresponded to the objective “Size” in VC+H (SIZE was always weighted between .04 and .10 throughout the exercise). Figure 7.7 depicts the difference in legibility between the two orientations.  96  Chapter 7. Empirical Evaluation  Figure 7.7: Lower-weighted objectives  • Test question: Which hotel has the overall worst SKYTRAIN-DISTANCE? Once again, as shown in the above example, skytrain-distance was weighted lower (always between .07 and .09) and thus less visible in VC+H. • Training question: For Television1, what is its strongest factor? Interestingly, most of the subjects who were trying to compare across objectives looked toward the display view (total score) rather than the chart. This is because all objectives are lined up closer to each other, and no missing values stand between them as they do in the chart. In some preliminary ad-hoc testing it was found that the slanting of objective names was potentially confusing to the user if placed between the chart and the display (See Figure 7.8). This may have resulted in subjects taking longer to match the alternative name to the appropriate score column in VC+V.  97  Chapter 7. Empirical Evaluation  a.  horizontal view includes the labels for the total score display.  possible confusion: lining up base of slanted text with chart column below.  does not have labels for total score display, subject must line up with chart and see what is listed at the bottom  b.  Figure 7.8: With VC+H (a), alternative labeling is on the score display, thus identifying the scores along the total display is quicker.  98  Chapter 7. Empirical Evaluation A major difference in the orientation is the persistent value function, but the results of the value function sensitivity analysis are fairly close between both interfaces. In our observations we noted that in general all subjects took longer to perform these tasks than the weighting sensitivity analysis: in the VC+H they had to recall what the value function was and how to access it, whereas subjects did not experience this problem in VC+V. They did, however, take longer to interact with the small display, and most subjects ended up opening the on-demand view.  In addition to the detailed task differences above, we noted some additional interaction issues, independent of display orientation. The pump task was performed incorrectly by several subjects: they understood the concept, but would turn on the pump and then slide the objective to the specified weighting. Possible solutions to this problem are a) have the same interaction but in different modes or b) different interactions without having to turn anything on and off. An additional problem was found in the reordering-by-objective feature: because the interaction was a double-click, pumping too fast (i.e., clicking on the header) resulted in error. Furthermore, leaving the pump function turned on proved as problematic. These should all be considered in the next iteration. The preferred method of looking up domain values was on-demand cell value (in VC+H, 2 subjects turned on the on-chart domain values, and in VC+V, 4 subjects used the summary window of domain values). We concur that the reason for this is the nature of the distinct task questions about the domain values. This was a question that did not require a comparison, so subjects were only required to look up at most one value at any given time.  7.4.5  Part A Sign Test  We have looked at the results of Part A overall, as well as individually by task. Although there are some interesting differences, none are statistically significant. The overall results for the testing phase is slightly in favor for VC+V, and a  99  Chapter 7. Empirical Evaluation closer look reveals that VC+V may in fact dominate. This conclusion can be supported by applying a two-tailed Sign Test [60] to assess whether VC+V is better than VC+H. This test measures the likelihood that the subjects performed better on the one implementation of VC over the other on m or more out of n independent measures under the null hypothesis that the two systems are equal. This test is insensitive to the magnitude of differences in each measure, noticing only which condition represents a better result. Similar studies that employ the sign test are found in Computational Linguistics [26], as well as some examples in HCI [8, 30]. In this step of our analysis we determined, for each task, what interface the subjects performed better. VC+V performed better on all five inspection tasks and also performed better on three out of the four sensitivity analysis tasks. We then applied a two-tailed Sign Test [60] to the obtained data (VC+V better 8 out of 9). The outcome of this test is that overall subjects performed significantly better with VC+V in the testing phase (p = 0.039). According to these tests we can conclude that even though there are no significant differences in training time between the two displays, subjects work better with the vertical interface after the initial training. For a similar analysis of results using the sign test, see [60].  7.5  Exploratory Study  In the first part of our study we looked closely at how subjects performed tasks that are important for effective analysis in decision-making, but they did not require to analyze their decision model in-depth. The exploratory study looks deeper into preference model analysis. A primary purpose of visualization is to generate insight [15]. We argue that the generation of insights leads to a better understanding of the domain and problem situation, thus aids in making better decisions. An effective visualization will aid the DM to see things that would otherwise go unnoticed, as well as enable her to view information about her preferences in a new light. 100  Chapter 7. Empirical Evaluation In our exploratory study we employ a simplified version of the evaluation protocol developed by Saraiya and North in [56]. The aim of this section is to evaluate ValueCharts in terms of the insight that they provide to the participants. As with Part A, we also sought to determine if there was a difference between the vertical and horizontal orientations. Insight Characteristics In Part B, we attempt to measure the amount of insight each subject gains from using ValueCharts for a particular decision-making scenario. Saraiya and North’s study for evaluating insight is very specific in the context of microbiological and microarray data [56]. Our work differs slightly from theirs: in their study the subjects had extensive domain knowledge, whereas our study is more general, as we approached this part in a more domain-independent manner since there are different domains that the subjects can choose from (See Chapter 7.5.1). We use their definition of an insight as an individual observation about the data by the participant, a unit of discovery. In terms of our model, we consider the DM’s preferences and weighting as part of the data observed. The “characteristics of an insight” from [56] were intended to serve as a set that could be applied in various domains. Since the characteristics were meant to be further defined and mapped to specific domains, some modifications and generalizations were made in our study. The following is our characterization of insight as applied to preferential choice: • Fact: The actual finding about the data (e.g. “Samsung [cell phones] are the smallest”) • Value: How to measure each insight? We determined and coded the value of each insight from 1 - 3, whereas simple observations of domain value and top ranking (e.g. “cheapest place is in East Van”) are fairly trivial,  101  Chapter 7. Empirical Evaluation and more global observations regarding relationships and comparison (e.g. “more expensive phones have all the features”) are more valuable. • Category: Insights were grouped into several categories: – Simple fact: an alternative rank or identification of domain value e.g. “This phone is fairly light”, “This phone is only [ranked] fourth for battery” – Sensitivity: how a change affects the results e.g. “This house again!”, “Now this phone is third” – Realization of personal preferences: users often stated that they made a realization about their preferences e.g. “it makes sense, because I really like hiking and nature”, “brand should be more important [to me]” These categories were defined after the experiment, and the grouping closely lends to the value coding.  7.5.1  Domain Data Sets  In order to ensure that the users had the capability to determine insightful facts about the information presented to them, it was important that they had a genuine interest in the domain that was studied. The participants were asked to choose 1 among 3 different decision problems. Each of the decisions included a scenario in the following domains: 1. House Rental Data was taken loosely from current postings on AMS Rentsline, where any missing information was fabricated. General information, such as Rent, Location, type, etc, were consistently available, but other more detailed information (bedroom size by sq-ft) was often fabricated. The scenario is that the DM goes to school at UBC and would like to move off campus. It is assumed that the DM is only considering Point Grey, Kitsilano, Downtown, and EastEnd. 102  Chapter 7. Empirical Evaluation The House Rental decision problem contained: • 13 objectives • 10 alternatives 2. Cell Phone Data was taken from Rogers Video website, and there were only a few cases of missing information. The information was narrowed down to 17 primitive objectives, and anything the participant was looking for (i.e. text-messaging) was assumed to be a feature included in all phones. The scenario is that the DM is looking for a phone from Rogers Wireless based on a 3-year plan (as prices were quoted). The Cell Phone decision problem contained: • 17 objectives • 12 alternatives 3. Tourism In this situation, the data was taken from Tourism Vancouver Official Visitor’s Guide ’05-06. The alternatives were narrowed down to those listed as being Downtown, East End, West End, and North Van. The scenario is that the DM is looking to take a visiting friend to a local tourist attraction. Alternatives were further categorized as type (scenic, historic, etc.), and indoor/outdoor. Cost was assumed as average/adult. The Tourism decision problem contained: • 17 objectives • 12 alternatives Each scenario was explained to the participants, and they were asked to choose which one they would like to work with. Further explanation of the data was provided as explained in the next section.  103  Chapter 7. Empirical Evaluation  7.5.2  Procedure  At this point of the experiment, subjects have already undergone considerable training and practise from Part A. Since the construction interface was not tested in the controlled study, experimenters worked with the subjects to build the initial decision model. Objectives were presented to them in a pre-existing hierarchy with all available factors, and were told to remove and rearrange as they pleased (additions were not allowed since data set was fixed and could not be extended). To set their initial preference model, they were instructed to go through the list of objectives and set the value function of each one to reflect their true preferences. Default functions were provided, where typically linear continuous functions were given (i.e. positive for battery talk time, negative for price), and each discrete objective was set with a best, worst, and 0.5 for others. Finally, the subjects ranked the objectives with the SMARTER weighting technique (see Chapter 6.2). Their resulting decision model was then presented on the ValueChart. The subject was asked to use the interface to analyze the decision model, perform any sensitivity analysis changes as they see fit, and view any information that they required. They were instructed to work with the interface to make a decision about the data, where the decision could be to select one or more preferred alternative. Subjects were asked to “think aloud” as they analyzed the preference model, being sure to let the experimenter know anything interesting that they saw. Notes were taken by the experimenter, and interaction logging was turned on once the ValueChart was created. The subject was asked to take as little or as much time as needed in order to reach a decision. If finished quickly, the experimenter would probe, but end the session if the subject was satisfied with the decision. The time for the experiment (total of both Part A and B) was 60 minutes, and if subjects were approaching the 60 minute mark, was warned by the experimenter but welcomed to stay until as long as the 75 minute mark.  104  Chapter 7. Empirical Evaluation At the end of the exercise the subject was asked what their decision was, and to keep in mind when answering the post-experiment questionnaire.  7.5.3  Results  It appeared that every subject had a genuine interest in the domain that they chose (10 cell phone, 6 tourism, and 4 house), but they varied in the degree of interest. Overall all subjects were able to use the tool and conclude on a best decision. Subjects went through the construction phase carefully. The time spent inspecting the interface (minus construction) ranged from 3-16 minutes. The number of insights ranged from 0 to 10. Comparison between interfaces Figure 7.9 summarizes two measures of insight gained and usage time, illustrating the two different interfaces. It shows a) mean number of insights acquired, b) the mean sum of value for all insight occurrences, and c) the average total time each subject spent using the tool until they felt that they reached a decision. Statistical analysis indicates that there are no significant differences, despite the fact that there appeared to be a great difference in the mean insights and value (49% and 34% more, respectively). Because of these noteworthy differences we also measured effect sizes (the magnitude of the differences) to determine the practical significance of the differences. Cohen’s d [20] provides a standardized measure of the mean difference between two treatments. In this measure d > 0.8 is considered to be a large effect, 0.8 > d > 0.5 to be a medium effect, and d < 0.5 to be a small effect. We found the effect sizes of the insight count and insight value to be 0.40 and 0.51 respectively. So, although our results are not statistically significant, according to Cohen’s criteria, using VC+V has a medium effect on the value of insights reported by our participants. Since the evaluation method is more qualitative and subjective than quantitative, general comparison of the tendencies in the results is also appropriate.  105  Chapter 7. Empirical Evaluation  Insight Count  3  sd=5.0  mean=3.5  Insight Value  Insight Count  4  Insight Value mean=4.7  5  sd=6.2  2 1 0  VC-H  VC-V  10 9 8 7 6 5 4 3 2 1 0  mean=8.8 sd=2.9 mean=5.9 sd=3.2  VC-H  VC-V  Total time mean=9.93 10.00 9.00 8.00  Time  7.00  mean=8.69  sd=2.8  sd=5.3  6.00 5.00 4.00 3.00 2.00 1.00  VC-H  VC-V  Figure 7.9: Insight Results  There were more insights counted for the vertical interface, which also fared better when value factor was considered. Looking more closely at the interaction logs reveal that subjects tended to perform more sensitivity analysis on VC+V, which in turn led to more insights on sensitivity. There were 89% more sensitivity analysis of value function performed on the vertical interface than the horizontal. We conclude that the reason for this is that the persistent view a) acts as a reminder of what the value function is and that it can be changed and b) is more inviting for users to directly manipulate value function. We hypothesize that there is a benefit from the persistent view of the component value functions, but may revisit the persistent sensitivity analysis technique in future iterations. More time was spent on the vertical interface. In contrast to time measurement in Part A that we used to gauge performance of lower-level tasks, more time spent performing the overall task of making a decision can not be viewed as negative. In fact, the general trend was that the more time spent by the  106  Chapter 7. Empirical Evaluation subject on the decision problem, more insights were reported. It should be noted that, regardless of the interface, the results were very mixed. Some subjects did not have any insights, and some had many. The standard deviation was high overall (see Figure 7.9). Individual differences were more apparent in this part (versus Part A) because subjects’ personalities could affect the amount of insights reported (a challenge of the think-aloud technique [25, 47]). In addition, the possible varying level of interest in each subject’s selected domains may contribute to this variance. Nonetheless, we believe that providing the subject with a selection of domains helped with degree of interest. A more extensive study might specify a single domain and recruit participants with a specific requirement (e.g. recruit participants who are in the market for a new cell phone, and plan to purchase or upgrade in the next month). Post-study Following the exploratory study, we completed the session by asking the subject a number of open questions and having them fill out a post-experiment questionnaire. They were asked to answer each question by selecting the degree of agreement of the statement from 1 to 5 where 1 is strongly disagree and 5 is strongly agree. Four questions were specific to the exercise they performed in PartB. All subjects were generally satisfied with the decision that they made (mean= 4.25), although their level of confidence was slightly lower overall (mean=3.95). A closer look shows that 4 out of 6 of the subjects who gave this a 3 or “neutral” rating had 3 or less insights, and the final decision of the remaining 2 were multiple options. This analysis further supports the assumptions made in Part B that more (insights, time, interaction) is better. Subjects were also asked their level of agreement to the statement “I learned a lot about my preferences in [selected domain]”. Interestingly, we found that there is a significant positive correlation between the rating of this question and insight.  107  Chapter 7. Empirical Evaluation Finally, subjects were asked the extent to which their final decision accurately reflects their initial preferences. It was not surprising to see that subjects who were surprised to see some results of their preference model analysis gave a lower rating (3 - neutral) for this question. Subject #8 was among the few who rated this question as neutral: she reported insights about how she understood that PNE (Pacific National Exhibition) was rated low because it was only available for specific seasons, but she did not change her preferences to reflect this realization (either by removing that objective from the decision model, weighting it lower, or changing the value function). Figure 7.10 illustrates the results of the 4 questions broken down by interface. I am satisfied with the decision I made  I am confident about the decision I made  8  8  7  7 VC-V mean=4.3 VC-H mean=4.2  5 4 3 2  overall mean = 4.25  6 subjects  subjects  6  3neutral  4agree  0  5 strongly agree  1 0  12strongly disagree disagree  3neutral  4agree  5 strongly agree  My solution reflects my intial preferences  7 VC-V mean=3.9 VC-H mean=4.0  overall mean = 3.95  VC-V mean=4.4 VC-H mean=3.8  6 subjects  subjects  overall mean = 3.95  8  8 7  4 3 2  3 1  21strongly disagree disagree  I learned a great deal about my preferences in <selected domain>  6 5  4 2  1 0  VC-V mean=4.0 VC-H mean=3.9  5  5 4 3  overall mean = 4.10  2 1 12strongly disagree disagree  3neutral  4agree  5 strongly agree  0  12strongly disagree disagree  3neutral  4agree  5 strongly agree  Figure 7.10: Results of Part B Post-Study questions  108  Chapter 7. Empirical Evaluation  7.6  Evaluation Summary  We addressed some challenges of information visualization evaluation (See Chapter 3) in several manners. First and foremost, we developed a taxonomy of tasks that should be used as a benchmark framework for design and evaluation of visualization techniques for preferential choice. In addition, we used a triangulation of methods that included a controlled experiment and an exploratory study in which we a) considered visual tasks beyond locate and identify, b) analyzed tasks individually, c) matched users with real data in realistic scenarios and d) included a measure of insight. We looked at ValueCharts in several angles with this evaluation. First, we assessed how the subjects performed on the low-level tasks. On average the subjects performed well in correctness, varying to some degree in length of time spent to complete the tasks. In turn, when asked to perform the high-level task of making a decision with our tool, the subjects reported that they were quite satisfied with their decision. These results corroborate our claim that if an interface supports the lower level tasks of PVIT well, then the interface also will enable the higher level tasks of the model. We pruned our tree to focus more on the basic tasks of inspection and sensitivity analysis, which more directly support the higher level task of decision-making. Since subjects were generally satisfied with the construction phase as well, it added to the success of ValueCharts and PVIT as tools for preferential choice. Although the construction interface was not evaluated formally, it played a big part in training and better understanding ValueCharts in the exploratory study. All subjects thought that the construction interface was helpful in their understanding of the decision process and of the ValueCharts display. Only 1 subject had some initial troubles with understanding the SMARTER weighting technique when constructing her model in Part B. Since the construction phase plays a large part in supporting the higher level cognitive task of learning (bridging the Worldview Gap), we can attribute the results of high rating of the amount learned (mean = 4.20) at least partly to the success of the construction 109  Chapter 7. Empirical Evaluation interface. We intend to conduct further studies of the construction interface in future evaluations. ValueCharts overall, regardless of which orientation, was very well-received. All subjects thought that it is useful, intuitive, easy to use and quick to learn. In particular, subjects rated the usefulness very high (mean = 4.40), and strongly agree that visualizing their preferences helps in their understanding of the decision (mean = 4.45). This is a useful tool for making decisions  Visualizing my decision model helps me understand it more clearly  8  8  7  7 6 VC-V mean=4.5 VC-H mean=4.3  5 4  overall mean = 4.40  3  subjects  subjects  6  2  VC-V mean=4.3 VC-H mean=4.6  overall mean = 4.45  4 3 2 1  1 0  5  12strongly disagree disagree  3neutral  4agree  0  5 strongly agree  1strongly disagree  2disagree  3neutral  4agree  5 strongly agree  I learned a great deal about how to analyze my decision model 8 7  subjects  6 VC-V mean=4.4  5 4 3  overall mean = 4.20  VC-H mean = 4.0  2 1 0  12strongly disagree disagree  3neutral  4agree  5 strongly agree  Figure 7.11: Results of general post-study questions  Some of the evidence that we have collected suggest that the vertical and horizontal ValueCharts designs are not equivalent interfaces since a) the Sign test indicates that subjects perform better on the VC+V than VC+H on low level tasks, and b) VC+V has a medium effect on insight value as we explored how subjects performed the higher level task of decision making. However, the lack of statistical significance for the difference in insights (count and value) indicates the need for a larger experiment.  110  Chapter 7. Empirical Evaluation Nonetheless, our evaluation results are very promising. What we intend to do is offer both orientations of ValueCharts. Since users did perform better with the vertical interface (according to the Sign test p = 0.019 for low level tasks and Cohen’s d = 0.51 for insight value) we find these results adequate to include VC+V as the default interface of ValueCharts.  111  Chapter 8  Conclusions and Future Work In this thesis we presented the Preferential choice Integrated Task model, and used it to redesign ValueCharts and to evaluate the design both analytically and empirically.  8.1  Conclusion  We identified the need for a comprehensive task framework that integrates principles from both Information Visualization and Decision Theory. We conducted a task analysis based on literature from these fields, and developed the Preferential choice Visualization Integrated Task model. The PVIT model assisted us substantially in the redesign of ValueCharts by enabling us to look closely at the previous design in comparison with other techniques. Furthermore, an empirical evaluation of our new design, based on PVIT, helped us explore the design more thoroughly, incorporating a controlled usability study and an insight-based observational study. Results are promising from the mixture of different evaluation techniques and indicate that the redesign was successful.  8.2  Future Work  Although our redesign incorporated most tasks of PVIT, the summary in Figure 6.9 also conveys that our system still lacks support for a number of tasks. Future 112  Chapter 8. Conclusions and Future Work work will address these shortcomings. In particular, we would like to investigate how to represent uncertainty by displaying missing data, consider methods to incorporate a computational display of sensitivity analysis, and also study ways to integrate known data exploration techniques into the construction phase. Furthermore, we intend to address some of the design issues encountered in the empirical evaluation as well as conduct further studies of the construction interface. Finally, we identified the need to consider conducting a more extensive experiment using a larger pool of subjects and focusing on a single domain with participants screened for specific requirements. Our Preferential choice Integrated Task model helped to identify shortcomings in our original design, guided our new redesign, and will lead the way into our future design. We hope that our model will be used as a guideline for future proposals of interactive visualization techniques for analysis of preference models.  113  Bibliography [1] Christopher Ahlberg. Spotfire: an information exploration environment. SIGMOD Rec., 25(4):25–29, 1996. [2] Christopher Ahlberg and Ben Shneiderman. Visual information seeking using the filmfinder. In CHI ’94: Conference companion on Human factors in computing systems, pages 433–434, New York, NY, USA, 1994. ACM Press. [3] Robert Amar and John Stasko. A knowledge task-based framework for design and evaluation of information visualizations. In INFOVIS ’04: Proceedings of the IEEE Symposium on Information Visualization (INFOVIS’04), pages 143–150, Washington, DC, USA, 2004. IEEE Computer Society. Best Paper. [4] Gennady L. Andrienko and Natalia V. Andrienko. Constructing parallel coordinates plot for problem solving. In P. Oliver A. Butz, A. Krueger and M. Zhou (Eds.), editors, 1st International Symposium on Smart Graphics, pages 9–14. ACM Press, 2001. [5] Gennady L. Andrienko and Natalia V. Andrienko. Interactive visual tools to support spatial multicriteria decision making. In UIDIS 2001, pages 127–131. IEEE Computer Society Press, 2001. [6] Natalia V. Andrienko and Gennady L. Andrienko. Informed spatial decisions through coordinated views. In Information Vizualization, volume 2, pages 270–285. Palgrave MacMillan, 2003.  114  Chapter 8. Conclusions and Future Work [7] Toshiyuki Asahi, David Turo, and Ben Shneiderman. Visual decisionmaking: using treemaps for the analytic hierarchy process. In CHI ’95: Conference companion on Human factors in computing systems, pages 405– 406, New York, NY, USA, 1995. ACM Press. [8] Ragnar Bade, Felix Ritter, and Bernhard Preim. Usability comparison of mouse-based interaction techniques for predictable 3d rotation. In Smart Graphics, pages 138–150, 2005. [9] F. Hutton Barron. The efficacy of smarter simple multi-attribute rating technique extended to ranking. Acta Psychologica, 93(1-3):23–36, 1996. [10] Jeanette Bautista and Micheline Manske. Ufinder: An interactive tool for data exploration and decision-making. Unpublished: class project for CPSC533C, 2004. [11] Valerie Belton. Visa visual interactive sensitivity analysis. SIMUL8 Corporation, 2007. http://www.simul8.com/products/visa.htm. [12] Valerie Belton and S.P Vickers. Use of a Simple Multi-Attribute Value Function Incorporating Visual Interactive Sensitivity Analysis for Multiple Criteria Decision Making, pages 319–334. Readings in Multiple Criteria Decision Aid. Springer Verlag, 1990. [13] Jim Blythe. Visual exploration and incremental utility elicitation. In Eighteenth national conference on Artificial intelligence, pages 526–532, Menlo Park, CA, USA, 2002. American Association for Artificial Intelligence. [14] Dominique Brodbeck and Luc Girardin. Visualization of large-scale customer satisfaction surveys using a parallel coordinate tree. In IEEE 2003 Symposium on Information Visualization, pages 197–201, 2003. [15] Stuart K. Card, Jock D. Mackinlay, and Ben Shneiderman. Readings in information visualization: using vision to think. Morgan Kaufmann Publishers Inc., 1999. 115  Chapter 8. Conclusions and Future Work [16] Giuseppe Carenini and John Loyd. Valuecharts: analyzing linear models expressing preferences and evaluations. In Proceedings of the working conference on Advanced visual interfaces, pages 150–157. ACM Press, 2004. [17] Giuseppe Carenini and David Poole. Constructed preferences and valuefocused thinking: Implications for ai research on preference elicitation. In AAAI02 Workshop on Preferences in AI and CP: symbolic approaches, 2002. [18] Vic Chung. Extending value charts to support step-wise linear utility sensitivity analysis. Unpublished: class project for CPSC448B, 2005. [19] Robert T. Clemen. Making Hard Decisions. Duxbury Press, 2nd edition, 1996. [20] Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences (2 ed.). Lawrence Earlbaum associates. [21] Brock Craft and Paul Cairns. Beyond guidelines: What can we learn from the visual information seeking mantra? In 9th Annual International Conference on Information Visualisation, 2005. To be published. [22] Alan Dix, Janet E. Finlay, Gregory D. Abowd, and Russell Beale. HumanComputer Interaction (3rd Edition). Prentice Hall, 2003. [23] J. S. Dyer. Remarks on the analytic hierarchy process. Manage. Sci., 36(3):249–258, 1990. [24] Ward Edwards and F. Hutton Barron. SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement. Organizational Behavior and Human Decision Processes, 60(4):306–325, 1996. [25] K. Anders Ericsson and Herbert A. Simon. Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA, 1984.  116  Chapter 8. Conclusions and Future Work [26] Barbara Di Eugenio, Michael Glass, and Michael J. Trolio. The DIAG experiments: Natural language generation for intelligent tutoring systems. In The Second International Natural Language Generation Conference. [27] Georges Trutschl.  Grinstein,  Ursa  Cvek,  Mark  Derthick,  and  Marjan  IEEE InfoVis 2005 Contest, Technology Data in the US.  http://ivpr.cs.uml.edu/infovis05. [28] Raimmo P. H¨am¨al¨ ainen and Hannu Lauri. Hipre 3+: Decision support software. System Analysis Laboratory, Helsinki University of Technology, 2007. http://www.eia.fi/hannul/hipre.html. [29] John S. Hammond, Ralph L. Keeney, and Howard Raiffa. Smart Choices: A Guide to Making Better Decisions. Harvard University Press, Boston, 1999. [30] Ken Hinckley, Randy Pausch, Dennis Proffitt, James Patten, and Neal Kassell. Cooperative bimanual action. In CHI ’97: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 27–34, New York, NY, USA, 1997. ACM Press. [31] HiView.  Hiview 3: enabling effective decisions.  Catalyze Ltd., 2007.  http://www.catalyze.co.uk/hiview/hiview.html. [32] Kasper Hornbaek, Benjamin B. Bederson, and Catherine Plaisant. Navigation patterns and usability of zoomable user interfaces with and without an overview. ACM Trans. Comput.-Hum. Interact., 9(4):362–389, 2002. [33] Logical Decisions Inc. Logical decisions for windows, version 5.1. Logical Decisions, Inc., 2007. http://www.logicaldecisions.com/. [34] Alfred Inselberg and Bernard Dimsdale. Parallel coordinates: a tool for visualizing multi-dimensional geometry. In Proceedings of the 1st conference on Visualization ’90, pages 361–378. IEEE Computer Society Press, 1990.  117  Chapter 8. Conclusions and Future Work [35] Piotr Jankowski, Natalia Andrienko, and Gennady Andrienko.  Map-  centered exploratory approach to multiple criteria spatial decision making. International Journal Geographical Information Science, 15:101–127, 2001. [36] Wesley Johnston. Model Visualization, pages 223–228. Information Visualization in Data Mining and Knowledge Discovery. Morgan Kauffman, 2001. [37] Ralph L. Keeney, John S. Hammond, and Howard Raiffa. Smart Choices: A Guide to Making Better Decisions. Harvard University Press, Boston, 1999. [38] Ralph L. Keeney and Howard Raiffa. Decisions with multiple objectives: Preferences and value tradeoffs. J. Wiley, New York, 1976. [39] R.L. Keeney. Value-Focused Thinking: A Path to Creative Decision Making. Harvard University Press, Cambridge, 1992. [40] Alfred Kobsa. An empirical comparison of three commercial information visualization systems. In INFOVIS ’01: Proceedings of the IEEE Symposium on Information Visualization 2001 (INFOVIS’01), page 123, Washington, DC, USA, 2001. IEEE Computer Society. [41] Robert Kosara, Helwig Hauser, and Donna L. Gresh. An interaction view on information visualization. In Eurographics 2003, 2003. [42] Robert Kosara, Christopher G. Healey, Victoria Interrante, David H. Laidlaw, and Colin Ware. User studies: Why, how, and when? IEEE Computer Graphics and Applications, 23(4):20–25, 2003. [43] Tom Lanning, Kent Wittenburg, Michael Heinrichs, Christina Fyock, and Glenn Li. Multidimensional information visualization through sliding rods. In AVI ’00: Proceedings of the working conference on Advanced visual interfaces, pages 173–180, New York, NY, USA, 2000. ACM Press.  118  Chapter 8. Conclusions and Future Work [44] Bongshin Lee, Catherine Plaisant, Cynthia Sims Parr, Jean-Daniel Fekete, and Nathalie Henry. Task taxonomy for graph visualization. In BELIV ’06: Proceedings of the 2006 conference on BEyond time and errors, pages 1–5, New York, NY, USA, 2006. ACM Press. [45] E. Morse, M. Lewis, and K. A. Olsen. Evaluating visualizations: using a taxonomic guide. Int. J. Hum.-Comput. Stud., 53(5):637–662, 2000. [46] J. Mustajoki and R.P. H¨ am¨al¨ ainen. Web-hipre - a java applet for ahp and value tree analysis. In 5th International Symposium on the Analytic Hierarchy Process (ISAHP’99), pages 411–416. IEEE Computer Society, 1999. [47] Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. Getting access to what goes on in people’s heads?: reflections on the think-aloud technique. In NordiCHI ’02: Proceedings of the second Nordic conference on Humancomputer interaction, pages 101–110, New York, NY, USA, 2002. ACM Press. [48] J. W. Payne, J. R. Bettman, and E. J.Johnson. The Adaptive Decision Maker. Cambridge University Press, 1993. [49] J. W. Payne, J. R. Bettman, and D. A. Schkade. Measuring constructed preference: Towards a building code. Journal of Risk and Uncertainty, 19:243–270, 1999. [50] Catherine Plaisant. The challenge of information visualization evaluation. In AVI ’04: Proceedings of the working conference on Advanced visual interfaces, pages 109–116, New York, NY,, 2004. ACM Press. [51] Catherine Plaisant, Jesse Grosjean, and Benjamin B. Bederson. Spacetree: Supporting exploration in large node link tree, design evolution and empirical evaluation. In INFOVIS ’02: Proceedings of the IEEE Symposium on Information Visualization (InfoVis’02), page 57, Washington, DC, USA, 2002. IEEE Computer Society. 119  Chapter 8. Conclusions and Future Work [52] Mari Poyhonen and Raimo P. Hamalainen. On the convergence of multiattribute weighting methods. European Journal of Operational Research, 129(3):569–585, 2001. [53] Jennifer Preece, Yvonne Rogers, and Helen Sharp. Interaction Design: Beyond Human-Computer Interaction. Wiley, 2002. [54] Pearl Pu and Boi Faltings. Enriching buyers’ experiences: the smartclient approach. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 289–296. ACM Press, 2000. [55] Pearl Huan Z. Pu and Pratyush Kumar. Evaluating example-based search tools. In EC ’04: Proceedings of the 5th ACM conference on Electronic commerce, pages 208–217, New York, NY, USA, 2004. ACM Press. [56] Purvi Saraiya, Chris North, and Karen Duca. An evaluation of microarray visualization tools for biological insight. In INFOVIS ’04: Proceedings of the IEEE Symposium on Information Visualization (INFOVIS’04), pages 1–8, Washington, DC, USA, 2004. IEEE Computer Society. [57] Ben Shneiderman. Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions in Graphics, 11(1):92–99, 1992. [58] Ben Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In VL ’96: Proceedings of the 1996 IEEE Symposium on Visual Languages, page 336, Washington, DC, USA, 1996. IEEE Computer Society. [59] Ben Shneiderman and Catherine Plaisant. Designing the User Interface : Strategies for Effective Human-Computer Interaction (4th Edition). Addison Wesley, 2004. [60] Sidney Siegel and N. John Jr. Castellan. Nonparametric statistics for the behavioral sciences. McGraw Hill, 1988.  120  Chapter 8. Conclusions and Future Work [61] Robert Spence and Lisa Tweedie. The attribute explorer: information synthesis via exploration. Interacting with Computers, 11(2):137–146, 1998. [62] Michael Spenke, Christian Beilken, and Thomas Berlage. Focus: the interactive table for product comparison and selection. In UIST ’96: Proceedings of the 9th annual ACM symposium on User interface software and technology, pages 41–50, New York, NY, USA, 1996. ACM Press. [63] Peter Todd and Izak Benbasat. Inducing compensatory information processing through decision aids theat facilitate effort reduction: An experimental assessent. Journal of Behavioural Dcision Making, 13:91–106, 2000. [64] Edward Tufte. Envisioning Information. Graphics Press, 1990. [65] Eliane R. A. Valiati, Marcelo S. Pimenta, and Carla M. D. S. Freitas. A taxonomy of tasks for guiding the evaluation of multidimensional visualizations. In BELIV ’06: Proceedings of the 2006 conference on BEyond time and errors, pages 1–6, New York, NY, USA, 2006. ACM Press. [66] Detlov von Winterfeldt and Ward Edwards. Decision Analysis and Behavioral Research. Cambridge University Press, 1986. [67] Colin Ware. Information visualization: perception for design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2000. [68] Stephen Wehrend and Clayton Lewis. A problem-oriented classification of visualization techniques. In VIS ’90: Proceedings of the 1st conference on Visualization ’90, pages 139–143, Los Alamitos, CA, USA, 1990. IEEE Computer Society Press. [69] Christopher Williamson and Ben Shneiderman. The dynamic Homefinder: evaluating dynamic queries in a real-estate information exploration system. In SIGIR ’92: Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, pages 338–346, New York, NY, USA, 1992. ACM Press. 121  Chapter 8. Conclusions and Future Work [70] Chauncey E. Wilson. Triangulation: the explicit use of multiple methods, measures, and approaches for determining core issues in product development. interactions, 13(6):46–ff, 2006. [71] Kent Wittenburg, Tom Lanning, Michael Heinrichs, and Michael Stanton. Parallel bargrams for consumer-based information exploration and choice. In UIST ’01: Proceedings of the 14th annual ACM symposium on User interface software and technology, pages 51–60, New York, NY, USA, 2001. ACM Press. [72] Enrique G. Zapatero, Charles H. Smith, and H. Roland Weistroffer. Evaluating multiple-attribute decision support systems.  Journal of Multi-  Criteria Decision Analysis, 6:201–214, 1997. [73] Michelle X. Zhou and Steven K. Feiner. Visual task characterization for automated visual discourse synthesis. In CHI ’98: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 392–399, New York, NY, USA, 1998. ACM Press/Addison-Wesley Publishing Co.  122  Appendix A  Additional Material A.1  Pre-study questionnaire  Subject #: Date: 1. Age group: ✷ 19 and under ✷ 20-29 ✷ 30-39 ✷ 40-49 ✷ 50+  2. Gender: ✷ male ✷ female  3. Computer use: ✷ rare ✷ less than 10 hours/week ✷ 10-19 hours/week ✷ 20-29 hours/week ✷ 30-49 hours/week ✷ 50+ hours/week  123  Appendix A. Additional Material 4. Experience with decision analysis ✷ first time ✷ have learned and understand concepts/theories ✷ regularly practice decision analysis methods and techniques  5. Experience with decision analysis tools ✷ none ✷ pros/cons list, pen & paper ✷ spreadsheet ✷ have used tools before ✷ use tools regularly  A.2  Post-study questionnaire  Subject #: Date: Section 2: To be completed after task study 1. With respect to your experience with ValueCharts, please indicate (by circling a number from 1-5) the extent to which you agree or disagree with the following statements. Where: 1 = Strongly Disagree  3 = Neutral  4 = Agree  2 = Disagree  5 = Strongly Agree  The decision a) I am satisfied with the decision I made. 1  2  3  4  5  b) I am confident about the decision I made. 124  Appendix A. Additional Material 1  2  3  4  5  c) I learned a great deal about how to analyze my decision model 1  2  3  4  5  d) I learned a lot about my preferences in <selected domain>. 1  2  3  4  5  4  5  The tool/process e) a useful tool for making decisions. 1  2  3  f) The system was flexible in allowing me to make changes during the decision process. 1  2  3  4  5  g) My solution reflects my initial preferences. 1  2  3  4  5  2. What I liked most about the system was:  3. What I liked least about the system was:  4. Other remarks:  125  Appendix A. Additional Material  A.3  Consent Form  126  Appendix A. Additional Material  127  Appendix A. Additional Material  A.4  BREB Approval  128  Appendix A. Additional Material  A.5  Training tasks  1. List the top 3 alternatives by total score. 2. For TV1, what is its strongest factor? 3. What is the actual size of TV3? 4. Increase the weighting of CONDITION to 25, decreasing PRICE accordingly. 5. List the top 3 alternatives by total score. 6. Which TV has the overall best COST? 7. What is the actual CONDITION of TV4? 8. Change the Value Function of CONDITION so that a TV rated as GOOD is valued at 0.7. 9. List the top 3 alternatives by total score. 10. For TV6, what is its strongest factor? 11. What are the best and worst CONDITIONS? 12. Increase the weighting of PRICE by 5, while decreasing ALL OTHER OBJECTIVES accordingly. 13. List the top 3 alternatives by total score. 14. Which TV has the least preferred price? 15. Is delivery included with TV6? 16. Change the Value Function of SIZE so that the preferability of 27¨ıs 1.0 and that of a 32¨ıs 0.0. 17. List the top 3 alternatives by total score. 18. Which TV has the overall worst quality? 19. What is the most preferred delivery type?  129  Appendix A. Additional Material  A.6  Testing tasks  1. List the top 3 alternatives by total score. 2. For Hotel3, what is its strongest factor? 3. What is the actual SIZE of the room at Hotel1? 4. Increase the weighting of INTERNET ACCESS to 30, decreasing AREA accordingly. 5. List the top 3 alternatives by total score. 6. Which hotel has the overall best LOCATION? 7. What is the RATE at Hotel4? 8. Change the Value Function of RATE so that anything $125 and lower is valued at 1.0. 9. List the top 3 alternatives by total score. 10. What alternative has the worst SKYTRAIN DISTANCE? 11. What area is Hotel1 in? 12. Increase the weighting of room SIZE to 10, while decreasing ALL OTHER FACTORS accordingly. 13. List the top 3 alternatives by total score. 14. For Hotel2, what is its strongest factor? 15. What is the most preferred type of a room’s INTERNET ACCESS? 16. item Change the Value Function of AREA so that the 2nd choice is EastEnd (0.5) and Uptown is the worst possibility (0). 17. List the top 3 alternatives by total score. 18. Which hotel has the overall worst LOCATION? 19. What are the best and worst SKYTRAIN distances?  130  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0051291/manifest

Comment

Related Items