Evaluating Design Review Policies in Santa Barbara, California by Jason Edward Smart B.A., The University of California, Santa Barbara, 2005 A PROJECT SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS (PLANNING) in THE FACULTY OF GRADUATE STUDIES School of Community and Regional Planning We accept this project as conforming to the required standard ...................................................... ..................................................... ..................................................... THE UNIVERSITY OF BRITISH COLUMBIA August 2009 © Jason Edward Smart, 2009 i Acknowledgements I am grateful for the input of my supervisor, Dr. Tim McDaniels, and second reader, Dr. Maged Senbel. Heather Baker and Jaime Limon of the City of Santa Barbara Community Development Department provided important assistance, as did the several Santa Barbara citizens who shared with me their time and insights. ii Executive Summary Project Purpose and Scope The purpose of this final Masters degree project is to provide information that could assist the City of Santa Barbara in deciding how to review its Neighborhood Preservation Ordinance/Single Family Design Guidelines Update (the NPO Update). The NPO Update was an effort to improve the city’s single-family residential housing design review policies and processes. The specific objectives of this project are: • Clarify the review decision problem. • Clarify the objectives of the NPO Update and the objectives of the NPO Update review. • Present possible alternative review topics and approaches and discuss their relative advantages and disadvantages. To address the above objectives, interviews were conducted with key stakeholders in Santa Barbara who were asked their opinions regarding the review. The project is also informed by city documents, relevant academic literature, and my personal experiences working as an intern for the city during a portion of the NPO Update process. This project does not attempt to determine whether the NPO Update has been successful, nor does it suggest changes to the City’s existing NPO policies and practices. NPO Update and Review Background The NPO Update Work Program states that the goal of the update was to “achieve quality, single-family residential project design compatible with existing neighborhoods through efficient design review” (City of Santa Barbara 2004b). Some new, expanded, or rebuilt homes in Santa Barbara were considered visually prominent and were larger than the surrounding homes in established neighborhoods. This phenomenon is sometimes referred to as “mansionization,” and the resulting homes are sometimes labeled “monster homes” or “McMansions” and may be criticized for their perceived aesthetic, privacy, or other impacts (Szold 2005). The NPO Update incorporated numerous changes to city regulations, guidelines, and review procedures, including the creation of maximum, mandatory floor to lot area ratio (FAR) requirements for some homes as well as a Single Family Design Board to review single-family home design. Upon approving iii these changes, the Santa Barbara City Council directed the city’s planning staff to conduct a review of the NPO Update in 2009. The City Council did not give staff any formal direction regarding the review. Thus there is a need to determine what such a review would entail and how it would be conducted. Monitoring and Evaluation Background The NPO Update review could be considered a policy or program “evaluation.” Weiss (1998, 4) defines “evaluation” as “the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.” To Seasons (2003), “monitoring” implies a continuous evaluation in which data is collected and interpreted on a regular basis. Seasons uses the term M&E (monitoring and evaluation) to encompass evaluative activities and observes that these activities can be informal and irregular in actual municipal planning practice. There are at least three general purposes of M&E, according to Patton and Sawicki. A first purpose may be to ensure that policies are being implemented as intended. A second review purpose could be to determine whether policies are having the desired impact and, if not, modify or terminate the policies in order to improve the outcomes. An additional evaluation purpose may be to learn from the policies and the policy-making process in order to improve future planning and decision-making efforts. An evaluation of the policy-making process can include both how a policy study was conducted and presented to decision makers, as well as how decision makers used the study (1993, 363-4). Municipal planners face challenges when evaluating policies or plans, including determining the objectives of the policies, and otherwise defining success. In addition, there can be difficulty establishing a causal relationship between planning activities and actual outcomes. Other factors constraining evaluation include limited resources such as time (Seasons 2003). Nevertheless, evaluation may still help to improve policy and program effectiveness. iv Project Approach and Methods This project applies a structured decision-making framework (as presented in Hammond, Keeney, and Raiffa 2002) to the problem. This framework contains the following elements: • Define the decision problem. • Clarify the goals and objectives. • Find alternatives to achieve the goals and objectives. • Understand how each alternative performs in terms of the goals and objectives. • Identify the tradeoffs between the goals and objectives associated with each alternative. This structured approach is often described as “rational,” although there are clearly limits to how rational a decision-making approach can be (Marsh 1994, 9). This type of approach helps divide a potentially complex decision into manageable components. It can be combined with other important decision-making approaches, such as responding to one’s instincts and others’ expectations. To understand stakeholders’ opinions regarding the review and address the elements of the above framework, interviews were conducted with key stakeholders in Santa Barbara. The interviewees included fifteen members of the city’s planning staff, design review boards, and NPO Update Steering Committee. These interviewees were selected because of their familiarity with the NPO Update. The standard set of interview questions can be found in Appendix A. Project Results The Decision Problem This project assumes the following decision problem statement: What review alternatives will best achieve the immediate and long-term success of the city’s NPO Update and residential design review policies? Success can be defined by the objectives discussed later in this report. The general decision of how to conduct a review can be broken down into the following components, or sub-decisions: 1. Determine what subjects or elements of the NPO Update the review should evaluate. v 2. Determine the appropriate role of the public in the review. 3. Determine the appropriate analytical approach or methods. Although these sub-decisions need to be addressed in the short term, they are connected to longer-term considerations as well. Several interviewees pointed out that it is likely too early to fully review the impacts or success of the NPO Update. The review activities conducted this year would form a crucial, but perhaps not the final, part of the city’s ongoing effort to improve the NPO Update. Thus a fourth sub-decision could be added to the above list: Determine the appropriate long-term review strategy or process for ensuring NPO Update success. Figure i illustrates the review decision problem and overall review process. The diamonds represent the sub-decisions that staff would have a role in making. Figure i. Review Decision Problem and the Review Process Objectives This project sought to identify consensus objectives that everyone can agree are worthy of achievement. Both objectives of the review and objectives of the NPO Update were identified. Two fundamental objectives of the review were identified: “Ensure NPO Update success” and “Minimize costs of the review.” Numerous means objectives, such as addressing stakeholders’ concerns, must be met in order to achieve the first objective. Six fundamental NPO Update objectives were identified based on city documents, public comment, and interviewee responses: 1. Maximize neighborhood compatibility of single-family homes. vi 2. Maximize clarity, consistency, and simplicity of design review. 3. Maximize social/community well-being. 4. Preserve the natural environment. 5. Ensure fairness. 6. Minimize administrative and other costs to all stakeholders. Alternatives Four types of alternatives were examined: what to review (i.e., which elements of the NPO Update to address and what information should be studied), public involvement methods, analytical methods, and long-term review strategies. The alternatives are not mutually exclusive. Different sets of criteria were used to rate each type of alternative. For example, as shown in Table i below, alternative public involvement methods were compared based on five criteria. Tables such as this are used throughout the discussion of alternatives to explore the tradeoffs. In Table i. Public Involvement Methods Compared Criteria Public Involvement Methods Discussions with key contacts Steering committee Public hearing Public workshop Surveys Relevant stages of review process Planning; understanding outcomes; long-term monitoring Making changes; planning Understanding outcomes; making changes Understanding outcomes; making changes; long-term monitoring Understanding outcomes; long-term monitoring Likelihood of acceptance Low if not supplemented by other approaches Moderate High Uncertain, depends on nature of workshop Low if not supplemented by other approaches Information depth vs. breadth Depth Depth Breadth Breadth Breadth Qualitative or quantitative focus Qualitative Qualitative Qualitative Either Quantitative Cost/difficulty Depends on scope High, potentially Low Medium Medium vii this example, the public hearing method has a high likelihood of acceptance and low cost/difficulty. Therefore, it is a compelling alternative for understanding NPO Update outcomes and identifying potential policy changes, as long as broad, qualitative information is desired. As for the other types of alternatives, several NPO Update elements appeared to meet at least three criteria indicating the elements may serve as sensible review priorities. These elements were: the Single Family Design Board and Guidelines, FAR square footage cutoffs, the 85% of maximum FAR trigger, FAR chart numbers, and private view guidelines and findings. Several possible indicators were identified, both qualitative and quantitative. Many of the qualitative indicators (e.g., level of general satisfaction) measure the achievement of the fundamental NPO Update objectives fairly directly, whereas the quantitative indicators (e.g., number of projects reviewed and built) do not - but still could provide insight into the design review process. Possible analytical methods include public input analysis, project data analysis, case studies of individual projects, site visits of projects built under the new rules, visual survey workshops, and computer simulations. Long-term review strategies could include: informal, ongoing review; rescheduling the review for later; follow-up meetings with stakeholders to solicit their concerns; exit surveys sent to applicants who complete the design review process; monitoring indicators; and annual site visits of completed projects. Conclusions and Recommendations A well-conducted review will help in ensuring the long-term success of the NPO Update. Deciding how to conduct the NPO Update review may be aided by a structured decision-making approach that focuses on the key review decisions, defines the fundamental objectives of the NPO Update, and selects alternatives that assist in achieving the fundamental objectives. This decision-making process should be informed by stakeholders’ values in order to ensure that a review is responsive to their needs. It is recommended that the city’s NPO Update review proceed with the following points in mind: • Using multiple strategies or methods in concert with one another can ensure that the full range of issues and stakeholders’ concerns are addressed and that different types of information inform the review, thereby increasing confidence in the review results. viii • Review findings should incorporate meaningful public input from all stakeholders. The success of the NPO Update may ultimately depend on whether community members feel that the NPO Update was reasonably successful at addressing their concerns. • There were two competing concerns prevalent during the NPO Update that a review would likely need to address in order to ensure that an acceptable compromise is in place. A complete review would thus likely address both neighbors’ concerns about the compatibility of single-family residential projects and applicants’ concerns about their ability to build reasonable projects. • While the city’s immediate review task may be a short-term project, it appears that some conclusions regarding the NPO Update’s success cannot yet be reached. It would thus be appropriate to consider how success would be reviewed in a long-term context. • The NPO Update review can be considered in the context of other priorities the city would like to address. Some of the interviewees felt that design and compatibility issues similar to those discussed during the NPO Update were now relevant for multi-family and downtown buildings. When considering the appropriate extent or scope of the NPO Update review, the urgency of addressing other planning issues can be considered. • Learning from the NPO Update can help to improve future planning efforts. Interviewee comments suggested that the main purpose of the immediate review should be to improve NPO Update policies as needed, but staff may, as a secondary focus, consider the lessons of the NPO Update process. A successful process can better achieve successful outcomes. ix Table of Contents 1 INTRODUCTION AND BACKGROUND........................................................................1 1.1 PROJECT PURPOSE AND SCOPE............................................................................................ 1 1.2 NPO UPDATE BACKGROUND................................................................................................. 2 1.3 NPO UPDATE REVIEW BACKGROUND.................................................................................... 4 2 MONITORING AND EVALUATION BACKGROUND....................................................4 2.1 MONITORING AND EVALUATION OVERVIEW ............................................................................ 4 2.2 DESIGN REVIEW POLICY EVALUATION EXAMPLES .................................................................. 6 3 PROJECT APPROACH AND METHODS.....................................................................8 3.1 STRUCTURED DECISION-MAKING APPROACH......................................................................... 8 3.2 STAKEHOLDER OPINIONS AND INTERVIEWS............................................................................ 8 4 PROJECT RESULTS ..................................................................................................10 4.1 THE DECISION PROBLEM..................................................................................................... 10 4.2 OBJECTIVES ....................................................................................................................... 12 4.2.1 Review Objectives ..................................................................................................... 13 4.2.2 NPO Update Objectives............................................................................................. 15 4.3 ALTERNATIVES.................................................................................................................... 20 4.3.1 Review Subjects ........................................................................................................ 20 4.3.2 Public Involvement Methods ...................................................................................... 28 4.3.3 Analytical Methods..................................................................................................... 32 4.3.4 Long-Term Review Strategies ................................................................................... 41 5 CONCLUSIONS AND RECOMMENDATIONS ...........................................................44 References .....................................................................................................................49 x Figures Figure 1. Review Decision Problem and the Review Process .......................................12 Figure 2. Potential Review Objectives............................................................................15 Tables Table 1. Potential Fundamental Objectives of NPO Update ..........................................18 Table 2. Potential NPO Update Means Objectives ........................................................19 Table 3. NPO Update Elements Rated on Review Priority Criteria ................................22 Table 4. Potential Indicators...........................................................................................26 Table 5. Public Involvement Methods Compared...........................................................29 Table 6. Project Characteristics for Use in Analysis.......................................................35 Table 7. Analytical Methods Compared .........................................................................40 Table 8. Long-Term Review Strategies Compared ........................................................44 Appendices Appendix A. Interview Script ..........................................................................................46 Appendix B. Additional Potential Indicators....................................................................47 Page 1 of 50 1 INTRODUCTION AND BACKGROUND The City of Santa Barbara, California, has spent several years aiming to improve its single- family residential housing design review policies and processes via a program called the Neighborhood Preservation Ordinance/Single Family Design Guidelines Update (abbreviated hereafter as the NPO Update). The NPO Update program comprehensively revised the city’s existing NPO, Single Family Design Guidelines, and design review procedures. Upon approving these changes, the Santa Barbara City Council directed the city’s planning staff to conduct a review of the NPO Update in 2009. The City Council did not give staff any formal direction regarding the review. Thus there is a need to determine what such a review would entail and how it would be conducted. Planning staff have noted that time constraints prevent them from devoting significant resources to conducting, or deciding how to conduct, the review. An external study such as this one may help staff to ensure that the review is innovative and comprehensive while conserving staff time. 1.1 Project Purpose and Scope The purpose of this final Masters degree project is to help determine how the City of Santa Barbara should review its NPO Update. The intended client is the City of Santa Barbara’s planning staff, which is responsible for planning and conducting the review. This project focuses on providing information that could assist the staff in making decisions regarding the review. Specifically, the project focuses on the following objectives: • Clarify the review decision problem. • Clarify the objectives of the NPO Update and the objectives of the NPO Update review. • Present possible alternative review topics and approaches and discuss their relative advantages and disadvantages. This project is comprised of two main parts. The first was an interview of key stakeholders in Santa Barbara in order to understand their opinions regarding the review. This report, the second part of the project, addresses the objectives listed above as informed by the interview results, city documents, relevant academic literature, and my personal experiences working as an intern for the city during a portion of the NPO Update process. Page 2 of 50 Although I helped to develop some of the policies that may be discussed as part of the NPO Update review, my goal in completing this project is not to support or oppose particular policies. This project does not attempt to determine whether the NPO Update has been successful, nor does it suggest changes to the existing NPO policies and practices. Rather, this project aims to provide information that can assist the city in conducting an intelligent, fair and effective review that helps to achieve successful policy outcomes. 1.2 NPO Update Background The City of Santa Barbara enacted the Neighborhood Preservation Ordinance in 1991 in order to ensure that single-family housing development would be compatible with surrounding neighborhoods. The NPO was a response to some new, expanded, or rebuilt homes that were considered very visually prominent and were larger than the surrounding homes in an established neighborhood. This process, sometimes referred to as “mansionization,” has occurred in other cities as well, and the resulting homes are sometimes labeled “monster homes” or “McMansions” (Szold 2005). In Santa Barbara, controversial homes were generally referred to as “incompatible” or “inappropriate” due to their perceived aesthetic or privacy impacts. Planning departments rely on several common tools for limiting mansionization, although the details vary by jurisdiction (Kendig 2005; City of Santa Barbara 2004a). One common response is to create floor to lot area ratio (FAR) regulations, which limit the allowable home size relative to the lot size. Other regulations control certain dimensions of the home or require design review for certain types of projects. Planning departments have also created voluntary design guidelines that encourage project applicants to design in accordance with the local context. Under the NPO, Santa Barbara began requiring certain single-family home projects to be submitted to the city’s Architectural Board of Review for design review. The board attempted to ensure projects complied with the city’s voluntary Single Family Design Guidelines. In 2002, several stakeholders expressed concerns that large, unaesthetic, and incompatible homes were still being built and that the city’s design review process was inefficient. In response, the city began the NPO Update process. The NPO Update Work Program states that the goal of the update was to “achieve quality, single-family residential project design compatible with existing neighborhoods through efficient design review” (City of Santa Barbara 2004b). Page 3 of 50 The NPO Update was extensive and involved several years of study. The project’s steering committee met over thirty times over the course of two years, and planning staff wrote ten issue papers analyzing relevant topics. The NPO Update was a comprehensive revision of the existing NPO, Single Family Design Guidelines, and design review procedures. It replaced the original NPO and Guidelines. Major changes included: • Maximum FAR requirements, with provisions to exceed the maximums with modifications in some cases. • Updated Single Family Design Guidelines to help guide project review. • Expanded application categories subject to design review, including new second- or third-story projects. These were often referred to as “triggers” of design review. • Creation of the Single Family Design Board (SFDB), which now reviews most single- family projects subject to design review. Previously the Architectural Board of Review reviewed single-family projects in addition to commercial and multi-family projects. • Revised findings required for project approvals. • Revised Hillside Special Design District boundaries. • Green building components required for large residences (City of Santa Barbara 2007). Aspects of the NPO Update, particularly the FAR component, were highly controversial and were frequently redrafted over the course of the update process. Some residents criticized the proposed FAR requirements as an unnecessary and unreasonable infringement on their property rights. These critics also argued that the FARs would prevent homeowners from providing sufficient living space for their families. Other residents supported FAR regulations and argued that they were needed in order for the city to meaningfully control home size and provide comfort that homes would be compatible with their neighborhoods. The City Council ultimately approved the NPO Update in May 2007. The approved changes included maximum FAR requirements, but only for two-story homes on lots smaller than 15,000 square feet. For homes on larger lots, FARs were implemented as guidelines only. In addition, homes with certain site and project characteristics were prevented from exceeding 85% of the maximum FAR requirements. Homes in excess of this threshold were assigned more stringent design review requirements. Page 4 of 50 1.3 NPO Update Review Background When approving the NPO Update in 2007, the City Council requested that it be reviewed two years later. Individual City Council members suggested certain items that they hoped the review would address1. However, the City Council did not provide any formal direction to staff. From talking to staff and design review board members, it is clear that there are some practical constraints on the review. For example, staff time is limited, although the specific amount of time available was not defined as of this writing. Also, there appear to be a few general expectations of the review, including that it will: 1) appropriately involve the public and relevant stakeholders; 2) address any problems with the NPO Update; and 3) present review results to the City Council. In general, though, staff appears to have flexibility and discretion regarding the review. Major changes to the city’s policies resulting from the review, particularly any changes to the city’s regulations, would need to be approved by the City Council. 2 MONITORING AND EVALUATION BACKGROUND 2.1 Monitoring and Evaluation Overview The NPO Update review could be considered a policy or program “evaluation.” Weiss (1998, 4) defines “evaluation” as “the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.” The term “evaluation” can describe various types of policy review. Patton and Sawicki (1993, 368) note that policy evaluation can consist of policy monitoring and/or ex-post policy analysis. Policy monitoring is the process of recording changes in key indicators after policy implementation. An ex-post policy analysis, on the other hand, determines the extent to which policy objectives were achieved. To Seasons (2003), “monitoring” implies a continuous evaluation in which data is collected and interpreted on a regular basis. Seasons uses the term M&E (monitoring and evaluation) to encompass evaluative activities and observes that these activities can be informal and irregular in actual municipal planning practice. The NPO Update review could thus be considered an M&E process, without necessarily implying a formal, intensive evaluation. 























































 1 The items included: the effectiveness of the proposed 20-closest-homes database; whether porches and carports should be included in FAR calculations; whether FARs should be used as guidelines or requirements for single- family homes in multi-family zones; and the frequency and nature of private view discussions at SFDB meetings. Page 5 of 50 When used by municipal planners, M&E should help to answer questions such as: How did our projects, policies, or plans perform? What happened and why? How can we improve our projects, policies, or plans (Seasons 2003)? There are at least three general purposes of M&E, according to Patton and Sawicki. A first purpose may be to ensure that policies are being implemented as intended. A second review purpose could be to determine whether policies are having the desired impact (that is, are the objectives being achieved?) and, if not, modify or terminate the policies in order to improve the outcomes. An additional evaluation purpose may be to learn from the policies and the policy-making process in order to improve future planning and decision-making efforts. An evaluation of the policy-making process can include both how a policy study was conducted and presented to decision makers, as well as how decision makers used the study (1993, 363-4). One of the first steps of an evaluation is to reinforce one’s understanding of the policies that are being evaluated. The NPO Update was complex, so this step could be useful. Answering questions such as the following can help in reinforcing understanding (Hatry, Winnie, and Fisk 1981): • What are the crucial components of the policy? • What is necessary for success? • How do we think the policy is supposed to work? One way to answer these questions is to create diagrams modeling the connections between a program’s actions and the desired outcomes. Additional steps common to evaluation are listed below (as adapted from Weiss 1998; Hatry, Winnie, and Fisk 1981; and Patton and Sawicki 1993). These steps are often iterative and may not occur in a rigid order. • Determine the objectives of the evaluation and the research questions. • Consider who participates and how. • Determine the goals of the policies. • Determine what to measure. • Select an evaluation approach and methods. Page 6 of 50 • Collect, analyze, and interpret data. • Adjust the policies based on the findings. Municipal planners face particular challenges when evaluating policies or plans, as one researcher found based on interviews with planning officials (Seasons 2003). One challenge is determining the objectives of the policies or otherwise defining success. The subjectivity of concepts such as success and failure has hindered planners’ ability or willingness to evaluate. In addition, there can be difficulty establishing a causal relationship between planning activities and actual outcomes. For example, home design is largely driven by market and cultural forces external to the planning process. Other factors constraining evaluation include limited resources such as time. Patton and Sawicki (1993, 375-6) argue that, as common as evaluation seems to be, it is often not conducted ideally. For example, evaluations can: fail to provide information about a program’s effects on the community; place too much attention on input variables such as dollars spent rather than on actual outcomes; collect data haphazardly; and focus only on those aspects of a program that are easily measured. Despite the practical limitations of M&E processes, they may still help to improve policy and program effectiveness (Seasons 2003). 2.2 Design Review Policy Evaluation Examples Other jurisdictions have evaluated their design review policies as well. The scope of this project did not include surveying other jurisdictions’ experiences, but the following examples are provided to illustrate potential evaluation approaches. Scarsdale, New York The Village of Scarsdale hired a consultant to evaluate newly created FAR policies for single- family homes. The review had three components: 1) a statistical comparison of home size before and after the FAR policies were enacted; 2) focused interviews with individuals who worked under or administered the regulations; and 3) computer simulations showing how projects built both before and after the regulations might appear if modified to fit precisely within the FAR maximums. The consultants concluded that average home size declined after the FARs went into effect (Janes and Heagney 2007a). Page 7 of 50 Lake Forest, Illinois The City of Lake Forest evaluates its design review policies informally on an ongoing basis. Lake Forest has found that incremental, ongoing improvements are more effective than conducting a more formal review every several years because incremental improvements are easier for stakeholders to understand and support. More formal reviews, planning staff believe, can easily become overly complex and extensive. The city’s review methods include surveying property owners after the completion of every project and holding a semi-annual meeting with architects, builders, and design professionals in order to solicit concerns and ideas for improvements. About every two to three years, the city’s design review board recommends code amendments to the City Council based on the findings of the ongoing review. The city makes less formal changes, such as to board procedures, in the interim (Czerniak, C., pers. comm.). Rochester, New York In Rochester, the city evaluated a new zoning code twenty-four months after its adoption. According to the city’s zoning director, this was too soon to fully analyze all aspects of the zoning code changes, as it takes much longer, perhaps ten to twenty years, to realize the results of a plan or zoning code. However, the evaluation did provide comfort that the city’s approach to the changes was conceptually correct and that the regulations and processes were reasonable and realistic. Rochester’s review relied on: direct observation; interviews with people who use the regulations; careful reviews of documents, approvals, findings, and decisions; and case studies of individual projects, including before-and-after comparisons using photos and graphics. The city found that quantitative numbers were useful in indicating a need for further analysis and to look deeper for causal relationships; however, it was important to also consider individual cases and experiences. The zoning director recommends that cities try to determine what to measure and how to do so, despite the difficulty of defining success and failure. In addition, evaluations should challenge the theories and objectives upon which the policies and procedures are based (Ientelucci 2005). Page 8 of 50 3 PROJECT APPROACH AND METHODS 3.1 Structured Decision-Making Approach The problem addressed by this project is a decision (that is, how to conduct a review), and this project applies a structured decision-making framework (as presented in Hammond, Keeney, and Raiffa 2002) to the problem. The results of this project are organized around this approach, which contains the following elements: • Define the decision problem. • Clarify the goals and objectives. • Find alternatives to achieve the goals and objectives. • Understand how each alternative performs in terms of the goals and objectives. • Identify the tradeoffs between the goals and objectives associated with each alternative. This structured approach is often described as “rational,” although there are clearly limits to how rational a decision-making approach can be (Marsh 1994, 9). This type of approach may appear formal and rigid to some people, but it helps divide a potentially complex decision into manageable components. It can supplement other important decision-making approaches, such as responding to one’s instincts and others’ expectations. 3.2 Stakeholder Opinions and Interviews This project assumes that any decisions regarding the review should be informed by stakeholders’ views. Thus it was important to try to understand stakeholders’ opinions when addressing each part of the above framework. The first method used to understand stakeholders’ views was to analyze comments made by the public and hearing body members regarding the NPO Update. In particular these comments yielded insight into the objectives of the NPO Update. The comments included oral comments at 29 Steering Committee meetings2 and two City Council hearings3. These comments do not represent every single statement made during the 























































 2 The meeting notes are available online at http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/. 3 These were the hearings held on August 8, 2006, and May 1, 2007. They can be viewed online at http://santabarbara.granicus.com/ViewPublisher.php?view_id=6. Page 9 of 50 NPO Update process, but they are likely to encompass the major ideas and concerns formally expressed during the process. These comments were analyzed by identifying any statements that seemed to express a speaker’s concerns or aspirations. Such statements can be rephrased as objectives (Hammond, Keeney, and Raiffa 2002). To understand stakeholders’ opinions regarding the review, interviews were conducted with key stakeholders in Santa Barbara. The interviewees included fifteen members of the city’s planning staff, design review boards, and NPO Update Steering Committee. These interviewees were selected because of their familiarity and experience with the NPO Update. The interviewees did not include members of the public who have not served on relevant city boards and committees, as this was beyond the scope of this project. The interviews do not necessarily represent the opinions of all relevant individuals. However, this sample represents a useful subgroup of people affected by and involved in the NPO Update. Among the interviewees were individuals who held opposing views regarding the NPO Update’s appropriate restrictiveness. The interview results provide a baseline of stakeholder opinions that can be augmented by the public participation elements the city incorporates into the review. Public involvement approaches are discussed further in section 4.3.2. At the start of each interview, interviewees were read a brief background statement regarding the city’s intent to review the NPO Update (see Appendix A) and the interview format. The questions and topics varied somewhat depending on each interviewee’s responses and knowledge areas. A standard set of questions attempted to provide input for each of the elements of the decision-making framework described earlier. These questions included: 1. What do you feel were the goals of the NPO Update? 2. How would you define a successful NPO Update? 3. What do you think should be the goals of a review of the NPO Update? 4. What factors do you think will determine the success or lack of success of the NPO Update? 5. What do you think a review of the NPO Update should measure or evaluate and how? 6. Do you have any other comments or ideas about how the NPO Update and how it might best be reviewed? Page 10 of 50 The interview questions had the advantage and disadvantage of being fairly open-ended. They were designed to solicit general ideas regarding the NPO Update and the review. This was considered appropriate because few decisions regarding the review had yet been made. The open-endedness may not have been desirable for a few interviewees who felt the questions were vague or difficult to distinguish. Some interviewees seemed most comfortable when discussing concrete topics such as the problems caused or mitigated by the NPO Update thus far and the particular topics or actions a review should include. The specific options inspired by the interviews and presented in this report could be discussed with stakeholders in more detail as part of the NPO Update review’s public participation process. In analyzing the interview responses, statements were coded according to each of the six questions as well as other topics including: review scope and purpose; public participation considerations; and suggested changes to the NPO Update. When coding the results, responses to the six questions were sometimes inferred based on statements made at an unrelated part of the interview. For example, while an interviewee might not mention “listening to the public” when asked what they felt should be the objectives of the review, later in the interview they might say, “It’s very important that we listen to the public to understand how they feel about the new policies.” This would thus be coded as a review objective. 4 PROJECT RESULTS 4.1 The Decision Problem The first step in this project’s decision-making framework is to define the decision problem, since the remaining steps follow from this. As noted earlier, the impetus to review the NPO Update was the City Council’s direction, upon adopting the NPO Update in 2007, that the NPO Update should be reviewed in two years’ time. Therefore, the immediate decision problem staff face is how to conduct a review in 2009 that meets stakeholders’ expectations. This broad decision problem can be divided into more concerete, useful components. Based on interviewees’ comments, it appears that, in any review conducted this year, staff would reasonably be expected to involve the public, address certain topics and subjects of concern, and provide some sort of analysis for City Council and the public to consider. Thus the general decision of how to conduct a review this year can be broken down into the following components, or sub-decisions: Page 11 of 50 • Determine what subjects or elements of the NPO Update the review should evaluate. • Determine the appropriate role of the public in the review. • Determine the appropriate analytical approach or methods. Although these sub-decisions are to be made in the short term, they are connected to longer-term considerations as well. It was clear from the stakeholder interviews that, while the immediate decision may be how to conduct a review this year, one could also define a broader decision such as: How can we ensure the long-term success of the city’s residential design review policies? Several interviewees pointed out that it is likely too early to fully review the impacts or success of the NPO Update. One respondent asked, “What kind of information can be obtained in just two years?” A review approach suggested by many interviewees was to analyze new single- family development, but conversations with staff suggest that very few projects have been completed under the new NPO rules so far. The review activities conducted this year would form a crucial, but perhaps not the final, part of the city’s ongoing effort to improve the NPO Update. Thus another component of the review decision could be stated as follows: Determine the appropriate long-term review strategy or process for ensuring NPO Update success. Based on the above considerations, the rest of this report assumes the following decision problem statement: What review alternatives will best achieve the immediate and long-term success of the city’s NPO Update and residential design review policies? Success is defined by the objectives discussed in the next section. The alternatives are described and evaluated in section 4.3 and are divided into the four components described earlier: review subjects, public involvement methods, analytical methods, and long-term review strategies. Figure 1 illustrates the review decision problem as defined above as well as the associated review process. The diamonds represent the decision components, or sub-decisions, that staff would have a role in addressing. The left of the diagram shows the initial decision to review the NPO Update. Next, staff decides how to conduct a review this year, based on public input, the objectives of the review and NPO Update, and constraints such as time and resources. This decision would consist of the three short-term sub-decisions described above. Based on these sub-decisions, staff conducts analysis, including gathering public input. The results of this Page 12 of 50 analysis yield staff recommendations regarding necessary policy changes. Then, according to City Council direction, staff implements particular changes and engages in the long-term review process deemed appropriate. This long-term process could repeat the decisions and actions shown in the diagram, as indicated by the dashed line. Thus the review decisions may have different answers when considered in the short term versus the long term. In the diagram, the review is shown as an ongoing process to improve the city’s residential design review policies as opposed to a one-off event. From this perspective, the NPO Update was also a review of the city’s design review policies, specifically the ones that existed under the original NPO. Figure 1. Review Decision Problem and the Review Process 4.2 Objectives The objectives4 of the review and the NPO Update can provide a basis for evaluating potential review approaches. As noted earlier, in a controversial planning process, stakeholders may disagree on the objectives. Defining the objectives can be difficult, but this project sought to identify consensus objectives that everyone can agree are worthy of achievement. While there may be disagreements regarding how these objectives should be prioritized or achieved, the consensus objectives at least provide a common reference for evaluating alternatives. 























































 4 For the purposes of this report, the term “objectives” is not meant to imply a distinct meaning from that of “goals,” and the terms can be used interchangeably. Page 13 of 50 The objectives identified in this report are categorized as either “fundamental” or “means” objectives. Means objectives help to achieve other objectives, whereas fundamental objectives reflect what one really wants to accomplish (Clemen and Reilly 2001, 46-50). Means objectives are not necessarily less important than fundamental objectives. Consider a means objective such as “limit regulatory loopholes.” Limiting loopholes is a means to an end or ends — e.g., ensuring that homes are compatible with the neighborhood — but it may be critical to achieving the fundamental objective(s). Distinguishing between the two types of objectives can help when evaluating alternatives, because it makes sense to try to evaluate alternatives in terms of one’s end goals to the extent possible. However, in some circumstances it can be sensible to use means objectives to evaluate review alternatives. Fundamental objectives are sometimes too abstract to help distinguish between options, so means objectives can substitute for the fundamental objectives. 4.2.1 Review Objectives Interviewees were asked what they thought should be the objectives of the review. A key theme among the responses was an emphasis on fixing any problems or otherwise adjusting and improving the NPO Update in order to achieve its objectives. Several interviewees emphasized determining whether specific outcomes, such as a reduction in incompatible homes, had been achieved. Other potential review objectives, such as ensuring the NPO Update is implemented as intended or learning from the NPO Update, were rarely mentioned. This does not mean that interviewees necessarily disagreed that those objectives would be appropriate; however, few interviewees mentioned them. Staff comments suggested that a review of the NPO Update’s implementation had proceeded informally ever since the NPO Update’s adoption. Once the new policies were implemented, staff had to clarify and defend policies to applicants. As a result, staff had to consider what was intended by the policies. Several policies were thereby modified, especially in the early stages of implementation. Among the interviewees who suggested learning from the NPO Update, it was felt that the lessons should be applied to related concerns such as downtown buildings’ size, bulk, and scale and multi-family residential policies. However, these concerns seemed to be framed as distinct problems to address rather than as objectives of the NPO Update review itself. Page 14 of 50 Interviewees’ suggested objectives most often fell under the following categories: • Identify and fix any problems. • Gather input from stakeholders and work to satisfy them. • Ensure the objectives of the NPO Update are achieved. • Ensure the NPO Update is successful in the long term. A common thread among these objectives is that they all determine or define the success of the NPO Update. The NPO Update would probably not be considered successful unless any problems associated with it were fixed and stakeholders were satisfied. Furthermore, success would probably depend on achieving the NPO Update’s objectives and doing so over the long term. It would appear that a fundamental objective of the review could be stated simply as: “Ensure the success of the NPO Update.” Almost all of the review objectives interviewees mentioned seemed to serve the ultimate purpose of NPO Update success, however success may be defined. One exception was the objective of minimizing the costs of the review (particularly in terms of staff time). Staff emphasized that this was a constraint. It could be considered a second fundamental objective of the review. Figure 2 illustrates relevant objectives of the review. Many of the objectives were taken directly from interviewees’ responses to questions regarding review objectives. Other objectives were inferred based on other statements made during the interviews. In the diagram, the two ovals represent the two fundamental objectives identified. The arrows between the text demonstrate how the review must achieve various means objectives in order to achieve NPO Update success. The means objectives form a network rather than a strict hierarchy. All of the objectives are potentially important, although some may be easier to achieve than others. For example, receiving sufficient public input may be easier than fully satisfying all stakeholders’ concerns. The objectives in the diagram relate to both what the review should address and how it should be conducted. In terms of what to review, identifying and addressing the following issues would contribute to maximizing NPO Update success: stakeholder concerns, undesirable or unintended consequences, controversial topics, and the issues most relevant to success. These issues could Page 15 of 50 overlap. For example, an unintended consequence of the NPO Update could also be controversial. Understanding the outcomes of the NPO Update could be particularly important, because understanding whether desired outcomes are achieved and why is critical to identifying whether adjustments are necessary and what types of adjustments may be appropriate. Other objectives in the diagram concern how to conduct the review. Many of these relate to gathering public input and engaging or satisfying stakeholders. An appropriate public involvement approach is necessary both to satisfy stakeholders (who expect to be consulted meaningfully) and to receive valuable input regarding NPO Update outcomes. The diagram also shows it is possible that the degree of NPO Update success may be affected somewhat by the rigor of the review (that is, whether it is thorough, analyzes the issues in sufficient depth, and has sufficient public involvement). Figure 2. Potential Review Objectives 4.2.2 NPO Update Objectives In Figure 2, one of the fudamental review objectives is to ensure NPO Update success. Logically speaking, a successful NPO Update would be one in which the fundamental objectives of the NPO Update were achieved. The review could ultimately be considered a means of achieving these fundamental objectives, and the success of both the review and the NPO Update would Page 16 of 50 depend on whether these NPO Update objectives are achieved. For this reason, it would be useful to evaluate review alternatives in terms of the fundamental NPO Update objectives to the extent possible. To do this, the NPO Update objectives must be identified. Three sources were used to infer NPO Update objectives: 1) city documents in which NPO Update objectives were explicitly stated (City of Santa Barbara 2004b, 2006a, 2006b); 2) public comments, as recorded in meeting notes and submitted letters; and 3) interviewees’ responses. Objectives may change as a process unfolds, and these three sources represent three different periods of the NPO Update process. The objectives in the city’s documents were defined at the beginning of the process. Public comments were recorded throughout the drafting of the new policies. The interviews gauged retrospective opinions regarding the objectives. The six fundamental objectives identified below were mentioned either explicitly or implicitly in all three sources. There was a great deal of overlap between the objectives identified in the three sources, although the city documents’ objectives were phrased more formally. A majority of interviewees mentioned objectives that fit in one of these two categories: 1) Address neighborhood compatibility, design quality, and development impact issues. Many interviewees identified mansionization issues such as these as the main catalyst for the NPO Update. 2) Make the design review process and rules clear, simple, and consistent. Interviewees said that there was a perception prior to the NPO Update that the existing rules and process were too subjective and complicated. This led to the approval of inappropriate homes as well as frustration among applicants and neighbors. Other objectives mentioned included: satisfying and engaging community members; ensuring design review rules were sufficiently rigorous; reducing controversy; improving the existing NPO; and addressing design review process issues such as appeals and Architectural Board of Review workload. At least two of the interviewees distinguished between the objectives they felt were driving the NPO Update and the objectives they personally embraced. These interviewees said that the NPO Update was not necessary in the first place because the existing rules and processes were sufficient. This is reflective of many of the public comments expressed during the NPO Update process. At Steering Committee meetings and City Council hearings, one group Page 17 of 50 of commenters seemed to agree that the NPO Update should enact new, stricter rules whereas another was opposed to such changes and questioned their necessity. Fundamental and means objectives Table 1 aggregates the fundamental objectives identified from all three sources. The table lists six fundamental objectives that can be divided into several sub-objectives that clarify what is meant by the fundamental objective. For example, one objective frequently expressed during the NPO Update process was to ensure the neighborhood compatibility of single-family homes. The objective of neighborhood compatibility has multiple elements such as: appropriate size, bulk, and scale; design quality; privacy; and view preservation. The second fundamental objective is maximizing the clarity, consistency, and simplicity of design review. This includes guidelines, regulations, terminology, calculations, and the design review process itself. Other expressed objectives might fall under the category of “social/community well-being.” This includes homeowners’ ability to accommodate the needs of their families as well as neighbors’ sense that they have influence and control over their own neighborhoods. Another objective could be natural environment preservation, as some people voiced concerns regarding homes’ impacts on the environment. Fairness was a significant element of NPO Update discussion. Many commenters emphasized the importance of their property rights and on making rules and processes fair. Also, the costs — whether measured in terms of time, money, or other burdens — facing applicants, hearing body members, and staff were clearly important. There are also two overarching objectives in the table, “Maximize quality of life in Santa Barbara” and “Satisfy the community/stakeholders.” All of the fundamental objectives and sub-objectives could be said to fall under these two broad objectives. Table 2 shows the means objectives identified as important in accomplishing the fundamental objectives described in Table 1. Each means objective is categorized under the most relevant fundamental objective. Several of the means objectives help to achieve multiple objectives and are listed accordingly. Page 18 of 50 Table 1. Potential Fundamental Objectives of NPO Update Overall objectives: Maximize quality of life in Santa Barbara. Satisfy the community/stakeholders. Fundamental objectives: 1. Maximize neighborhood compatibility of single-family homes. 1.1. Minimize size, bulk, and scale impacts of homes. 1.2. Maximize design quality of homes. 1.3. Protect neighbors’ privacy. 1.4. Protect public views. 2. Maximize clarity, consistency, and simplicity of design review. 2.1. Ensure clear, simple, and consistent design guidelines and regulations. 2.2. Ensure clear, simple, and consistent terminology and calculations. 2.3. Ensure a clear, simple, and consistent design review process. 3. Maximize social/community well-being. 3.1. Minimize conflict and controversy. 3.2. Maximize neighbors’ comfort and sense of influence. 3.3. Maximize residents’ long-term ties to neighborhood. 4. Preserve the natural environment. 4.1. Minimize homes’ construction impacts. 4.2. Minimize resource use of homes after construction. 4.3. Minimize development of open space. 5. Ensure fairness. 5.1. Respect property rights. 5.2. Maximize residents’ ability to build homes that meet their needs. 5.3. Maximize fairness of NPO Update outcomes. 6. Minimize administrative and other costs. 6.1. Minimize costs to project applicants. 6.2. Minimize costs to hearing body members. 6.3. Minimize costs to city staff. Page 19 of 50 Table 2. Potential NPO Update Means Objectives Fundamental Objectives Associated Means Objectives 1. Maximize neighborhood compatibility of single-family homes. • Ensure design creativity and flexibility. • Recognize design differences between neighborhoods. • Ensure neighborhoods’ architectural diversity. • Minimize the “spilldown” of homes on sloped lots. • Reduce homes’ size/bulk/scale. • Ensure design review guidelines are sufficient. • Develop a better understanding of neighborhood compatibility. 2. Maximize clarity, consistency, and simplicity of design review. • Specify what is allowed and what is not. • Make approval standards clear. • Consistently apply calculation methods. • Simplify the NPO municipal code. 3. Maximize social/community well-being. • Promote the diversity of neighborhood residents. • Maintain a long-term, year-round neighborhood population. • Minimize the “flipping” of homes. • Ensure adequate neighborhood parking. • Strengthen Good Neighbor Policies. 4. Preserve the natural environment. • Limit the effects of grading on hillside environments. • Promote sustainable home designs. 5. Ensure fairness. • Avoid overly restrictive regulations. 6. Minimize administrative and other costs. • Avoid drastic changes to regulations. • Reduce the design review process time. • Avoid long design review board hearing schedules. • Avoid delays in application approvals. • Improve the effectiveness and efficiency of the review process. Multiple fundamental objectives • Address issues associated with the NPO Update since it was adopted. • Ensure appropriate opportunities for public input. • Respond to steering committee and public input. • Limit loopholes of regulations. • Prevent “piecemealed” projects. • Ensure design review rules and policies are appropriately restrictive. • Avoid unintended consequences. • Avoid creating unnecessary regulations. • Improve noticing requirements. • Reduce the number of project appeals, particularly frivolous appeals. • Ensure projects trigger the appropriate level and type of design review. • Ensure the NPO Update is implemented effectively. • Ensure design review boards have appropriate analytical tools. • Ensure design review board members are properly trained. • Ensure an ability to recruit design review board members. • Respect the talents and contributions of design review board members. • Calculate FARs, building height, and slope appropriately. • Ensure project designs contribute to wildfire prevention. • Ensure appropriate application submittal requirements. • Establish time limits for NPO findings. • Revise Hillside Design District boundaries. • Flag site parcel constraints earlier in the review process. Page 20 of 50 4.3 Alternatives This section identifies and evaluates review options available to the city. These alternatives are divided into four types: what to review (that is, the topics and indicators, or subjects, of the review), public involvement methods, analytical methods, and long-term review strategies. The alternatives are not mutually exclusive. The alternatives are evaluated in terms of the review objectives, including, where possible, the NPO Update objectives identified above. 4.3.1 Review Subjects One of the decisions staff would need to make is to determine what to review. This decision could include which elements of the NPO Update to address (e.g., FAR numbers? Carports?) as well as what information should be studied (e.g., the number of appeals? The length of time for project approval?). The possible elements to address are discussed in the “NPO Update elements” section, and the information to study is discussed in the “Indicators” section. Which elements and indicators are addressed would depend on the scope of the review. Ideally a review would be broad (in terms of addressing all of the relevant elements of the NPO Update) and deep (analyzed fully) enough to maximize the likelihood of NPO Update success. A comprehensive, rigorous review would reduce the likelihood that unaddressed issues will interfere with the NPO Update’s long-term success. At the same time, a review with a broad and deep scope would pose costs to staff and limit their ability to address other important projects. There may be an inherent tradeoff between the two fundamental review objectives, because maximizing the probability of NPO Update success entails costs. Ultimately the scope would depend on staff resources and stakeholders’ expectations. Interviewee responses did not suggest a consensus regarding the scope. While several interviewees felt the review should be minor (e.g., “Make a few tweaks.”), others felt it would be worthwhile to re-examine major components of the NPO Update, such as the principle of regulating design using FARs. If the elements and indicators are prioritized, staff can focus on as many as constraints allow. Accordingly, the alternative elements and indicators discussed in this section are evaluated based on factors that can help to prioritize among them. Page 21 of 50 One way of prioritizing the elements and indicators is to determine which ones are most directly relevant to the fundamental objective of NPO Update success. In other words, which factors determine success? Interviewees were asked which factors they felt would contribute to NPO Update success or lack thereof. The determinants of success mentioned were: • The clarity, simplicity, and consistency of regulations, guidelines, and procedures. • The nature of completed projects (“on-the-ground results”). • Stakeholders’ satisfaction. • Whether there are any loopholes. • Stakeholders’ inclusion in the NPO Update process and the review process. • The functioning of the SFDB and associated guidelines. • The number and nature of appeals. • FAR chart formulas and the cutoff between standards and guidelines. NPO Update elements Table 3 lists various NPO Update elements and shows whether they meet certain criteria that may suggest their priority in a review. Not every possible element of the NPO Update is included in the table, only those that meet at least one of five criteria: • The element was mentioned by City Council members two years ago as a potential review topic. • The element appeared to be a subject of concern for at least three interviewees. Three is an arbitrary number, but it seemed to be a useful threshold because most of the elements that were raised as concerns more than once were mentioned by about three interviewees. Interviewee comments were coded as “concerns” if they raised specific suggestions (such as, “The carport loophole should be fixed.”) or expressed uncertainty as to whether problems existed (such as, “Is the Built Green program working as intended?”). • The element was mentioned as a determinant of success by at least one interviewee. • The element appeared to be a controversial topic during the NPO Update, based on a review of meeting notes. Page 22 of 50 • The element was described as major in staff’s May 1, 2007, Council Agenda Report summarizing the NPO Update changes. Table 3. NPO Update Elements Rated on Review Priority Criteria Review Priority Criteria NPO Update Elements City Council review topic? Interviewee topic of concern? Interviewee determinant of success? Controversial topic? Major element? Single Family Design Board √ √ √ √ Single Family Design Guidelines √ √ √ FAR square footage cutoffs √ √ √ √ 85% of FAR trigger √ √ √ FAR chart numbers √ √ √ Private view guidelines & findings √ √ √ 20 closest homes tool √ Porch regulations √ Carport regulations √ FAR guidelines in multi-family zones √ Design review triggers √ Project approval findings √ Hillside Design District boundaries √ Green building requirements √ Noticing requirements √ = NPO Update element meets at least three of the review priority criteria Page 23 of 50 These five criteria were selected because they represent means objectives (e.g., “Address City Council topics.”) identified as relevant to the fundamental review objective of ensuring NPO Update success. The elements were rated based on these criteria rather than on the fundamental objectives of the NPO Update, because they appeared more applicable. The various NPO Update elements are generally relevant to all of the fundamental objectives, so the specific choice of elements to review would probably not directly affect the achievement of the fundamental objectives. The table presents just one way of rating the NPO Update elements. It is not meant to definitively determine which elements should be reviewed, but it may be a useful starting point for making this decision. The table does not consider the magnitude with which elements meet the criteria. For example, while the FAR chart numbers and 85% FAR trigger may have raised concerns among the same number of interviewees, it is possible that the concerns regarding one of those elements are stronger than the concerns regarding the other. Thus, if interviewees were asked to rank the elements based on the criteria, the results might be different. This table can serve as a framework that staff modifies based on staff’s own ratings and discussions with stakeholders. In the table, the shaded rows indicate NPO Update elements that meet at least three of the criteria. Six NPO Update elements met three or more criteria: the creation of the SFDB; the revisions to the Single Family Design Guidelines; the choice of FAR square footage cutoffs; the 85% FAR trigger; the FAR chart numbers; and private view guidelines and findings. Three of the criteria — “interviewee topic of concern,” “interviewee determinant of success,” and “controversial topic” — appear to be correlated in the table. Elements meeting one of these three criteria tend to meet the other two as well. This makes sense; topics that were originally controversial can easily concern interviewees still and would logically be critical to satisfying stakeholders and thus achieving NPO Update success. About half of the major NPO Update elements appeared to be controversial and/or concerning, whereas the other half were not. The elements that meet only one of the criteria may still be worth reviewing. For example, noticing requirements only meet the “interviewee concern” criterion, but this suggests there may be Page 24 of 50 unresolved problems to address. Also, the elements that only meet the “City Council” criterion might require at least a basic level of review, since the City Council is a crucial stakeholder. Indicators Indicators “reveal changes that represent a bigger question or problem” (Guijt 2000). Indicators could be a useful part of the review, because they can reveal potential problems and measure progress toward certain objectives. Indicators may not show whether a policy is responsible for achieving objectives, but they can determine a need for analysis and to look deeper for causal relationships (Ientelucci 2005). If an indicator’s performance is clearly unacceptable, then it suggests a need for further review. Conversely, good performance on an indicator may suggest no changes are needed. Another reason for using indicators is to track outcomes over time and ensure that no problems emerge. As discussed earlier, success is not easily defined in many planning problems, but indicators can serve as partial definitions. Indicators can be either quantitative or qualitative. Quantitative indicators have long been popular because they provide hard numbers, but qualitative indicators are increasingly commonly used by municipalities and can provide an important “reality check” and texture that quantitative data does not (Seasons 2003). Also, quantitative indicators can measure seemingly qualitative impacts, and vice versa. There are important limits to the proper use of indicators. First, indicators can be subject to multiple interpretations. A large number of modifications, for example, could suggest that regulations are too strict, but it could also be interpreted as evidence that modifications are too readily granted. Second, an observed change in an indicator does not indicate that a particular policy is responsible for the change, unless there is no other reasonable explanation for the change. For example, a reduction in average project approval time might be caused by a more efficient design review process, or there may be economic or other factors prompting applicants to propose smaller, less controversial, and more quickly reviewable projects. Third, the explanation for an indicator’s performance can be more informative than the indicator itself. Indicators do not explain how or why outcomes are achieved. In some cases, indicator performance that seems favorable (e.g., fewer hearings per project) may have unfavorable explanations (e.g., an inappropriately lenient design review board). Fourth, it is important not to let indicators replace the original objectives. The focus should be on achieving objectives rather Page 25 of 50 than on improving performance on an indicator. These and other limitations are substantial (Perrin 1998), but indicators are nevertheless useful and common. Potential indicators were identified from two interview questions. The answers to both of these questions were often easily rephrased as potential indicators. First, interviewees were asked how they would define the success of the NPO Update. The most common definitions of success were variations of “Everyone is happy/satisfied.” Interviewees emphasized the importance of there being a consensus that the NPO Update policies represented a reasonable compromise. Many interviewees said they did not feel it was possible to fully satisfy all stakeholders, but that people would hopefully at least agree that the outcomes were fair enough. The definitions of success tended to be outcome-oriented. Examples include, “The design review process is simpler” and “There is less controversy.” Definitions focused on the NPO Update process (e.g., “Everyone was able to contribute input,” or, “All the relevant issues were studied thoroughly”) were not mentioned. Interviewees’ definitions of success were usually qualitative — for example, “Neighbors feel they have more control” — rather than quantitative. One interviewee stated, “Success is not: All homes are less than 85% of the maximum FAR.” Interviewees’ reluctance to define success quantitatively appeared to reflect a consensus that objectives such as neighborhood compatibility are subjective and qualitative in nature. When interviewees were asked the second question (“What should the review measure or evaluate?”), many similar qualitative indicators were mentioned. While few quantitative indicators were suggested as definitions of success, they were frequently suggested as responses to this second question. Table 4 presents the potential indicators. Indicators are listed in this table only if statements from multiple interviewees seemed to suggest them. The table displays both qualitative and quantitative indicators. For each indicator, the “Indication of” column describes what the indicator might reasonably be expected to indicate. For example, the number of appeals of SFDB decisions can indicate whether the decisions are perceived as striking the right balance between applicants’ and neighbors’ interests. Fundamental NPO Update objectives relevant to each indicator are listed in the “Related Objectives” column. Additional indicators inferred from at least one interviewee are presented in Appendix B. The city can generate other indicators based on additional interviews, brainstorming, or public comments and participation. Page 26 of 50 Table 4. Potential Indicators Indicator Indication of Related Objectives1 Qualitative indicators Level of general satisfaction Objective achievement Overall Level of controversy or conflict Objective achievement Overall; 3.2 – Minimize controversy Perception of projects’ neighborhood compatibility Objective achievement 1 – Neighborhood compatibility Projects’ compatibility with guidelines Whether projects are meeting guidelines 1 – Neighborhood compatibility Neighbors’ ability or sense of ability to participate in design review process Objective achievement 3.3 – Neighbors’ comfort and influence Perception of fairness Objective achievement 5 – Fairness Consensus among all stakeholders that NPO Update was successful (enough) Objective achievement Overall Quantitative indicators Number of projects reviewed and built Whether homes are being approved and built Overall; 5 – Fairness Number of projects reviewed and approved within various FAR ranges Effects on home size; fairness of regulations; SFDB functioning Overall; 5 – Fairness Number of appeals and/or % of projects appealed Level of controversy; satisfaction with SFDB decisions 6 – Costs; 3.2 – Conflict / controversy; 5 - Fairness Length of time for project approval Costs to applicants; efficiency of design review process 6 – Costs; 2 – Clarity et al. Number of hearings per project SFDB functioning; applicants’ responsiveness to SFDB input; design review process efficiency 6 – Costs; 2 – Clarity et al. Number of projects meeting various other project characteristics Types of projects being built; restrictiveness and fairness of process 1 – Neighborhood compatibility; 4 – Environment; 5 – Fairness Number of complaints Concerns regarding particular NPO Update elements Overall 1 As described in Table 1 Page 27 of 50 As the “Indication of” column suggests, many of the qualitative indicators measure the achievement of the fundamental NPO Update objectives fairly directly. This is because many of the objectives are dependent on people’s qualitative opinions. For example, because “neighborhood compatibility” is a highly subjective concept, a reasonable measure of whether “neighborhood compatibility” is accomplished is simply whether stakeholders feel it is. Thus the indicator, “perception of projects’ neighborhood compatibility.” Many of the qualitative indicators are essentially the fundamental objectives restated and thus indicate whether people feel these objectives are accomplished. The quantitative indicators, on the other hand, measure objectives’ achievement more indirectly. They could be useful, though, for their insight into the design review process. Most of the identifiable quantitative indicators concern statistics regarding various project characteristics. For example, many interviewees suggested measuring how many projects were reviewed and built both overall and within certain FAR ranges. Other common project characteristics mentioned were the length of time for project approval and number of hearings per project. These types of indicators reflect what actually gets built, which is obviously relevant to the fundamental objectives. They also reflect the general functioning of the SFDB and the rest of the design review process by revealing whether the regulations appear to be overly easy or difficult to comply with and whether the process is easier or less time- consuming than before. The number of possible indicators related to project characteristics is very large. An indicator could be created for virtually any project characteristic that seems to merit tracking. To determine which indicators to measure, the indicators could be evaluated using the two fundamental review objectives. However, at this point it is difficult to say how the indicators would perform on these two objectives. In terms of the “costs” of each indicator (whether measured in terms of difficulty or workload), this would likely depend on how the indicators are measured. This is discussed further in section 4.3.3. Some of the indicators may be difficult to measure in the short term, because not enough projects have been reviewed and built to yield sufficient data. However, they may be well worth measuring once further data is available. In terms of the second review objective, ensuring the success of the NPO Update, any of the indicators could potentially be useful. While the qualitative indicators are useful in directly measuring the achievement of NPO objectives, the quantitative indicators could be used to track Page 28 of 50 information that would help the city understand the NPO Update and thereby make any necessary improvements. Thus the important question may not be which indicators are the most useful, but which are the most useful for the particular purpose, subject, or alternative being studied. 4.3.2 Public Involvement Methods Public input can be important both for making an informed, high-quality decision and building acceptance among stakeholders (Thomas 1995, 36). Public input could play a part in four stages of the review process: 1. Determining how the review should be conducted (i.e., planning the review). 2. Understanding outcomes thus far, such as whether the NPO Update is meeting its objectives and why or why not. 3. Making changes to policies if needed. 4. Ongoing, long-term monitoring of the NPO Update’s success. Interviewees emphasized the importance of public involvement. Many interviewees commented that public involvement would be necessary to determine whether the NPO Update achieved its objectives. There was a sense that success would depend on stakeholders’ experiences and impressions. There was a consensus that it was important to hear from people who were affected by the NPO Update (particularly applicants and neighbors), as well as the SFDB that applies the new policies to projects. There was no consensus as to the appropriate method for involving the public. One respondent suggested reconstituting the steering committee that helped to craft the NPO Update policies. Many interviewees seemed to assume there would be a public hearing to solicit input. Five possible public involvement methods were identified: 1) discussions with key contacts; 2) steering committee; 3) public hearing; 4) public workshop; and 5) surveys. These five methods are discussed below and are compared in Table 5. The five methods can be compared using several factors. First, different methods appear to be relevant to different stages of the review process. For example, a steering committee would seem particularly appropriate if there is a need Page 29 of 50 to make changes to existing policies or craft new policies. A second basis for comparison is the likelihood that stakeholders would accept the method. Methods are unlikely to be acceptable unless all relevant stakeholders feel they have a meaningful opportunity to provide input. While any of the methods could be acceptable in combination with others, some, such as a survey approach, may not satisfy stakeholders who wish to be able to give input to decision makers more directly. The table also distinguishes between the methods’ breadth or depth of information and qualitative versus quantitative focus. Methods like interviews with key contacts can yield relatively in-depth information that can help in understanding the nuances of the NPO Update. However, it is time-consuming to involve several stakeholders in such an in-depth method. Other methods, such as surveys, are better at yielding a breadth of information from many individuals, but the ability to obtain in-depth information may be limited. Surveys and public workshops are likely to have a relatively quantitative focus, while other methods like public hearings would easily yield stakeholders’ qualitative impressions. The final basis for comparison in the table is the methods’ cost or difficulty. Table 5. Public Involvement Methods Compared Criteria Public Involvement Methods Discussions with key contacts Steering committee Public hearing Public workshop Surveys Relevant stages of review process Planning; understanding outcomes; long-term monitoring Making changes; planning Understanding outcomes; making changes Understanding outcomes; making changes; long-term monitoring Understanding outcomes; long-term monitoring Likelihood of acceptance Low if not supplemented by other approaches Moderate High Uncertain, depends on nature of workshop Low if not supplemented by other approaches Information depth vs. breadth Depth Depth Breadth Breadth Breadth Qualitative or quantitative focus Qualitative Qualitative Qualitative Either Quantitative Cost/difficulty Depends on scope High, potentially Low Medium Medium Page 30 of 50 Method 1: Discussions with key contacts This method would entail interviewing, holding meetings with, or simply having conversations with, key contacts. The key contacts would be important stakeholders or representatives of stakeholder groups. These key contacts are probably particularly useful for deciding how to conduct the review, sensing the general level of community satisfaction, and identifying major concerns. This method may be controversial if it is the only one used, because it may easily exclude people who have important opinions. The more people who are included in the discussions, and the more in-depth the discussions, the more useful the method can be, but the tradeoff is the increased time and effort required. One-on-one discussions with key contacts may yield more in-depth, forthcoming, and useful information but are more time-consuming than group meetings. Method 2: Steering committee This method entails forming an advisory committee similar to the one that helped to craft the NPO Update policies. The most logical use of such a committee would be to craft new policies in response to perceived problems with the existing ones. If major problems are revealed during the course of the review, such a steering committee may be necessary and appropriate. Otherwise, it is a potentially costly approach, as the scope of issues addressed by a steering committee can expand easily and become time-consuming. Method 3: Public hearing A key purpose of a public hearing is to give members of the public a chance to comment on any relevant issues, and to do so directly to relevant decision makers. This method helps to ensure that people’s voices are heard, literally. It is likely to be highly accepted by stakeholders, as it is a commonly used method. It is also fairly easy to conduct, and it allows for numerous people to participate. The combination of a high likelihood of acceptance and low cost/difficulty makes this method a compelling alternative for understanding NPO Update outcomes and identifying potential policy changes as long as broad, qualitative information is desired. However, one interviewee was concerned that such hearings should not be the only method used because they can potentially draw opposing parties who repeat similar comments. While hearings can be useful in receiving a large quantity of public input, not all of it may be unique. Furthermore, the Page 31 of 50 nature of public hearings tends to limit the time of speakers’ comments and thereby limit the depth of information obtained. Method 4: Public workshop Public workshops allow for different types of public input than a public hearing. Often they are used to solicit public input regarding particular options. Public workshops can include various exercises where participants share their opinions. At the beginning of the NPO Update process, planning staff held workshops where participants were asked to rate the neighborhood compatibility of homes and to fill in a survey about their opinions regarding the city’s residential development. For the review, the city could hold similar workshops where participants rate new development and answer survey questions tailored to the qualitative indicators discussed earlier. Whether workshops are acceptable would depend on the nature of the workshop and whether participants felt it provided an opportunity to provide relevant input. Workshops could be used for multiple stages of the review process: understanding outcomes, monitoring indicators over time, or even developing indicators. Method 5: Surveys Surveys can be a part of public workshops, or they can be used independently. One way to understand the outcomes of the NPO Update is to send a survey to relevant stakeholders, who may include: applicants who have experienced the new policies, people on staff’s “NPO Update Interested Parties” list, and relevant organizations such as homeowners’ or realtors’ associations. Surveys can also be used to track opinions over time. For example, Lake Forest, Illinois, uses exit surveys of all applicants to solicit opinions regarding the design review process. Surveys can give a large number of people an opportunity to provide input. The results of surveys can be systematically analyzed in order to gather statistics, but response rates to surveys can be low. A low response rate may suggest that people have few concerns regarding the NPO Update, but it may also be due to a poor survey design that seems confusing or irrelevant to respondents. Surveys may not be acceptable to some stakeholders unless supplemented by other methods that allow for more direct, free-form responses than surveys may allow. Page 32 of 50 4.3.3 Analytical Methods It appears that staff would be expected as part of the review this year to analyze the outcomes of the NPO Update thus far. The general purpose of such analysis would be to present data and insight that could serve as a foothold for City Council’s or staff’s decision making. This analysis could determine: • What outcomes have resulted. • How and why particular outcomes resulted. • Whether NPO Update objectives have been achieved. • Whether particular policies are responsible for achieving NPO Update objectives. Such analysis may be statistical and quantitative or simply just a summary of qualitative impressions and public input. Particular analytical methods are described below. There also exists a general analytical technique that could apply across methods: comparing before-and- after data, i.e., data before and after the NPO Update went into effect. The point of this technique is to measure net outcomes, which are the outcomes attributable to the NPO Update policies. It would be hard to prove that the NPO Update is responsible for any particular outcomes, considering the other forces acting upon development. However, some interviewees did feel before-and-after comparisons would be valuable. Even if causality cannot be proven, the before- and-after data could at least suggest that a problem is not worsening. The simplest way to do these comparisons is to measure a variable over time. If, for example, home size is showing a particular trend prior to policy implementation, and the trend changes after implementation, one might consider the NPO Update to be one possible explanation for that change. The degree to which policies could be considered responsible depends on the likelihood of alternative explanations for any observed change. The ideal before-and-after comparison is one in which one group being studied is subject to the policy of interest and the other is not. If the group subject to the policy shows a change whereas the other does not, then the reviewer could consider the policy to be one possible explanation. The degree to which the policy can be considered responsible depends on how similar the two groups truly are. Page 33 of 50 Alternative analytical methods Following are different analytical methods staff can use in conducting the review. These methods were identified from interviewee comments, as well as background research and brainstorming. Method 1: Public input analysis The simplest form of analysis may be reporting the results of whatever public input staff gathers. This may involve categorizing and counting public comments and then presenting the results to decision makers. For example, staff might determine which concerns regarding the NPO Update were mentioned most frequently. Public input can also be used to measure qualitative indicators such as the ones described in section 4.3.1. These qualitative indicators may be measured informally. Staff and other stakeholders may develop overall impressions of indicator performance based on public comment, conversations, interviews, etc. For example, the level of controversy before and after the NPO Update may seem self-evident to stakeholders and not require the support of explicit data. Alternatively, staff can attempt to provide City Council and the public with explicit measures of performance based on its own judgments or on quantitative data. A key consideration would be the expectations City Council and other stakeholders have for answering qualitative questions such as, “Are people satisfied?” If staff is not expected to interpret and summarize public sentiment on these qualitative questions, very little analytical work would be involved. Alternatively, the qualitative indicators could be measured quantitatively. For example, one can measure perceived neighborhood compatibility using a numeric scale, such as one to five, with each number representing a defined degree of controversy. Doing so would involve asking relevant stakeholders to participate in a rating exercise, such as on a survey or at a workshop. This quantitative approach would require more effort than forming a qualitative judgment, but it would also yield concrete data that may be desired. Method 2: Project data analysis Interviewees often suggested that the review should measure quantitative project data. Hard data is often valued by stakeholders and decision makers, and analyzing project data is the logical way to answer some basic questions (e.g., “Are homes getting smaller?”) and more complex questions (such as, “Are homes’ footprints increasing due to the greater difficulty of building Page 34 of 50 second-story homes?”). Project data analysis focuses on important aspects of the NPO Update: project proposals and the resulting homes. This method would be useful in addressing particular concerns regarding NPO Update elements. For example, interviewees had questions regarding FAR guidelines and whether they were being adhered to on large lots. This could be determined by studying project data. Project data analysis could be considered a valuable, yet incomplete part of a review process. It cannot provide the type of insights derived from learning about individuals’ personal experiences. Furthermore, while this type of analysis helps with understanding the basics of the design review process, the results need to be carefully interpreted. For example, if projects are always under the 85% of maximum FAR threshold, it may mean that the FAR threshold is too generous to have any effect, or it may mean that exceeding the 85% FAR is very cumbersome. Also, project data analysis may not always be necessary. One interviewee suggested that if stakeholders have no serious concerns regarding the existing policies, it may not be practical to analyze the policies and project data further. The amount of effort required for project data analysis would depend on the ease of collecting the data. Some project data may be relatively easy to analyze, but it appears that the city’s project permitting database is not always conducive to analysis. Project information can be listed inconsistently in the database. Furthermore, the database does not have fields for all information, so analyzing some project characteristics would require individually researching each project in order to record the relevant data. Table 6 shows project characteristics that may be useful in analyzing project statistics. Project characteristics include aspects of the site itself (such as the lot size), development aspects (e.g., the square footage added), and aspects of the design review process (e.g., the number of hearings the project receives). The development characteristics can be measured either as proposed or as approved. The approved information should be more similar to what actually gets built, but the proposed information may be more easily discernable in the city’s database. Staff could collect this data by compiling a spreadsheet with each characteristic filled in for each project. Doing so for every project would be time-consuming, but once complete, it could be very useful in answering in-depth questions. Updating the city’s database to better capture this information may be very difficult in the short term but could make future analysis easier and more reliable. Page 35 of 50 Table 6. Project Characteristics for Use in Analysis Site Characteristics Development Characteristics Design Review Process Characteristics • Lot size • Zone (single-family, multi-family) • Location (hillside vs. infill) • Slope • Scope (new home, addition, deck, fence/wall) • Number of stories • Total home square footage • New square footage • FAR • % of max. FAR • Grading • Building height • Garage square footage • Apparent height • Footprint/lot coverage • Design review level (full, consent, admin) • Number of hearings • Length of time for approval • Number of public comments received • Any modifications requested or granted? • Any appeals? This data would provide a foundation for a potentially large degree of analysis. The possible types of analysis could fall into five categories, described below: • Indicator tracking / before-and-after comparisons • Correlations (i.e., what characteristics are associated with good/bad outcomes?) • Triggers (i.e., which design review triggers are causing projects to be reviewed?) • Constraints (i.e., to what extent do the regulations appear to be constraining projects?) • Potential loopholes Indicator tracking / before-and-after comparisons The project data could be used to track quantitative indicators. The indicators could be used to identify concerns or to try to see whether a policy is responsible for observed changes. Variables related to the design review process are the most likely to be determined by NPO Update policies. The new SFDB, for example, is entirely a result of the NPO Update. Staff can measure whether, relative to the Architectural Board of Review, there have been changes in the length of time for project approval, the number of hearings per project, or average hearing time per project. Page 36 of 50 Correlations If staff gathers enough project information, they can search for correlations between project characteristics. It may be particularly useful to discover any correlations between controversial projects and particular project characteristics in order to find ways to limit the controversy of future projects. While controversy cannot be measured directly, projects subject to appeals could clearly be considered controversial. The number of public comments a project receives could serve as a reasonable proxy for controversy as well. Triggers A major part of the NPO Update was revising the “triggers,” or project characteristics, that result in design review. The NPO Update also created several triggers that, if met, reduce the allowable square footage to 85% of the maximum FAR. Project statistics can help to understand how the design review process is operating by showing why projects receive certain types of review. Studying triggers can answer questions such as, “What percentage of projects subject to design review are triggered by fence or wall triggers?” Also, staff can determine which triggers tend to reduce projects’ allowable square footage to the 85% threshold. Constraints It may be desirable to determine how regulations are constraining single-family home design. This approach was used by Scarsdale, New York, to review its FAR regulations (Janes 2006). The main effect of FAR regulations, for example, is to prevent homes from exceeding a certain size. Therefore, it would be expected that FARs are indeed constraining home size. If most of the homes built are far smaller than the FARs would allow, it would suggest that FARs are not a main constraint on home size. However, if homes tend to be slightly smaller than the FARs would allow, FARs could reasonably be considered a constraint on home size. This is especially true if homes built before the NPO Update were more frequently proposed at sizes higher than the current regulations would allow. Constraints can also be measured by determining how many projects are requesting modifications. The city could then compare how many projects are constrained by one or more of the regulations, both before and after the NPO Update. The before-and-after comparison would help to determine the extent to which the NPO Update as a whole has served to constrain home development. Page 37 of 50 Another form of studying constraints is to determine whether the design review process itself, as opposed to regulatory limits, is constraining development. Some interviewees were curious whether applicants were trying to avoid design review under the new process due to a perception it is more cumbersome. One big change implemented by the NPO Update was the triggering of all second-story projects for design review. Therefore, staff could measure the ratio of one-story to two-story homes both before and after the NPO Update. A dramatic increase in one-story homes relative to two-story homes might reasonably be interpreted as a sign that applicants were indeed avoiding design review. If many projects are constrained by the regulations or design review process, it may be interpreted as a sign that the new rules are too restrictive. Very few constrained projects might suggest the opposite. However, the appropriate number of projects subject to constraints would be highly subjective. Potential loopholes Some interviewees were concerned that certain exemptions from FAR calculations — including carports and certain porches — were serving as loopholes. One way to measure this would be to determine how many projects would cross 85% or 100% FAR requirement thresholds if these features had been included in FAR calculations. If the percentage of projects in this category seems high, or is beyond a certain threshold such as more than half of projects, it would suggest that the exemptions are indeed significant. This could be complicated to analyze given the way the city’s database is configured, but it might be worth it given that City Council members and other stakeholders expressed concerns regarding loopholes. Also, note that this technique does not indicate whether an exemption is appropriate, only whether it is potentially influential. Method 3: Case studies of project approval process Staff could conduct case studies of individual projects to explore the design review process in detail. This method might entail a mix of analyzing meeting notes and project data and interviewing SFDB members, staff, and applicants. Interviewees mentioned several recent projects that they felt were highly controversial, difficult to review, or not particularly aesthetic. These projects could serve as case studies that staff analyze in depth to determine why the process was difficult and why the outcomes were dissatisfying. Case studies could be used not just for controversial projects, but also in general to understand how the SFDB is operating. Page 38 of 50 Interviewees suggested questions such as, “What about the project changed as a result of SFDB review?” Whereas the project data analysis method is concerned with analyzing project statistics, this method would look in depth at individual projects to determine nuances that statistics alone would not reveal. Method 4: Site visits of projects built under new rules Some interviewees felt that site visits were a highly valuable part of the NPO Update process because they allowed participants to understand how and why certain homes appear incompatible with the neighborhood. Seeing homes in their actual context can be valuable. Staff could thus organize visits to homes built under the new NPO rules, and participants could rate the homes’ neighborhood compatibility. These ratings could be a sense of compatibility overall and/or whether the homes comply with the city’s Single Family Design Guidelines. Compliance with neighborhood compatibility guidelines was an indicator mentioned by several interviewees. Site visits could also be one component of a case study approach in which individual projects are analyzed in depth. A downside of site visits is that they may not be very useful in the short term, assuming few projects have been built under the new regulations. Many interviewees said that it would be difficult to form overall conclusions about the NPO Update until numerous homes have actually been built under the new rules. Therefore, this method may be relevant in the long term. Method 5: Visual survey workshops Visual survey workshops were used during the NPO Update process to rate homes’ neighborhood compatibility. Participants rated photographs of homes using a multiple-choice survey, rating each home as either “very appropriate,” “may be appropriate,” “neutral,” “may not be appropriate,” or “not appropriate.” This method could be repeated following the NPO Update. An advantage of doing so would be that it could compare the compatibility of homes built both before and after the NPO Update in a systematic way using public or SFDB members’ input. A disadvantage is that photographs, or other representations of homes’ appearance, may be no substitute for visiting the homes in their neighborhood context. Also, small changes in the questions’ wording could dramatically affect the results. In order for the results to be directly comparable to the workshops conducted earlier, the surveys should be very similar. Preferably, Page 39 of 50 several of the homes rated in the earlier workshops would be rated in the new workshops as well in order to calibrate the results. Method 6: Computer simulations Another method is to use computer simulations to try to estimate how projects’ visually apparent bulk may be affected by the new regulations. Actual projects can be represented on the computer using CAD, and then their dimensions can be modified or measured in order to understand regulations’ effects. Scarsdale, New York, used this method to determine the effectiveness of its FAR regulations at controlling visual bulk (Janes and Heagney 2007b). Scarsdale’s consultants selected projects with high FARs built before the regulations went into effect and modified their designs using the computer to make the homes fit within the FAR maximums. They then calculated the area of the homes’ visual outlines, as viewed from a uniform public viewpoint, both before and after the redesigns, in order to calculate the FARs’ effects on bulk. They also simulated redesigns of homes built after the new regulations went into effect in order to show how they could be altered to increase the appearance of bulk. An advantage of this method is that it can be used to explore what types of designs are possible under certain combinations of regulations and site constraints. Thus it can be used for what-if scenarios that may be valuable if the city feels it is necessary to consider changes to the existing regulations. Also, when used to study FAR regulations, the method provides a quantitative measure of FARs’ effects on visual appearance, which may be desired. However, computer simulations may not be an appropriate substitute for seeing actual homes. Also, Santa Barbara’s FARs were not intended solely to regulate the visual appearance of bulk. As a method of evaluating FAR regulations, the method excludes a range of issues relevant to homes’ neighborhood compatibility and aesthetics, such as the neighborhood context and the homes’ scale, materials, and architectural style. Furthermore, the city’s concept of bulk, as defined in the Single Family Design Guidelines, relates to the building’s qualitative massing rather than the visible quantitative volume (City of Santa Barbara 2008). It is also a potentially expensive method, since the city may need to outsource this work to a consultant. Page 40 of 50 Comparing the analytical methods Table 7 compares the six analytical methods described above based on four criteria: cost/difficulty, short-term feasibility, the methods’ relevant uses, and interviewees’ general opinions regarding the method. These criteria were selected as proxies for the fundamental objectives, as it appears any of the methods could potentially help in achieving the fundamental objectives. In the short term, case studies, project data analysis, and public input analysis are all possible. Numerous interviewees suggested both project data analysis and public input analysis. Case studies were also suggested, though less often. Public input analysis would also be relatively easy to conduct and is thus an obvious choice, assuming the city wishes to measure qualitative indicators. Project data analysis is immediately relevant and would be useful for measuring quantitative indicators, although it could potentially be more painstaking depending on the type of analysis and the data limitations. Case studies could be conducted now as well, with moderate difficulty, if there is a need to learn from specific projects. Table 7. Analytical Methods Compared Analytical Methods Criteria Public input analysis Project data analysis Case studies Site visits Visual survey workshops Computer simulations Cost / difficulty Low-Medium Potentially high Medium Medium Medium High Short- term feasibility High High High Low Low Low Inter- viewee opinion Widely suggested Widely suggested Not mentioned Suggested Mentioned Not mentioned Relevant uses Understanding general impressions and concerns; measuring qualitative indicators Generating statistics; finding correlations; analyzing NPO Update elements; measuring quantitative indicators Analyzing functioning of design review process Understanding on-the-ground results; rating perceived neighborhood compatibility Rating perceived neighborhood compatibility What-if scenarios Page 41 of 50 The other three methods have potential uses over the long term. Computer simulations would probably be uniquely expensive and would thus be justified only if the city needs to conduct what-if scenarios. Site visits and/or visual survey workshops would be appropriate for measuring the key objective of neighborhood compatibility. This was such a prominent objective that it would perhaps be necessary to eventually, if not at present, analyze whether homes built under the new rules are indeed perceived as more compatible. While interviewees mentioned both site visits and workshops as possible options, only site visits were unequivocally endorsed. Thus they would appear to have an advantage. 4.3.4 Long-Term Review Strategies While the city may be able to get a preliminary sense this year of whether NPO Update objectives are on track to being accomplished, it may take longer to reach conclusions regarding the NPO Update’s success. As noted earlier, there may not yet be enough evidence to conclude how NPO Update policies are affecting development. Therefore, the city may wish to develop a long-term review strategy. This can serve two purposes. First, it can set aside for later those elements that cannot yet be reviewed. Second, it can be an ongoing way to ensure the success of the NPO Update. Even if it seems very likely after this year’s review that the NPO Update will be a success, implementing a long-term review strategy will help to ensure that any problems that develop later are addressed. The city would presumably not choose a long-term strategy until after the immediate review task is completed and staff has reported review results to City Council. Perhaps at that time, staff can present strategies such as the ones described in this section and make recommendations regarding them. Description of strategies Strategy 1: Informal, continuous review Under this strategy, city staff would not plan any formal long-term review efforts but would make any ongoing, probably minor adjustments based on any problems that are brought to staff’s attention. This could be considered a default or status-quo strategy that staff would probably follow at a minimum regardless of how the review unfolds this year. This strategy may seem appropriate if the review conducted this year reveals few problems and the city feels it cannot prioritize further formal review efforts. Page 42 of 50 Strategy 2: Schedule another review for a later date Staff could schedule a later review — for example, two years from now — at which point more data may be available for forming conclusions about the NPO Update’s ultimate impacts. This may be relevant if it is very hard to determine whether the NPO Update is on the right track or not. It may also be appropriate if major problems are revealed and major changes to the city’s policies need to be made as a result. In that case, the later review would effectively be a review of those changes. The difficulty or cost of this strategy would depend on the nature of the rescheduled review. Strategy 3: Regularly scheduled follow-up meetings with stakeholders Using this strategy would entail meeting regularly, perhaps annually, with stakeholders to solicit any concerns. This strategy is appropriate if there is a desire to receive regular feedback from important stakeholders such as architects, neighbors, etc. Such meetings would probably only be able to address relatively minor issues and would therefore be most appropriate only after any major issues regarding the NPO Update have been resolved. Important considerations with this strategy would be deciding who is invited to the meetings and how the meetings would be facilitated. Strategy 4: Exit surveys of applicants Staff could offer surveys to every applicant once the applicant’s project has received its building permit (or been denied a permit). Questions could target specific aspects of the design review process such as the SFDB’s review or interactions with staff. This strategy would provide ongoing feedback that could be used to improve the design review process. Survey responses could serve as indicators that can be used to analyze performance over time. Exit surveys would have a narrow focus. They would only target one segment of the community (applicants) and would only address their experiences with the design review process. However, such experiences could be considered an important focus of the NPO Update. If this year’s review determines that the design review process has become more onerous for applicants, as some interviewees believed may be the case, this would be an appropriate method for analyzing improvement on this particular issue. However, because of its specific focus, it should probably not be used as the sole indicator of success. Page 43 of 50 Strategy 5: Indicator monitoring Although it may not be possible to measure some indicators as part of the immediate review, staff could develop a list of indicators that the public and staff feel should be measured at regular intervals. Other strategies, such as surveys or meetings, could be used to determine performance on the indicators. For example, staff could maintain a database, such as of project information, and calculate annual statistics that serve as indicators. Doing so would likely require an initial investment of time and effort to ensure that there is a systematic process for collecting information over time, thereby simplifying future data analysis. This would potentially be a costly strategy, but would be useful if there is a desire for continuous status reports regarding certain indicators. It could also be helpful in understanding the long-term effects of the NPO Update. Indicators could be used later to analyze the NPO Update’s ultimate success. Strategy 6: Annual site visits of projects built within the past year Each year, staff, the SFDB, and other interested parties could visit homes that have been completed within the past year. Afterward, SFDB members and the public could comment at an SFDB meeting regarding what they learned from the visits. These insights could be used to develop ideas for how to improve policies and procedures. This strategy is obviously relevant for understanding the built results of the NPO Update policies. This strategy would require some effort but would be a way for the SFDB to evaluate its own performance and brainstorm ways to improve the process. As a standalone strategy, it would not address applicants’ experiences with the design review process, so it may be appropriately combined with a strategy such as exit surveys that can address these concerns. Comparing the strategies Table 8 compares the strategies according to three criteria: cost/difficulty, whether each strategy could produce qualitative or quantitative data, and the scenarios under which each strategy would be most relevant. The appropriate strategy would depend on the results of the short-term review. If the review activities conducted this year determine that the NPO Update is going very well, a less thorough strategy such as the informal Strategy 1 may be considered sufficient. This strategy would also be appropriate if it appears that few staff resources can be dedicated to ongoing review activities. If, on the other hand, there remains considerable uncertainty regarding the Page 44 of 50 success of the NPO Update, review activities may need to be scheduled for a later date. This is particularly true if the current NPO policies require major adjustments. The remaining strategies may be valuable if the policies do not seem to require major changes yet are not deemed entirely successful thus far and merit further review effort. Follow-up meetings could gather general stakeholder feedback, while exit surveys would focus more specifically on applicants’ experiences with the design review process. Annual site visits, on the other hand, would address the actual built results of the process. Indicator monitoring could report data over time and thus help in determining long-term effects. Table 8. Long-Term Review Strategies Compared Long-Term Review Strategies Criteria 1: Informal 2: Reschedule 3: Follow-up meetings 4: Exit surveys 5: Indicator monitoring 6: Annual site visits Cost / difficulty Low Depends on review type Medium Medium Medium- High Medium Qual. vs. quant. Qual. Either Qual. Either Either Qual. Relevant scenarios Few problems observed; large focus on minimizing review costs Few conclusions reached; major problems require changes to policies Desire for ongoing stakeholder feedback; no major unresolved issues Desire for ongoing feedback from applicants; analyzing performance Desire for ongoing status reports; less focus on minimizing review costs; analyzing long-term effects Desire to understand built results 5 CONCLUSIONS AND RECOMMENDATIONS A well-conducted review will help in ensuring the long-term success of the NPO Update. Deciding how to conduct the NPO Update review may be aided by a structured decision-making approach that focuses on the key review decisions, defines the fundamental objectives of the NPO Update, and selects alternatives that assist in achieving the fundamental objectives. This decision-making process should be informed by stakeholders’ values in order to ensure that a Page 45 of 50 review is responsive to their needs. It is recommended that the city’s NPO Update review proceed with the following points in mind: • Using multiple strategies or methods in concert with one another can ensure that the full range of NPO issues and stakeholders’ concerns are addressed and that different types of information inform the review, thereby increasing confidence in the review results. • Review findings should incorporate meaningful public input from all stakeholders. Design review outcomes such as neighborhood compatibility are difficult to define objectively, and NPO Update success may ultimately depend on whether community members feel the NPO Update was reasonably successful at addressing their concerns. • There were two competing concerns prevalent during the NPO Update that a review would likely need to address in order to ensure that an acceptable compromise is in place. A complete review would thus address both neighbors’ concerns regarding neighborhood compatibility and applicants’ concerns regarding their ability to build reasonable projects. • While the city’s immediate review task may be a short-term project, it appears that some conclusions regarding the NPO Update’s success cannot yet be reached. It would thus be appropriate to consider how success would be reviewed in a long-term context. • The NPO Update review can be considered in the context of other priorities the city would like to address. Some of the people interviewed for this project felt that design and compatibility issues similar to those discussed during the NPO Update were now relevant to multi-family residential housing and downtown, mixed-use buildings. When considering the appropriate extent or scope of the NPO Update review, the urgency of addressing other planning issues can be considered. • Learning from the NPO Update can help to improve future planning efforts. Interviewee comments suggested that the main purpose of the immediate review should be to improve NPO Update policies as needed, but staff may, as a secondary focus, consider the lessons of the NPO Update process. A successful process can lead to successful outcomes. For example, one interviewee stated that site visits to actual homes were crucial in determining that FARs should apply only to two-story homes. Had the city not chosen the right process, the NPO Update may have included inappropriate policies less likely to achieve a successful outcome such as neighborhood compatibility. Page 46 of 50 Appendix A : Interview Script As you may know, the City is currently implementing its Neighborhood Preservation Ordinance / Single Family Design Guidelines Update (in other words, the NPO Update), and the City intends to conduct a review of the NPO Update this year. I’d like to ask you a few questions about the NPO Update and its review. The purpose of the questions is to learn your personal opinions regarding the NPO Update and its review. The questions are intended to be open-ended, so there’s no right or wrong way to answer. Do you have any questions at this point? 1. What do you feel were the goals of the NPO Update? 2. How would you define a successful NPO Update? 3. What do you think should be the goals of a review of the NPO Update? 4. What factors do you think will determine the success or lack of success of the NPO Update? 5. What do you think a review of the NPO Update should measure or evaluate and how? 6. Do you have any other comments or ideas about how the NPO Update and how it might best be reviewed? Page 47 of 50 Appendix B : Additional Potential Indicators Indicator Indication of Related Objectives1 Qualitative indicators SFDB members’ satisfaction with own decisions SFDB functioning Overall; 1 – Neighborhood compatibility Simplicity and clarity of design review Objective achievement 2 – Clarity et al. Degree to which guidelines are understood by applicants and public Familiarity with guidelines; ease of understanding guidelines 2 – Clarity et al. Ability of homeowners to build in a way that meets their needs Objective achievement 5.2 – Ability to meet needs Level of neighbors’ comfort or fear regarding neighborhood development Objective achievement 1 – Neighborhood compatibility; 3.3 – Neighbors’ comfort and influence Degree of project change between hearings SFDB functioning 6 – Costs Level of neighbors’ sense of control over own neighborhoods Objective achievement 3.3 – Neighbors’ comfort and influence Degree to which SFDB hearings are civil Level of conflict and controversy at hearings 3.1 – Conflict and controversy Community-wide level of frustration Whether problems and concerns have been addressed 3 – Community well-being Ability to move forward and focus on other issues Whether problems and concerns have been addressed Overall Homes meet indicators of sustainability Sustainability of homes 4 – Environmental preservation Quantitative indicators Number of documents distributed Number of people obtaining guidelines Overall Degree of change resulting from SFDB review SFDB functioning; degree to which applicants’ initial proposals conform with regulations Overall; 1 – Neighborhood compatibility Page 48 of 50 Indicator Indication of Related Objectives1 Degree to which applicants’ proposals already comply with regulations and guidelines Design review consistency and clarity; strictness of regulations and SFDB Overall; 2 – Clarity et al.; 6 - Costs Number of public comments per meeting Neighbors’ sense of influence; controversy 3.3 – Neighbors’ comfort and influence Number of modifications requested and/or granted Regulations’ strictness; ease of obtaining modifications 5 – Fairness; 6 – Costs Average hearing length SFDB functioning; controversy 6 - Costs Average hearing time per project SFDB functioning; controversy 6 - Costs Number of private view comments at SFDB hearings Private view impacts and controversy City Council review question; 1 – Neighborhood compatibility 1 As described in Table 1 Page 49 of 50 References City of Santa Barbara. Planning Division, Community Development Department. 2008. Single family residence design guidelines. February 28, 2008. Retrieved July 20, 2009, from http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/. ——— . 2007. Subject: Single Family Design Guidelines (SFDG)/Neighborhood Preservation Ordinance (NPO) introduction. Council Agenda Report. May 1, 2007. Retrieved July 20, 2009, from http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/. ——— . 2006a. Subject: SFDG/NPO Update Package transmittal memo. Transmittal memo. May 4, 2006. Retrieved July 20, 2009, from http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/NPO_Archives_2 005.htm. ——— . 2006b. Subject: Single Family Design Guidelines and Neighborhood Preservation Ordinance Update Package. Transmittal memo. August 8, 2006. Retrieved July 20, 2009, from http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/. ——— . 2004a. Issue Paper D: Part I. Floor to lot area ratio and lot coverage issues and options. Accessible at http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/npo_issue_papers .htm. ——— . 2004b. Neighborhood Preservation Ordinance work program detail. Accessible at http://www.santabarbaraca.gov/Resident/Major_Planning_Efforts/NPO/NPO_Archives_2 004.htm. Clemen, R. and Reilly, T. 2001. Making hard decisions with DecisionTools®. Pacific Grove, CA: Duxbury. Guijt, I. 2000. Methodological issues in participatory monitoring and evaluation. In Learning from change: Issues and experiences in participatory monitoring and evaluation, ed. Estrella, M. et al., 201-16. London, UK: ITDG Publishing. Hammond, J., Keeney, R., and Raiffa, H. 2002. Smart choices. Paperback ed. New York, NY: Broadway Books. Hatry, H., Winnie, R., and Fisk, D. 1981. Practical program evaluation for state and local governments, 2nd ed. Washington, D.C.: Urban Institute Press Page 50 of 50 Ientelucci, A. 2005. Monitoring and evaluating a zoning code. Zoning Practice, July 2005. American Planning Association. Janes, G. 2006. Deliverable 1: Statistical analysis of development activity. Review of floor area ratio regulations and design guidelines. Environmental Simulation Center report. Retrieved September 13, 2008, from http://www.simcenter.org/Scarsdale/scarsdale.html. Janes, G. and Heagney, B. 2007a. Final report. Review of floor area ratio regulations and design guidelines. Environmental Simulation Center report. Retrieved September 13, 2008, from http://www.simcenter.org/Scarsdale/scarsdale.html. ——— . 2007b. Measuring the visual effectiveness of the FAR: Part 1, documentation and narrative. Review of floor area ratio regulations and design guidelines. Environmental Simulation Center report. Retrieved September 13, 2008, from http://www.simcenter.org/Scarsdale/scarsdale.html. Kendig, L. 2005. Too big, boring, or ugly: planning and design tools to combat monotony, the too-big house, and teardowns. Planning Advisory Service Report, 528. American Planning Association. Marsh, J. 1994. A primer on decision making: How decisions happen. New York, NY: The Free Press. Patton, C. and Sawicki, D. 1993. Basic methods of policy analysis and planning, 2nd ed. Upper Saddle River, NJ: Prentice Hall. Perrin, B. 1998. Effective use and misuse of performance measurement. American Journal of Evaluation 19(3):367-79. Seasons, M. 2003. Monitoring and evaluation in municipal planning. Journal of the American Planning Association 69(4):430-40. Szold, T. 2005. Mansionization and its discontents: Planners and the challenge of regulating monster homes. Journal of the American Planning Association 71(2):189-202. Thomas, J.C. 1995. Public participation in public decisions. San Francisco, CA: Jossey-Bass. Weiss, C. 1998. Evaluation: Methods for studying programs and policies, 2nd ed. Upper Saddle River, NJ: Prentice Hall.