REAL WORLD RESEARCH IN PRODUCT EVALUATION AND SUSTAINABABLE DEVELOPMENT TO REACH SCALE Susan Murcott1,3, Jeff Asher2, Dan Frey1, Jarrod Goentzel1, Jennifer Green1, Bish Sanyal1 1 Massachusetts Institute of Technology, USA 2 Consumer Reports, Yonkers, NY, USA 3 murcott@mit.edu  Abstract:  Low-income consumers aspire to a better life that humanitarian products offer. International aid agencies, non-governmental organizations, governments and social entrepreneurs promote and disseminate millions of products to alleviate poverty.  But many of these products fail to deliver -- either to perform consistently, or if they survive in the marketplace, they fail to reach scale. Preconditions to impact, sustained use and scale are rigorous product evaluations that are trusted, affordable and comprehensible. Massachusetts Institute of Technology (MIT) launched the Comprehensive Initiative on Technology Evaluation (MIT-CITE), a five-year USAID-funded project to develop a methodology, called the “3S’s” which has guided the 2014 CITE Household Water Filter (HWF) Evaluation in Ahmedabad, India. Three expert sub-groups investigated different dimensions of the HWF product ecosystem:   • Suitability (S1) Team • S1-Lab: Technical Performance at Consumer Reports in Yonkers, New York. • S1-India: Technical Performance of water filters in Ahmedabad, India households • Scalability Team – evaluation of the commercial HWF product supply chain and capacity to scale up in India based on availability, affordability and aftermarket indicators.  • Sustainability Team – integrating social, economic, behavioral and product usability criteria.  Findings and Lessons Learned: CITE developed a decision support tool for users and institutional purchasers of HWF products targeted to low-income consumers. Using a methodology patterned after Consumer Reports, we have done a comparative, multi-objective evaluation of more than 100+ HWF products in Ahmedabad. Results cover three product categories: particle removal filters, gravity non-electric water filters and reverse osmosis systems. Findings are discussed. 1 INTRODUCTION This paper presents the results of the MIT-CITE Household Water Filter (HWF) evaluation. It is the second evaluation completed by CITE and an integral part of the on-going effort to develop a methodology that tests, compares and reports on humanitarian products for the poor (CITE, 2015). The overarching aim is to identify the most promising, proven breakthrough innovations to accelerate development and improve the lives of millions (Global Development Lab, 2015). EESD’15    The 7th International Conference on Engineering Education for Sustainable Development Vancouver, Canada, June 9 to 12, 2015  101-1  2 COMPREHENSIVE TECHNOLOGY EVALUATION METHODOLOGY A sustainable development technology evaluation methodology, focusing on three main evaluation components, the “3S’s,” guided this evaluation. Three expert sub-groups investigated different dimensions of the HWF product ecosystem, as follows:  • Suitability (S1) Team • S1-Lab: Technical Performance at Consumer Reports Lab in Yonkers, New York. • S1-India: Technical Performance of water filters in Ahmedabad, India households • Scalability (S2) Team – evaluation of the commercial HWF product supply chain and capacity to scale up existing HWF products in India based on availability, affordability and aftermarket indicators.  • Sustainability (S3) Team – integrating social, economic, behavioral and product usability criteria.  CITE identified more than 100 models of HWFs in India from nine major brands, plus minor brands and locally assembled “Dolphin” brands, in three product categories: conventional particle filters (CPF), gravity non-electric (GNE) and reverse osmosis (RO) -- according to multiple Suitability (product performance), Scalability (supply chain) and Sustainability (user behavior) criteria to assess which ones performed best, and which are recommended for scale up. We limited ourselves to filters commonly found in homes and the market, as opposed to specialty items, (e.g. arsenic or fluoride filters, water softeners, carbon and mineral filters).  The teams collaborated throughout the project, while providing expertise within their disciplinary areas. There were three unifying methodological elements: (1) Filters for the Poor: Target water filters for low-income households, while considering the entire spectrum of income levels and filter products found in the Ahmedabad market. (2) Three Filter Categories: Focus on three categories of commonly available household water filters as a simplification of a larger universe of options (e.g. disinfection options, community treatment systems, water softeners, etc.). (3)  Multi-Criteria Analysis and Consumer Reports-Style Rating Charts: Use a multi-criteria approach to analyze performance and summarize results using Consumer Reports (CR)-style ratings charts.  3 INDIVIDUAL TEAM’S METHODS AND RESULTS  3.1 Suitability (S1-CR) Consumer Reports Lab Team – Methods and Results  S1-CR Methods: The test parameters included: E.coli,  turbidity, total dissolved solids (TDS), percent recovery (RO filters only) and filter lifetime, with the “end-of-life” defined as flow rates of less than 1L/hr for GNE filters or 100 ml/minute for RO filters. Each filter model was tested either once (in the case of lengthy end-of-life tests) or with multiple test samples. The challenge water for all filters had 40±10 NTU turbidity and 1500±150mg/L TDS, following the WHO Scheme’s “Harmonized Test Protocol” (World Health Organization, 2014).   S1-CR Results: Fifteen models were tested on the basis of the three filter categories. A Ratings Chart was created for the RO and GNE category filters (Figure 1). There was insufficient data to include cloth and jali mesh filters from the conventional particle removal category into this Figure 1 ratings chart. The chart shows the top performers in the two product categories – RO and GNE.  In addition, it shows the “value for money,” i.e. the relative performance increase as a function of price. This allows those making purchasing decisions to balance the trade-offs between overall performance and price, as well as between the various attributes. The user of this chart may select a model by the highest overall score, or the score as a function of price, or by critical factors relating to their specific needs. Among all the ratings charts generated by the CITE team, this ratings chart most closely approximates the typical Consumer Reports-style of model testing, evaluation and ratings. 101-2     Figure 1: S1-CR Ratings Chart for Gravity Non-Electric (GNE) Filters and Reverse Osmosis (RO) Filters Notes:   Overall Score ranges from 0 to 100, with 0 as low and 100 as high. [1] – The exchange rate used for this calculation is 60 INR per USD. [2] – Operating cost is the Total Cost of Ownership (TCO), which averages the initial purchase price plus the cost of the replacement parts for a household consuming 25 liters per day over the five-year lifetime of the device. [3] – Gravity Non-electric (GNE) filters as a product category cannot reduce hardness [4] – Multiple samples never met the minimum flow rate of 1 liters per minute  Note: There was not sufficient data to generate a S1-CR Ratings Chart for the cloth and jali filters.101-3   3.2 Suitability (S1-India) Team Methods and Results  S1-India Methods: CITE’s S-1 India team located seven filter brands in households in statistically valid sample sizes  -- between 20 to 160 units of a given brand. S1-India’s five test parameters were E.coli, total coliform, turbidity, TDS and hardness. Piped water supply in the home was tested for residual chlorine at the tap only (“in” water).  Otherwise, we always analyzed water “in” and “out” of the filter. S1-India team covered 38 of 64 wards in Ahmedabad in all quadrants of the city. All households were supplied from the municipal piped water source. It was these existing filters units, which were used, not new ones, from which we collected our samples.   S1-India Results: Five water quality parameters were simplified into three key performance attributes -- total coliform (TC) contamination, total coliform (TC) reliability and turbidity reduction -- in order to create the S1-India Ratings Chart shown in Figure 2. Overall scores ranged from 23 to 36 out of a total of 0 to 100. Compared to the scores of the other S teams, these were the lowest scores for all brands and categories. This may have been due to the fact that filters were used, as opposed to brand-new, and products were tested in real household settings, as opposed to “ideal” lab settings.  Despite the low overall scores, five brands: Dolphin RO, Eureka Forbes RO, Kent RO, Hindustan Lever (Pureit) and the jali mesh scored “excellent” on one attribute, and Eureka Forbes RO and Hindustan Lever (Pureit) scored “excellent” on two attributes, but poor or fair on the third, turbidity reduction. Not shown in the chart, Conventional Particle Filters (CPF) and GNE filters had no impact in lowering TDS, while RO filters were rated “excellent” on this performance attribute.     Figure 2: Ratings Chart (S1-India) TC Contamination, TC Reliability, and Turbidity Reduction  3.3 Scalability (S2) Team Methods and Results S2 Methods: For each product, the Scalability team determined a composite scalability score based on nine attributes characterizing three key aspects: availability, affordability and after-market support. Empirical data from field surveys of over 108 retailers and semi-structured interviews of distributors and manufacturers in the supply chain provided the evidence to assess and score each attribute.    S2 Results: The underlying values of nine attributes were combined in weighted sums to determine the overall Scalability score, as shown in Figure 3. The Scalability ratings chart differentiates categories and brands, highlighting inherent tradeoffs in business models. For example, brands with excellent availability are generally more expensive, with lower ratings on initial investment; yet these products perform well on financing that enable consumer investment and offer better maintenance plans to reduce ongoing costs. The locally assembled Dolphin RO offers excellent availability, in part due to entrepreneurial initiative, while remaining reasonably affordable with good after-market support, especially in comparison with other RO brands. Eureka Forbes offers the best combination of affordability and after-market support, but low availability, notably in rural areas. It is important to note that Eureka Forbes also offers direct door-to-door sales, which is a channel not captured by retailer surveys. The availability assessment could have been more comprehensive if all original equipment manufacturers (OEMs) had shared more information on 101-4   their sales channels. The weight assigned to each attribute is listed above it as a percentage in the second row in Figure 3. Overall, Affordability is weighted 50% of the Scalability Score, with Availability and After Market each contributing 25%.    Figure 3: Scalability Comparative Ratings Chart  3.4 Sustainability Team Methods and Results S3 Methods: Consistent with the WHO’s International Household Water Treatment and Safe Storage Network’s evaluation protocols, Sustainability in this CITE context is defined as “the successful adoption of a water filter product and its correct, consistent and continuous use over time” (WHO, 2011). Our study specifically targeted consumer knowledge, decision-making and perceptions of HWFs. We performed in-depth surveys of 263 households in Ahmedabad, India. To calculate comparative indicators for each filter, the Sustainability team developed a weighted criteria evaluation matrix. This matrix considered 11 indicator/sub-criteria groups under four macro-criteria: social, economic, perceived benefits and usability.  S3 Results: The Sustainability Ratings Chart comparing six filter categories is shown in Figure 4. This chart includes the scores for all 11 indicators and the weightings for the indicators can be changed at both the macro-criteria level (i.e. Social, Economic, Perceived Benefits and Usability) as well as at the sub-criteria level (i.e. observability, affordability, dependability, etc.). As currently weighted, the overall Integrated Sustainability score for the conventional particle filters (cloth, jali, and cloth/jali) averaged 74/100 points, while the more advanced GNE and RO filters categories performed worse overall at an average of 52/100. The cloth and jali filter scores were driven lower by poor performance in the Improvement in Water Quality and Dependability indicators. The GNE and RO categories had low scores in the “Time/Money” indicator, which is consistent with their higher costs.  101-5   Figure 4 Sustainability Ratings Chart by Category (4a) and By Brand (4b)    Figure 4a: Sustainability Ratings Chart by Category   Figure 4b: Sustainability Ratings Chart by Brand 101-6   4 FINDINGS 4.1 Suitability--S1-CR Findings  • RO: Based on the product end-of-life tests, the locally assembled Dolphin RO filter (Clean Water model) was as effective as a branded RO (Tata Swach Platina model) which, however, was three times more expensive. Unfortunately, RO systems waste water, typically in the ratio of 3:1 of waste to clean water.  • GNE: There are big differences among the GNE filters regarding their effectiveness in terms of flow rate, turbidity removal, lifetime, and E.coli removal. In general, the higher the flow rate, the lower the turbidity and E.coli removal. Like the cloth/jali filters, the GNE filters were not effective in removing TDS.  • Cloth/Jali: Cloth filters had limited effectiveness in reducing turbidity, little impact on removing E.coli (less than 25%) and no impact on removing total dissolved solids (TDS) (0%). Jali mesh type filters likewise had limited effectiveness in reducing turbidity (5%) and none on removing E.coli (0%). 4.2 Suitability--S1-India Findings  • Water quality, even from piped sources, is not consistently reliable in terms of its safety. • Ahmedabad’s government has done a noteworthy job at provision of piped water for the poor. Other cities in India can learn from this experience.  • HWFs and other household-level interventions can complement piped water supply, but the preferred intervention for the urban poor is improved water supply services. HWFs for the poor should be targeted to households with unimproved water supplies or those households facing emergency conditions. • Reverse Osmosis is a popular water filter system widely perceived as the best. But most, with the possible exception of local Dolphin filters, are not an affordable option for the poor.  • Mismatch between the water quality “problem” and the household water filter “solution” requires better communication of drinking water data and matching with appropriate solutions.  This relates to the fact that Ahmedabad’s water is a complex mix of groundwater and surface water, making it challenging for the general public to understand what is needed to “fix” the problem. What is needed on the part of the government and industry is reliable public information on water quality that is transparent and comprehensible. Hence there may be an important role for low-cost water testing, whether by the government, industry, NGOs or the citizens themselves.  4.3 Scalability--S2 Findings • Availability results point to an interesting paradox. The products that are most affordable are not available in rural areas, while the high-end models are prevalent. The critical issue in reaching the people of Gujarat with HWFs is a rural distribution strategy for the most affordable models. This is an area where supply chains need to be strengthened in order to provide safe drinking water for the broader population.  • Water filter options are available to reach a significant percentage of Gujarat’s population.  On a total cost of ownership basis, many are affordable for the poor, in part due to the nature of the locally assembled (“Dolphin”) models that rely on entrepreneurial investment.  4.4 Sustainability--S3 Findings • Water filter use seems to be dependent on use of water filters in the past (especially growing up) and peer effects (Figueroa & Kincaid, 2010) • GNE and RO filter users self-report a greater improvement in quality from pre- to post-filtered water than cloth and/or jali users. GNE and RO users perceive their unfiltered water as worse relative to cloth/jali users and they perceive their filtered water as better. 101-7  • The decision to buy a filter is a social activity.  • While most respondents were confident that they could correctly treat their water to make it safe for drinking, those who were not confident primarily used cloth and/or jali.  • Knowledge about HWFs designed specifically for poor households is low among that market segment. Because purchase decisions are often a social activity, technology adoption proves difficult when one’s peers do not have knowledge about a particular product. • Cost is a key barrier to use among non-users.  • Respondents valued tasty, “sweet” water, while the primary metric for water quality was taste among GNE and RO filter users and appearance among cloth/jali and non-users. 5 IMPLICATIONS OF THE FINDINGS CITE has developed the 3S evaluation methodology to test, compare, and report on products for the poor and to identify the most promising, proven breakthrough innovations in order to accelerate development and scale to reach millions of people. Implications of the findings from the household water filter evaluation in Ahmedabad, India are summarized below:  • Water quality, even from piped sources, is not consistently reliable in terms of its safety. • Cloth and jali mesh, which are widely used by the poor in Ahmedabad, are not effective in providing protection against the risk of microbial contaminants, which are leading causes of diarrhea, cholera, typhoid, and other water-borne diseases. Cloth/jali are not recommended. • Awareness about low-cost filter alternatives to cloth and jali that are in the market (e.g., Tata Swach, HUL Pureit) is very low among bottom of the pyramid consumers • Other types of CPF or GNE filters (e.g. ceramic, biosand, and siphon) were not found in the Ahmedabad market and may be additional appropriate options.  • Cost is a key barrier to filter use among non-users. • Dolphin ROs, assembled by local entrepreneurs, offer the same high technical performance as nationally branded RO models at half the cost. They also have the highest availability in rural areas. • Supply chains are not reaching rural towns with affordable GNE models to provide safe drinking water for this population.  The poor should be informed about these key findings, especially the uncertain quality of piped supplies and the ineffectiveness of cloth or jali mesh filters. Entrepreneurs should be informed about the Dolphin business model that scaled in this market and about opportunities that may exist in India for alternate CPF or GNE products, particularly if they reach rural areas.  References CITE (2015). Water Filter Evaluation Integrated Report. 3Ss: Suitability, Scalability, Sustainability. MIT: Cambridge MA. http://cite.mit.edu (accessed 1-20-15). Figueroa, M., & Kincaid, D. (2010). Social, Cultural and Behavioral Correlates of Household Water Treatment and Storage. Johns Hopkins Bloomberg School of Public Health. Baltimore:C.C.Program WHO (2011). Evaluating Household Water Treatment Options: Heath-based targets and Microbiological Performance Specifications. ISBN: 978 92 4 154822 9 http://www.who.int/water_sanitation_health/publications/2011/evaluating_water_treatment.pdf?ua=1 (accessed 1-20-15). For “correct, consistent and continuous use,” see pp. 6 & 55.  WHO (2014). WHO International Scheme to Evaluate Household Water Treatment Technologies Harmonized Testing Protocol: Technology Non-Specific. World Health Organization. Geneva, Switzerland. 2014. http://www.who.int/household_water/scheme/HarmonizedTestProtocol.pdf?ua=1  101-8