Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Zhang, Junbin
Master of Applied Science - MASc
University of British Columbia
Numerous static taint analysis techniques have recently been proposed for identifying information flows in mobile applications. These techniques are often optimized and evaluated on a set of synthetic benchmarks, which makes the comparison results difficult to generalize. Moreover, the techniques are commonly compared under different configuration setups, rendering the comparisons inaccurate. In this paper, we provide a large, controlled, and independent comparison of the three most prominent static taint analysis tools: FlowDroid, Amandroid, and DroidSafe. We align the configuration setup for the tools and evaluate them on both a set of common benchmarks and on real applications from the Google Play app store. We compare the results of our analysis to the results reported in previous studies, identify main reasons for inaccuracy in existing tools, and provide suggestions for future research.
Analyzing Android Taint Analysis Tools: FlowDroid,Amandroid, and DroidSafebyJunbin ZhangB.Eng., Sun Yat-Sen University, 2017B.Eng., The Hong Kong Polytechnic University, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of Applied ScienceinTHE FACULTY OF APPLIED SCIENCE(Electrical & Computer Engineering)The University of British Columbia(Vancouver)July 2020© Junbin Zhang, 2020The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Analyzing Android Taint Analysis Tools: FlowDroid, Amandroid, and DroidSafesubmitted by Junbin Zhang in partial fulfillment of the requirements forthe degree of Master of Applied Sciencein Electrical & Computer EngineeringExamining Committee:Julia Rubin, Assistant Professor, Electrical & Computer Engineering, UBCSupervisorKarthik Pattabiraman, Associate Professor, Electrical & Computer Engineering, UBCSupervisory Committee MemberPhilippe Kruchten, Professor Emeritus, Electrical & Computer Engineering, UBCSupervisory Committee MemberiiAbstractNumerous static taint analysis techniques have recently been proposed for identi-fying information flows in mobile applications. These techniques are often opti-mized and evaluated on a set of synthetic benchmarks, which makes the compar-ison results difficult to generalize. Moreover, the techniques are commonly com-pared under different configuration setups, rendering the comparisons inaccurate.In this paper, we provide a large, controlled, and independent comparison of thethree most prominent static taint analysis tools: FLOWDROID, AMANDROID, andDROIDSAFE. We align the configuration setup for the tools and evaluate them onboth a set of common benchmarks and on real applications from the Google Playapp store. We compare the results of our analysis to the results reported in pre-vious studies, identify main reasons for inaccuracy in existing tools, and providesuggestions for future research.iiiLay SummaryAndroid applications often send out userâs sensitive information without usersknowledge. To protect users privacy, numerous techniques are proposed to detectdata leakages by analyzing a mobile application without ever running it. However,these techniques are often compared under different setups, rendering the compar-isons inaccurate. In addition, existing comparisons are limited to predicting thetoolsâ performance only on a set of synthetic benchmarks.This thesis compares three most prominent techniques, FLOWDROID, AMAN-DROID, and DROIDSAFE with aligned configuration setup. We first evaluate thesetechniques on the same set of commonly-used benchmark apps and compare ourresults to those reported in previous studies. We then evaluate the techniques onreal apps from the popular Google Play app store. We identify strengths and weak-nesses of each technique and provide suggestions for future research.ivPrefaceThis thesis is part of a research project carried out in collaboration with YingyingWang, Lina Qiu, and Julia Rubin (research advisor). It builds up on a previouspublication:[30] Lina Qiu, Yingying Wang, and Julia Rubin, Analyzing the analyzers: FLOW-DROID/ICCTA, AMANDROID, and DROIDSAFE. In Proc. of ISSTAâ18, pages176-186, 2018. ACM SIGSOFT Distinguished Paper Award.In this thesis, Chapter 2 was mainly adopted from Qiu et al. [30] with a new para-graph written by me for native code analysis. All the other chapters were myoriginal work, which underwent major extension from Qiu et al. [30]. Specifically,I was responsible for:â¢ Repeating all experiments reported by Qiu et al. [30] on the latest version ofFLOWDROID, AMANDROID, and DROIDSAFE and the latest version of thebenchmark suites DroidBench and ICC-Bench ;â¢ Analyzing of toolsâ failures on benchmarks, which includes three new andone corrected causes of failures not identified by Qiu et al. [30];â¢ Evaluating tools on Google Play apps, leading to the discovery of five ad-ditional causes of failures not covered by existing benchmarks, e.g., in han-dling Android framework methods, callbacks, and reflection.After I conducted the first round of ground truth identification and result analy-sis in Chapters 3 to 5, the results for benchmark apps (Chapters 3.3 and 4) werevmainly cross-validated by Lina Qiu and myself; the results on Google Play apps(Chapters 3.4 and 5) were mainly cross-validated by Yingying Wang and myself.Julia Rubin was the research advisor on this project and was involved in conceptformation and manuscript edits.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAcronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Research Questions and Main Findings . . . . . . . . . . . . . . 21.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Background: Taint Analysis . . . . . . . . . . . . . . . . . . . . . . 62.1 Sensitivities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Implicit Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Java-specific Features . . . . . . . . . . . . . . . . . . . . . . . . 8vii2.4 Android-specific Features . . . . . . . . . . . . . . . . . . . . . . 93 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.1 Tool Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Tool Configuration Parameters . . . . . . . . . . . . . . . . . . . 143.3 Evaluation Methodology for Benchmark Apps . . . . . . . . . . . 173.3.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.3.2 Sources and Sinks . . . . . . . . . . . . . . . . . . . . . 183.3.3 Expected Results . . . . . . . . . . . . . . . . . . . . . . 193.3.4 Metrics and Measures . . . . . . . . . . . . . . . . . . . 213.4 Evaluation Methodology for Google Play Apps . . . . . . . . . . 223.4.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.4.2 Sources, Sinks, and Expected Results . . . . . . . . . . . 243.4.3 Metrics and Measures . . . . . . . . . . . . . . . . . . . 263.5 Limitations and Threats to Validity . . . . . . . . . . . . . . . . . 274 Results: Benchmark Apps (RQ1-RQ3) . . . . . . . . . . . . . . . . . 284.1 Tools Performance (RQ1) . . . . . . . . . . . . . . . . . . . . . . 284.1.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 284.1.2 Execution Time . . . . . . . . . . . . . . . . . . . . . . . 324.2 Comparison with Earlier Experiments (RQ2) . . . . . . . . . . . 324.2.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 324.2.2 Execution Time . . . . . . . . . . . . . . . . . . . . . . . 354.3 Causes of Inaccuracy (RQ3) . . . . . . . . . . . . . . . . . . . . 355 Results: Google Play Apps (RQ4) . . . . . . . . . . . . . . . . . . . 445.1 Tools Performance . . . . . . . . . . . . . . . . . . . . . . . . . 445.1.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 445.1.2 Execution Time . . . . . . . . . . . . . . . . . . . . . . . 475.2 Causes of Inaccuracy . . . . . . . . . . . . . . . . . . . . . . . . 486 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566.1 Internal Tool Evaluation . . . . . . . . . . . . . . . . . . . . . . 566.2 External Tool Evaluation . . . . . . . . . . . . . . . . . . . . . . 58viii6.3 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 617.1 Discussion and Lessons Learned . . . . . . . . . . . . . . . . . . 617.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65ixList of TablesTable 3.1 Selected tools. . . . . . . . . . . . . . . . . . . . . . . . . . . 12Table 3.2 Configuration decisions of FLOWDROID, AMANDROID, andDROIDSAFE. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Table 3.3 List of source and sink methods for benchmark apps (SS-Bench ). 19Table 3.4 Differences in expected results for DroidBench and ICC-Bench . 20Table 3.5 List of the selected Google Play apps, and number of the ex-pected flows. . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Table 3.6 List of source and sink methods for Google Play apps (SS-GP ). 25Table 4.1 Benchmark apps: comparisons of precision, recall, and F- mea-sures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Table 4.2 FN/FP breakdown by target criteria, at benchmark level. . . . . 36Table 4.3 Failures grouped by target criteria. . . . . . . . . . . . . . . . 43Table 5.1 FLOWDROID and AMANDROID results on Google Play apps. . 45Table 5.2 Google Play apps: comparisons of precision, recall, and F-measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Table 5.3 FN/FP breakdown by target criteria, for Google Play apps. . . . 48xList of FiguresFigure 2.1 Context and object sensitivity. . . . . . . . . . . . . . . . . . 7Figure 3.1 The Google Play apps selection process. . . . . . . . . . . . . 23Figure 4.1 Flows with ICC-related sinks. . . . . . . . . . . . . . . . . . 30Figure 4.2 Placement of entry methods. . . . . . . . . . . . . . . . . . . 31Figure 4.3 U-DB10.6 â Sources or sinks invoked using reflection. . . . . 37Figure 4.4 U-DB7.24 â Flows through casted variables. . . . . . . . . . 38Figure 4.5 Simplified Jimple code for Figure 4.4. . . . . . . . . . . . . . 38Figure 4.6 U-DB2.12 â Android framework method modeling (AMANDROIDmistakenly propagates taint from one parameter to another). . . . . 40Figure 4.7 U-DB4.15 â Location-related flow. . . . . . . . . . . . . . . . 40Figure 4.8 U-DB4.16 â Callback Intent handling. . . . . . . . . . . . . . 41Figure 4.9 U-DB10.7 â Reflection via Object.getClass() method. . 42Figure 5.1 U-DB2.12 â Android framework method modeling (FLOWDROIDdoes not propagate taint from a parameter to the receiver object). . . 49Figure 5.2 U-DB2.12 â Android framework method modeling (FLOWDROIDmistakenly propagates taint from a parameterâs field to the return value). 50Figure 5.3 U-DB4.17 â Missing callbacks. . . . . . . . . . . . . . . . . 51Figure 5.4 U-DB4.18 â Callback methods registered in fragments. . . . . 52Figure 5.5 U-DB10.8 â Reflective invocation of class constructors. . . . . 53Figure 5.6 U-DB7.25 â Context sensitivity. . . . . . . . . . . . . . . . . 54Figure 5.7 U-DB7.26 â Flows triggered from multiple call sites. . . . . . 55xiAcronymsAAPT Android Asset Packaging ToolAPI Application Programming InterfaceAPK Android PackageCPU Central Processing UnitFN False NegativeFP False PositiveGB GigabyteHTTP Hypertext Transfer ProtocolIAC Inter-app CommunicationIC3 Inter-component Communication Analysis with COALICC Inter-component CommunicationJNI Java Native InterfaceMIME Multipurpose Internet Mail ExtensionsRAM Random-access MemorySMS Short Message ServiceTP True PositiveUI User InterfaceURI Uniform Resource IdentifierXML Extensible Markup LanguagexiiAcknowledgmentsFirst and foremost, I would like to express my gratitude to my supervisor, JuliaRubin, for her patient guidance, constructive advice, and persistent encouragementduring my journey of this masterâs degree. I am also thankful for my committeemembers in carefully reviewing this thesis and providing insightful feedback.Moreover, I would like to thank my collaborators and colleagues, YingyingWang and Lina Qiu, for giving me invaluable help and suggestions on this research.Their previous wonderful work inspired the idea for the current project.It is also my fortune to have all colleagues and friends, both in the ReSeSSResearch Lab and during my study throughout the masterâs degree. Thanks for allyour support in my life.Last but not the least, it is hard to express my appreciation to my belovedparents, for giving me endless love and patience.xiiiChapter 1Introduction1.1 OverviewStatic taint analysis is commonly used for identifying information flows in mo-bile applications and numerous tools for performing such analysis have being pro-posed [4, 8, 12, 13, 21, 25, 41, 46]. The authors of these tools typically tune andevaluate them on a set of common benchmarks, such as DroidBench 1 and ICC-Bench 2. Our experience, however, shows that the tools are often compared un-der different setups, making the comparison inaccurate. The tools are also mostlycompared on a set of synthetic benchmarks, making it difficult to predict the per-formance of tools on real apps.Some authors [4, 24, 41, 43] evaluate a particular tool on FDroid3 and GooglePlay4 apps. Others [9, 21, 28, 32, 33] compare the tools with each other but, aswith benchmark apps, do not align the tool setups. Moreover, they mostly focus onreporting the number of detected flows and runtime failures, without consideringthe underlying reasons leading to the incorrect results and without analyzing falsenegatives.We address these limitations in our work, performing an independent, con-trolled, and large-scale study comparing the three most-popular open-source static1https://github.com/secure-software-engineering/DroidBench2https://github.com/fgwei/ICC-Bench3https://f-droid.org/4https://play.google.com/store1taint analysis tools to each other: FLOWDROID, AMANDROID, and DROIDSAFE.We focus on comparing the performance of the tools in terms of accuracy and exe-cution time. We configure the tools to use a similar setup and we evaluate the toolson more than 180 benchmark applications and 20 applications from the GooglePlay store, for which we carefully collected the expected flows. We compare theresults of our study with the results in earlier reports, discuss the strengths andweaknesses of each tool, and the implications of our findings. As we are not in-volved in the development of these tools, our analysis is completely independent.Moreover, we make the results of our analysis, the set of subject applications thatwe used, and our configuration setup available to the community, making our studyreplicable and reproducible.1.2 Research Questions and Main FindingsThis thesis aims at answering the following research questions:RQ1: How well do the tools perform on the benchmark applications?RQ2: Can we replicate the results of earlier experiments?RQ3: What are the main causes of inaccuracy for the benchmark applications?RQ4: Do the results on the benchmarks applications generalize to the Google Playapplications?To answer RQ1, we extracted the configuration options used in the empirical eval-uation of FLOWDROID [21], AMANDROID [41, 43], and DROIDSAFE [13]. Welearned that none of the reports provides detailed information on all of the follow-ing: (a) the parameters used for configuring the tools; (b) the list of the sourcesand sinks used; (c) the exact set of DroidBench and ICC-Bench apps used in theevaluation.For example, FLOWDROID has 44 configuration parameters, AMANDROID has11, and DROIDSAFE has 57; the exact values of even the most significant of thoseparameters are not reported. While the DROIDSAFE authors provided precise in-formation about the set of sources and sinks they used, they did not list the names ofthe benchmark applications used for evaluation. Both FLOWDROID and AMAN-DROID listed the names of the benchmark applications; yet, we were unable tomap those names to the applications in the current version of the benchmark due2to naming inconsistencies. We thus performed our own experiments, evaluatingeach tool with a common configuration setup we have chosen, on the same set ofapplications, and using the same set of sources and sinks.We used the DroidBench and ICC-Bench benchmark suites. Our results showthat on benchmark suites, FLOWDROID has the highest accuracy, DROIDSAFEcomes second, and AMANDROID has the lowest accuracy.For RQ2, our findings show that the accuracy of the tools in our experimentoften differs from the results reported in earlier studies. For instance, the authorsof AMANDROID report an F-measure of 81% [41] and 96% [43] when executed onthe DroidBench benchmark suite; yet, we were unable to replicate these results,observing an F-measure of 55% for our experimental setup.We manually analyzed the false positive (FP) and false negative (FN) resultsreported by each tool (RQ3). In a nutshell, we observed that all tools have majorissues in handling reflection, albeit for different reasons. In addition, FLOWDROIDfails to accurately parse and track ICC Intents involving complex string analysisand list management operations; AMANDROID does not accurately handle Androidframework methods, lifecycle and callback methods, and location-related flows.For RQ4, we investigated whether FLOWDROID and AMANDROID are able toidentify the flow between the login credentials entered by the user and an Internettransmission operation. We excluded DROIDSAFE from this analysis as the toolauthors explicitly state that the tool is not designed to run on Google Play apps5and our experiments concur with that statement.The rationale behind selecting the login case study is that applications haveto send user credentials out to the Internet to support the login functionality. Wecan thus use this information transmission as an expected flow in a real-world appand check how well the tools perform for identifying this flow. We also manuallyinspected all other flows the tools found in each application, identified FP results,and analyzed the reasons behind the toolsâ FPs and FNs.Our analysis on 20 applications from Google Play shows that in all but twoapplications the tools cannot find the expected login-related flow when configuredto run for three days per app. FLOWDROID crashes with exceptions in 11 out of5https://mit-pac.github.io/droidsafe-src/320 applications, mostly when analyzing certain Inter-component Communication(ICC) flows and reflective calls. AMANDROID fully analyzes nine applications andproduces partial results for another nine applications, timing out on some of theseapplicationsâ components.For the apps the tools analyzed, both tools have difficulties detecting particularcallback methods and reflective calls, as Google Play apps rely on callback/reflec-tion mechanisms not covered in benchmarks. While FLOWDROIDâs modeling ofAndroid framework methods is accurate for all methods used in the benchmarks,it does not cover methods used in Google Play apps, leading to both FP and FNresults. Interestingly, AMANDROID has no framework-method-related failures onGoogle Play apps even though it had some failures on the benchmark apps. Whileframework method modeling strategies the tools employ differ, both have weak-nesses, albeit in different cases. We also observed that when a flow is triggeredfrom multiple locations within an app, AMANDROID reports the same flow multi-ple times but does not provide any additional information to the user, making theseduplicated reports indistinguishable.As these failures were not explicitly covered by existing benchmarks, we cre-ated a sample app isolating each failure and added it to our own benchmark suite,called UBCBench [45]. We intend to contribute this suite to DroidBench ; we arealso working with the authors of FLOWDROID and AMANDROID to fix some ofthe failures (DROIDSAFE is no longer supported).1.3 ContributionsThis thesis reports on an extension of the earlier study published at ISSTAâ18 [30].The ISSTA paper compared FLOWDROID, AMANDROID, and DROIDSAFE underthe same setup on benchmark apps and reported on the results of our findings. Thefollow-up work described in this thesis performs the same experiments using thelatest versions of the tools and benchmark suites. It also includes a new experimentconducted on the Google Play apps and comparing the tools in a realistic use case.The contributions this thesis makes are described below:â¢ It provides an in-depth independent analysis that compares static taint anal-ysis tools for Android applications [21, 32, 41, 43] under a similar setup, on4both benchmark and Google Play apps, and puts the results in the context ofreports from earlier studies.â¢ It identifies the main strengths and weaknesses of each tool.â¢ It provides detailed information about the chosen configuration setup, se-lected sources and sinks, and applications used for analysis, making the re-sults replicable and reproducible.â¢ It extends the benchmarks used for comparing Android-specific static taintanalysis tools with previously uncovered cases.â¢ It makes our experimental setup and results publicly available, to supportreplication and reproducibility [45].1.4 StructureThis thesis is structured as follows:â¢ Chapter 2 provides the necessary background on taint analysis;â¢ Chapter 3 outlines our study design, including the selection and configura-tion of the tools and subject apps, our definition of expected results, andmetrics we used for tool evaluations;â¢ Chapters 4 and 5 present the results of our analysis for the benchmarks andGoogle Play apps;â¢ Chapter 6 discusses the related work;â¢ Chapter 7 summarizes the study with lesson learned, outlines future work,and finalizes the thesis with conclusions.5Chapter 2Background: Taint AnalysisWe now introduce the basic concepts of static taint analysis â a popular informationflow analysis technique which tracks the flow of sensitive information from a set ofsensitive sources to sensitive sinks. In our context, sources define the informationwe want to protect on a mobile device (e.g., phone number, contacts, location,and unique device identifiers) and sinks define points of unwanted informationrelease (e.g., methods related to the Internet and SMS transmission). If data froma sensitive source reaches a sink, taint tracking identifies the path from the sourceto the sink as an instance of data leakage.Taint analysis can be implemented both statically and dynamically; this thesisfocuses on static taint analysis techniques, i.e., those that track taint propagation byanalyzing the code of an Android application without ever running it. Examples oftools implementing static taint analysis include FLOWDROID, AMANDROID, andDROIDSAFE. The tools mostly differ in design decisions they take for making theanalysis accurate and scalable at the same time. Below, we briefly discuss fourmain dimensions for such decisions.2.1 SensitivitiesTo handle aliasing and virtual dispatch constructs, typical static analysis for Javaprograms applies some degree of context, object, and field sensitivity; flow andpath sensitivities are used to control the order of statements and their correspon-61 class A{ void s ink ( S t r i n g ) { . . .} }2 void foo ( ) {3 A a1 = new A ( ) ; a1 . s ink ( â t a i n t e d â ) ;4 }5 void bar ( ) {6 A a2 = new A ( ) ; a2 . s ink ( â un ta in ted â ) ;7 }Figure 2.1: Context and object sensitivity.dence to branches of the program:A context-sensitive analysis considers the calling context, i.e., a sequence ofcall sites, when analyzing the target of a method call. Specifically, in a k-call-site-sensitive analysis, the context of a called method includes the current call site ofthe method and the call sites of the caller methods, up to a pre-defined depth k [37].For the example in Figure 2.1, foo() and bar() are two different call sites for themethod call sink(), which are used by a context-sensitive analysis to differentiatethe flows to this method.An object-sensitive analysis uses object abstractions, i.e., allocation sites, ascontext [37]. Specifically, the analysis qualifies the method local variables withthe allocation site of the receiver object of the method call. For the example inFigure 2.1, object sensitive analysis uses the heap addresses of objects a1 and a2to differentiate the calls to sink(). That is, context- and object-sensitive analysesuse different program elements â call sites vs. allocation sites â as differentiatingcontextual information.A field-sensitive analysis distinguishes different fields of the same abstract ob-ject, instead of lumping all fields together [29, 37].A flow-sensitive analysis takes the order of statements into account [37]. Forexample, for a list of statements x=1; y=x+1; x=2, a flow-sensitive analysis willbe able to determine that y=2, whereas a flow-insensitive analysis will concludethat y=2 or y=3.A path-sensitive analysis collects path information which indicates the feasi-bility of a path. For instance, for a branch condition x > 0, the analysis wouldassume x > 0 on the target of the branch and x <= 0 on the fall-through path.72.2 Implicit FlowsImplicit flows are flows in which sensitive data indirectly impacts the observed out-put by affecting which branch to take in the control flow [34]. A typical example is:if taint then output = 1 else output = 0, where taintwould affectwhich branch to take, then further affect the value of the variable output. Con-ceptually, an implicit flow tracker taints all data items that are dependent on thetaint from the conditional, hence can result in large numbers of false-positives.2.3 Java-specific FeaturesAs Android applications are generally written in Java, analysis tools need to han-dle Java-specific features, such as reflection and exceptions. Reflection refers tothe ability to dynamically access members and type information of an object, oftenbased on string representations of the memberâs or typeâs name. Android devel-opers heavily rely on reflection, e.g., for backward compatibility, generality, andsometimes for hiding sensitive information flows [23]. Accurately resolving re-flective calls poses a challenge to the soundness of static analysis. To address thisproblem, some tools consider all possible resolutions for a reflective call. Othersrestrict the resolution by type of the variable, its scope, etc. [5].Exceptions can be thrown by a statement or expression, and then read by thecatch code block either within the method or up in the call stack. As exceptionscan be loaded dynamically and the type of exception determines the code blockto execute, precisely handling exceptions is challenging [15]. Some classic frame-works like SPARK [19] and PADDLE [44] use an over-approximation approach tohandle exceptions, assigning all exceptions thrown in the program to a single globalvariable; the variable is then read at the exception catch site. That is, this approachassumes that all possible exceptions are thrown and ignores the information aboutwhat exceptions can propagate to a catch site [16]. Other frameworks, such asSOOT [40], remove unrealizable exception edges from the intraprocedural controlflow graphs [15].82.4 Android-specific FeaturesEven though Android applications are often written in Java, several Android-specificfeatures should be taken into account to achieve accurate results.Android modeling: To track taint when an Android application interacts withthe Android execution environment, tools typically either (a) conservatively as-sume that the return value of all Android framework methods is tainted if any ofthe parameters is tainted or (b) precisely model (a subset of) the framework meth-ods. The latter is performed either manually or by automated analysis of Androidbinary distribution libraries [2].Application lifecycle: An Android application is composed of four types ofcomponents, namely, Activity, Service, Broadcast Receiver, and Content Provider.Each component has its own lifecycle methods, which are called by the Androidsystem to start/stop/resume the component. An application also contains callbackmethods which are triggered in response to system and user events (e.g., locationchange and button click). Static analysis tools identify and model lifecycle andcallback methods to ensure the correct propagation of taint. To extract the set ofsuch methods, the tools rely on information from the Android Open Source Projectand also analyze application code and configuration files, such as manifest andlayout XML.Inter-component Communication (ICC): The multi-component model of An-droid allows different components to communicate with each other and exchangeinformation, usually via explicit or implicit Intents. The former explicitly definetarget components to be invoked while the latter rely on Android to find the com-ponents which implement the requested functionality. ICC handling is a challengedue to the difficulty to precisely match the invoked components in a static man-ner [7]. Tools like EPICC [25] and IC3 [26] extract ICC information, enablingICC-aware information flow analysis. PRIMO [27] overlays a probabilistic modelof ICC on top of these static analysis results, further improving the accuracy ofICC detection.Inter-app Communication (IAC): Similar to ICC, IAC looks into how sensitivedata is transferred between components of different applications installed on thesame device. Tools such as APKCOMBINER [20] aim at reducing an IAC problem9to an ICC problem by combining different applications into a single APK on whichexisting tools can perform inter-app analysis.Native code: Android applications can include code written in C and C++ andtriggered via Java Native Interface (JNI)1. As taint can propagate via such nativeC and C++ code, a static taint analysis tool needs to be able to track data flow in Cor C++ code. AMANDROID is currently the only tool that supports such tracking,which relies on inter-language JN-SAF [42] framework for âstitchingâ data flowsfound in Java and native code.1https://developer.android.com/training/articles/perf-jni10Chapter 3Study DesignIn this chapter, we describe our method for selecting and configuring the static taintanalysis tools for the study (Chapters 3.1 and 3.2). We then outline the method-ology we used for evaluating the tools, for benchmarks and Google Play apps,separately (Chapters 3.3 and 3.4), describing how we selected the subject applica-tions, the set of sources and sinks that we used, and our specification of expectedresults.3.1 Tool SelectionLi et al. [22] performed a systematic literature review and identified 38 static taintanalysis tools for Android applications. From this list, we selected all open-sourcetools cited more than 100 times on Google Scholar1 as of June 2017. The names,citation counts, and versions for the selected tools â FLOWDROID, ICCTA, AMAN-DROID, and DROIDSAFE, for June 2017 and December 2019, are listed in Ta-ble 3.1.The last column of the table lists the release date of the latest publicly avail-able version of each tool as of December 2019; we used this version in our ex-periments. Starting from release 2.0, FLOWDROID is integrated with ICCTA andleverages it for processing ICC flows. We thus performed our experiments with thecombined version. Experimental data for running FLOWDROID without ICCTA,1https://scholar.google.com11Table 3.1: Selected tools.Tool# Citation(Jun. 2017)# Citation(Dec. 2019)VersionFLOWDROID 547 13992019-Jan-21 (2.7.1)ICCTA 129 410AMANDROID 148 359 2018-Dec-28 (3.2.0)DROIDSAFE 100 323 2016-Jun-22as well as experiments with older versions of these tools, are available in our onlineappendix [45]. Next, we briefly describe each of the analyzed tools.FLOWDROID [4] is a flow-, context-, field-, and object-sensitive static anal-ysis tool for Android applications, which is built on top of SOOT [40] and DEX-PLER [6]. It precisely models the Android lifecycle and handles data propagationvia callbacks of UI objects. FLOWDROID can only resolve reflective calls whosearguments are constant strings; it bases its exception handling mechanism on thatof SOOT. In the original design, the tool did not support implicit flows tracking [4],but newer versions of FLOWDROID are able to handle implicit flow [2].For modeling Android framework methods, FLOWDROID relies on STUB-DROID [3] which, given a binary distribution of the Android framework, performstaint analysis for the most common framework method and computes method sum-mary that captures the propagation of taint within the method. For methods thatcannot be analyzed with STUBDROID, e.g., those containing native code, FLOW-DROID applies a conservative strategy, dividing the framework methods into fourtypes: generation, exclude, kill, and default [2]. When a receiver object, at leastone of the method parameters, or the fields of the receiver object/method param-eters are tainted, the methods in the generation type will have their receiver andthe return value, as well as all their fields, tainted; no taint will be propagated formethods of the exclude type; all taints will be removed for methods of the kill type.If a method is not assigned with any of these types, the default rule will apply,propagating the taint to the method return value and its fields (but not to the re-ceiver object). That is, only for the methods in the generation type FLOWDROIDwill taint the receiver parameter and its fields; it does not taint any other method12parameters in any of the conservative strategies.ICCTA [21] extends FLOWDROID with the analysis of inter-component com-munication. It leverages existing ICC extraction tools, specifically, EPICC [25] andIC3 [26], augmenting them with application-level instrumentation of ICC-relatedmethods for extracting precise ICC flows. It also leverages APKCOMBINER [20],which generates an integrated APK for two or more applications. As a result, IC-CTA builds a complete intra-component, inter-component, and inter-applicationmodel. In our experiments, we used FLOWDROID version 2.7.1 from January20192, which is integrated with ICCTA and is the latest version of the tool at thetime of writing.AMANDROID [41, 43] implements a flow- and context-sensitive intra-componentdata flow analysis. On top of an inter-procedural control flow graph and data flowgraph, AMANDROID builds a data-dependency graph for each component and thengenerates a summary table documenting possible component communication con-nections. AMANDROID precisely models a subset of Android framework methodsand applies a conservative strategy for the remaining ones. Interestingly, AMAN-DROIDâs strategy is different from that of FLOWDROID: in addition to the receiverobject and the return value of a method, AMANDROID also taints all method pa-rameters and their fields. However, it is doing so only when the method receiver orparameters themselves are tainted, not when one of their fields is tainted. Strate-gies applied by both tools lead to false positive and false negative results, albeit indifferent cases, as we show in our evaluation.Based on proprietary inter-component and inter-application flow analysis, AMAN-DROID provides support for both ICC and IAC detection. However, AMANDROIDhas limited capacity to handle exceptions and reflection, and it cannot handle im-plicit flows [41, 43]. In our study, we used version 3.2.0 of AMANDROID releasedon December 29, 20183. This is the latest version available today. Yet, it can onlyhandle applications targeting API level 25 and below.DROIDSAFE [13] implements an object-sensitive and flow-insensitive anal-ysis. It builds a comprehensive Android execution model that contains analysisstubs for most of the Android framework methods. This allows the tool to pre-2https://github.com/secure-software-engineering/FlowDroid/releases/tag/v2.7.13https://bintray.com/arguslab/maven/argus-saf/3.2.013cisely track flows through Android APIs. Yet, it limits the analysis to a particularAndroid version. For reflection, DROIDSAFE uses string analysis to replace reflec-tive calls with direct calls to the target method, when possible. Yet, the tool doesnot have fully-sound handling of reflection. It uses a proprietary model to handleICC and IAC flows. Implicit flows were not supported in the original version of thetool [13], but the latest version of the source code contains an option to enable im-plicit flows. The DROIDSAFE authors officially stopped supporting the tool sinceJune 22, 2016. Therefore, in our study, we considered the latest version that wasavailable as of June 20164, which supports API level 19 and below.3.2 Tool Configuration ParametersEach of the tools provides numerous configuration parameters: FLOWDROID has44 parameters, AMANDROID has 11, and DROIDSAFE has 57. Because the resultsof the analysis largely depend on the tuning of each tool, comparing tools withdifferent configuration setup is not meaningful.We investigated the tool documentation and configuration setup used in previ-ous studies [13, 21, 41, 43], to align the tools along the main configuration dimen-sions. We observed that most of the studies do not document the selected configu-ration parameters, making the comparison inaccurate and irreplicable. We also ob-served that for some tools, a number of important design decisions are hard-codedand cannot be configured at all, e.g., object-sensitivity of FLOWDROID, and thatsome configuration choices are not documented, e.g., field sensitivity of DROID-SAFE and AMANDROID.In our attempt to align the tools around the same parameters, we created andran tests to identify the design choice implemented by each tool and align differenttools to apply the same decisions, when possible. Our final set of configurationchoices is described below.1. Sensitivities. The first five rows of Table 3.2 list five types of sensitivity-relatedconfigurations discussed in Chapter 2, namely, field, context, object, flow, and pathsensitivities; we document our decisions in the second column of Table 3.2. Theremaining columns of the table describe the default sensitivity choice implemented4https://github.com/MIT-PAC/droidsafe-src14Table 3.2: Configuration decisions of FLOWDROID, AMANDROID, andDROIDSAFE.Configuration SelectedFLOWDROID AMANDROID DROIDSAFEDflt. Config? Dflt. Config? Dflt. Config?SensitivityField 3 3 yes 3 no 3 noContext 3 3 yes 3 yes 3a yesObject 3 3 no 3 no 3 yesFlow 3 3 yes 3 no 7 noPath 7 7 no 7 no 7 noImplicit flows 7 7 yes 7 no 7 yesJava-specificReflection 3 7 yes 3 no 3 noException 3 3 yes 3 no 3 yesAndroid-specificICC 3 7 yes 3 no 3 noIAC 7 7 yesb 7 yes 7 yesNative code 7 7 no 3 yes 7 noUI elementsdetection7 3 yes 3 no 7 noa DROIDSAFEâs context sensitivity is for static methods only.b FLOWDROID requires APKCOMBINER for IAC analysis.by each tool and whether this choice can be configured by the user.As some tools do not provide an option to enable/disable certain sensitivitytypes, we had to pick their default, arriving at field-sensitive, object-sensitive, andpath-insensitive analysis. Flow sensitivity (row 4) can only be configured in FLOW-DROID; AMANDROID supports flow sensitivity and DROIDSAFE does not. Assuch, we were unable to fully align the tools for this configuration option. More-over, DROIDSAFE is context-sensitive for static methods only (row 2). We stillopted for enabling context sensitivity, even though other tools provide context sen-sitivity for all rather than only static methods, as this option is central for obtainingaccurate analysis results. Both FLOWDROID and AMANDROID allow to set a con-text sensitivity depth, but use different default values: five and one, respectively.We followed the default setup of FLOWDROID and set the context sensitivity depthto five, to obtain more accurate results; DROIDSAFE does not allow to set thisparameter.2. Implicit flows. FLOWDROID and DROIDSAFE support implicit flow tracking;this option is disabled by default in both tools. AMANDROID does not document15whether it supports implicit flow tracking or not. In our communication with theAMANDROID authors, they confirmed that the tool cannot handle implicit flows.Therefore, we disabled implicit flow tracking for all tools (row 6).3. Java-specific features. FLOWDROID, AMANDROID, and DROIDSAFE all ex-plicitly report that they do not have a fully-sound handling of reflections. Moreover,support for resolving reflective calls differs among tools. Yet, handling reflection isenabled in AMANDROID and DROIDSAFE by default, and we enabled this optionin FLOWDROID as well (row 7).For FLOWDROID and DROIDSAFE, exception handling is enabled by default,though it can be disabled. AMANDROID reports on a limited capability to handleexceptions and provides no configuration parameter regarding exception handling.As exception tracking is an important property of static analysis and all tools en-abled this option by default, we proceed with that choice (row 8).4. Android-specific features. As discussed in Chapter 3.1, FLOWDROID allowsto use STUBDROID to precisely model a subset of Android framework methodsand use a conservative strategy to cover the remaining ones. We opted for usingthis strategy. Android modeling is enabled by default and is not configurable forAMANDROID and DROIDSAFE.We configured FLOWDROID to use the IC3 model for ICC flow extraction:according to earlier studies, this model has higher accuracy than EPICC [26]. Wedid not use PRIMO because it relies on the statistical similarity between the an-alyzed applications, which is absent in the benchmark suites. In addition, by de-fault, the FLOWDROID version that we used applies âpurificationâ of ICC-relatedflows. This means that the tool deviates from the standard taint analysis semanticsand implements additional logic when handling sinks involved in ICC communi-cation, such as startActivity(). After running a number of experiments, weconfirmed that the purification functionality is not fully implemented and alignedwith its documented behavior. In our communication with the FLOWDROID au-thors, they advised us to disable this functionality using noiccresultspurifyparameter and evaluate the tool against the standard taint flow semantics. We thusswitched the ICC purification off. AMANDROID and DROIDSAFE both implementproprietary ICC handling mechanisms and do not provide any configuration op-16tions. We relied on the default behavior of these tools for our study (row 9).FLOWDROID can detect inter-application flows when augmented with AP-KCOMBINER [20]. AMANDROID and DROIDSAFE can also detect IAC flows, butthat option is disabled by default. As the benchmark suite we used contains onlytwo IAC cases and we did not consider IAC in our Google Play experiment, wedisabled IAC tracking in all tools (row 10).AMANDROID is the only tool that supports native code analysis. For consis-tency with the other tools, we disabled this option in AMANDROID (row 11).Finally, both FLOWDROID and AMANDROID are able to detect flows from sen-sitive UI elements, such as password fields. This option is configurable in FLOW-DROID but not in AMANDROID. However, DROIDSAFE does not implement thecorresponding feature. We thus disabled sensitive UI detection in FLOWDROID(row 12) and filtered out all flows from sensitive UI elements reported by AMAN-DROID in our analysis.5. Other configuration parameters. Upon reviewing additional configuration pa-rameters, we observed that most of the remaining FLOWDROID and DROIDSAFEparameters deal with disabling advanced analysis functionality (which we decidedto keep) and provide supportive functions, such as an option to generate .json re-ports in DROIDSAFE. The remaining AMANDROID options control input locationin the file system, debug choices, and other parameters that do not affect the coreflow detection functionality. We left the default values for all these parameters.3.3 Evaluation Methodology for Benchmark AppsWe now discuss our selection of benchmark applications, taint sources and sinkswe used in our analysis, our definition of expected results, and study metrics andmeasures.3.3.1 SubjectsDroidBench and ICC-Bench are the most commonly used benchmarks for compar-ing the tools [4, 13, 21, 28, 32, 41, 43]. These benchmarks are publicly available;the FLOWDROID, ICCTA, and DROIDSAFE teams contributed to the DroidBenchtest suite, and AMANDROID team created ICC-Bench . As all teams participated in17the creation of these benchmarks and also used them for evaluating their tools, thisselection does not unintentionally benefit any of the tools.We chose the latest version of the DroidBench benchmark suite, version 3.05,which covers numerous categories of Android analysis problems, including intra-component and inter-component communication, handling of reflection, sensitivitytypes and more. We used 158 apps from this suite, excluding 28 apps that focuson testing configuration options that we disabled, i.e., tracking of implicit flows,inter-application flow detection, native code analysis, and sensitive UI elementsdetection. In addition, we excluded three applications related to dynamic classloading, as all three tools explicitly state that they cannot handle that scenario. Theexact list of the excluded benchmarks is found in our online appendix [45].The ICC-Bench test suite containing 24 benchmarks was originally developedby the AMANDROID team to test ICC-related capabilities of taint analysis tools.We used the latest version of this suite at the time of writing: version 2.06. ICC-Bench applications use Android API level 25. As DROIDSAFE only supports APIlevel up to 19, we did not run the tool on ICC-Bench . All DroidBench applicationsare compatible with the API level 19 and thus DROIDSAFE, as well as other tools,runs on these applications successfully.Our benchmark test suite thus consists of 182 applications in total from Droid-Bench (158) and ICC-Bench (24), covering 13 DroidBench categories and fourICC-Bench categories. The full list of the applications we analyzed is availableonline [45].3.3.2 Sources and SinksIn earlier work, Arzt et al. [4] and Li et al. [21] used the list of sources and sinksgenerated by the SUSI project [31]; the authors of AMANDROID confirmed thatthey used the sources and sinks marked in each of the benchmarks [43]. The list ofsources and sinks in DROIDSAFE is not configurable without modifying the sourcecode of the tool; Gordon et al. [13] thus ran their experiments with all sources andsinks hard-coded in the tool (4,051 sources and 2,116 sinks in total).To use the same sources and sinks for all the tools, we inspected the headers and5https://github.com/secure-software-engineering/DroidBench/tree/develop6https://github.com/fgwei/ICC-Bench18Table 3.3: List of source and sink methods for benchmark apps (SS-Bench ).SignatureSourcesandroid.telephony.TelephonyManager: java.lang.String getDeviceId()android.telephony.TelephonyManager: java.lang.StringgetSimSerialNumber()android.location.Location: double getLatitude()android.location.Location: double getLongitude()android.telephony.TelephonyManager: java.lang.String getSubscriberId()Sinksandroid.telephony.SmsManager: voidsendTextMessage(java.lang.String,java.lang.String,java.lang.String,android.app.PendingIntent,android.app.PendingIntent)android.util.Log: int i(java.lang.String,java.lang.String)android.util.Log: int e(java.lang.String,java.lang.String)android.util.Log: int v(java.lang.String,java.lang.String)android.util.Log: int d(java.lang.String,java.lang.String)java.lang.ProcessBuilder: java.lang.Process start()android.app.Activity: void startActivityForResult(android.content.Intent,int)android.app.Activity: void setResult(int,android.content.Intent)android.app.Activity: void startActivity(android.content.Intent)java.net.URL: java.net.URLConnection openConnection()android.content.ContextWrapper: void sendBroadcast(android.content.Intent)comments of all 182 benchmark applications and extracted the sources and sinksused in these applications (5 sources and 11 sinks listed in Table 3.3). We namedthis list of sources and sinks SS-Bench . We configured both FLOWDROID andAMANDROID to use SS-Bench . For DROIDSAFE, we confirmed that the sourcesand sinks from this list are the subset of sources and sinks considered by the tool.Furthermore, for a fair comparison between the tools, we ignored flows related toother sources and sinks, if such flows were reported by DROIDSAFE.3.3.3 Expected ResultsWith our selection of sources, sinks, and configuration parameters discussed ear-lier, the expected flows for each benchmark application might deviate from theresult specified by the benchmark designers, e.g., because a source extracted fromone benchmark application could affect another application. We thus manually an-alyzed each of the benchmark applications, extracting all flows expected under our19Table 3.4: Differences in expected results for DroidBench and ICC-Bench .Category Benchmark# Exp. FlowOriginal OursDroidBench1. AliasingMerge1 0 1StrongUpdate1 1 02. Android-specificApplicationModeling1 1 0PrivateDataLeak3 2 14. Callbacks LocationLeak3 1 25. Emulator Detection IMEI1 2 08. Inter ComponentCommunicationActivityCommunication2 1 2ActivityCommunication3 1 2ActivityCommunication4 1 2ActivityCommunication5 1 2ActivityCommunication6 1 2ActivityCommunication7 1 2ActivityCommunication8 1 2BroadcastTaintAndLeak1 1 2ComponentNotInManifest1 0 1IntentSource1 2 0UnresolvableIntent1 2 39. Lifecycle ActivityEventSequence2 1 011. Reflection ICCActivityCommunication2 1 2AllReflection 1 2OnlyIntent 1 2OnlyIntentReceive 1 2OnlySMS 1 2OnlyTelephony 1 2OnlyTelephony Dynamic 2 3OnlyTelephony Reverse 1 2OnlyTelephony Substring 1 2SharedPreferences1 1 2ICC-Bench1. ICC Handlingicc explicit src nosink 0 1icc explicit src sink 1 2icc stateful 3 2icc explicit1 1 23. Mixed icc rpc comprehensive 3 220configuration setup. To ensure validity, the manual analysis was performed inde-pendently and in parallel by me and another member of the research group. Thedisagreements in manual analysis (for seven out of 182 cases, a disagreement rateof 4%) were discussed in a meeting with me and two other members of the researchgroup, towards reaching a common resolution.A full list of all expected flows is available online [45]. In 33 cases listed inTable 3.4, our expected results deviated from those documented in the benchmarkitself. For example, in benchmark ActivityCommunication2, one additional flow isexpected because of an additional sink, startActivity(Intent), which wasadded as it appeared in multiple benchmarks, e.g., DroidBench IntentSink2; inbenchmark IMEI1, no flows are expected because we disabled implicit flow track-ing for all tools. It should be noted that, as discussed in Chapter 3.3.1, we excludedfrom our analysis benchmarks that were solely designed to check the features wedisabled, e.g., ImplicitFlow1-6 that examine implicit flows tracking.3.3.4 Metrics and MeasuresSimilar to the process of establishing the ground truth for our experiments, me andanother member of the research group manually inspected the flows identified byeach tool, comparing them to the expected results. The goal of this analysis was toidentify expected flows detected correctly by a tool: true positives (TP); unexpectedflows mistakenly identified by the tool: false positives (FP); and expected flowsmissed by a tool: false negatives (FN). Similar to the process of identifying theexpected flows, the differences in the results of this manual analysis (for six out of182 cases, a disagreement rate of 3%) were resolved in a discussion involving meand two other members of the research group and, when required, the authors ofthe tools.We then calculated the precision, recall, and F-measure for each tool, as de-scribed below.â¢ Precision: the fraction of correctly reported flows out of the total number ofreported flows, calculated as TPTP+FP .â¢ Recall: the fraction of correctly reported flows out of the total number ofexpected flows, calculated as TPTP+FN .21â¢ F-measure: the weighted harmonic mean of the Precision and Recall, calcu-lated as 2ÃPrecisionÃRecallPrecision+Recall .We also measured the execution time of each tool and reported on the averageexecution time from five consecutive runs. We ran all experiments on a Ubuntu16.04 server with a 12-core CPU and limited the RAM allocation for each experi-ment to 64 GB.3.4 Evaluation Methodology for Google Play AppsManually establishing ground truth for Google Play applications with a variety ofsources and sinks is a challenging task as Google Play applications are generallylarge and obfuscated. We thus opted to perform a realistic case study where wefocused on applications that perform user authentication and tracked the flow ofthe input user credential into internet-related sinks. We selected this case studybecause (a) to correctly perform user authentication, an app has to send out thecredentials to a third-party server, which guarantees the presence of the expectedflow and (b) detecting this type of flows is important in the security setup, e.g., toensure that all passwords are sent over the network encrypted [11]. The goal ofour case study is thus to check whether the tools can detect these flows and, if not,analyze why not.As discussed in Chapter 1, DROIDSAFEâs authors explicitly state that the toolis not designed to work on Google Play application: âPlease do not expect torun DroidSafe on large apps from the Google Play store, and expect DroidSafe tocomplete and/or give you accurate results.â Moreover, the tool only supports appswith API level up to 19, which are not common at the time of writing. We thusexclude this tool, focusing our analysis of Google Play apps on FLOWDROID andAMANDROID only.Below, we discuss our criteria for selecting subject applications and our choiceof sources, sinks, and expected results. We also discuss the metrics we applied andthe measures that we performed.22188 apps 51 apps 20 appsUnsupported API Level FilteringAuthenticationFunctionalityIdentificationIn-appLoginIdentificationTop 5,569 appsFigure 3.1: The Google Play apps selection process.3.4.1 SubjectsFigure 3.1 shows an overview of our Google Play app selection process. We tooktop 100 apps from each of the 58 free app categories in the Canadian Google PlayStore as of December 2019, arriving at 5,569 apps (some categories contain lessthan 100 apps). We used Android Asset Packaging Tool (AAPT)7 to extract theAPI level (i.e., targetSdkVersion) of each app and filtered out apps of API levelsthat are not supported by the tools. Specifically, as AMANDROID only supportsAPI levels up to 25, we selected apps with API level 25 or below when comparingthe tools. This selection resulted in 188 apps.Next, me and another member of the research group used one device each andexecuted every app for around 20 minutes, to check whether the app contains anyauthentication functionality and, if so, to validate that the authentication function-ality is working. After cross-validating the results, we confirmed reliable authen-tication mechanism in 51 of the apps. There were no disagreements between meand another member of the research group in this case.We then analyzed the bytecode of the identified 51 applications to ensure theflow between username/password and internet-related sinks is fully implementedwithin the code of the applications the tools are able to analyze. That excluded(1) seven apps that implement login functionality using WebView â a widget thatallows user to input login credentials in a web page; (2) five apps built using cross-platform frameworks such as Unity8 and React Native9; and (3) 16 apps that dele-7https://developer.android.com/studio/command-line/aapt28https://docs.unity3d.com/Manual/android-GettingStarted.html9https://facebook.github.io/react-native/23Table 3.5: List of the selected Google Play apps, and number of the expectedflows.App ID App Name API Level # Exp. Flow1 com.echangecadeaux 14 12 com.rtp.livepass.android 17 13 com.tripadvisor.tripadvisor 19 14 ca.intact.mydrivingdiscount2115 com.asiandate 16 ca.passportparking.mobile.passportcanada2217 com.aldiko.android 18 com.aldiko.classic 19 com.passportparking.mobile.parkvictoria 110 com.passportparking.mobile.toronto 111 tc.tc.scsm.phonegap 112 com.onetapsolutions.morneau.activity23313 net.fieldwire.app 114 com.ackroo.mrgas 24 1515 com.airbnb.android25116 com.bose.gd.events 217 com.phonehalo.itemtracker 318 com.viagogo.consumer.viagogo.playstore 119 com.yelp.android 120 onxmaps.hunt 1gate the login functionality to third-party OAuth service providers, such as Googleor Facebook. We further excluded three apps that were highly obfuscated and wecould not identify the authentication type and flow.Our final app selection consists of 20 Google Play apps listed in Table 3.5 andincludes apps such as Tripadvisor, Airbnb, and Yelp. The distribution of apps thatwe excluded at each stage of our subject selection process is found in the onlineappendix [45]. In what follows, we refer to the analyzed apps by their app ids listedin the first column of Table 3.5, e.g., app #1 refers to com.echangecadeaux.3.4.2 Sources, Sinks, and Expected ResultsMe and another member of the research group further proceeded with a manualanalysis of the selected apps to identify expected flows from the password field24Table 3.6: List of source and sink methods for Google Play apps (SS-GP ).Signature # AppSourcesandroid.widget.EditText: android.text.Editable getText() 20Sinksorg.apache.http.client.HttpClient: org.apache.http.HttpResponseexecute(org.apache.http.client.methods.HttpUriRequest)4java.net.URL: java.net.URLConnection openConnection() 3okhttp3.Call: okhttp3.Response execute() 3okhttp3.Call: void enqueue(okhttp3.Callback) 2org.apache.http.impl.client.DefaultHttpClient:org.apache.http.HttpResponseexecute(org.apache.http.client.methods.HttpUriRequest)2com.squareup.okhttp.Call: com.squareup.okhttp.Response execute() 1cz.msebera.android.httpclient.client.HttpClient:cz.msebera.android.httpclient.HttpResponse exe-cute(cz.msebera.android.httpclient.client.methods.HttpUriRequest)1java.io.OutputStreamWriter: void write(java.lang.String) 1java.io.PrintWriter: void write(java.lang.String) 1java.net.HttpURLConnection: int getResponseCode() 1java.util.zip.GZIPOutputStream: void write(byte[]) 1org.apache.http.client.HttpClient: org.apache.http.HttpResponseexecute(org.apache.http.client.methods.HttpUriRequest,org.apache.http.protocol.HttpContext)1to the internet. To ensure validity, me and another member of the research groupconducted manual checks independently and cross-validated their findings. Theyhad disagreements in two out of 20 apps (10% disagreement rate); like with thebenchmark applications, the disagreements are resolved in a meeting with me andtwo other members of the research group. In the end, we were able to identify theflow in all 20 apps, resulting in one expected flow per app.We used textual input fields, namely EditText.getText(), as the source forour password-related flows. We then followed the flow to identify internet-relatedAPIs that send the userâs password over the network and considered them as sinks.While doing our analysis, we noticed that one app also writes the password tothe log file, for debugging purposes. This is a concerning behavior and we added25the corresponding flow and the sink to our ground truth results. The list of theidentified sources and sinks is shown in Table 3.6. We refer to this list as SS-GPand configured the tools to use this list. The last column of Table 3.6 lists thenumber of apps containing each source and sink.As source EditText.getText() can cover user inputs other than the pass-word, tools may detect more valid flows than the 20 login-related ones. Hence, foreach app, we analyzed all flows identified by the tools and categorized them intoTPs and FPs. We added all identified TPs to our expected results, which resultedin additional 19 flows in four apps: two flows in app #12, 14 flows in app #14, oneflow in app #16, and two flows in app #17. The total number of expected flowswe identified is shown in the fourth column of Table 3.5. We could not detect allpossible flows in the applications due to their size and complexity.3.4.3 Metrics and MeasuresAs with the benchmark apps, me and another member of the research group in-dependently identified FP and FN results based on our ground truth and cross-validated their findings. The disagreements (in one app, 5% disagreement rate)were resolved in a discussion with me and two other members of the researchgroup. We then calculated precision, recall, and F-measure for each tool; we alsomeasured the execution time of each tool and reported on the average executiontime from five consecutive runs. All experiments were run on the same Ubuntu16.04 server with a 12-core CPU; we limited the RAM allocation for each experi-ment to 256 GB.We set a 72-hour timeout for each app. AMANDROID also provides the optionto set a maximum running time for each component in an app. To prevent the toolfrom spending the entire allocated time on one component and being unable toreach the remaining components, we set a per-component limit as 72 hours dividedby the number of components declared in an appâs manifest file. As the number ofcomponents in our selected apps ranges from 18 to 314, the timeout set for eachcomponent ranges from 14 to 240 minutes, which is larger than 10 minutes set bythe authors of AMANDROID in their experiments [41].263.5 Limitations and Threats to ValidityThe main threat to the validity of our results stems from the manual analysis weperformed to identify the expected results for each benchmark and Google Playapp, the FP and FN flows in each tool, and the causes of these flows. We miti-gated this threat in three ways: first, me and another member of the research groupperformed the analysis independently and then cross-checked each otherâs results.Second, we constructed and ran our own test cases, to confirm each hypothesis fora possible cause of a FP/FN result. We also reached out to the authors of each toolto resolve cases where a definite conclusion could not be reached.As the set of sources and sinks in DROIDSAFE is not configurable without codemodifications, we ignored flows detected by DROIDSAFE when these flows did notinvolve sources and sinks in our set. While this part of the process was automated,we could have mistakenly missed important flows. Again, we mitigated this threatby independently inspecting the results of each tool and comparing it to our groundtruth by at least two members of the research group. Yet, we acknowledge thatrunning the tool with an extended set of sources and sinks could alter its behavior.Finally, as DROIDSAFE does not support applications above API level 19 andICC-Bench applications require API level 25, we did not report DROIDSAFE re-sults on ICC-Bench . Moreover, limited API support of AMANDROID constrainedthe range of Google Play apps we tested the tool on. As we only run each tool onapps it can support, we believe the experiment is valid.27Chapter 4Results: Benchmark Apps(RQ1-RQ3)We now answer RQ1âRQ3 introduced in Chapter 1.4.1 Tools Performance (RQ1)To answer RQ1, we ran FLOWDROID, AMANDROID, and DROIDSAFE, on the182 benchmark apps using the configuration setup described in Chapter 3.2, andmeasured the accuracy and execution time of each tool.4.1.1 AccuracyThe first row of Table 4.1 (âOur current experimentâ) summarizes the precision,recall, and F-measure obtained in our experiments with benchmarks, which wereport separately for DroidBench and ICC-Bench . FLOWDROID achieves thehighest accuracy on DroidBench benchmarks (F-measure of 77%), followed byDROIDSAFE (73%) and AMANDROID (55%). FLOWDROID also performs betteron ICC-Bench : 79% vs. 67% for AMANDROID. We did not run DROIDSAFE onthis benchmark, as discussed in Chapter 3.3.1.For six benchmark apps, FLOWDROID crashed with exceptions; for one app,DROIDSAFE did not finish after two hours time limit. As all these benchmarksinclude expected flows, we counted these flows as FNs for the corresponding tools.28Table 4.1: Benchmark apps: comparisons of precision, recall, and F-measures.Source Benchmark Suite Tool Precision Recall F-measure1. OurcurrentexperimentDroidBench (158 appsfrom v3.0)FLOWDROID 88 68 77AMANDROID 63 49 55 (61)DROIDSAFE 73 73 73 (87)ICC-Bench (24 appsfrom v2.0)FLOWDROID 100 65 79AMANDROID 77 59 67 (92)2. The priorwork [30]DroidBench (109 appsfrom v2.0)FLOWDROID 90 81 85AMANDROID 67 69 68DROIDSAFE 76 85 80 (92)ICC-Bench (24 appsfrom v2.0)FLOWDROID 100 76 87AMANDROID 85 100 923. Arzt et al.[4]DroidBench (35 appsfrom v1.0 developed byFLOWDROIDâs authors,no implicit flows)FLOWDROID(no ICCTA)86 93 894. Li et al.[21]22 ICC-related appsdeveloped by ICCTAâsauthors and ICC-Bench(9 apps from v1.0)FLOWDROID 97 97 97AMANDROID 79 52 635. Wei et al.[41]DroidBench (39 appsfrom v1.0)FLOWDROID(no ICCTA)86 72 78AMANDROID 87 75 81ICC-Bench (16 appsdeveloped byAMANDROIDâs authors)FLOWDROID(no ICCTA)75 45 56AMANDROID 100 100 1006. Wei et al.[43]DroidBench (18 appsfrom v2.0, ICC-related)FLOWDROID 86 83 85AMANDROID 96 96 96DROIDSAFE 85 96 90ICC-Bench (24 appsfrom v2.0)FLOWDROID 97 90 93AMANDROID 97 100 98DROIDSAFE 10 3 57. Gordonet al. [13]DroidBench (94 appsfrom v1.2)FLOWDROID 73 81 76DROIDSAFE 88 94 91DroidBench (40 appsdeveloped byDROIDSAFEâs authors)FLOWDROID 79 35 48DROIDSAFE 100 100 100291 class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 void onCreate ( Bundle s ta te ) {4 . . .5 S t r i n g imei = getDeviceId ( ) ; / / source6 I n t e n t i = new I n t e n t ( this , A n o t h e r A c t i v i t y . class ) ;7 i . pu tEx t ra ( â imei â , imei ) ;8 s t a r t A c t i v i t y ( i ) ; / / s ink , leak9 }10 }11 class A n o t h e r A c t i v i t y extends A c t i v i t y {12 @Override13 void onCreate ( Bundle s ta te ) {14 . . .15 I n t e n t i = g e t I n t e n t ( ) ;16 S t r i n g imei = i . ge tS t r i ngEx t ra ( â imei â ) ;17 Log . i ( âTAGâ , imei ) ; / / s ink , leak18 }19 }Figure 4.1: Flows with ICC-related sinks.When manually classifying the remaining flows detected by each tool into TPs,FPs, and FNs, we discovered that, by design, AMANDROID and DROIDSAFEdeviate from the âstandardâ taint analysis semantics and implement additionallogic when handling ICC-related flows. By inspecting the source code of AMAN-DROID, we noticed that starting from version 3.1.2, the tool simply filters out intra-component flows for ICC-related sinks, e.g., startActivity(Intent), eventhough the tool can successfully detect them. For the example in Figure 4.1, thesource in line 5 flows through the sink in line 8 and the sink in line 17. In thiscase, AMANDROID detects both flows but only reports the second one (line 17),while our expected results assume a âstandardâ taint analysis semantics and thustwo flows.Similarly, DROIDSAFE also uses its own proprietary logic to handle ICC flows.Based on DROIDSAFEâs documentation and our experiments with multiple exam-ples, we learned that the logic applied by DROIDSAFE is more advanced than thatof AMANDROID. Specifically, it only filters intra-component flows for explicit In-tents; for implicit Intents, the tool always reports the intra-component part of ICCflow as well, as components from other applications could also register and receivethis Intent.301 class Act iv i t yWi thRunnab le extends A c t i v i t y {2 @Override3 void onCreate ( Bundle s ta te ) {4 . . .5 Executors . newCachedThreadPool ( ) . execute (6 new MyRunnable ( getDeviceId ( ) ) / / source7 ) ;8 }9 class MyRunnable implements Runnable {10 S t r i n g dev ice Id ;11 MyRunnable ( S t r i n g dev ice Id ) { th is . dev ice Id = dev ice Id ; }12 @Override13 void run ( ) { Log . d ( â Ac t i v i t yWi thRunnab le â , dev ice Id ) ; } / / s ink , leak14 }15 }Figure 4.2: Placement of entry methods.We also found that DROIDSAFE does not report the exact path of the detectedflows, making the flow analysis more difficult. Instead, it reports the source andsink methods of the identified flow, as well as the location of the entry method, i.e.,the method in which the source is defined. We used this information to identify theflows and match them to our ground truth results.Yet, the tool sometimes reports entry methods incorrectly. For the example inFigure 4.2, DROIDSAFE correctly reports a flow from getDeviceId() (line 6) toLog.d() (line 13), but considers MyRunnable: void run() being the entrymethod, which is incorrect. Overall, there are 17 output record with the correctsource and sink methods but with the entry method pointing to the sink rather thanthe source; in five cases, the tool produced duplicated flows, one with the entrymethod correctly pointing to the source and another â to the sink; in one additionalcase, the entry method pointed to a different method altogether.Originally, we categorized flows with incorrect entry points as FPs and alsoconsidered as FNs the correct flows missed by the tool, e.g., the flow with voidonCreate() (line 3) as the entry method in Figure 4.2. Yet, as the tool still reportsthe correct sources and sinks for all these cases, we also recalculated the precision,recall, and F-measure for DROIDSAFE while ignoring the placement of entry meth-ods and duplicated flows, i.e., ignoring 23 FPs and 18 FNs results. Likewise, toincrease fairness, we also accounted for the special semantics of ICC flows em-ployed by AMANDROID and DROIDSAFE as described above, ignoring 26 FNs for31AMANDROID and 5 FNs for DROIDSAFE.The adjusted numbers are shown in parentheses in Table 4.1: the F-measureof AMANDROID on the DroidBench and ICC-Bench becomes 61% and 92%, andthe one for DROIDSAFE on DroidBench reaches 87%. With the overall accuracygoes up after the adjustment, DROIDSAFE now has the highest accuracy (87%) onDroidBench , followed by FLOWDROID (77%). AMANDROID still has the lowestaccuracy (61%) on this benchmark suite. Yet, AMANDROID now performs betteron ICC-Bench : 92% vs. 79% for FLOWDROID.4.1.2 Execution TimeWe measured the execution time of each tool on the DroidBench applications only,as DROIDSAFE does not run on ICC-Bench . We averaged execution time from fiveconsecutive runs, and further averaged the execution time for all analyzed bench-marks. To ensure a fair comparison, we configured FLOWDROID and AMAN-DROID to use the same set of sources and sinks that DROIDSAFE uses, as the setof sources and sinks in DROIDSAFE cannot be configured. Our results show thatall the analyzed tools processed benchmark applications within a relatively shorttime, with FLOWDROID being the fastest: 12 seconds, followed by AMANDROID:16 seconds, and DROIDSAFE: 149 seconds. These statistics exclude six benchmarkapps for which FLOWDROID crashed and one outlier app for which DROIDSAFEdid not finish after two hours.4.2 Comparison with Earlier Experiments (RQ2)To answer RQ2, we extracted the accuracy and execution time reported by previousstudies as well as the prior work [30], compared with our results reported in theprevious subchapter, and outlined possible reasons for differences in results.4.2.1 AccuracyComparison with the prior work [30]. The precision, recall, and F-measureobserved in the prior work [30] are shown in the second row of Table 4.1; theseresults are higher than the results observed in our current experiments (the first rowof the table). The main reason for the difference is the upgrade of the DroidBench32benchmark from version 2.0 that we used earlier to version 3.0 that is used inthe current experiments (there are no updates in the ICC-Bench version). TheDroidBench upgrade included 49 new and one modified benchmark apps, whichexposed additional weaknesses in all three tools. When excluding the new andmodified DroidBench benchmarks, FLOWDROID achieves the F-measure of 84%and AMANDROID achieves the F-measure of 66%, which are comparable with 85%and 68%, respectively, reported in the earlier work; DROIDSAFE version did notchange, so the F-measure becomes 80%, as before.The remaining differences stem from the updates of the tools themselves; weused the latest versions of the tools available in December 2019: FLOWDROIDversion 2.7.1 (vs. version 2.0 in the earlier work) and AMANDROID version 3.2.0(vs. 3.1.1). A detailed analysis of all problems identified in the current versionof the tools is given in Chapter 4.3. Here, we focus on upgrade-related issues weidentified during our detailed comparisons between the two versions of the tools.We reported these issues to the toolsâ authors and some are already fixed. Thecomplete list of benchmark applications on which tools report different results isavailable in our online appendix [45].For FLOWDROID, we identified several issues related to the use of STUB-DROID, which caused reduced accuracy on the DroidBench suite, and issues re-lated to the handling of Intents that involve Services and Broadcast Receivers,which caused reduced accuracy on the DroidBench (ICC category) and ICC-Benchsuite. The patches we provided to fix these issues were already accepted by theFLOWDROID authors; we confirmed the fixes and were informed that they will bereleased in the next version of the tool.The authors of AMANDROID stated that the version upgrade was performedto integrate AMANDROID with the native code analysis framework JN-SAF [42],which defines a unified representation for data flows in Java and native code, sothey can be âstitchedâ together. This integration introduced a few issues and ledto decreased accuracy in the DroidBench suite. We reported these issues to theAMANDROID authors and are working with them on a fix.Comparison with other reports. We list the tool accuracy reported by other stud-ies conducted by the toolâs authors themselves [4, 13, 21, 41, 43] in rows 3-7 of33Table 4.1. We noticed that the accuracy observed in our experiments is mostlylower than that reported in earlier studies. For example, Li et al. [21] (row 4)reported a very higher accuracy of 97% for FLOWDROID, compared with 77% onDroidBench and 79% on ICC-Bench in our study. As our experiments include sub-stantially larger benchmark suites (e.g., 158 DroidBench and 24 ICC-Bench appsvs. 31 ICC-related apps in Li et al. [21] experiment), we observe more failures andcannot compare our results to other studies directly.Interestingly, there is also a high variability in the results reported by differentstudies themselves. This is mainly because each paper only evaluated the toolson a subset of benchmarks, focusing on a specific set of challenges. Even thoughboth Li et al. [21] (row 4) and Wei et al. [43] (row 6) evaluated FLOWDROID onICC-related benchmark apps, they still used a different subset of apps: the for-mer contributed 22 ICC-related apps and used 9 apps from ICC-Bench while thelater selected 18 ICC-related apps from DroidBench and 24 apps from ICC-Bench ,most of which were not available in 2015. In fact, the comparison often involvesbenchmarks contributed by the toolsâ authors, e.g., Gordon et al. [13], Li et al.[21], Wei et al. [41], showing a high accuracy for a particular tool on that subset ofbenchmarks.Moreover, the results reported by Arzt et al. [4] (row 3) cannot be directly com-pared with those of Li et al. [21] (row 4), because the latter work introduced theintegration of FLOWDROID with ICCTA. Even though both Arzt et al. [4] (row3) and Wei et al. [41] (row 5) ran FLOWDROID without ICCTA integration, they,again, used a different subset of benchmarks: the former work excluded all bench-marks with implicit flows while the latter rather selected benchmarks related toICC communication.In addition to the different benchmarks and parameters used to configure theevaluated tools, the set of sources and sinks also differs between the studies (e.g.,DROIDSAFE uses a proprietary and non-configurable set of sources and sinks),further hindering the comparison. Our independent and large-scale experimentcovering the entire set of benchmark apps under the common setup is necessary toassess and compare the performance of the tools.344.2.2 Execution TimeWith respect to the performance evaluation, our results are consistent with otherreports showing that FLOWDROID is faster than AMANDROID, which is, in turn,faster than DROIDSAFE [13, 21, 30]. We do not perform a numerical comparisonof execution times with other experiments, as these experiments were performedusing a different hardware setup, such as CPU and RAM size, and different con-figuration selections.4.3 Causes of Inaccuracy (RQ3)In this subchapter, we analyze the failures we observed in each of the tools. Westart by identifying the target criterion for each benchmark â a particular aspect oftaint analysis that the benchmark app is designed to test. For example, the Acces-sArray1 benchmark app tests whether the analysis distinguishes between differentarray positions. We mark this criterion with the index of its corresponding bench-mark, i.e., DB3.1 . Numbers prefixed with DB indicate target criteria extractedfrom DroidBench (126 target criteria in total, grouped into 13 categories); num-bers prefixed with ICC indicate target criteria extracted from ICC-Bench (18 targetcriteria, grouped into four categories).A particular benchmark app can evaluate multiple target criteria, e.g., bothreflection and location handling. For example, in benchmark EventOrdering1, thedesigners tested whether the analysis tool is able to take into account differentrepeating runs of the same activity. At the end of the first run, they stored the taintedvariable in SharedPreferences and then retrieved it in the next run. WhileFLOWDROID and AMANDROID both failed for this app, the underlying reasonis not that they cannot track the repeating runs of an activity (criterion DB9.16)but rather that the tools cannot model SharedPreferences (criterion DB8.16).We distilled the SharedPreferences failure into a separate test case, which weadded to UBCBench .Moreover, we identified six additional target criteria not explicitly mentionedby the existing benchmarks. We marked them with the âU-â prefix and assignedthem the next available id in their corresponding category, e.g., U-DB10.6 for thecriterion we added to the Reflection category DB10.*. We used these new criteria35Table 4.2: FN/FP breakdown by target criteria, at benchmark level.ToolFN FP# Breakdown # BreakdownDroidBenchFLOWDROID 50 1x DB2.8 ;1x DB7.10 ;1x DB7.13 ;1x DB7.15 ;1x DB7.16 ;1x DB8.2 ;1x DB8.6 ;1x DB8.8 ;5x DB8.16 ;2x DB8.18 ;1x DB9.3 ;1x DB9.8 ;1x DB9.14 ;1x DB9.17 ;1x DB9.18 ;2x DB9.19 ;1x DB12.6 ;7x BUG1 ;1x BUG2 ;1x EX ;4x UN ;1xU-DB7.24;13xU-DB10.6 ;15 1x DB2.1 ;1x DB3.1 ;1x DB3.2 ;1x DB3.5 ;1x DB3.8 ;1x DB3.9 ;1x DB4.14 ;1x DB6.7 ;1x DB7.4 ;1x DB7.21 ;1x DB7.22 ;1x DB7.23 ;3x DB13.1 ;AMANDROID 69 1x DB2.7 ;1x DB2.8 ;1x DB2.11 ;1x DB3.6 ;1x DB3.7 ;1x DB3.10 ;1x DB4.4 ;1x DB4.5 ;1x DB7.5 ;1x DB7.6 ;1x DB7.10 ;1x DB7.12 ;1x DB7.16 ;3x DB8.16 ;1x DB8.17 ;1x DB9.1 ;1x DB9.5 ;1x DB9.8 ;5x DB9.9 ;1x DB9.14 ;1x DB9.17 ;1x DB10.2 ;1x DB10.3 ;1x DB10.4 ;5x DB10.5 ;20x DB11.1 ;1x DB12.6 ;2x UN ;10x U-DB4.15 ;1x U-DB7.24 ;45 1x DB3.1 ;1x DB3.2 ;1x DB3.8 ;1x DB3.9 ;2x DB4.12 ;1x DB4.14 ;2x DB7.4 ;1x DB7.21 ;1x DB8.2 ;4x DB13.1 ;22x U-DB2.12 ;7x U-DB4.15 ;1x U-DB4.16 ;DROIDSAFE 20 1x DB2.6 ;1x TO ;1x UN ;17x U-DB10.7 ;20 5x DB1.1 ;1x DB2.1 ;1x DB3.1 ;1x DB3.2 ;1x DB3.8 ;1x DB3.9 ;2x DB4.12 ;1x DB4.14 ;2x DB7.4 ;1x DB8.18 ;1x DB9.2 ;3x DB13.1 ;ICC-BenchFLOWDROID 12 1x ICC1.6 ;1x ICC2.4 ;1x ICC2.5 ;3x BUG2 ;6x EX ;0 noneAMANDROID 0 none 6 1x ICC2.2 ;1x ICC4.1 ;2x U-DB2.12 ;2x U-DB4.16 ;to explain some of the failures and also added the corresponding six benchmarkapplications to UBCBench . In total, we added 15 new benchmarks to UBCBench :nine for untangling benchmarks criteria and six for the new target criteria we iden-tified. We did not use these new benchmarks in our experiments, to focus theanalysis on DroidBench and ICC-Bench only.361 public class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 public void onCreate ( Bundle ins tance ){4 . . .5 TelephonyManager manager = . . . ;6 Class c = Class . forName ( â andro id . te lephony . TelephonyManager â ) ;7 Method m = c . getMethod ( â getDeviceId â ) ;8 S t r i n g imei = ( S t r i n g ) m. invoke ( manager ) ; / / source ( r e f l e c t i o n )9 Log . v ( â imei â , imei ) ; / / s ink , leak10 }11 }Figure 4.3: U-DB10.6 â Sources or sinks invoked using reflection.Each analyzed benchmark application contains none to multiple expected flows.In our suite, there are 192 expected flows â 158 for the DroidBench and 34 for theICC-Bench apps. Table 4.2 shows the number of false positive and false negativeresults reported by each tool, when evaluated against the set of expected flows. Weaggregated the results for each tool on a particular benchmark suite by their failingcriteria. That is, we inspected each FP and FN flow observed in our experiment tofind the reason for the failure and indexed it with the appropriate criterion. We ad-justed for special semantics AMANDROID and DROIDSAFE employ, as discussedin Chapter 4.1, i.e., we report here the FP and FN flows that correspond to thenumbers in parentheses in Table 4.1.In seven cases, we were unable to recover the reasons for the failures. Thesecases are indicated by UN in Table 4.2. In addition, DROIDSAFE timed out inone case, which we denoted by TO. FLOWDROID exceptions discussed in Chap-ter 4.1.1 are marked as EX . Bugs related to FLOWDROIDâs integration withSTUBDROID and its handling of ICC flows in Services and Broadcast Receiversdiscussed in Chapter 4.2.1 are marked as BUG1 and BUG2, respectively. Thefull indexed list of benchmarks, failing criteria for each benchmark and tool, andthe UBCBench test suite are available online [45]. We now discuss the main rea-sons for the failures of each tool.FLOWDROID does not resolve reflective calls when identifying source and sinkmethods but rather relies on API signatures. For the example in Figure 4.3, thegetDeviceId() source method is invoked reflectively in line 8. FLOWDROIDwill miss this source and, thus, will miss the leakage to the sink in line 9. That371 public class M a i n A c t i v i t y extends A c t i v i t y implements View . OnCl ickL is tener {2 @Override3 protected void onCl ick ( View v ) {4 . . .5 S t r i n g imei = getDeviceId ( ) ; / / source6 ( ( Button ) v ) . se tH in t ( imei ) ;7 Log . d ( âTAGâ , ( ( Button ) v ) . ge tH in t ( ) . t o S t r i n g ( ) ) ; / / s ink , leak8 }9 }Figure 4.4: U-DB7.24 â Flows through casted variables.1 public class M a i n A c t i v i t y extends A c t i v i t y implements View$OnCl ickListener {2 public void onCl ick ( andro id . view . View ) {3 View $r1 ;4 . . .5 S t r i n g $r3 ; / / t a i n t ed by getDeviceId ( )6 Button $r4 ;7 CharSequence $r5 ;8 $r4 = ( Button ) $r1 ;9 v i r t u a l i n v o k e $r4. ($r3 ) ;10 $r4 = ( Button ) $r1 ;11 $r5 = v i r t u a l i n v o k e $r4. () ;12 $r3 = i n t e r f a c e i n v o k e $r5.<CharSequence : S t r i n g t o S t r i n g ( ) >() ;13 s t a t i c i n v o k e <Log : i n t v ( S t r ing , S t r i n g )>( âTAGâ , $r3 ) ; / / s ink14 return ;15 }16 }Figure 4.5: Simplified Jimple code for Figure 4.4.behavior leads to 13 FN results stemming from six separate benchmarks (out of 10in the Reflection ICC category, DB11.*). As none of these benchmarks specificallyfocuses on the issue of sources and sinks defined through reflection but rather in-cludes other âcomplexitiesâ such as string manipulation and ICC flow, we created anew target criterion (U-DB10.6) and the corresponding benchmark in UBCBenchto pinpoint this issue. Interestingly, AMANDROID also cannot handle this case, butDROIDSAFE is successful in identifying such sources and sinks.We also discovered that FLOWDROID fails to propagate taint when the involvedobjects are accessed via casting (U-DB7.24). For the example in Figure 4.4, aView passed as a parameter to the method is, in fact, a Button. Button-specificmethods, setHint() and getHint(), are accessed via casting: in line 6, the hintis set to the tainted imei variable and then the value of the hint is leaked in line 7.38To explain the reasons behind this failure, Figure 4.5 shows the relevant portionof the Jimple code [39] that corresponds to the bytecode in Figure 4.4 (FLOW-DROID performs its analysis on Jimple rather than bytecode level). When thetainted value in $r3 is assigned to a field of $r4 (via the method setHint()in line 9), FLOWDROID triggers backward alias analysis to identify all other vari-ables aliased by $r4, as these valuables should be tainted as well. The analysisreaches $r1 (line 8) but fails to backward-propagate the taint, as the assignmentin this case is performed via a casting. As a result, when $r1 is assigned to $r4again (line 10), $r4 is not tainted. Hence, the taint does not reach the sink (line13), resulting in an FN.As this issue was, again, discovered in a benchmark whose goal is to test call-back handling, we created a separate benchmark, U-DB7.24, to focus specificallyon the casting problem and added it to UBCBench . AMANDROID also fails on thisbenchmark albeit for a different reason: it treats casted variables as two completelyseparate entities; DROIDSAFE manages to handle this case successfully.Another major reason for FLOWDROID failures is the ICC handling: there areten failures in the general ICC category (DB8.*) and three in ICC-Bench ( ICC2.4 ,ICC2.5 , and ICC1.6). These failures are related to Intent tracking through listoperations, using complex operations like string manipulations for defining In-tents and their actions, and the SharedPreferences problems discussed above(DB8.16). The tool also fails to handle advanced ICC involving URIs and MIMEformat and ICC-related communication between activities.The remaining, less major failures are from the General Java and Threadingcategories (eight failures in DB7.* and one in DB12.6), Lifecycle and Callbacks(seven failures in DB9.* and one in DB4.14), Array and Lists (five failures inDB3.*), Field and Object Sensitivity (DB6.7), Unreachable Code (three failuresin DB13.*), Android-specific categories (two failures in DB2.*).AMANDROID reports 24 FPs related to its conservative modeling of Android frame-work methods: as discussed in Chapter 3.1, when one of the parameters of amethod is tainted, AMANDROID propagates the taint to other parameters as well.For the example in Figure 4.6, the method Toast.makeText(Context, Char-Sequence, int) in line 7 is an Android framework method that presents a pop-391 public class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 public void onCreate ( Bundle ins tance ){4 . . .5 S t r i n g imei = getDeviceId ( ) ; / / source6 i n t l eng th = Toast .LENGTH SHORT;7 Toast . makeText ( this , imei , l eng th ) . show ( ) ;8 Log . i ( âTAGâ , â â + leng th ) ; / / s ink , no leak9 }10 }Figure 4.6: U-DB2.12 â Android framework method modeling (AMANDROIDmistakenly propagates taint from one parameter to another).1 class M a i n A c t i v i t y extends A c t i v i t y {2 LocationManager locManager ;3 Loca t i onL i s tene r l o c L i s t e n e r = new Loca t i onL i s tene r ( ) {4 double l a t ;5 @Override6 void onLocationChanged ( Locat ion loc ) {7 l a t = loc . ge tLa t i t ude ( ) ; / / source8 Log . d ( âTAGâ , l a t ) ; / / s ink , leak9 }10 } ;11 . . .12 }Figure 4.7: U-DB4.15 â Location-related flow.up on the screen.1 All the parameters of this method are independent and there isno data flow between them. However, AMANDROID propagates the taint from thesecond parameter, imei, tainted in line 5, to the third parameter, length. As aresult, the tool reports an FP flow to the sink in line 8. As this issue is observed in13 benchmarks aiming at evaluating exception and lifecycle handling, we createda new target criterion and the corresponding benchmark in UBCBench focusingsolely on this issue (U-DB2.12).AMANDROID also reports 7 FPs and 10 FNs that stem from handling location-related flows (U-DB4.15). This is because the tool hard-codes the Locationparameter of the onLocationChanged() callback as a source, and does not con-sider location sources specified by the user like getLatitude()/getLongitude().For the example in Figure 4.7, AMANDROID will not report the expected flow from1https://developer.android.com/guide/topics/ui/notifiers/toasts401 class M a i n A c t i v i t y extends A c t i v i t y {2 . . .3 @Override4 void o n A c t i v i t y R e s u l t ( i n t requestCode , i n t resul tCode , I n t e n t resu l tDa ta ) {5 Log . d ( âTAGâ , resu l tDa ta ) ; / / s ink , no leak6 }7 }Figure 4.8: U-DB4.16 â Callback Intent handling.the specified source getLatitude() to the sink Log.d() in line 8, but will reporta (false-positive) flow from the callback parameter loc of onLocationChanged()(line 6) to the sink Log.d(). This behavior is troublesome because there are caseswhere the Location object is accessed, but the retrieved data is not sensitive,e.g., loc.getTime(); hence, AMANDROIDâs conservative handling of locationsources results in many FP flows. We focused on this issue in the benchmark appU-DB4.15.Likewise, AMANDROID always considers Intent parameters of callbacks assources. For the example in Figure 4.8, AMANDROID considers the Intent pa-rameter resultData as a source, and then propagates a flow to the sink in line 5,which is incorrect. We mark this issue as U-DB4.16 in Table 4.2; it leads to threeFP results.Another major reason for failures in AMANDROID is its limited handling ofreflection, as confirmed by the toolâs authors. That results in 28 FNs (DB10.* andDB11.*). AMANDROID also has ten FNs in Lifecycle category (DB9.*) and fivefailures in Callbacks (DB4.*). That is mainly because the tool does not modellifecycle methods of the Application class and does not correctly handle certainlifecycle methods of Activity and fragment, e.g., onRestoreInstanceState()of Activity and onAttach() of fragment. The tool also fails to track Intentsthrough string manipulation and SharedPreferences, leading to seven ICC-related failures (DB8.* , ICC2.2 , and ICC4.1).Similarly to FLOWDROID, the tool also does not handle taint propagationthrough casted variables (U-DB7.24). Other failures include General Java andThreading (eight failures in DB7.* and one in DB12.6), Arrays and Lists (sevenfailures in DB3.*), Unreachable Code (four failures in DB13.*), and Android-411 public class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 protected void onCreate ( Bundle savedInstanceState ) {4 . . .5 Class c = getClass ( ) ;6 Method m = c . getMethod ( â get Imei â ) ;7 m. invoke ( this , nul l ) ;8 }9 public S t r i n g get Imei ( ) {10 S t r i n g imei = getDeviceId ( ) ; / / source11 Log . i ( âTAGâ , imei ) ; / / s ink , leak12 }13 }Figure 4.9: U-DB10.7 â Reflection via Object.getClass() method.specific categories (three failures in DB2.*).DROIDSAFE has 17 FNs in handling reflections, all due to a singular reason: whilethe tool can handle the reflective method Class.forName(String), it cannotidentify the class retrieved via the method Object.getClass() and further failsto resolve reflective calls to methods in the retrieved class. We confirmed thisissue with a new benchmark in Figure 4.9 and marked it as U-DB10.7 . In thisexample, DROIDSAFE fails to recognize the class MainActivity obtained bygetClass() in line 5. As a result, it misses reflective calls to invoke getImei()(line 6 and 7), and further misses the leakage inside this method in line 11. In-terestingly, FLOWDROID can handle this case while DROIDSAFE can handle thereflective calls in Figure 4.3 causing FLOWDROID to fail.DROIDSAFE also has five failures related to flow insensitivity, which is due tothe toolâs design (DB1.1). Five failures are related to the handling of Callbacks,Lifecycle, and ICC (DB4.* , DB8.18 , DB9.2). These are because DROIDSAFEassumes all possible callback orders, even those that cannot occur at runtime. Italso conservatively assumes that Intents which cannot be resolved statically, e.g.,those that depend on user input, can target all components, leading to FPs. Addi-tional failures are due to the handling of Arrays and Lists (four failures in DB3.*),Unreachable Code (three failures in DB13.*), Android-specific (two failures inDB2.*), and General Java categories (two failures DB7.*).42Table 4.3: Failures grouped by target criteria.Target Criteria FLOWDROID AMANDROID DROIDSAFEFlow Sensitivity (DB1.1) 0 0 5Android-specific(DB2.* , U-DB2.12)2 27 2Arrays and Lists (DB3.*) 5 7 4Lifecycle and Callbacks(DB4.* , DB9.* , U-DB4.*)8 35 4Object Sensitivity (DB6.7) 1 0 0General Java (DB7.* , U-DB7.*) 9 9 2ICC (DB8.* , ICC*.*) 13 7 1Reflection(DB10.* , DB11.* , U-DB10.*)13 28 17Threading (DB12.6) 1 1 0Unreachable Code (DB13.*) 3 4 3Total 55 118 38Summary: Table 4.3 summarizes the criteria where at least one of the toolsfails and lists the total number of failures per criterion for each tool. Overall,the major issue of all the tools is in reflection handling, albeit for different rea-sons. FLOWDROID also fails to accurately parse and track ICC Intents involv-ing complex string analysis and list management operations. AMANDROID hasmajor issues in handling Android framework methods, location-related flows,and lifecycle and callback methods. Once these major problems are fixed, thetools will have a much higher overall accuracy.43Chapter 5Results: Google Play Apps (RQ4)In this chapter, we answer RQ4 introduced in Chapter 1: do the results on thebenchmark applications generalize to the Google Play apps? We start by report-ing the performance of FLOWDROID and AMANDROID on the Google Play apps(Chapter 5.1). We then analyze the reasons for inaccuracy, identify strengths andweaknesses of each tool, and highlight the differences between the tool results onthe benchmark and Google Play apps (Chapter 5.2). We run the tools on 20 appswith API level 25 or lower, as discussed in Chapter 3.4.5.1 Tools Performance5.1.1 AccuracyTable 5.1 shows the results of the analysis for each of the apps. The first four rowsof the table list the apps with at least one flow detected by at least one of the tools.The remaining rows show apps with no flow detected. We mark with cases wherea tool completed the app analysis in full and with â cases where the tool failed toanalyze an app due to a runtime exception ( EX ), out of memory exception ( OM ),or timeout ( TO ).FLOWDROID finished analyzing nine out of 20 apps and resulted in runtime ex-ceptions in the remaining apps. We identified three cases in which the tool throwsexceptions: when handling ICC flows in Services (apps #2, #7, #8, and #15), when44Table 5.1: FLOWDROID and AMANDROID results on Google Play apps.(â â indicates the tool fully analyzed the app, â â indicates the tool partially analyzed the app,and â â indicates the tool failed to analyze the app. k is the depth of context sensitivity selectedfor AMANDROID.)App ID # Exp. FlowFound FlowsFLOWDROIDAMANDROID(k = 5)AMANDROID(k = 1)12 3 0 0 TO 43(3 TP, 40 FP)14 15 17 0 83(15 TP, 2 FP) (5 TP, 78 FP)16 2 0 0 5(1 TP, 4 FP)17 3 0 EX 10 10(2 TP, 8 FP) (2 TP, 8 FP)4, 18 1 0 0 06, 10, 11 1 0 0 03 1 0 0 0 TO2 1 0 EX 0 05, 7, 8 1 0 EX 0 09, 13, 15, 19, 20 1 0 EX 0 01 1 0 EX 0 0 OMhandling ICC-related methods with no parameters, e.g., PendingIntent.send()(apps #1 and #17), and when calling the underlying SOOT framework to resolve pa-rameters of the reflective Method.invoke(Object, Object[]) method (apps#5, #9, #13, #19, and #20). We reported these failures to the FLOWDROID authors,discussed possible fixes, and believe that these fixes will be available in the nextrelease of the tool.AMANDROID finished analyzing only three apps in full (apps #2, #4, #18) andtimed-out for the remaining apps. The tool works in two phases: it first analyzeseach app component individually and then combines the results of each componentanalysis to identify the flows. Due to that design, even if particular app componentstimed out, the tool is still able to use the results of remaining components to build(a partial list of) flows. That was the case for 16 applications, which we mark witha symbol in Table 5.1. In one additional app (#12), the tool timed out in thesecond phase, after executing that phase for 68.7 hours.45Table 5.2: Google Play apps: comparisons of precision, recall, and F-measures.App IDFLOWDROID AMANDROID (k = 5) AMANDROID (k = 1)P R FM P R FM P R FM12 0 0 0 0 0 0 7 100 1314 88 100 94 0 0 0 6 33 1016 0 0 0 0 0 0 20 50 2917 0 0 0 20 67 31 20 67 31Compared with the results reported AMANDROID authors when running theirtool on real apps [41], we observed substantially more component-level timeouts(80% in our case vs. 10% on 853 Google Play and malware apps analyzed inthat study), even though we allocated a longer execution time to each component.Specifically, in our case, we proportionally divided the three-day execution timebetween all app components, arriving at per-component execution times between14 and 240 minutes, depending on the number of components in an app, whilethe authors of the tool uniformly allocated 10 minutes per component. Yet, in ourexperiments, we ran the tool with a context-sensitivity depth of five (k = 5), toalign it with the equivalent configuration used by FLOWDROID, while the authorsof AMANDROID used k = 1.To ensure that we do not disadvantage the tool in our experiments, we re-ranAMANDROID with a k = 1 setup and report the results in the fifth column of Ta-ble 5.1. Under this setup, the tool finished analyzing six additional apps (#5, #7,#8, #12, #14, #17); it timed-out after running the second analysis phase for 57.7hours for one app that was partially analyzed with k = 5 (app #3), and ran out ofmemory with 256 GB allocation for one additional app (#1).Overall, FLOWDROID was able to detect flows in only one of the analyzedapps (#14) and AMANDROID with the k = 5 configuration â in only one app aswell (#17). With k = 1, AMANDROID detects the same flows in this app and alsoflows in three additional apps (#12, #14, #16). The number of FP and FN flowsper app that we obtained in our manual analysis are also listed in Table 5.1, for theapps in the first four rows, where at least one flow per app was detected by at leastone tool. Table 5.2 further summaries the precision, recall, and F-measure of the46tools on these four apps.The precision and recall of FLOWDROID on the app it was able to analyze (app#14) is very high: 88% and 100%, respectively. The tool was also able to detect thetarget login-related flow in that app. AMANDROID suffers from low precision inboth k = 5 and k = 1 configurations for all apps, and from low recall in all but oneapp (#12). Like FLOWDROID, it is able to detect login-related flows in app #14 andalso in app #12. In Chapter 5.2, we discuss the causes for the low accuracy for bothtools, focusing on k = 1 configuration for AMANDROID, as the results produced bythis configuration strictly subsume the results produced with k = 5.5.1.2 Execution TimeTo report execution time, similar to the benchmarks, we only consider apps thateach tool analyzed in full, i.e., those marked with in Table 5.1, and averagedtheir execution times from five consecutive runs. There are only two apps thatboth FLOWDROID and AMANDROID analyzed in full under the same configurationsetup, i.e., with k = 5: apps #4 and #18. For these two apps, FLOWDROID spent2 minutes on #18 and 1.3 hours on #4, on average (0.7 hours on average per app)while AMANDROID spent 4.5 and 7.2 hours per app, respectively (5.8 hours onaverage).When considering all nine apps for FLOWDROID and three apps for AMAN-DROID that these tools finished analyzing in full, FLOWDROID spent 4.3 hours onaverage per app (min: 4 minutes, max: 20.2 hours, median: 1.3 hours) and AMAN-DROID spent 10.6 hours on average per app (min: 4.5 hours, max: 20.1 hours,median: 7.2 hours). Further, for the nine apps AMANDROID was able to analyzein full with the k = 1 setup, it spent 4.0 hours on average per app (min: 6 minutes,max: 23.9 hours, median: 1.5 hours). Finally, when only considering the threeapps that AMANDROID fully analyzed with both k = 1 and k = 5, we observe thatk = 1 ran substantially faster than k = 5: 1.1 hours vs. 10.6 hours on average perapp. We do not measure the execution time of the tools on other apps, as the toolseither threw exceptions or raised timeout.To summarize, our results are consistent with previous reports on Google Playapps: FLOWDROID is faster than AMANDROID [21, 28]. Also, as expected, AMAN-47Table 5.3: FN/FP breakdown by target criteria, for Google Play apps.FNFPLogin-related Other# Breakdown App ID # Breakdown AppID# Breakdown App IDFLOWDROID8 1x U-DB2.12 12 3 2x U-DB2.12 12 2 2x U-DB2.12 143x U-DB4.17 6, 10, 11 1x UN 161x U-DB4.18 31x DB8.16 42x U-DB10.8 16, 18AMANDROID16 5x U-DB4.17 5, 6, 9, 10, 11 10 10x UN 14 130 88x U-DB7.25 12, 14, 16, 176x U-DB4.18 2, 7, 8, 13, 17, 19 42x U-DB7.26 12, 14, 171x DB8.16 42x U-DB10.8 16, 182x TO 15, 20DROID with k = 1 is faster than with k = 5, as it trades accuracy for execution time.Allocating more time to the tool is unlikely to improve the detection accuracy, asmost runs terminate within a day and run out of time/memory otherwise. Similar toour experiment on benchmark apps, we did not perform a numerical comparison ofexecution times with other work due to different hardware setups and configurationselections.5.2 Causes of InaccuracyTable 5.3 shows the detailed distribution of the FN and FP results for FLOWDROIDand AMANDROID, separately, for the apps these tools analyzed in full ( ) or par-tially ( ), i.e., nine apps for FLOWDROID and 18 apps for AMANDROID with k = 1.For the FN results, we report on login-related flows (one per app) separately fromother expected flows. Similar to the benchmarks, we assign a failing criterion foreach FN and FP result: for each failure, we first checked whether the failure occursdue to a reason already captured by a certain target criterion from the benchmarkapps. If so, we marked the failure with that target criterion. Otherwise, we created anew target criterion (U-DB4.17 , U-DB4.18 , U-DB7.25 , U-DB7.26 , U-DB10.8).We further provide the ids of the apps for which each failure occurs.For example, FLOWDROID has eight login-related FN results, as it is able todetect the expected login flow in one out of nine apps it finished analyzing (app481 public class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 protected void onCreate ( Bundle savedInstanceState ) {4 pwd = getText ( ) . t o S t r i n g ( ) ; / / source5 HttpPost req = new HttpPost ( ) ;6 req . s e t E n t i t y (new S t r i n g E n t i t y (pwd) ) ;7 H t t p C l i e n t h t t p C l i e n t = . . . ;8 h t t p C l i e n t . execute ( req ) ; / / s ink , leak9 }10 }Figure 5.1: U-DB2.12 â Android framework method modeling (FLOWDROIDdoes not propagate taint from a parameter to the receiver object).#14). It has three additional FNs (in apps #12,#16) and two FPs (in app #14). Wenow discuss the main reasons for the observed FN and FP results.FLOWDROID missed one login-related flow (in app #12) due to the incorrect mod-eling of Android framework methods. We mark this failure with U-DB2.12 , as inthe benchmark apps. It also leads to two additional FN flows in the same app. List-ing 5.1 shows a snippet of code producing such FN result: in this case, an Androidframework method HttpPost.setEntity(HttpEntity) is used to set the con-tent of an HTTP request (line 6). It is parameterized with the StringEntity ob-ject, which is tainted because the pwd field is tainted (line 4). As the frameworkmethod is not supported by STUBDROID and is not explicitly classified in any ofthe conservative framework method modeling strategies described in Chapter 3.1,FLOWDROID applies the default strategy: when a receiver object, at least one ofthe method parameters, or the fields of the receiver object/method parameters aretainted, the taint is propagated to the methodâs return value. As the default strat-egy does not propagate the taint to the receiver object req, the tool misses theflow to the HttpClient.execute(...) sink (line 8). To fix the failure, theHttpPost.setEntity(HttpEntity) method should have been assigned thegeneration type, which would propagate the taint to req.Another case of framework method misclassification, this time leading to twoFP results (both in app #14) is exemplified in Figure 5.2: Context.getString(int)is an Android framework method that allows developers to retrieve a string re-source, e.g., the app name (line 7). The pwd field of the methodâs receiver ob-ject, this, is tainted (line 6). However, this method is, again, not supported by491 public class M a i n A c t i v i t y extends A c t i v i t y {2 S t r i n g pwd = â â ;3 @Override4 protected void onCreate ( Bundle savedInstanceState ) {5 . . .6 th is . pwd = getText ( ) . t o S t r i n g ( ) ; / / source7 S t r i n g name = th is . g e t S t r i n g (R. s t r i n g . app name ) ;89 P r i n t W r i t e r w r i t e r = . . . ;10 w r i t e r . w r i t e (name) ; / / s ink , no leak11 }12 }Figure 5.2: U-DB2.12 â Android framework method modeling (FLOWDROIDmistakenly propagates taint from a parameterâs field to the return value).STUBDROID as it contains native code and is not explicitly classified in any of theconservative framework method modeling strategies. Applying the default strat-egy in this case propagates the taint to the methodâs return value, which is furtherassigned to the variable name (line 7). The tainted name variable flows to the sinkPrintWriter.write(...) (line 10). Yet, this is an FP result as the return stringof the Context.getString(int) is not, in fact, affected by the tainted value.To fix the failure, this method should have been assigned to the exclude type, whichwould not propagate the taint.Interestingly, while FLOWDROID has no failure of this kind on the benchmarkapps, Google Play apps contain Android framework methods not covered by thebenchmarks, which expose additional weaknesses and the need to tune FLOW-DROID to better represent the entire range of Android framework methods. Incontrast, AMANDROID has FPs due to inaccurate modeling of Android frameworkmethods in the benchmark apps. Yet, its strategy of not propagating the taint fromthe fields of method parameters allows it to avoid the FP flow in Figure 5.2. AMAN-DROID avoids the FN flow in Figure 5.1 as its conservative strategy propagates thetaint between the method parameters.That is, for framework methods that are not accurately modeled by the tools,FLOWDROID and AMANDROID apply different types of conservative strategies,both with their own strengths and weaknesses. Even though propagating the taintfrom a method parameter to the method receiver (like in Figure 5.1) can be solvedby FLOWDROIDâs generation conservative modeling strategy, none of the toolâs501 public class M a i n A c t i v i t y extends A c t i v i t y {2 i n t i d = R. layou t . a c t i v i t y m a i n ;3 @Override4 protected void onCreate ( Bundle savedInstanceState ) {5 . . .6 setContentView ( i d ) ;7 }8 / / Cal lback def ined i n a layou t f i l e9 public void bu t t onC l i c k ( View view ){10 S t r i n g pwd = getText ( ) . t o S t r i n g ( ) ; / / source11 P r i n t W r i t e r w r i t e r = . . . ;12 w r i t e r . w r i t e (pwd) ; / / s ink , leak13 }14 }Figure 5.3: U-DB4.17 â Missing callbacks.strategies supports propagating taint between explicit method parameters. AMAN-DROID supports this behavior, which is required, for example, for the android.database.DatabaseUtils::appendValueTo- Sql(StringBuilder sql,Object value) framework method. At the same time, FLOWDROID is able topropagate taint from fields of a method parameter, which is required for meth-ods such as java.lang.StringBuffer::append(StringBuffer) becausethe value of the input buffer is stored in the field called value and can be tainted.AMANDROID cannot support this behavior without precisely modeling the API.The lack of accurate modeling of Android framework method is a major weaknessaffecting the effectiveness of the tools.Additional four FNs, all leading to misclassifications of login-related flows infour different apps, relate to callback method analysis. Figure 5.3 shows a simpli-fied version of the code leading to the failure marked as U-DB4.17 , which we ob-served in apps #6, #10, and #11. In this code snippet, R.layout.activity mainis a unique identifier of a file defining the UI elements and their corresponding call-back methods for an activity. The identifier is assigned to a variable id (line 2); thatvariable is further used in onCreate(...) method to set the layout for the ac-tivity (line 6). The layout file defines a callback method buttonClick(), whichis called when a button is clicked. The implementation of this method contains aflow from the source pwd (line 10) to the sink PrintWriter.write(...) (line12).511 public class MyFragment extends Fragment implements View . OnCl ickL is tener {2 @Override3 public View onCreateView ( L a y o u t I n f l a t e r i n f l a t e r , ViewGroup conta iner , BundlesavedinstanceState ) {4 . . .5 Button but ton = . . . ;6 but ton . se tOnCl i ckL is tener ( th is ) ;7 }8 @Override9 public void onCl ick ( View v ) {10 S t r i n g pwd = getText ( ) . t o S t r i n g ( ) ; / / source11 P r i n t W r i t e r w r i t e r = . . . ;12 w r i t e r . w r i t e (pwd) ; / / s ink , leak13 }14 }Figure 5.4: U-DB4.18 â Callback methods registered in fragments.To detect the expected flow, FLOWDROID needs to detect the callback methodfirst. It does that by identifying and parsing all layout resource files passed as aparameter to setContentView(int). However, we observed that FLOWDROIDcan only handle layout files when the value of the parameter is a primitive integerrather than a variable. Unlike in this app, all benchmarks and several Google Playapps use R.layout.activity main, which is a final static variable that is re-placed by its integer value at compile time, directly as an input to the setContent-View(int) method. FLOWDROID does not perform any analysis on the app to re-trieve the value of the id variable and thus cannot detect the layout file. As a result,it misses the callback and its related flows.Moreover, while FLOWDROID models the lifecycle methods of fragments âreusable UI parts defined within an Activity â it does not collect callbacks dynam-ically registered in fragmentsâ lifecycle methods. For example, the onClick()callback method in Figure 5.4 (line 9) is dynamically registered in the fragmentâsonCreateView(...) lifecycle method (line 6). FLOWDROID is unable to col-lect this callback method and considers it as unreachable. As a result, it cannotidentify the flow inside this callback method: from the source getText() (line10) to the sink PrintWriter.write(...) (line 12). We marked this issue asU-DB4.18 in Table 5.3, which leads to one FN (in app #3). This is a serious lim-itation while approximately 90% of the current Google Play apps contain at leastone fragment [18], DroidBench only contains two benchmark apps (FragmentLife-521 public class MyClass {2 public MyClass ( ) {3 S t r i n g pwd = getText ( ) . t o S t r i n g ( ) ; / / source4 P r i n t W r i t e r w r i t e r = . . . ;5 w r i t e r . w r i t e (pwd) ; / / s ink , leak6 }7 }8 public class M a i n A c t i v i t y extends A c t i v i t y {9 @Override10 protected void onCreate ( Bundle savedInstanceState ) {11 . . .12 Class<?> c l s = Class . forName ( â MyClass â ) ;13 Const ruc tor c s t r = c l s . ge tCons t ruc to r ( ) ;14 MyClass ob j = ( MyClass ) c s t r . newInstance ( th is ) ;15 }16 }Figure 5.5: U-DB10.8 â Reflective invocation of class constructors.cycle1 and FragmentLifecycle2) focusing on fragment lifecycle method modeling.Another major source of failures leading to two FNs and undetected login-related flows (in apps #16 and #18) is due to using reflective calls to invoke classconstructors. Figure 5.5 shows an example where class MyClass (lines 1-7) isinstantiated via reflection (lines 12-14). Because FLOWDROID cannot handle thereflective method Class.getConstructor(), it cannot reach the constructormethod and misses the flow inside that method. We mark this issue as U-DB10.8 .We identified this issue while analyzing FN results in applications using aframework called ButterKnife1, which simplifies the code for registering callbacksof UI elements. ButterKnife automatically generates a helper class for each Activ-ity and relies on reflective calls similar to the one in Figure 5.5 to invoke the con-structor of the helper class and to perform callback registrations. As ButterKnife isused by many Google Play applications, lack of support for reflective constructorshinders the applicability of FLOWDROID in practice.Finally, one additional FN for FLOWDROID (in app #4) is due to a reasonwe already observed in benchmarks: lack of support for SharedPreferences(DB8.16). In one case (app #16), we could not recover the reason for the failure.We mark this case as UN in Table 5.3.1https://github.com/JakeWharton/butterknife531 public class M a i n A c t i v i t y extends A c t i v i t y {2 P r i n t W r i t e r w r i t e r = . . . ;3 @Override4 protected void onCreate ( Bundle savedInstanceState ) {5 . . .6 S t r i n g pwd = getText ( ) . t o S t r i n g ( ) ; / / source7 S t r i n g a = m1(pwd) ;8 S t r i n g b = m1( â safe â ) ;9 w r i t e r . w r i t e ( b ) ; / / s ink , no leak10 }11 public S t r i n g m1( S t r i n g s ) {12 S t r i n g c = m2( s ) ;13 return c ;14 }15 public S t r i n g m2( S t r i n g s ) {16 return s ;17 }18 }Figure 5.6: U-DB7.25 â Context sensitivity.AMANDROID shared the callback method collection issues (U-DB4.17 and U-DB4.18;apps #2, #5-#11, #13, #17, and #19) and reflective constructor call issues (U-DB10.8 ;apps #16 and #18) with FLOWDROID, though the number of failures are differentas the tools finished analyzing different sets of apps. In addition, as FLOWDROID,AMANDROID has one FN related to handling of SharedPreferences (DB8.16;app #4). All these prevent the tool from identifying the login-related flows.The majority of FP results reported by AMANDROID relate to inaccurate han-dling of context-sensitivity when using k = 1. This leads to 88 FPs marked asU-DB7.25 . Due to the lack of existing benchmarks for testing context-sensitivity,we created a test case to showcase this issue: in Figure 5.6, the method m1()is called with the tainted variable pwd (lines 6â7) and with an insensitive string"safe" (line 8). The m1() further passes its input parameter to the method m2(),assigns the return value to the variable c, and further returns c (lines 12-13). Themethod m2() simply returns its input parameter (line 16). A context-insensitiveanalysis will analyze both methods m1() and m2() only once and will concludethat their return values can be tainted, as they are indeed tainted when the methodsare called with the pwd parameter (line 7). That will lead to an undesired FP flowof the variable b to the writer.write(b) sink (line 9). With k = 1, the toolcannot further distinguish between the calling context of m2(), as it has only one541 public class M a i n A c t i v i t y extends A c t i v i t y {2 @Override3 protected void onCreate ( Bundle savedInstanceState ) {4 . . .5 leak ( ) ;6 leak ( ) ;7 }8 private void leak ( ) {9 S t r i n g pwd = getText ( ) . t o S t r i n g ( ) ; / / source10 P r i n t W r i t e r w r i t e r = . . . ;11 w r i t e r . w r i t e (pwd) ; / / s ink , leak12 }13 }Figure 5.7: U-DB7.26 â Flows triggered from multiple call sites.calling site â m1(). Only when k â¥ 2, the tool can correctly separate the m1()->m2() calls issued from the call site in line 7 from that in line 8, eliminating the FPresult. As expected, context-sensitivity also makes the analysis more expensive.Additional FPs of AMANDROID, which we marked as U-DB7.26 in Table 5.3,stem from the tool reporting certain flows multiple times. Looking into the reasonsfor such a behavior, we conjecture that duplications occur when a flow is triggeredfrom multiple call sites. For example, method leak() in Figure 5.7 is calledtwice: in line 5 and then in line 6. The method itself contains a flow from thesource in line 9 (pwd) to the sink in line 11 (PrintWriter.write(...)). Inthis case, AMANDROID reports two identical flows; we believe, one for each callsite. Yet, the tool does not provide any additional information allowing the user todistinguish between the flows. We thus counted all but one such duplicated flow asFPs, resulting in 42 FP flows.Finally, in two apps (#15 and #20), even though the tool partially analyzed theapps, it timed out on components that contained the expected flows, resulting intwo FNs denoted by TO in Table 5.3. In 10 cases, all from the same app (#14), wewere unable to recover the reasons for the non-login-related FN flows.Summary: Our experiments identified several major issues not covered by ex-isting benchmarks, such as inaccurate handling of Android framework methods,callbacks, fragments, and reflective calls. In addition, the applicability of exist-ing tools for real-world applications is hindered by a large number of runtimeexceptions, timeouts, and out-of-memory errors.55Chapter 6Related WorkOur discussion of related work focuses on the (a) internal evaluation of the toolswe selected for our study, (b) external evaluation of the tools, and (c) surveys onAndroid static taint analysis tools.6.1 Internal Tool EvaluationWe start by discussing the evaluation of the tools performed by their authors. Arztet al. [4] compared the precision and recall of FLOWDROID with two commer-cial tools, IBM APPSCAN SOURCE1 and FORTIFY SCA2 on DroidBench version1.0, which contained 39 benchmarks at that time. They provided a detailed kit ex-plaining how to reproduce their experiments3. The authors also ran FLOWDROID(without ICCTA) on 500 most popular Google Play applications and 1,000 knownmalware apps from the VirusShare dataset4. They concluded that FLOWDROIDwas able to detect leakages of sensitive information into log files. However, theauthors did not report on the number of apps the tool successfully analyzed and thenumber of flows the tool reported per app. They also did not classify all flows intoFP and FN results.1https://www.ibm.com/us-en/marketplace/ibm-appscan-source2https://software.microfocus.com/en-us/solutions/enterprise-security3https://github.com/secure-software-engineering/soot-infoflow-android/wiki/PLDIâ14-Artifact-Evaluation4https://virusshare.com56Li et al. [21] conducted a comparison between FLOWDROID without ICCTA,FLOWDROID with ICCTA, DIDFAIL [17], and AMANDROID on 22 ICC-relatedbenchmarks the authors developed and nine benchmarks from ICC-Bench . Theauthors also ran FLOWDROID (with ICCTA) on 15,000 randomly selected GooglePlay applications and showed that the tool reported 2,395 ICC-related leakages in337 apps. Like in the previous study, the authors did not further check the validityof the reported flows. When comparing the execution time of FLOWDROID (withICCTA) and AMANDROID on 50 randomly selected Google Play applications, theauthors showed that FLOWDROID was faster than AMANDROID, which concurswith our findings.Wei et al. [41] compared AMANDROID with EPICC [25] and FLOWDROID,also focusing on ICC handling ability. The subject programs consisted of 39 ap-plications from DroidBench , 16 applications from ICC-Bench , and another fourproprietary test cases. Later, the same authors upgraded the tool and conducted anew comparison [43], focusing on 18 DroidBench and 24 ICC-Bench applicationsrelated to ICC and IAC. In these two reports, the authors also evaluated AMAN-DROID on 753 and 2,300 Google Play apps, respectively, showing that the tool isable to detect data leakages in practice. Again, they did not provide an in-depthanalysis of the validity of the flows reported by the tool.Finally, Gordon et al. [13] compared DROIDSAFE with FLOWDROID using 94applications from DroidBench , 40 additional benchmarks developed by the teamand later contributed to DroidBench , and 24 applications from a proprietary bench-mark. The authors did not run DROIDSAFE on any Google Play apps, stating thetool is not designed for this purpose.A detailed comparison with all these works is presented in Chapter 4.2. In anutshell, executing the tools on different subsets of benchmark applications andwith different versions of the tools makes the results incomparable. Moreover,the detailed tool configuration setup used in these experiments is rarely described.Experiments with Google Play apps focus on evaluating the scalability rather thanthe accuracy of the tools. In our work, we make a particular effort to align the toolconfigurations and also focus on a larger common set of benchmarks. As our teamwas not involved in developing these tools, we provide an independent assessmentof their accuracy. Furthermore, we perform an in-depth analysis to investigate the57reasons leading to FP and FN results on both the benchmark and Google Play appsfor each tool.6.2 External Tool EvaluationThe closest to ours is the work by Pauck et al. [28], which proposed a frameworkfor automatically evaluating reports produced by Android taint analysis tools andused the framework for comparing the accuracy and execution time of six statictaint analyzers (including those we selected). The authors evaluated the tools onDroidBench version 3.0, ICC-Bench version 2.0, and the DIALDroid-Bench [10]benchmark consisting of 30 apps collected from Google Play in June 2015. Forthe DIALDroid-Bench apps, the authors manually analyzed all flows reported byat least two tools and determined whether these flows are TP or FP results; theyidentified 22 TP and 4 FP flows, which formed the ground truth of their experiment.Unlike us, the authors ran each tool with its default configuration. They also didnot analyze the reasons for failures in details, untangle the benchmarks, or proposeadditional new benchmarks. Moreover, in our work, we manually identified theexpected analysis result, which allowed us to include many expected flows thatare missed by all tools. Our analysis thus identified new failures missed by earlierwork, such as those related to modeling Android framework methods, handlingreflective constructor calls, callbacks in fragments, and more.Bonett et al. [9] proposed a framework that leverages mutation analysis to dis-cover undocumented flaws of Android static taint analysis tools. The authors usedthe framework to generate and inject 2,026 mutants (i.e., data leakages) to sevenopen-source Android apps, showing that DROIDSAFE can detect the highest num-ber of leakages, followed by FLOWDROID and AMANDROID. The authors furthershowed that all tools miss flows due to inaccurate handling of callback methodsand FLOWDROID and AMANDROID further have issues related to multi-threading.Our work differs as we focused on real rather than injected leakages, which al-lowed us to identify issues such as unsupported reflective constructor calls used bythe ButterKnife framework.Rodriguez and Kouwe [33] ran FLOWDROID, AMANDROID, and DROIDSAFEon 48 benign and 48 malicious apps randomly selected from the AndroZoo dataset [1]58â a repository that crawls apps from Google Play and alternative markets. The au-thors observed a large number of crashes and timeouts when running the tools,looked into the reasons for such crashes, and concluded that AMANDROID andDROIDSAFE have de-compilation issues causing these tools to miss Android APIs.Yet, some of these failures are expected, as the tools do not support the newest An-droid API levels. In our work, we only ran the tools on apps with supported APIlevels. Moreover, while Rodriguez and Kouwe assumed that the tools should reportat least one flow in malicious apps and no flows in benign apps, we disagree withthis assumption, investigating every single flow reported by each tool.Luo et al. [24] aimed at providing insights into what can be done to increasethe precision of taint analysis. They proposed a tool, named COVA, to computepartial path constraints that capture the circumstances under which taint flows mayactually occur in practice. The authors further used the tool to conduct a qualitativestudy on taint flows reported by FLOWDROID on Google Play apps. They showedthat many flows occur only under specific conditions, e.g., environment settings,user interaction, I/O. Unlike our work, the authors did not conduct a comparativeanalysis evaluating multiple tools, focusing on the recommendations of includingsuch path constraints in future taint analysis approaches.6.3 SurveysSeveral papers survey Android-specific static analysis techniques for identifyingpermission and privacy leakage detection [36], robustness against obfuscation [14],and intra- and inter-application communication vulnerabilities [7]. Sufatrio et al.[38] provide a taxonomy of security vulnerabilities in Android. Li et al. [22] andSadeghi et al. [35] perform large-scale systematic literature reviews on static analy-sis tools for Android. They also propose a taxonomy, classifying Android securityassessment mechanisms and research approaches.Apart from the conceptual surveys discussed above, Reaves et al. [32] also aug-ment their survey with an empirical experiment comparing the surveyed static anal-ysis tools. This work evaluated usability, performance, and precision of the toolsfor DroidBench applications, mobile bank applications, and the top 10 most popu-lar financial applications in Google Play. The primary emphasis of this experiment59is on investigating the usability and accessibility of the tools, the understandabil-ity of the results, and the usefulness of the documentation provided by the tools.They reported on the number of applications that each tool was able to process,without a detailed analysis of the accuracy of each tool. Moreover, experimentswith different tools were conducted using the default configurations of those tools,which hinders the validity of the results. In contrast, our study starts by aligningconfigurations of the tools and reports not only the overall accuracy for each toolbut also the reasons for all failures, summarizing the main weaknesses of each tool.To the best of our knowledge, our study is the first large-scale, in-depth, compara-tive analysis of these tools under the same setup. Moreover, our configuration andresults are publicly available, to allow reproducibility.60Chapter 7Summary and ConclusionIn this chapter, we summarize the study with discussion and lessons learned (Chap-ter 7.1). We then provide promising directions for future study (Chapter 7.2) andfinalize this thesis with conclusions (Chapter 7.3).7.1 Discussion and Lessons LearnedThe absence of accurate information on tool configuration as well as sources, sinks,and benchmarks apps used for the analysis, hinders the replicability of earlier stud-ies. When each of the analyzed tools was introduced, it reportedly outperformedthe others. That is because the tools were evaluated under different configurationsetups, with different sets of sources and sinks, and only on a selected subset ofbenchmarks. To further add to the confusion, as our experience analyzing ICCcommunication shows, the tools also have different interpretations of Android-specific flow semantics. Comparing the tools to each other under the same setuphelps identify their strengths and weaknesses, e.g., in Android framework methodmodeling approaches implemented by FLOWDROID vs. AMANDROID.Our experience evaluating the tools on Google Play apps shows that the highaccuracy on the benchmarks does not necessarily translate to a high accuracy onGoogle Play apps. While the tools are tuned to perform well on the benchmarkapps, they miss cases not covered by the benchmarks, e.g., certain types of reflec-tive calls, callbacks, and Android framework methods. Our study helped identify61and fix several such cases. Additional evaluations of the tools on real applications,with a detailed analysis of each failure, as we did in Chapter 5.2, is a difficult yetproductive direction for future work, which will help improve toolsâ applicabilityin practice.Furthermore, based on our study, we observed that the current way of measur-ing tools accuracy is sub-optimal: as multiple FP and FN flows might be causedby the same underlying reason, just counting FP and FN flows can produce falseimpressions with respect to the real tool accuracy. In fact, such counting mightproduce accuracy metrics that look artificially low, even though the tool outper-forms others in many aspects. To mitigate this problem, we suggest to (a) simplifythe benchmarks and ensure they focus, as much as possible, on one particular as-pect of the analysis and (b) investigate reasons behind each failure, as we did inChapters 4.3 and 5.2.In our work, we separated tangled benchmarks and extracted new benchmarksrepresenting issues found in the analyzed Google Play apps. We included newlyidentified benchmarks in UBCBench and augmented this suite with tests of dif-ferent types of sensitivities. We clearly articulated potential reasons for failures ineach of the benchmarks and made our setup and experimental data publicly avail-able for others to build on.We hope our work will benefit tool builders, as it highlights some of the mainfactors impeding successful adoption of the tools in practice: scalability, handlingof callbacks and reflections, and accurate support of Android framework methods.We also hope our work will help tool users to better interpret the results of analysisand tune the tools for their needs.7.2 Future WorkThere are several promising directions of research arising from this study. Firstly,our work reveals that existing benchmark suites do not fully cover situations ap-peared in Google Play apps, e.g., the usage of certain Android framework methods.As tool builders still rely on benchmark suites as an indicator of the performanceof their proposed tool, improving existing benchmark suites is important. Whileour work provides an approach to extract new benchmarks from Google Play apps,62it is a compelling research topic to find a systematic way in generating more com-prehensive benchmark suites. For example, performing a large scale quantitativeanalysis to estimate the prevalence of different Android framework methods in realapps, thus deciding what kinds of methods should be tested by benchmark suites,could be an encouraging subject of future work.Our study also shows that evaluating tools on Google Play apps is beneficial inhelping tool buildersâ to improve their tools. Yet, building ground truth for GooglePlay apps is difficult. Though our work mitigates this problem by manually identi-fying potential flows for a specific case study, an automatic approach is demandedto cover more situations in Google Play apps. Using dynamic analysis to identifypotential flows is an idea worth trying.Furthermore, the research presented in this thesis raises issues that hinder thetools to perform taint analysis on Google Play apps, e.g., in handling Androidframework methods, callbacks, and reflection. Proposing new techniques focusingon resolving these issues, and also validating if these techniques are applicablein practice, can potentially shine further light for the design of future static taintanalyzers.7.3 ConclusionThis thesis reports on the results of a large-scale, controlled, and independent ex-periment we conducted to evaluate the most prominent static taint analysis tools forAndroid applications: FLOWDROID, AMANDROID, and DROIDSAFE. We alignedthe tools along the same configuration setup, used them to analyze 182 benchmarkapplications, compared our results to those reported in earlier work, and discussedthe identified differences. We also extracted ground truth flows from 20 GooglePlay applications and used FLOWDROID and AMANDROID to analyze these appli-cations as well. We inspected all reported false-positive and false-negative flows toidentify the main reasons for inaccuracy in each tool. Furthermore, we identifiedseveral deficiencies in existing benchmarks, such as missing checks and dependentchecks that are encoded in a single benchmark app.Our work emphasizes the importance of providing detailed information aboutan experimental setup, which is required to ensure the correctness and reproducibil-63ity of reported comparisons. We make our entire configuration setup, including thetool configuration parameters, the set of sources and sinks, the precise version ofeach benchmark application used, the list of selected Google Play applications, andour expected results available to other researchers.64Bibliography[1] K. Allix, T. F. Bissyande´, J. Klein, and Y. L. Traon. AndroZoo: CollectingMillions of Android Apps for the Research Community. In Proc. ofMSRâ16, 2016.[2] S. Arzt. Static data flow analysis for Android applications. PhD thesis,Darmstadt University of Technology, Germany, 2017.[3] S. Arzt and E. Bodden. StubDroid: Automatic inference of precise date-flowsummaries for the Android framework. In Proc. of ICSEâ16, pages 725â735,2016.[4] S. Arzt, S. Rasthofer, C. Fritz, E. Bodden, A. Bartel, J. Klein, Y. Le Traon,D. Octeau, and P. McDaniel. FlowDroid: Precise context, flow, field,object-sensitive and lifecycle-aware taint analysis for Android apps. In Proc.of PLDIâ14, pages 259â269, 2014.[5] P. Barros, R. Just, S. Millstein, P. Vines, W. Dietl, M. dâAmorim, and M. D.Ernst. Static analysis of implicit control flow: Resolving Java reflection andAndroid intents (T). In Proc. of ASEâ15, pages 669â679, 2015.[6] A. Bartel, J. Klein, Y. Le Traon, and M. Monperrus. Dexpler: ConvertingAndroid Dalvik bytecode to Jimple for static analysis with Soot. InProceedings of the ACM SIGPLAN International Workshop on State of theArt in Java Program Analysis, pages 27â38, 2012.[7] S. Bhandari, W. B. Jaballah, V. Jain, V. Laxmi, A. Zemmari, M. S. Gaur,M. Mosbah, and M. Conti. Android inter-app communication threats anddetection techniques. Computers & Security, 70:392â421, 2017.[8] A. Bianchi, J. Corbetta, L. Invernizzi, Y. Fratantonio, C. Kruegel, andG. Vigna. What the app is that? deception and countermeasures in theAndroid user interface. In Proc. of S&Pâ15, pages 931â948, 2015.65[9] R. Bonett, K. Kafle, K. Moran, A. Nadkarni, and D. Poshyvanyk.Discovering flaws in security-focused static analysis tools for Android usingsystematic mutation. In Proc. of USENIX Securityâ18, pages 1263â1280,2018.[10] A. Bosu, F. Liu, D. D. Yao, and G. Wang. Collusive data leak and more:Large-scale threat analysis of inter-app communications. In Proc. of ASIACCSâ17, pages 71â85, 2017.[11] S. Fahl, M. Harbach, T. Muders, Baumga¨rtner, B. Freisleben, and M. Smith.Why Eve and Mallory love Android: An analysis of Android SSL(in)security. In Proc. of CCSâ12, pages 50â61, 2012.[12] Y. Fratantonio, A. Machiry, A. Bianchi, C. Kruegel, and G. Vigna. CLAPP:Characterizing loops in Android applications. In Proc. of ESEC/FSEâ15,pages 687â697, 2015.[13] M. I. Gordon, D. Kim, J. H. Perkins, L. Gilham, N. Nguyen, and M. C.Rinard. Information flow analysis of Android applications in DroidSafe. InProc. of NDSSâ15, 2015.[14] J. Hoffmann, T. Rytilahti, D. Maiorca, M. Winandy, G. Giacinto, andT. Holz. Evaluating analysis tools for Android apps: Status quo androbustness against obfuscation. In Proc. of CODASPYâ16, pages 139â141,2016.[15] J. Jorgensen. Improving the precision and correctness of exception analysisin Soot. Technical Report 2003-3, McGill University, Canada, 2003.[16] G. Kastrinis and Y. Smaragdakis. Efficient and effective handling ofexceptions in Java points-to analysis. In Proc. of CCâ13, pages 41â60, 2013.[17] W. Klieber, L. Flynn, A. Bhosale, L. Jia, and L. Bauer. Android taint flowanalysis for app sets. In Proc. of SOAPâ14, pages 1â6, 2014.[18] D. Lai and J. Rubin. Goal-driven exploration for Android applications. InProc. of ASEâ19, 2019.[19] O. Lhota´k and L. Hendren. Scaling Java points-to analysis using SPARK. InProc. of CCâ03, pages 153â169, 2003.[20] L. Li, A. Bartel, T. F. Bissyande´, J. Klein, and Y. Le Traon. ApkCombiner:Combining multiple Android apps to support inter-app analysis. In Proc. ofSECâ15, pages 513â527, 2015.66[21] L. Li, A. Bartel, T. F. Bissyande´, J. Klein, Y. Le Traon, S. Arzt, S. Rasthofer,E. Bodden, D. Octeau, and P. McDaniel. IccTA: Detecting inter-componentprivacy leaks in Android apps. In Proc. of ICSEâ15, 2015.[22] L. Li, T. F. Bissyande´, M. Papadakis, S. Rasthofer, A. Bartel, D. Octeau,J. Klein, and Y. Le Traon. Static analysis of Android apps: A systematicliterature review. Information & Software Technology, 88:67â95, 2017.[23] L. Li, J. Gao, M. Hurier, P. Kong, T. F. Bissyande´, A. Bartel, J. Klein, andY. Le Traon. AndroZoo++: Collecting millions of Android apps and theirmetadata for the research community. The Computing Research Repository,abs/1709.05281, 2017.[24] L. Luo, E. Bodden, and J. Spâ´ath. A qualitative analysis of Androidtaint-analysis results. In Proc. of ASEâ19, 2019.[25] D. Octeau, P. McDaniel, S. Jha, A. Bartel, E. Bodden, J. Klein, andY. Le Traon. Effective inter-component communication mapping inAndroid: An essential step towards holistic security analysis. In Proc. ofUSENIX Securityâ13, pages 543â558, 2013.[26] D. Octeau, D. Luchaup, M. Dering, S. Jha, and P. McDaniel. Compositeconstant propagation: Application to Android inter-componentcommunication analysis. In Proc. of ICSEâ15, pages 77â88, 2015.[27] D. Octeau, S. Jha, M. Dering, P. McDaniel, A. Bartel, L. Li, J. Klein, andY. Le Traon. Combining static analysis with probabilistic models to enablemarket-scale Android inter-component analysis. In Proc. POPLâ16, pages469â484, 2016.[28] F. Pauck, E. Bodden, and H. Wehrheim. Do Android taint analysis toolskeep their promises? In Proc. of ESEC/FSEâ18, pages 331â341, 2018.[29] D. J. Pearce, P. H. Kelly, and C. Hankin. Efficient field-sensitive pointeranalysis of C. ACM Transactions on Programming Languages and Systems,30(1), 2007.[30] L. Qiu, Y. Wang, and J. Rubin. Analyzing the analyzers: FlowDroid/IccTA,AmanDroid, and DroidSafe. In Proc. of ISSTAâ18, pages 176â186, 2018.[31] S. Rasthofer, S. Arzt, and E. Bodden. A machine-learning approach forclassifying and categorizing Android sources and sinks. In Proc. ofNDSSâ14, 2014.67[32] B. Reaves, J. Bowers, S. A. Gorski III, O. Anise, R. Bobhate, R. Cho,H. Das, S. Hussain, H. Karachiwala, N. Scaife, B. Wright, K. Butler,W. Enck, and P. Traynor. *droid: Assessment and evaluation of Androidapplication analysis tools. ACM Computing Surveys, 49(3):55, 2016.[33] S. A. Rodriguez and E. v. d. Kouwe. Meizodon: Security benchmarkingframework for static Android malware detectors. In Proc. of CECCâ19,2019.[34] A. Russo, A. Sabelfeld, and K. Li. Implicit flows in malicious andnonmalicious code. Logics and Languages for Reliability and Security, 25:301â322, 2010.[35] A. Sadeghi, H. Bagheri, J. Garcia, and S. Malek. A taxonomy andqualitative comparison of program analysis techniques for securityassessment of Android software. IEEE Transactions on SoftwareEngineering, 43(6):492â530, 2017.[36] S. Schmeelk, J. Yang, and A. V. Aho. Android malware static analysistechniques. In Proc. of CISRâ15, pages 5:1â5:8, 2015.[37] Y. Smaragdakis and G. Balatsouras. Pointer analysis. Foundations andTrends in Programming Languages, 2(1):1â69, 2015.[38] Sufatrio, D. J. J. Tan, T.-W. Chua, and V. L. L. Thing. Securing Android: Asurvey, taxonomy, and challenges. ACM Computing Surveys, 47(4):58:1â58:45, 2015.[39] R. Vallee-Rai and L. Hendren. Jimple: Simplifying Java bytecode foranalyses and transformations. Technical Report 1998-4, McGill University,Canada, 1998.[40] R. Valle´e-Rai, P. Co, E. Gagnon, L. Hendren, P. Lam, and V. Sundaresan.Soot - a Java bytecode optimization framework. In Proc. of CASCONâ99,1999.[41] F. Wei, S. Roy, X. Ou, and Robby. Amandroid: A precise and generalinter-component data flow analysis framework for security vetting ofAndroid apps. In Proc. of CCSâ14, pages 1329â1341, 2014.[42] F. Wei, X. Lin, X. Ou, T. Chen, and X. Zhang. JN-SAF: Precise and efficientNDK/JNI-aware inter-language static analysis framework for securityvetting of Android applications with native code. In Proc. of CCSâ18, pages1137â1150, 2018.68[43] F. Wei, S. Roy, X. Ou, and Robby. Amandroid: A precise and generalinter-component data flow analysis framework for security vetting ofAndroid apps. ACM Transactions on Privacy and Security, 2018.[44] J. Whaley, D. Avots, M. Carbin, and M. S. Lam. Using datalog with binarydecision diagrams for program analysis. In Proc. of APLASâ05, pages97â118, 2005.[45] J. Zhang, L. Qiu, Y. Wang, and J. Rubin. Supplementary materials.https://resess.github.io/PaperAppendices/StaticTaint/index.html, 2020.[46] M. Zhang, Y. Duan, H. Yin, and Z. Zhao. Semantics-aware Androidmalware classification using weighted contextual API dependency graphs.In Proc. of CCSâ14, pages 1105â1116, 2014.69
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Analyzing android taint analysis tools : FlowDroid, Amandroid, and DroidSafe
Text