Multimodal Human Brain Connectivity Analysis based onGraph TheorybyChendi WangB.A.Sc, Biomedical Engineering, Southeast University, 2010M.A.Sc, Biomedical Engineering, Southeast University, 2013A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Electrical and Computer Engineering)The University of British Columbia(Vancouver)April 2018c© Chendi Wang, 2018AbstractBillions of people worldwide are affected by neurological disorders. Recent stud-ies indicate that many neurological disorders can be described as dysconnectivitysyndromes, and associated with changes in the brain networks prior to the devel-opment of clinical symptoms.This thesis presents contributions towards improving brain connectivity analy-sis based on graph theory representation of the human brain network. We proposenovel multimodal techniques to analyze brain imaging data to better understand itsstructure, function and connectivity, i.e., brain connectomics.Our first contribution is towards improving parcellation, i.e., brain networknode definition, in terms of reproducibility, functional homogeneity, leftout datalikelihood and overlaps with cytoarchitecture, by utilizing the neighbourhood in-formation and multi-modality integration techniques. Specifically, we embed neigh-borhood connectivity information into the affinity matrix for parcellation to amelio-rate the adverse effects of noise. We further integrate the connectivity informationfrom both anatomical and functional modalities based on adaptive weighting foran improved parcellation.Our second contribution is to propose noise reduction techniques for brain net-work edge definition. We propose a matrix completion based technique to combatfalse negatives by recovering missing connections. We also present a local thresh-olding method which can address the regional bias issue when suppressing the falsepositives in connectivity estimates.Our third contribution is to improve the brain subnetwork extraction by us-ing multi-pronged graphical metric guided methods. We propose a connection-fingerprint based modularity reinforcement model which reflects the putative mod-iiular structure of a brain graph. Inspired by the brain subnetwork’s biological na-ture, we propose a provincial hub guided feedback optimization model for morereproducible subnetwork extraction.Our fourth contribution is to develop multimodal integration techniques to fur-ther improve brain subnetwork extraction. We propose a provincial hub guidedsubnetwork extraction model to fuse anatomical and functional data by propagatingthe modular structure information across different modalities. We further proposeto fuse the task and rest functional data based on hypergraphs for non-overlappingand overlapping subnetwork extraction.Our results collectively indicate that combing multimodal information and ap-plying graphical metric guided strategies outperform classical unimodal brain con-nectivity analysis methods. The resulting methods could provide important insightsinto cognitive and clinical neuroscience.iiiLay Summary3.6 million people in Canada (more than 10% of the total population) are affectedby neurological diseases, such as Parkinson’s disease, Alzheimer’s disease andother dementias. There are changes in the anatomy of the human brain and changesin the way how brain works before a patient develops clinical symptoms. In thisthesis, we present new methods to study human brain images to better understandits anatomy and how different parts of the brain work together to execute certainfunctions. Using multiple sources of brain images, we model the brain as a graph,by defining different parts of the brain as nodes and identifying the relations be-tween those parts. The relations between brain parts are studied by how stronglythey are connected to each other, and how they work together. These methods willimprove our understanding of how human brain works and furthermore how thebrain is affected by diseases.ivPrefaceThe data used in research presented herein was approved by the Ethics Board fromthe Open Access Dataset.This thesis is based on the following papers, resulting from collaboration be-tween multiple researches.Studies described in Chapter 2 have been published in:[P1] [1] (ISBI’15) C. Wang, B. Yoldemir, and R. Abugharbieh. Improved func-tional cortical parcellation using a neighborhood-information-embedded affinitymatrix. In IEEE International Symposium on Biomedical Imaging, pages 1340–1343,2015.[P2] [2] (MICCAI’15) C. Wang, B. Yoldemir, and R. Abugharbieh. Multi-modal cortical parcellationbased on anatomical and functional brain connectivity.In International Conference on Medical Image Computing and Computer-AssistedIntervention, pages 21–28. Springer, 2015. Poster presentation - Acceptance rate:∼ 33%. Early Acceptance. Student travel award.[P3] [3] (Journal’18) C. Wang, B. Ng, and R. Abugharbieh. Multimodal wholebrain parcellation, submitted, xx(x):xxxx–xxxx, 2018.Studies described in Chapter 3 have been published in:[P4] [4] (MICCAI’17 Workshop) C. Wang, B. Ng, A. Amir-Khalili, and R.Abugharbieh. Recovering missing connections in diffusion weighted mri usingmatrix completion. In International Conference on Medical Image Computingand Computer-Assisted Intervention Workshop on Computational Diffusion MRI.Springer, 2017.[P5] [5] (MICCAI’16) C. Wang, B. Ng, and R. Abugharbieh. Modularity rein-forcement for improving brain subnetwork extraction. In International Conferencevon Medical Image Computing and Computer-Assisted Intervention, pages132–139.Springer, 2016. Poster presentation - Acceptance rate: ∼ 30%. Student travelaward.Studies described in Chapter 4 have been published in:[P5] [5] (MICCAI’16) C. Wang, B. Ng, and R. Abugharbieh. Modularity rein-forcement forimproving brain subnetwork extraction. In International Conferenceon Medical Image Computing and Computer-Assisted Intervention, pages132–139.Springer, 2016. Poster presentation - Acceptance rate: ∼ 30%. Student travelaward.[P6] [6] (IPMI’17) C. Wang, B. Ng, and R. Abugharbieh. Multimodal brainsubnetwork extraction using provincial hub guided random walks. In Interna-tional Conference on Information Processing in Medical Imaging, pages 287–298.Springer, Cham, 2017. Oral presentation - Acceptance rate: ∼ 15%. Scholarshipfor Junior Scientists for Underrepresented Populations.Studies described in Chapter 5 have been published in:[P6] [6] (IPMI’17) C. Wang, B. Ng, and R. Abugharbieh. Multimodal brainsubnetwork extraction using provincial hub guided random walks. In Interna-tional Conference on Information Processing in Medical Imaging, pages 287–298.Springer, Cham, 2017. Oral presentation - Acceptance rate: ∼ 15%. Scholarshipfor Junior Scientists for Underrepresented Populations.[P7] [7] (Journal’18) C. Wang and R. Abugharbieh. High order relation in-formed task and rest data integration for subnetwork extraction based on hyper-graph, submitted, xx(x):xxxx–xxxx, 2018.[P8] [8] (Journal’18) C. Wang and R. Abugharbieh. Co-activated clique basedmultisource overlapping subnetwork extraction, submitted, xx(x):xxxx–xxxx, 2018.All listed publications were revised and edited by all co-authors.The contribution statements are as the following:In all the listed publications, Wang et al.. [1–8], the primary author, I was themain contributor to the algorithmic idea conception, design, implementation, andtesting of proposed methodology, and the writing effort under the supervision ofDr. Rafeef Abugharbieh. I also presented the oral presentation for Wang et al.. [6]and the poster presentation for Wang et al.. [1, 2, 5].In Wang et al.. [1] and Wang et al.. [2], I was responsible for concept formationviand design for the idea, data downloading, processing and analysis, experimentaldesign, as well as manuscript composition. Dr. Burak Yoldemir under the super-vision of Dr. Abugharbieh helped with preprocessing of the data and providedsuggestions on the manuscript revision.In Wang et al.. [3, 5], I was responsible for concept formation and designfor the idea, data preprocessing and analysis, and manuscript composition. Dr.Bernard Ng helped immensely with his valuable input on experimental design andmanuscript edits.In Wang et al.. [6], I was responsible for concept formation and design, datapreprocessing and analysis, and manuscript composition. Dr. Bernard Ng helpedwith the ideas for experimental design.In Wang et al.. [4], Dr. Bernard Ng contributed the general matrix comple-tion idea and helped with brainstorming the validation schemes, Dr. Alborz Amir-Khalili under the supervision of Dr. Abugharbieh contributed the rank range selec-tion idea, and I contributed the neighborhood information filling idea. I was respon-sible for processing and analysis of the data, implementation of the methodology,validation experimental design, and manuscript composition. Both Dr. Bernard Ngand Dr. Alborz Amir-Khalili contributed to manuscript edits.In [7] and [8], I was responsible for concept formation and design, data prepro-cessing and analysis, experimental design, and manuscript composition.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiList of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Thesis Objectives . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Imaging for Brain Connectomics . . . . . . . . . . . . . . . . 41.3.1 Definition of Brain Connectomics . . . . . . . . . . . 41.3.2 Imaging Resolution in Brain Connectomics . . . . . . 51.3.3 Functional Imaging for Macroscale Connectomics . . 51.3.3.1 Modalities for Functional Imaging . . . . . 51.3.3.2 Functional Magnetic Resonance Imaging (fMRI)Experimental Design . . . . . . . . . . . . 7viii1.3.4 Structural Imaging for Macroscale Connectomics . . . 71.4 State-of-the-art Techniques in Brain Connectivity Analysis . . 81.4.1 Brain Network Analysis for Connectivity Studies . . . 81.4.1.1 Model-dependent Methods for Seed-based Net-work Analysis . . . . . . . . . . . . . . . . 81.4.1.2 Model-free Methods for Whole Brain NetworkAnalysis . . . . . . . . . . . . . . . . . . . 91.4.2 Graph Representation in Brain Connectomics . . . . . 101.4.2.1 Existing Studies of Graph Use for Brain Con-nectomics . . . . . . . . . . . . . . . . . . 101.4.2.2 Definition of Nodes: Parcellation-based BrainConnectome . . . . . . . . . . . . . . . . . 111.4.2.3 Definition of Edges: Estimating Connections Be-tween Brain Regions . . . . . . . . . . . . . 121.4.2.3.1 Estimating Anatomical Connectivity 131.4.2.3.2 Estimating Functional Connectivity 171.4.2.3.3 Thresholding . . . . . . . . . . . 191.4.2.4 Network Measures: Graphical Metrics for BrainConnectomics . . . . . . . . . . . . . . . . 191.4.2.5 Module Detection: Brain Subnetwork Extrac-tion . . . . . . . . . . . . . . . . . . . . . . 201.4.2.5.1 Anatomical Subnetworks . . . . . 221.4.2.5.2 Functional Subnetworks . . . . . . 221.4.3 Multimodal Fusion in Brain Connectivity Analysis . . 251.4.4 Deep Learning in Brain Connectivity Analysis . . . . 261.5 Current Challenges Addressed in This Thesis . . . . . . . . . 281.6 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 291.6.1 How Can We Improve Parcellation? . . . . . . . . . . 301.6.2 How Can We Achieve Noise Reduction When Construct-ing Edges? . . . . . . . . . . . . . . . . . . . . . . . 301.6.3 How Can We Improve Subnetwork Extraction? . . . . 301.6.4 How Can Multimodal Information Help with SubnetworkExtractions for Brain Connectivity Analysis? . . . . . 30ix1.7 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . 301.7.1 Brain Network Node Definition . . . . . . . . . . . . 311.7.2 Brain Network Edge Estimation . . . . . . . . . . . . 311.7.3 Graphical Metric Guided Subnetwork Extraction . . . 321.7.4 Multimodal Fusion for Subnetwork Extraction . . . . 321.8 Materials and Experimental Setup . . . . . . . . . . . . . . . 331.8.1 Overview of Currently Available Open Access Neuroimag-ing Datasets . . . . . . . . . . . . . . . . . . . . . . . 331.8.2 Human Connectome Project . . . . . . . . . . . . . . 331.8.3 Data Preprocessing . . . . . . . . . . . . . . . . . . . 341.8.3.1 Data Preprocessing for Functional Connectivity 341.8.3.2 Data Preprocessing for Anatomical Connectiv-ity . . . . . . . . . . . . . . . . . . . . . . 352 Neighbourhood Information Embedding and Multimodal Integra-tion for Improved Parcellation . . . . . . . . . . . . . . . . . . . 362.1 Neighborhood Connectivity Informed Parcellation . . . . . . . 372.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 372.1.2 Neighborhood Information Embedded Affinity Matrix 372.1.2.1 Gaussian Kernel Affinity Matrix . . . . . . 382.1.2.2 Proposed Affinity Matrix . . . . . . . . . . 382.1.3 Experiments and Results . . . . . . . . . . . . . . . . 402.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . 432.2 Multimodal Connectivity Fusion for Parcellation . . . . . . . 442.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 442.2.2 Proposed Multimodal Integrated Parcellation Framework 442.2.3 Anatomical and Functional Connectivity Estimation . 462.2.4 Distribution Normalization . . . . . . . . . . . . . . . 472.2.5 Multimodal Connectivity Estimation using Adaptive Weight-ing . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.2.6 Clustering and Region Level Extension . . . . . . . . 492.2.7 Design of Evaluation Metrics . . . . . . . . . . . . . 502.2.7.1 Reproducibility . . . . . . . . . . . . . . . 50x2.2.7.2 Functional Homogeneity . . . . . . . . . . 512.2.7.3 Leftout Data Likelihood . . . . . . . . . . . 512.2.7.4 Overlaps with Cytoarchitecture . . . . . . . 522.2.7.5 Subnetwork Extraction . . . . . . . . . . . 522.2.8 Experiments and Results . . . . . . . . . . . . . . . . 532.2.8.1 Reproducibility . . . . . . . . . . . . . . . 532.2.8.2 Homogeneity . . . . . . . . . . . . . . . . 572.2.8.3 Leftout Data Likelihood . . . . . . . . . . . 582.2.8.4 Cytoarchitecture . . . . . . . . . . . . . . . 612.2.8.5 Subnetwork Structure . . . . . . . . . . . . 632.2.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . 652.2.9.1 Purposes and Key Challenges of Multimodal Par-cellation . . . . . . . . . . . . . . . . . . . 652.2.9.2 Multimodal Parcellation Improves Reproducibil-ity and Data Likelihood, and Maintains Func-tional Homogeneity . . . . . . . . . . . . . 652.2.9.3 Linking Parcels to Prior Knowledge . . . . 662.2.9.4 Region Level vs. Whole-brain Parcellation . 662.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673 Noise Reduction for Brain Network Edge Building . . . . . . . . 693.1 Matrix Completion to Combat False Negatives . . . . . . . . . 693.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 693.1.2 Low Rank Matrix Completion for Connectivity Estima-tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.1.3 Rank Range Search and Aggregation . . . . . . . . . 713.1.4 Negative Entries Filling using Neighborhood Information 723.1.5 Experiments . . . . . . . . . . . . . . . . . . . . . . 733.1.5.1 Materials . . . . . . . . . . . . . . . . . . . 733.1.5.1.1 Synthetic Data . . . . . . . . . . . 733.1.5.1.2 Real Data . . . . . . . . . . . . . 733.1.5.2 Results . . . . . . . . . . . . . . . . . . . . 753.1.5.2.1 Recovery Accuracy . . . . . . . . 76xi3.1.5.2.2 Intelligence Quotient (IQ) Prediction 773.1.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . 813.2 Local Thresholding to Suppress False Positives . . . . . . . . 823.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 823.2.2 Local Thresholding . . . . . . . . . . . . . . . . . . . 843.2.3 Experiments . . . . . . . . . . . . . . . . . . . . . . 853.2.3.1 Materials . . . . . . . . . . . . . . . . . . . 853.2.3.1.1 Synthetic Data . . . . . . . . . . . 853.2.3.1.2 Real Data . . . . . . . . . . . . . 853.2.3.2 Results . . . . . . . . . . . . . . . . . . . . 873.2.3.2.1 Synthetic Data . . . . . . . . . . . 873.2.3.2.2 Real Data . . . . . . . . . . . . . 883.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . 903.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 904 Graphical Metric Guided Subnetwork Extraction . . . . . . . . 924.1 Modularity Reinforcement Subnetwork Extraction . . . . . . . 934.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 934.1.2 Modularity Reinforcement Model . . . . . . . . . . . 934.1.3 Subnetwork Extraction Based on Graph Cuts . . . . . 944.1.4 Experiments . . . . . . . . . . . . . . . . . . . . . . 944.1.4.1 Materials . . . . . . . . . . . . . . . . . . . 944.1.4.2 Results . . . . . . . . . . . . . . . . . . . . 944.1.4.2.1 Synthetic Data . . . . . . . . . . . 954.1.4.2.2 Real Data . . . . . . . . . . . . . 964.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . 984.2 Provincial Hub Guided Random Walker Based Subnetwork Ex-traction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 1004.2.2 Resemblance Between Provincial Hubs and the Seeds 1014.2.3 Overview of Random Walker Model . . . . . . . . . . 1024.2.4 Feedback Informed Optimization Model . . . . . . . . 1034.2.4.1 Relating to Random Walker (RW) with Prior Model 103xii4.2.4.2 Multi-seed Model . . . . . . . . . . . . . . 1044.2.4.3 Subnetwork Assignment Confidence Based onPosterior Probability . . . . . . . . . . . . . 1054.2.4.4 Framework of the Feedback Informed Optimiza-tion Model . . . . . . . . . . . . . . . . . . 1064.2.5 Experiments . . . . . . . . . . . . . . . . . . . . . . 1074.2.5.1 Materials . . . . . . . . . . . . . . . . . . . 1074.2.5.1.1 Synthetic Data . . . . . . . . . . . 1074.2.5.1.2 Real Data . . . . . . . . . . . . . 1084.2.5.1.3 Parameter Setting . . . . . . . . . 1084.2.5.2 Results . . . . . . . . . . . . . . . . . . . . 1084.2.5.2.1 Synthetic Data . . . . . . . . . . . 1084.2.5.2.2 Real Data . . . . . . . . . . . . . 1094.2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . 1094.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1105 Multimodal/Multisource Brain Subnetwork Extraction . . . . . 1115.1 Multimodal Random Walker based Subnetwork Extraction . . 1125.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . 1125.1.2 Multimodal Provincial Hub Guided Random Walker Model 1135.1.3 Subnetwork Assignment Confidence and Overlapping Sub-network Exploration Based on Posterior Probability . . 1155.1.4 Experiments . . . . . . . . . . . . . . . . . . . . . . 1155.1.4.1 Materials . . . . . . . . . . . . . . . . . . . 1155.1.4.1.1 Synthetic Data . . . . . . . . . . . 1155.1.4.1.2 Real Data . . . . . . . . . . . . . 1155.1.4.2 Results . . . . . . . . . . . . . . . . . . . . 1165.1.4.2.1 Synthetic Data . . . . . . . . . . . 1165.1.4.2.2 Real Data . . . . . . . . . . . . . 1175.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . 1195.2 Fusion of Task and Resting State Functional Connectivity for Sub-network Extraction Based on Hypergraph . . . . . . . . . . . 122xiii5.2.1 Related Work - Relationship between Task and RestingFunctional Connectivity . . . . . . . . . . . . . . . . 1225.2.2 Related Work - Multilayer Brain Network Analysis . . 1235.2.3 High Order Relation Informed Subnetwork Extraction 1235.2.3.1 Framework . . . . . . . . . . . . . . . . . . 1245.2.3.2 Notation Overview of Hypergraph . . . . . 1245.2.3.2.1 Notations . . . . . . . . . . . . . 1245.2.3.2.2 Graphcut of the Hypergraph . . . 1265.2.3.3 Task Activation Detection - Node Definition inthe Hypergraph . . . . . . . . . . . . . . . 1275.2.3.4 Strength Informed Weighted Multi-task Hyper-graph . . . . . . . . . . . . . . . . . . . . . 1285.2.3.4.1 Pairwise Nodal Connection Strength Es-timation . . . . . . . . . . . . . . 1285.2.3.4.2 Proposed Strength Informed WeightedHypergraph . . . . . . . . . . . . 1295.2.3.5 Multisource Integration of Rest and Task fMRI 1305.2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . 1315.2.4.1 Materials . . . . . . . . . . . . . . . . . . . 1325.2.4.2 Similarity of Functional Connectivity (FC) Be-tween Resting State and Task Data . . . . . 1335.2.4.3 Modularity Q Value . . . . . . . . . . . . . 1345.2.4.4 Inter-subject Reproducibility of Subnetwork Ex-traction . . . . . . . . . . . . . . . . . . . . 1355.2.4.5 Biological Meaning . . . . . . . . . . . . . 1365.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . 1415.2.5.1 Hypergraph Encodes Higher Order Nodal Rela-tionship . . . . . . . . . . . . . . . . . . . 1415.2.5.2 Multisource Integration Improves Subnetwork Ex-traction . . . . . . . . . . . . . . . . . . . . 1415.2.5.3 Limitations and Future Directions . . . . . . 1425.3 Clique Based Multisource Overlapping Brain Subnetwork Extrac-tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142xiv5.3.1 Related Work - Overlapping Brain Subnetwork Extraction 1425.3.2 Co-activated Clique Based Multisource Overlapping Sub-network Extraction . . . . . . . . . . . . . . . . . . . 1445.3.2.1 Clique Identification Based on Task Co-activation 1465.3.2.2 Clique Property Computation . . . . . . . . 1495.3.2.3 Core Clique Identification . . . . . . . . . . 1505.3.2.4 Clique Based Overlapping Subnetwork Extrac-tion . . . . . . . . . . . . . . . . . . . . . . 1505.3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . 1515.3.3.1 Materials . . . . . . . . . . . . . . . . . . . 1525.3.3.2 Comparison with Existing Overlapping Subnet-work Extraction Methods . . . . . . . . . . 1535.3.3.2.1 Group-level Subnetwork Extraction Re-producibility . . . . . . . . . . . . 1535.3.3.2.2 Subject-wise Subnetwork Extraction Re-producibility . . . . . . . . . . . . 1545.3.3.3 Biological Meaning - Analyzing Function Inte-gration . . . . . . . . . . . . . . . . . . . . 1545.3.3.4 Comparison Between the Subnetwork Overlapsand the RW Posterior Probability . . . . . . 1575.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . 1595.3.4.1 Benefits of Clique Identification Based on TaskCo-activation . . . . . . . . . . . . . . . . . 1595.3.4.2 Multisource Information Integration Improves theOverlapping Subnetwork Extraction . . . . 1605.3.4.3 Overlapping Subnetwork Identification Correspondswith the RW Posterior Probability . . . . . . 1605.3.4.4 Other Considerations . . . . . . . . . . . . 1615.3.4.5 Limitations and Future Work . . . . . . . . 1615.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1616 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163xv6.1 Our Contributions in Graphical Framework for Human Brain Con-nectivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . 1636.1.1 Definition of Nodes: Parcellation-based Brain Connectome 1636.1.2 Definition of Edges: Estimating Connections Between BrainRegions . . . . . . . . . . . . . . . . . . . . . . . . . 1646.1.3 Network Measures: Graphical Metrics for Brain Connec-tomics . . . . . . . . . . . . . . . . . . . . . . . . . . 1656.1.4 Multimodal/Multisource Fusion to Improve Brain Connec-tivity Analysis . . . . . . . . . . . . . . . . . . . . . 1666.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 168Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171xviList of TablesTable 2.1 Details on existing anatomical and functional parcellations. 61xviiList of FiguresFigure 1.1 Prevalence and total Disability-Adjusted Life Years (DALYs)for selected diseases and neurological disorders. Data are adoptedfrom [9, 10]. . . . . . . . . . . . . . . . . . . . . . . . . 2Figure 1.2 Projected indirect economic costs due to working-age deathand disability, by select neurological condition and age group,Canada, 2011, 2021, and 2031 . . . . . . . . . . . . . . . 3Figure 1.3 Spatial-temporal resolution plots for commonly used functionalneuroimaging methods in the year of 2014. VSD, voltage-sensitive dye; TMS, transcranial magnetic stimulation; 2-DG,2-deoxyglucose. Image courtesy of [11]. . . . . . . . . . . 6Figure 1.4 Connectivity matrix based on graph theory created by RolfKo¨tter and projections derived from tract tracing, one of thefirst attempts to obtain a large-scale anatomical connectome ofthe mammalian brain. Image courtesy of [12]. . . . . . . . 11Figure 1.5 Cartoon illustrations of ambiguities in mapping diffusion toaxon geometry. Different axon geometries can lead to a sim-ilarly oriented tensor. The tensor’s principal direction is thesame for all cases, but modeling multiple Principal DiffusionDirection (PDD)s helps distinguish a few of the cases. Model-ing fiber fanning separates the top two cases. Further model-ing the polarity of a fanning can help separate all cases. Imagecourtesy of [13]. . . . . . . . . . . . . . . . . . . . . . . 14xviiiFigure 1.6 Deterministic and probabilistic tractography. Pyramidal tractstreamlines based on deterministic (left) and probabilistic (right)approaches (results superimposed on Fractional Anisotropy (FA)image). Color bar indicates confidence about the presence ofthe tract. Image courtesy of [14]. . . . . . . . . . . . . . . 15Figure 1.7 Typical workflow for Anatomical Connectivity (AC) estima-tion based on a Diffusion-weighted Magnetic Resonance Imag-ing (dMRI) experiment. Diffusion weighted images are ac-quired and preprocessed to reconstruct the diffusion Orienta-tion Distribution Function (ODF)s. Fiber tracts are then gen-erated using tractography. AC matrix can indicate the fibercounts between each brain pair. MR scanner image: A patientis loaded into an MRI machine, by Morsa Images/Getty Im-ages, https://www.verywell.com/diffusion-weighted-mri-3146133,retrieved on August 30, 2017. . . . . . . . . . . . . . . . 16Figure 1.8 Example of false negatives in in vivo tractography of the cere-bellum. The dotted green lines indicate the incomplete re-construction of the superior cerebellar tracts where streamlinesstop before crossing to the contralateral side as expected fromknown post-mortem neuroanatomy (false negatives). Imagecourtesy of [15]. . . . . . . . . . . . . . . . . . . . . . . 17Figure 1.9 Example of ambiguities near the gray-while matter interface.Both axon configurations lead to the same diffusion profile andtracking results, but with different connectivities. Image cour-tesy of [13]. . . . . . . . . . . . . . . . . . . . . . . . . . 17Figure 1.10 Typical workflow for FC estimation based on a fMRI experi-ment. Functional Magnetic Resonance Imaging (MRI) imagesacquired when the subjects are at task conditions or rest. Voxeland regional time courses are extracted after image preprocess-ing. FC matrix can indicate covariance between each brainpair. MR scanner image: A patient is loaded into an MRI ma-chine, by Morsa Images/Getty Images, https://www.verywell.com/diffusion-weighted-mri-3146133, retrieved on August 30, 2017. . . 18xixFigure 1.11 Examples of the connectivity matrices after graph density thresh-olding. Image courtesy of [16]. . . . . . . . . . . . . . . . 19Figure 1.12 Exemplar network parameter measures. Image courtesy of[17]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 1.13 Subnetwork structure based on functional connectivity. Sub-graphs from three thresholds are shown for the areal (spheres)and modified voxel-wise graphs (surfaces). Subnetworks suchas default mode subnetwork, frontoparietal executive controlsubnetwork, and attention subnetwork comprise of spatiallydistributed brain regions. Image courtesy of [18]. . . . . . 23Figure 1.14 The functional coactivation network based on meta-analysisof task-related fMRI studies has similar modularity to a func-tional connectivity network based on resting-state fMRI data.Co-activation and connectivity networks plotted in anatomicalspace. The edges are defined by the minimum spanning treefor illustrative purposes. The size of the nodes is proportionalto their strength, and their color corresponds to module mem-bership. Image courtesy of [19]. . . . . . . . . . . . . . . 24Figure 2.1 Illustration of Interior Point (IP) (red dot) and Boundary Point(BP) (green and blue dots). The neighbors of the latter have aheterogeneous cluster membership structure whereas all neigh-bors of the former belong to the same cluster. . . . . . . . 39Figure 2.2 Theoretical illustration of Neighborhood-information-embeddedMultiple Density (NMD) affinity matrix. Left is traditionalGaussian kernel, and the right is NMD. The affinity distribu-tion, shown in the red bars, expands to a wider range whenNMD is used. . . . . . . . . . . . . . . . . . . . . . . . . 40xxFigure 2.3 Inter-subject test-retest reliability quantified using Dice Simi-larity Coefficient (DSC) in Inferior Parietal Lobule (IPL) andCingulate Cortex (Cg) for the Human Connectome Project (HCP)data. As reflected in the box plots, our method consistentlyoutperforms the state-of-art method. The blue rectangle spansthe first quartile to third quartile. The red line indicates the me-dian and the black whiskers indicate the minimum and maxi-mum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Figure 2.4 Qualitative parcellation results for IPL and Cg using the HCPdata. In each area - Top: Using the multiple density in (2.2);Bottom: Using our proposed affinity matrix NMD in (2.4).The 1st column shows the group maps while the 2nd and 3rdcolumns are the subject-specific maps with the highest andlowest DSC with group maps, respectively. The significantdifferences between the subject-specific map having the low-est DSC (right) and the group map (left) are highlighted witharrows. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Figure 2.5 Flowchart of proposed approach for multimodal brain parcel-lation. AC and FC estimates are generated for each voxel toderive voxel-by-voxel connectivity matrices followed by a dis-tribution normalization. The resulting matrices are integratedto generate multimodal connectivity estimates. Normalizedcuts (Ncuts) is applied on the integrated similarity matrices,to generate whole-brain parcellation. . . . . . . . . . . . . 45Figure 2.6 Histograms of FC and AC derived from the group data of 77HCP subjects. The difference between the two distributionscould potentially cause bias if the FC and AC are naively fused. 48xxiFigure 2.7 Group-wise parcellation and Subject-group consistency. Boththe parcellations based on the proposed multimodal approach(left two) and unimodal FC based parcellations (right two) aredisplayed at 512-parcel scale in 2D space. Second and fourthcolumn: Subject level parcellation having the lowest DSC withthe group parcellation. The proposed multimodal parcellationsattain higher subject-group consistency (a & b), compared tounimodal FC based parcellations (c & d), which we highlightcertain areas using blue circles. . . . . . . . . . . . . . . . 54Figure 2.8 Intra-subject test-retest reliability using the DSC as the evalu-ation criterion. The number of parcels M was set to 256, 512and 1024, respectively. The blue rectangle spans the first quar-tile to third quartile. The red line indicates the median andthe black whiskers indicate the minimum and maximum. Mul-timodal parcellations achieved significantly higher test-retestreliability than those generated based on fMRI data alone. 55Figure 2.9 The inter-subject test-retest reliability (range from low red tohigh green) of each brain region of the Harvard-Oxford atlas(only shown on sagittal slices X=88mm, 72mm and 56mm asexemplars). Regions, such as the Lingual Gyrus (LG), LateralOccipital Cortex (LOC), Postcentral Gyrus (PG), Frontal Pole(FP), and Superior Frontal Gyrus (SFG) show relatively lowerreproducibility. . . . . . . . . . . . . . . . . . . . . . . . 55Figure 2.10 Subject-group consistency. Comparison between the groupparcellation and the subject parcellation of frontal pole havingthe lowest DSC with the group parcellations. . . . . . . . 56xxiiFigure 2.11 Stability with respect to the number of subjects in generatinggroup parcellations. We compared the multimodal group par-cellations generated from all 77 subjects against NS subjectswith NS set between 5 and 75 at interval of 5 subjects. Stabil-ity was estimated using the Dice coefficient. The number ofparcels was set to 512. The stabilities of the group parcella-tions as measured using the Dice coefficient plateaued at 0.96after NS = 50. . . . . . . . . . . . . . . . . . . . . . . . . 57Figure 2.12 Functional homogeneity comparisons between multimodal andunimodal parcellations. In order to show consistent gain ineach case of comparisons, plots of Multimodal Connectivity(MC)-FC, MC-AC, MC-physical distance (DC) and MC-randomparcellation (RM) are provided. Positive FC pc ratio and neg-ative mean distance indicate higher functional homogeneitygained from multimodal parcellations. Number of parcels wereset to 512. Multimodal parcels achieved significantly higherfunctional homogeneity in both FC pc ratio and mean distance,see text for details. . . . . . . . . . . . . . . . . . . . . . 59Figure 2.13 Existing anatomical and functional parcellations . . . . . 59Figure 2.14 Functional homogeneity comparison between multimodal par-cellations and existing atlases. Number of parcels for multi-modal parcellation was set to the exact number of parcels in thecontrasted atlas. Contrasted atlases included Harvard-Oxford(HO) (112 parcels), Automated Anatomical Labeling (AAL)(116 parcels), Gordon’s (333 parcels), Shen’s (268 parcels),and Craddock (190 parcels). Multimodal approach achievedsignificantly higher homogeneity against the contrasted atlasesexcept for Gordon’s. . . . . . . . . . . . . . . . . . . . . 60xxiiiFigure 2.15 Leftout log-data likelihood comparison between multimodaland unimodal parcellation. Number of parcels was set to 256,512 and 1024, respectively. (a) The likelihood of MC washigher than FC and DC, and significantly higher than AC atthe scale of 256 parcels. (b) The likelihood of MC was higherthan FC and AC at the scale of 512 parcels. (c) The likeli-hood of MC was significantly higher than FC, AC and DC atthe scale of 1024 parcels. RM performs better due to the reg-ular sampling of volume of interest from parcels with roughlyequal size. . . . . . . . . . . . . . . . . . . . . . . . . . . 62Figure 2.16 Comparison of the parcel boundaries to exemplar cytoarchitec-tonic areas. The areas in cytoarchitectonic map are bordered inblack dotted line, and the parcels having at least 70% overlapwith the cytoarchitectonic areas are displayed using solid colorbased on the ‘RGBYR’ colormap in Caret [20]. . . . . . . 63Figure 2.17 Extracted subnetworks. Subnetworks, such as Visual Network(light green in Occipital Cortex), Ventral Motor and Sensorysystem and Auditory system (orange), Dorsal Motor and Sen-sory system (spring green), and the Default Mode network(dark red) have been found in (b). . . . . . . . . . . . . . 64Figure 3.1 Eigenspectrum of an exemplar AC . . . . . . . . . . . . . 72Figure 3.2 Matrix recovery on an exemplar synthetic dataset. MCmedFill(i) more accurately recovered the ground truth connections (inred) than the contrasted methods (c-h). . . . . . . . . . . . 74Figure 3.3 Matrix recovery using MCmedFill (a) on an exemplar syn-thetic dataset. The weights in each column of the low rank ma-trix Y align with the ground truth subnetwork assignment. (b)An example of thresholded weights for MCmedFill shown. 75Figure 3.4 Rank selection method. Our criteria of using recovery accu-racy on randomly removed AC entries displays clear normal-ized root-mean-squared-error (NRMSE) minima for rank se-lection. . . . . . . . . . . . . . . . . . . . . . . . . . . . 76xxivFigure 3.5 Matrix recovery on synthetic data. Mean values are indicatedas black diamonds and labeled at the bottomn of each subfig-ure. MCmedFill achieved significantly higher accuracy thancontrasted methods based on Wilcoxon signed rank test. . 78Figure 3.6 Matrix recovery on real data. Mean values are indicated asblack diamonds and labeled at the bottomn of each subfigure.MCmedFill achieved significantly higher accuracy than con-trasted methods based on Wilcoxon signed rank test. . . . 79Figure 3.7 IQ prediction on real data. (a) MCmedFill achieved signifi-cantly higher correlation coefficient between observed and pre-dicted IQ scores than contrasted methods. (b) Brain regionsubnetwork weights along columns of Y (thresholded for clearervisualization) were found to resemble known brain networks,e.g., Frontal-Parietal network. . . . . . . . . . . . . . . . 80Figure 3.8 Signal dropouts in the orbitofrontal cortex (OFC) caused byfield inhomogeneities (at 1.5 T). Image courtesy of [21]. . 82Figure 3.9 Schematic illustrating local thresholding (minimal spanningtree and k-nearest neighbors (MST-KNN)) and global thresh-olding methods. Image courtesy of [22]. . . . . . . . . . . 83Figure 3.10 Schematic illustrating our proposed local thresholding using asmall scale example having two subnetworks with each subnet-work having a provincial hub (blue) and linked by a connec-tor hub (orange), shown in (a). In (b), warmer color indicateshigher connectivity and black dots indicate the ground truthadjacency matrix. We denote C¯ as global thresholded, and Cˆas local thresholded connectivity matrix. At a graph density of0.25, the Global Thresholding (GT) generated isolated node 2in (c), while our Local Thresholding (LT) preserved two edgeslinked to node 2 in (d). . . . . . . . . . . . . . . . . . . . 86Figure 3.11 Will90fROI atlas [23] with 90 region of interest (ROI)s beingassigned to 14 well-established brain systems. . . . . . . . 87xxvFigure 3.12 Subnetwork extraction accuracy using synthetic dataset usingdifferent thresholding schemes. The bar chart indicates the av-erage DSC over 100 synthetic dataset across a graph densityrange of [0.005, 0.5] at an interval of 0.01. . . . . . . . . . 88Figure 3.13 Circular plots of the whole brain connections based on the14 subnetwork structure using different thresholding schemes.Exemplar results shown here are thresholded at the graph den-sity of 0.02. The thickness of the connections indicate thestrength of the connections. . . . . . . . . . . . . . . . . 89Figure 4.1 Schematic illustrating our method using small scale examplehaving two subnetworks with each subnetwork having a provin-cial hub (blue) and linked by a connector hub (orange), shownin (a). In (b), warmer color indicates higher connectivity andblack dots indicate the ground truth adjacency matrix. We de-note C¯ as global thresholded, and Cˆ as local thresholded con-nectivity matrix. At a graph density of 0.25, the GT generatedisolated node 2 in (c), while our LT preserved two edges linkedto node 2 in (d). Refining the graph (c) and (d) suppressed thebetween-network edges (edges between nodes 6 and 7 & nodes6 and 9) to be the lowest connectivity in (e) and (f). . . . . 96Figure 4.2 Subnetwork extraction on synthetic data at graph densities from0.005 to 0.5 at interval of 0.01. Our proposed strategy attainedthe highest DSC overall. . . . . . . . . . . . . . . . . . . 97Figure 4.3 Subnetwork extraction on real data at graph densities from0.05 to 0.5 at interval of 0.05. Our proposed strategy attainedthe highest DSC overall. . . . . . . . . . . . . . . . . . . 98xxviFigure 4.4 Subnetwork visualization. 11 subnetworks were extracted fromgraphs with a density of 0.2. (a) Well-established brain sys-tems [23]. (b) Two subnetwork formed by isolated nodes andfalse inclusion of premotor-related regions into auditory sys-tem was observed using global thresholding. (c) Local thresh-olding failed to detect one region of known visual systems andfalsely detected four unrelated regions into dorsal default modesystem. (d) Our strategy LTMR correctly detect most of thesubnetworks found in [23]. . . . . . . . . . . . . . . . . . 99Figure 4.5 Augmented graph model for our augmented RW with priormodel. The use of prior is equivalent to using M labeled “float-ing” augmented nodes (dash-line circles in black) that corre-spond to each label and are connected to each blue node withblack dash lines. . . . . . . . . . . . . . . . . . . . . . . 103Figure 4.6 Illustration of robustness of using multiple seeds by mitigat-ing the problems caused by noisy connections between sub-networks. Noisy connections between subnetworks could af-fect the power of single seed by creating strong “false” walks(thick purple line connecting the blue seed to a node whichshould be in the red subnetwork). By using multi-seeds (thesmaller floating dots around the big dot in the center), we canpreserve several reliable connections (the dash lines connect-ing those smaller dots to the nodes) to battle against possiblefalse positive connections. . . . . . . . . . . . . . . . . . 105Figure 5.1 Schematic illustration of multi-modal Random Walker (mmRW),where the graph weights in the previous iteration t−1 are de-fined by FC edges in blue and the seeds are found using theprovincial hubs derived from AC (shown as red dash circles),then the weights in the current iteration t are defined by ACshown in red edges and the seeds are found using the provin-cial hubs derived from FC (shown as blue dash circles). . . 114xxviiFigure 5.2 Subnetwork extraction accuracy on synthetic data. Our mmRWapproach achieved significantly higher DSC than unimodal andexisting multimodal methods. . . . . . . . . . . . . . . . 117Figure 5.3 Subnetwork extraction overlap to well-established brain sys-tem [23] on real data from HCP. Our mmRW approach achievedsignificantly higher DSC than contrasted methods. . . . . 118Figure 5.4 Inter-subject reproducibility on real data from HCP. Multi-modal RW approach achieved comparable DSC to unimodalRW approach, but significantly higher DSC than existing mul-timodal methods. . . . . . . . . . . . . . . . . . . . . . . 119Figure 5.5 Subnetwork extraction results on an exemplar brain graph. (a-b) Probabilities of a node being assigned to a given subnetwork(color-coded in 14 subnetworks), the larger size of a node in-dicates a higher probability. (c-e) Exemplar subnetworks iden-tified by mmRW. . . . . . . . . . . . . . . . . . . . . . . 120Figure 5.6 Hypergraph and its corresponding simple graph and incidencematrix. Left: an hyperedge set E = {e1,e2,e3,e4} and a nodeset V = {v1,v2,v3,v4,v5,v6,v7}. Middle: the correspondingsimple graph. Right: the incidence matrix H of the hypergraphon the left, with the entry (vi,e j) being set to 1 if vi is in e j,and 0 otherwise. . . . . . . . . . . . . . . . . . . . . . . 125Figure 5.7 Subject-wise level modularity Q values using Method (1)-(6).For method (5) and (6), parameter γ and ω were selected at thehighest inter-subject reproducibility. . . . . . . . . . . . . 135Figure 5.8 Subject-wise level inter-subject reproducibility of subnetworkextraction using Method (1)-(6). For method (5) and (6), pa-rameter γ and ω were selected at the highest inter-subject re-producibility. . . . . . . . . . . . . . . . . . . . . . . . . 136Figure 5.9 Visualization of subnetworks extraction using methods (1)-(6).The brain is visualized in the axial view. The mass centerof each ROI is plotted in the Montreal Neurological Institute(MNI) space and colorcoded by the membership of seven sub-networks. . . . . . . . . . . . . . . . . . . . . . . . . . . 138xxviiiFigure 5.10 The schematic illustration of multisource clique based overlap-ping subnetwork extraction approach. . . . . . . . . . . . 145Figure 5.11 Group-level Subnetwork Extraction reproducibility based ondata from two different sessions. Multisource Clique-basedSubnetwork Extraction (MCSE) outperforms all other contrastedapproaches. . . . . . . . . . . . . . . . . . . . . . . . . . 153Figure 5.12 Subject-wise level inter-subject reproducibility of subnetworkextraction. Our proposed MCSE approach outperforms ex-isting state-of-the-art overlapping community detection meth-ods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154Figure 5.13 Visualization of Task activation and overlapping subnetworksextracted from our proposed approach and contrasted threeother methods. The brain is visualized in the axial view. Ourproposed MCSE approach outperforms existing state-of-the-art overlapping community detection methods by detecting well-known hubs which reside within subnetwork overlaps. . . 155Figure 5.14 Overlapping confidence of interacting nodes in blue versus in-divual nodes in red derived by MCSE with Ct-r. The probabil-ity of a node being assigned into subnetworks was derived bythe RW based approach [6]. . . . . . . . . . . . . . . . . 159xxixList of AcronymsAAL Automated Anatomical LabelingAC Anatomical ConnectivityACC Anterior Cingulate CortexAD Alzheimer’s diseaseBOLD Blood Oxygenated Level DependentBP Boundary PointCBP connectivity based parcellationCg Cingulate CortexCIS Connected Iterative ScanCNN Convolutional Neural NetworksCOREG CO-training with REGularizationCPM Clique Percolation MethodCSA Constant Solid AngleCSORD Coupled Stable Overlapping Replicator DynamicsDAE Deep Auto-EncoderDALYs Disability-Adjusted Life YearsxxxDBN Deep Belief NetworkDC physical distanceDMN Default Mode NetworkdMRI Diffusion-weighted Magnetic Resonance ImagingDNNs Deep Neural NetworksDOF degree of freedomDSC Dice Similarity CoefficientDTI Diffusion Tensor ImagingEC Effective ConnectivityECN Executive Control NetworkEEG ElectroencephalographyEPI Echo-Planar ImagingFA Fractional AnisotropyFC Functional ConnectivityfMRI Functional Magnetic Resonance ImagingGBD Global Burden of DiseaseGLM General Linear ModelGMM Gaussian Mixture ModelGROUSE Grassmannian Rank-One Update Subspace EstimationGT Global ThresholdingHARDI High Angular Resolution Diffusion ImagingHCP Human Connectome ProjectxxxiHO Harvard-OxfordICA Independent Component AnalysisIP Interior PointIPL Inferior Parietal LobuleIQ Intelligence QuotientLECN Left Executive Control NetworkLMaFit Low-Rank Matrix FittingLT Local ThresholdingMC Multimodal ConnectivityMCNF Matrix Completion with Nonnegative FactorizationMCSE Multisource Clique-based Subnetwork ExtractionMEG MagnetoencephalographyMITK Medical Imaging Interaction ToolkitmmRW multi-modal Random WalkerMNI Montreal Neurological InstituteMRI Magnetic Resonance ImagingMST-KNN minimal spanning tree and k-nearest neighborsMVSC MultiView Spectral ClusteringNcuts Normalized cutsNMD Neighborhood-information-embedded Multiple DensityNMF Non-negative Matrix FactorizationNMI Normalized Mutual InformationxxxiiNP Non-deterministic Polynomial-timeNRMSE normalized root-mean-squared-errorODF Orientation Distribution FunctionOSLOM Order Statistics Local Optimization MethodPCA Principal Component AnalysisPCC Posterior Cingulate CortexPD Parkinson’s diseasePDD Principal Diffusion DirectionPDF Probability Density FunctionPET Positron Emission TomographyRBM Restricted Boltzmann MachinesRD Replicator DynamicsRM random parcellationROI region of interestRQ Research Questionsrs-fcMRI Resting State Functional Connectivity based on MRIRW Random WalkersMRI Structural Magnetic Resonance ImagingSNR signal-to-noise ratioSORD Stable Overlapping Replicator DynamicsSVR Support Vector Regressiont-fcMRI Task Functional Connectivity based on MRIxxxiiiTR repetition timeWHO World Health Organization3D three-dimensionalxxxivAcknowledgmentsI would like to thank my advisor, Dr. Rafeef Abugharbieh, for supervising methroughout my doctoral studies. Her guidance and support have encouraged me tomake it through every tough moment. I owe particular thanks to Dr. Bernard Ngfor his guidance to research. Furthermore, I would like to thank my friends insideand outside of BiSICL for their support. Last but not least, I would like to thankmy parents and my boyfriend, Logan, for always being there for me.xxxvChapter 1IntroductionIf the human brain were so simple that we could understand it, wewould be so simple that we couldn’t. — Emerson W. PughThe brain is the command center for the human nervous system and controlsmost of the activities of the body. It is one of the human body’s most complexand magnificent organs. It contains hundreds of billions of neurons (grey matter),which are connected by trillions of connections (synapses), and billions of nervefibers (white matter). The brain’s connectivity pattern is represented by anatomicallinks, such as synapses, nerve fiber pathways, or functional links which revealthe information flow between different regions. Neural activities are constrainedby connectivity and the interrelations between anatomical and functional aspects.Brain connectivity is thus crucial to elucidating how neural networks behave toenable complex cognitive processes.1.1 MotivationAccording to the latest Global Burden of Disease (GBD) study, published by theWorld Health Organization (WHO) in October, 2016, neurological and mental dis-orders are leading causes of suffering and disability, affecting 2.26 billion peopleworldwide [10]. To quantify the effects of different diseases, the GBD study de-signed a metric termed Disability-Adjusted Life Years (DALYs) to measure thenumber of years of life lost as a result of both premature death and disability. Based1Figure 1.1: Prevalence and total DALYs for selected diseases and neurologi-cal disorders. Data are adopted from [9, 10].on the latest GBD data for 2015, neurological and mental disorders cause 250 mil-lion DALYs, and this demonstrated a 36% increase since 1990 [9]. A comparisonof the prevalence and DALYs for selected common diseases is given in Figure 1.1.In Canada, based on Mapping Connections report released by Neurological HealthCharities Canada in September, 2014, 3.6 million people (more than 10% of thetotal population) are affected by neurological disorders [24]. By 2031, the numberof Canadians with neurological conditions will increase, and the number of peoplewith Alzheimer’s disease (AD) and other dementias, Parkinson’s disease (PD) andtraumatic brain injury is expected to double [24]. This report, through microsimu-lation results, predicts that Canadians born during the current decade (2010-2020)who will be affected by neurological disorders will lose between 14 and 41 yearsof fully healthy lives [24]. Having a neurological condition affects one’s generaland mental health, causes impairments in function that impact on the quality of2Figure 1.2: Projected indirect economic costs due to working-age death anddisability, by select neurological condition and age group, Canada,2011, 2021, and 2031life, and induces a tremendous economic cost. The mapping connections studypredicted that these costs are expected to progressively increase as the populationlife span extends (Figure 1.2). Recent studies indicate that many neurological dis-orders can be described as dysconnectivity syndromes, associated with changesin the structural and functional brain networks prior to the development of clin-ical symptoms [25]. Our enhanced understanding of brain network organizationcan help explain network changes in neurological diseases. As a result, there is agenuinely urgent need to comprehend the brain’s connectivity and the interactionof its regions, and to reveal these relationships to diseases of current significantfundamental importance.31.2 Thesis ObjectivesThe overarching research objective of this thesis is to develop methods for a multi-modal analysis of human brain imaging data to better understand the brain’s struc-ture, function and connectivity, i.e., brain connectomics. By studying the rela-tionship between brain structure and function, we propose multimodal fusion ap-proaches for human brain connectivity analyses based on graph theory. Our ratio-nales are that, integrating information from various sources would be beneficial bycombining their respective strengths; additionally graphical metrics should play animportant role in brain network analysis.In this thesis, we focus on studying the healthy brains to build a baseline ofhow regular brain network works. Our aim is to discover the intrinsic features ofthe brain without being biased by discriminative patterns in certain diseases. Theimproved understanding of normal brain-network organization could make it pos-sible to explain network changes in a wider range of neurological and psychiatricdiseases. It is important to study healthy brains and uncover network changes priorto the development of clinical symptoms, which enables us to predict the diseaseonset and perform interventions for at risk subjects.The resulting methods we develop could provide important new insights intocognitive and clinical neuroscience, where the findings may be applied to neuro-logical disease research in revealing biomarkers for early diagnoses, prognoses,neurosurgical planning and therapeutic interventions. We believe that our researchwill enable further exploration of linking brain network features to the stages ofdisease severity, our thoughts on promising directions will be summarized in theConclusion (Chapter 6).1.3 Imaging for Brain Connectomics1.3.1 Definition of Brain Connectomics“Connectomics” is the study of brain connectivity, a term which was introducedin 2005 to refer to the complete mapping of structural connections among neuronsand brain areas [26]. The term has recently come to include the mapping of bothanatomical and functional connectivity [14].41.3.2 Imaging Resolution in Brain ConnectomicsThe connectome can be analyzed by different scales associated with levels of spa-tial resolution in brain imaging. This resolution ranges from the microscale atthe micrometer resolution level, which maps individual neurons and their synapticconnections; mesoscale at the hundreds of micrometers resolution level, which en-compasses neuronal populations, formed by local circuits (e.g. cortical columns)that link hundreds of individual neurons; and the macroscale at the millimeter res-olution division, which parcellates large brain systems into anatomically or func-tionally distinct brain areas. Currently, macroscale connectomics is the most com-monly used scale due to its lower computational and analytical demands, whichmake comprehensive mapping for noninvasive and in vivo imaging of the whole-brain connectome feasible [14].1.3.3 Functional Imaging for Macroscale ConnectomicsThe functional perspective of macroscale connectome is coded for by the temporaldependence between the activities from diverse brain areas.1.3.3.1 Modalities for Functional ImagingDifferent modalities are available for macroscale functional imaging. Electroen-cephalography (EEG) and Magnetoencephalography (MEG) are well suited to study-ing the temporal dynamics of direct neuronal activities within milliseconds of res-olution. However, it is difficult to uniquely identify the location of underlyingbioelectric/biomagnetic activity based on the scalp topology of measured signals,which is termed as the ill-posed inverse problem [27]. Positron Emission Tomog-raphy (PET) is an imaging technique that detects molecular biological details us-ing radiolabelled molecular probes, based on the assumption that areas of highradioactivity are associated with brain activity [28]. Functional Magnetic Reso-nance Imaging (fMRI) is an in vivo imaging technique measuring changes in brainfunction via Blood Oxygenated Level Dependent (BOLD) contrast [29]. fMRI hashigh spatial (millimeters) and intermediate temporal (hundreds of milliseconds orseconds) resolution. Spatial-temporal resolution plots for commonly used func-tional neuroimaging methods have been illustrated in Figure 1.3. In fMRI read-5Figure 1.3: Spatial-temporal resolution plots for commonly used functionalneuroimaging methods in the year of 2014. VSD, voltage-sensitive dye;TMS, transcranial magnetic stimulation; 2-DG, 2-deoxyglucose. Imagecourtesy of [11].ings, the hemodynamic response, i.e., variations in blood oxygenation during neu-ral activity, causes magnetization changes that can be detected in a Magnetic Reso-nance Imaging (MRI) scanner [29]. Specifically, neuronal firing consumes oxygen,which increases local magnetic inhomogeneity, and, hence, decreases BOLD sig-nals near the brain areas involved. When an inflow of oxygenated blood rushesinto these brain areas in response to the oxygen demand, the relative decrease indeoxygenated blood reduces local magnetic inhomogeneity, which increases theBOLD signals [30]. The BOLD-fMRI, which employs high localization power, isan indirect measure of brain activity through oxygenation and blood flow [31].Among the modalities used for macroconnectomics, we have chosen the BOLD-fMRI due to its widespread availability, superior spatial resolution, good temporalresolution, and safe and noninvasive sampling fashion [14].61.3.3.2 fMRI Experimental DesignThere are broadly two types of fMRI experiments for functional connectivity stud-ies. For the Task Functional Connectivity based on MRI (t-fcMRI), subjects areasked to perform certain cognitive tasks which help us identify and character-ize functionally distinct regions in the human brain. The data for Resting StateFunctional Connectivity based on MRI (rs-fcMRI), are obtained when subjects arescanned without performing any explicit task requirements. Hence, the latter re-flects spontaneous neural activities in the absence of bias towards any particularcognitive or motor task demands [32].In this thesis, these two types are considered as two different sources, with thers-fcMRI being used for standard connectivity analyses, and the t-fcMRI for themultisource fusion of rest and task data using both the connectivity derived fromthe task data and the activation magnitude or location of important brain regions.1.3.4 Structural Imaging for Macroscale ConnectomicsThe structural perspective of the macroscale connectome is an organizational de-scription of the network of elements and connections forming the human brain.This structural or physical pattern of connections is relatively fixed at shorter timescales. There have been attempts to map the large-scale structural architecture ofthe human cerebral cortex, such as by exploiting correlations in cortical volumeor thickness using Structural Magnetic Resonance Imaging (sMRI) [33]. Corticalthickness correlations have been postulated as being indicators for the presence ofphysical structural connections. However, this approach provides highly indirectinformation about connection patterns. It is worth mentioning that, currently, onlyinvasive tracing studies are able to illustrate direct axonal connections. Diffusion-weighted Magnetic Resonance Imaging (dMRI) has been primarily used to uncoverrelatively direct in vivo markers of fiber tracts with whole-brain coverage.The dMRI estimates white matter integrity and connectivity by examining thetranslational displacement of water molecules [34]. Water molecules are in con-stant thermal motion, constrained by physical boundaries. In fibrous tissues, suchas axons in white matter, water diffuses more rapidly in the direction aligned withthe structure and more slowly in the perpendicular direction. Measurements of7this anisotropic diffusion thus reveal micro-structural properties of the underly-ing tissue. In practice, images sensitized to different diffusion directions are ac-quired, followed by a fiber reconstruction step to estimate the voxel-wise three-dimensional (3D) diffusion Probability Density Function (PDF) [35]. Based on thePDF, tractography is a methodology capable of generating a continuous, smoothrepresentation of the white matter fiber trajectories.1.4 State-of-the-art Techniques in Brain ConnectivityAnalysis1.4.1 Brain Network Analysis for Connectivity StudiesThe human brain can be regarded as being a network where units, or nodes, rep-resent different specialized regions, and edges represent communication pathways.Brain network analysis methods for connectivity studies can be classified into twotypes: model-dependent methods for seed-based network analysis, and model-freemethods for whole brain network analysis [36].1.4.1.1 Model-dependent Methods for Seed-based Network AnalysisModel-dependent analysis is centered on a seed region, which may be identifiedeither as a lesion, a region of interest (ROI) involved in a specific pathway, or afunction of a brain disorder [37]; it can also be selected from a traditional task-dependent activation map acquired from meta-analysis results from previous fMRIexperiments [36]. In the seed-based analysis, signals from only the voxels withinthe seed are used to compute the functional correlations between time series of theseed region and other voxels using fMRI data; they can also be utilized to track thefiber tracts passing the seed region using dMRI data [37]. This provides a preciseand detailed description of regions of specific connectivity in particular areas ofinterest. For example, Marwan et al.. extracted six seed regions to study howchronic pain disrupts the Default Mode Network (DMN) [38]. This is one of thewell-known networks which is more active at rest than during task performance,suggesting the existence of a resting state in which the brain remains active inan organized manner. The seed-based methods have the advantage of producing8straightforward results. However, these types of methods only examine limitedconnectivity information concerning selected ROIs, and the results might dependon the selected seeds and are thus greatly dependent upon the experience and priorknowledge of the researcher [37].1.4.1.2 Model-free Methods for Whole Brain Network AnalysisModel-free methods aim at identifying whole brain networks in a data-driven man-ner. The related methods are used to discover general patterns of connectivityacross brain regions without defining a prior ROIs [36]. Based on connectivityinformation, methods such as matrix factorization, clustering, and graph-theoreticapproaches have been applied to whole brain network analysis. For example, ma-trix factorization techniques such as Principal Component Analysis (PCA) [39],Independent Component Analysis (ICA) [40] and Non-negative Matrix Factoriza-tion (NMF) [41] have been applied to whole brain network analysis. The maindrawback of such methods is the lack of interpretability of the resulting compo-nents (independent or principle components), since they contain a more complexrepresentation of the data [36]. Clustering methods such as k-means [42], hierar-chical clustering [43] and Laplacian clustering [44] target at maximizing the sim-ilarities between the data points within connected sub-clusters, i.e., subnetworks[36]. In graph-theoretic approaches, a graph is an intuitive and straightforwardrepresentation of a brain which is organized by grey matter regions (nodes) con-nected by white matter tracts (edges). Graph theory provides a theoretical frame-work for examination of the topology of networks and reveals both the local andglobal organization of brain networks [18, 45].In order to explore the connectivity of the whole brain network (connectomics),we chose to focus on the graph theory, which provides a compelling framework forthe analysis of large-scale brain network architecture. The graph theory framework,which has been widely used in cognitive and clinical neuroscience [46], enablescomprehensive studies on brain connectivity from three levels: candidate circuitanalysis (similar to seed based approaches), connectome-wide analysis (similar tomatrix factorization applications) and topological analysis (unique to graph theorytechniques) [47].91.4.2 Graph Representation in Brain ConnectomicsIn the graphical representation of the brain connectome data, the brain network canbe defined as a graph G = (V ;E) with V being the set of nodes reflecting the brainregions, and E corresponding to the degree of interactions between the brain unitsmodeled by edge weights.1.4.2.1 Existing Studies of Graph Use for Brain Connectomics“Network neuroscience” has been proposed to refer to research where neurosciencecomponents are studied mathematically in the form of networks or graphs [46].Graphs have been used in early neuroscience research on non-human brain net-works, such as the seminal work examining the organization of large-scale con-nectivity networks in the Caenorhabditis elegans, which possesses 302 neuronsand 6,393 connections [48]. “CoCoMac” comprises approximately 40,000 exper-imental findings on anatomical connections in the macaque brain, as derived fromneuroanatomical tract tracing studies [12], Figure 1.4. Most of these studies wereconducted by graph-theoretical analyses of the network structures of primates.Recent studies also conceptualize the human brain as a graph organized acrossdifferent spatial and temporal scales, i.e., a distributed complex system using graphtheory whose integrated function underlies human behaviour and cognition [16,45]. There has been a great and rapidly growing number of studies on both healthyand disease brains using graph theory during the past decade, detailed in reviewpapers ([49–51] for healthy brain, and [47, 51, 52] for disease brain).These studies have indicated that healthy brain systems have features of com-plex networks, such as small worldness, modularity and rich club of hubs [16].Brain graph organization in neurological disorders almost always reflects a devia-tion from the normal pattern, which is characterized by high local connectedness(indicated by high clustering coefficient), high integration (short path length), hi-erarchical modularity, and hub nodes that are interconnected in a rich club [52].A significant number of evidence indicates that graph theory enables us to dis-cover the mechanisms of normal brain organization and functions, uncover devel-opmental mechanisms leading to abnormal brain network organization, and trackthe progression of disease in neurological disorders [47].10(a) Macaque connectivity matrix (b) Projections in selected areas inmacaque brainFigure 1.4: Connectivity matrix based on graph theory created by Rolf Ko¨tterand projections derived from tract tracing, one of the first attempts toobtain a large-scale anatomical connectome of the mammalian brain.Image courtesy of [12].1.4.2.2 Definition of Nodes: Parcellation-based Brain ConnectomeIn graph analysis frameworks, valid node definition, i.e., parcellation, is criticalfor the accurate mapping of inter-regional connectivity. Inconsistent or imprecisenode identifications will have a huge impact on subsequent analyses [53]. Accurateparcellation enables an efficient comparison of results across studies, and it acts asa foundation for illuminating the structural and functional organization of the brain;and as a means to reduce data complexity while increasing statistical sensitivity andpower [54]. Ideal brain nodes should represent homogeneous brain regions, retainfunctional heterogeneity with the other nodes, and ensure spatial contiguity [16].Whole-brain parcellation still remains challenging in the neuroimaing field [16]due to multiply intertwined problems. First, what properties define a brain unit areunclear. For MRI data, a voxel is often considered a brain unit [55], but this defi-nition solely depends on the scan protocol, e.g., a 1 vs. 2 mm. acquisition wouldresult in “neuron patches” of different sizes being defined as brain units. Anotheralternative is random parcellation, which typically produces roughly equisizedparcels, but does not guarantee within-parcel homogeneity [56]. This disadvantagecan be partially alleviated by the generation of a very large number of parcels; how-11ever, this in turn significantly increases computational complexity in subsequentanalysis steps. A common and likely more neuroanatomically-meaningful strategyis to group voxels based on certain neurobiological properties, such as gyral-basedarchiteture [57] (well-known examples include the Harvard-Oxford (HO) atlas),cyto-architecture [58], myelo-architecture [59], chemo-architecture [60], corticalfolding [61], shared brain function [62], regional homogeneity [63], meta-analyticactivation modeling [64] and connectivity patterns, i.e., connectivity based par-cellation (CBP) [65–67]. With today’s wide availability of MRI scanners, moststrategies are now designed around using only MRI-based features for parcellation[56, 65, 66]. In particular, the majority of recent works focus on CBP, which doesnot require generating a very large number of parcels, since within-parcel homo-geneity is inherently enforced, facilitating the investigation of brain networks atcomputationally feasible resolutions. Hence, we will focus on CBP approaches tomeet the challenging demands for parcellation.A related problem is the choice of the number of parcels. Often, this num-ber is arbitrarily set [53]. More principled strategies include using criteria, suchas predictability on leftout data [68] or stability over data splits [69]. Based onarchitectonic [70] and functional [71] information, five hundred parcels appear asprobable. Others argue that this choice should be application-specific [72]. For ex-ample, estimating a 500×500 connectivity matrix given the presence of 100 timesamples would result in a high degree of estimation errors [73].1.4.2.3 Definition of Edges: Estimating Connections Between Brain RegionsThe edge which constitutes a continuous measurement of connectivity betweenbrain regions is usually defined by three broad classes of brain connectivity: anatom-ical, functional and effective [46]. In the sense of Anatomical Connectivity (AC),edges can be connection probability between two regions of tractography-deriveddatasets [74]. On the other hand, Functional Connectivity (FC) is most commonlycomputed using the Pearson’s correlation coefficient between regional activity timecourses. Lastly, Effective Connectivity (EC) infers the causal interactions betweentwo brain regions by estimating spectral coherence or Granger causality measures[75]. We focus on AC and FC in this thesis for the reason that current tempo-12ral resolutions of available fMRI data might not suffice to perform accurate ECestimations.1.4.2.3.1 Estimating Anatomical Connectivity The underlying principle of esti-mating anatomical connectivity from tractography-derived data is to rely on waterdiffusion as an indirect probe of axon geometry [13]. By tracking the motion ofwater, one can map the orientation(s) of fibers passing through each voxel of whitematter [14]. Since each voxel contains thousands of axonal fibers, one is first toinfer a probability function for each voxel as the fiber Orientation DistributionFunction (ODF) , which captures the different fiber orientations present [14]. Asimple model that approximates the fiber ODF is Diffusion Tensor Imaging (DTI),which uses a 3× 3 positive semi-definite matrix to provide an ellipsoid represen-tation (the major eigenvector) of the water-diffusion profile for a given voxel [34].Based on the assumption that the direction of least hindered diffusion, or Prin-cipal Diffusion Direction (PDD), is aligned with that of the axons; tractographyalgorithms based on DTI, such as “streamline” tractography, have been proposedto estimate tract trajectories by connecting the ellipsoids “pointing” at each otherfrom voxels [76]. 3D trajectories, referred to as streamlines, are used to trace puta-tive white-matter paths. PDD provides a good estimate of fiber orientations whenthe axons are aligned in parallel within a voxel. In reality, fibers are known tofan/merge, cross, kiss and bend within a single voxel (Figure 1.5), which leads toa heterogeneity not accounted for by a simple ellipsoidal ODF [13]. More elab-orate models for approximating fiber ODF have been used to extend the conceptof a single PDD to mixtures of PDDs to include features like fanning and polar-ity (Figure 1.5), based on High Angular Resolution Diffusion Imaging (HARDI)techniques [35]. This process is defined as qBall imaging reconstruction since theHARDI data are usually acquired on one or multiple spherical shells in q-space[35]. These more complex ODFs can be estimated by a linear radial projection ofthe PDF [77] or by the projection in a cone of Constant Solid Angle (CSA) [78].It has been demonstrated that using the projection of CSA provides a sharper rep-resentation of the diffusion process, resulting in a clearer reconstruction of certainfiber bundles [78]. Using more sensitive ODF models does not guarantee moreaccurate tractography results. For these more complex ODF models, streamlining13Figure 1.5: Cartoon illustrations of ambiguities in mapping diffusion to axongeometry. Different axon geometries can lead to a similarly orientedtensor. The tensor’s principal direction is the same for all cases, butmodeling multiple PDDs helps distinguish a few of the cases. Modelingfiber fanning separates the top two cases. Further modeling the polarityof a fanning can help separate all cases. Image courtesy of [13].follows the same principle, but with multiple peak orientations available at eachvoxel rather than a single PDD, which enables streamlines with different orienta-tions to pass through the same voxel. However, these streamline algorithms arealso prone to errors since they require the estimation of the number of bundles ina voxel to determine the number of ODF peaks. Measurement noise usually in-duces spurious variations in the generated streamlines [14]. Additionally, use of afixed step size in the generation of streamlines (despite local variations in anatomywhere all the configurations exit within a voxel) aggregates measurement errorwhen traditional tractography approaches are deterministic. Probabilistic tractog-raphy algorithms model these errors or noise by estimating the uncertainty of localfiber orientations in the entire process [14]. Uncertainty in voxel-wise fiber ori-entation can be quantified regarding the locations of streamlines [79], Figure 1.6.Nevertheless, these algorithms are mostly very computationally expensive, whichlimits their use for whole-brain connectivity. An alternative method that bypasses14Figure 1.6: Deterministic and probabilistic tractography. Pyramidal tractstreamlines based on deterministic (left) and probabilistic (right) ap-proaches (results superimposed on FA image). Color bar indicates con-fidence about the presence of the tract. Image courtesy of [14].the potential source of error of deterministic tractography at a reasonable compu-tation speed is global tractography [80], where the decisions are made on a globallevel, aggregating diffusion data across voxels to infer subvoxel features. Globaltractography jointly considers all fiber trajectories in determining the most plausi-ble fiber configuration [81]. Streamline tractography tends to terminate tracking atlocations such as crossing fiber, whereas global tractography exploits the geometryof the adjacent fibers by examining the whole-brain fiber configuration simulta-neously to compensate for ambiguous local information. This strategy alleviateserror propagation along tracts and reveals a greater number of known tracts missedwith conventional approaches, such as bilateral tracts that connect the two hemi-spheres [82]. Collectively, more accurate models and better signal descriptions arenecessary to overcome the limitations of the state-of-the-art fiber reconstructionand tractography methods [83].In order to quantify the connectivity strength for AC estimation based on fibertracts reconstructed from tractography results, metrics such as uncertainty, fibercounts, fiber density, fiber length, and FA have been used [84]. We note that the un-certainty of streamline trajectories is not easily equal to connectivity strength. Theapproximation that stronger connections are expected to have lower uncertainty intheir fiber trajectories can break in cases where locally nondominant pathways havegreater degrees of uncertainty [14]. Fiber count, arguably the most common ACmetric, measures the number of fiber tract streamlines connecting pairs of brain15Figure 1.7: Typical workflow for AC estimation based on a dMRI experi-ment. Diffusion weighted images are acquired and preprocessed to re-construct the diffusion ODFs. Fiber tracts are then generated using trac-tography. AC matrix can indicate the fiber counts between each brainpair. MR scanner image: A patient is loaded into an MRI machine,by Morsa Images/Getty Images, https://www.verywell.com/diffusion-weighted-mri-3146133, retrieved on August 30, 2017.regions. Fiber density (fiber count / the number of voxels in brain regions), is avariant of fiber count conducted by normalizing the fiber count by the total volumeof the brain region pairs interconnected in numbers of voxels. Mean fiber length(i.e. the average length in mm for all streamlines belonging to a fiber tract) is some-times used to correct the bias that longer tracts may have larger accumulated error,leading to lower fiber counts. The average FA along streamlines connecting brainregions has also been proposed as a proxy for connection strength [85].Based on the previous work in our lab [86], we have chosen to use the CSA-ODF estimation and global tractography based on HARDI data. We use the nor-malized fiber count to quantify the AC due to its intuitive interpretation of connec-tion strength. Steps in a typical dMRI experiment for AC estimation are shown inFigure 1.7.AC estimated from tract strength based on dMRI data particularly suffers fromfalse negative connections, Figure 1.8. The false positives and negatives here arenot this way in the common statistical sense; instead, they are misestimates ofconnectivity due to modeling errors [15]. Due to partial volume effects from thelimited spatial resolution, problems such as ambiguous diffusion direction at cross-ing fibers during tractography can cause the premature termination of tracts withconventional streamline algorithms [87]. Even when this problem is resolved withbetter ODF modeling, the ambiguity in diffusion direction near the gray-white mat-16Figure 1.8: Example of false negatives in in vivo tractography of the cere-bellum. The dotted green lines indicate the incomplete reconstructionof the superior cerebellar tracts where streamlines stop before cross-ing to the contralateral side as expected from known post-mortem neu-roanatomy (false negatives). Image courtesy of [15].Figure 1.9: Example of ambiguities near the gray-while matter interface.Both axon configurations lead to the same diffusion profile and trackingresults, but with different connectivities. Image courtesy of [13].ter interface introduces great uncertainty concerning the fiber endpoint locations[13], Figure 1.9, which can also lead to fiber tract pre-mature termination resultingin false negatives.1.4.2.3.2 Estimating Functional Connectivity Functional Connectivity reflectsthe co-activation patterns between distributed and often spatially remote brain re-gions during both spontaneous and task-evoked brain activity [46].Different approaches for estimating FC measure different aspects of connec-17Figure 1.10: Typical workflow for FC estimation based on a fMRI ex-periment. Functional MRI images acquired when the subjects areat task conditions or rest. Voxel and regional time courses areextracted after image preprocessing. FC matrix can indicate co-variance between each brain pair. MR scanner image: A patientis loaded into an MRI machine, by Morsa Images/Getty Images,https://www.verywell.com/diffusion-weighted-mri-3146133, retrievedon August 30, 2017.tivity. The most widely used Pearson’s correlation quantifies marginal (direct andindirect) dependencies, while partial correlation measures conditional (direct) de-pendencies. Other metrics include spectral coherence and more complex metricsusing such as phase-locking [88]. Simulation studies have demonstrated that corre-lation metrics perform notably better than complex metrics [88]. Real data studieson test-retest analyses have shown that correlation is more stable than coherence[89]. We have thus chosen to use Pearson’s correlation in this thesis for FC esti-mation. Steps in a typical fMRI experiment for FC estimation are shown in Fig-ure 1.10.False positive correlation is one major problem when estimating FC [90]. fMRImeasurement of brain activity is indirect and error-prone, because it captures theaverage effect of many spikes and does not resolve cortical columns or individ-ual neurons, resulting in structured noise and signal convolutions [91]. Hence, theinherently low signal-to-noise ratio (SNR) of the BOLD signal together with addi-tional confounds such as scanner drift, the subject’s head motions and other phys-iological noise (cardiac and respiratory cycles, arterial CO2 concentration, bloodpressure/cerebral autoregulation, and vasomotion) pose major challenges to theinterpretation of fMRI data, resulting in many false positive correlations in FC es-timations [90].18(a) density = 0.75 (b) density = 0.20Figure 1.11: Examples of the connectivity matrices after graph densitythresholding. Image courtesy of [16].1.4.2.3.3 Thresholding Typically, measurements of both anatomical and func-tional connectivity strength are formed into a connectivity matrix, Figure 1.7 andFigure 1.10. The connectivity matrix can be thresholded and binarised to create anunweighted graph presented by a binary adjacency matrix. The choice of thresh-olds is non-trivial: different thresholds will create graphs of distinct connectiondensity (or network sparsity), see the example in Figure 1.11. Hence, networkproperties are often analyzed over a range of reasonable thresholds. Meanwhile,methods deploying weighted graphs are emerging and increasingly being appliedto brain network analysis [16]. In this scenario, a positive correlation between re-gional activity time courses represents integration or cooperation, while a negativecorrelation suggests competition or segregation [92]. Most classical graph theoreticmetrics are based on only positively weighted connections due to the difficulty ininterpreting negative connectivity [93].Overall, it is important to leverage these different types of connectivity to de-fine weighted edges, which potentially enables the full richness of the availablemultimodal data to be exploited for connectomics studies [94].1.4.2.4 Network Measures: Graphical Metrics for Brain ConnectomicsGiven brain connectivity matrices, brain networks can be quantitatively examinedfor certain commonly used network measures of anatomical and functional con-19nectivity. Network measures are often represented in two ways. One is the mea-surement of individual network elements, i.e., nodes or edges, quantifying the con-nectivities linked to these elements to reveal the manner in which they are con-nected within the brain network. Subsequently, the measurement values of eachmetric comprise a distribution, which renders a global description of the network[17]. This distribution is typically represented by the mean value and shape, whichcan be used to estimate the types and properties of the network. The widely usedmeasures detect various aspects of functional segregation and integration, quan-tify the importance of individual brain regions, and examine networks’ resilienceagainst attacks [17]. Segregation and integration are two principles that link differ-ent modes of brain connectivity [95]. Segregation refers to the presence of special-ized brain regions with distinct functions forming segregated cortical areas, whileintegration indicates the collaboration of distributed neuronal populations enablingthe emergence of coherent cognitive and behavioural states [95]. For example, theclustering coefficient is a good measurement of functional segregation, quantifyingthe presence of clusters or modules within a network. The average shortest pathlength between all pairs of nodes in the network, known as the network’s char-acteristic path length, is one of the most commonly used measures of functionalintegration. On the other hand, in order to study the information flow or resilienceof the brain network, we focus on hubs, i.e., highly connected nodes which reflectthe importance of certain core brain regions. They either connect primarily withother nodes in the same group (provincial hubs) or with nodes that belong to dif-ferent groups (connector hubs). Figure 1.12 illustrates some basic measures, anda comprehensive list of mathematical definitions of the measures can be found in[17].1.4.2.5 Module Detection: Brain Subnetwork ExtractionAmong the above measures, the modular structure (community structure) is ofparticular interest; it is from this structure that we can infer information aboutbrain subnetworks. The modular structure is extracted by subdividing a networkinto groups of nodes with the maximal possible within-group links and minimalbetween-group links using community detection methods [96]. The term mod-20Figure 1.12: Exemplar network parameter measures. Image courtesy of [17].ularity quantifies the degree to which the network may be subdivided into suchclearly delineated and non-overlapping groups [97]. Typically, the optimal modu-lar structure is extracted using optimization algorithms based on modularity. Thenotable modularity maximization approach [97] is known to be sufficiently fastfor smaller networks. Another heuristic method based on modularity optimization[98] performs much faster for larger networks and is also able to detect a hierarchyof modules. There are other approaches for discovering overlapping modular net-work structure, where single nodes may belong in more than one specific module.For example, a new definition of modularity has been proposed based on unbiasedcluster coefficients to discover overlapping subnetworks in resting state connectiv-ity [99]. Although being able to uncover important subnetworks, the modularitymeasurement suffers from a resolution limit [100], which implies that the modulesidentified from simply maximizing their modularity might contain some smallerand more finely grained modules. This problem can be tackled by choosing differ-ent resolution parameters.Except from modularity maximization, seed-based approach, ICA and graphpartitioning approaches have been commonly used to discover subnetworks [101].Despite its intuitiveness and simplicity, the drawback of seed-based approach isthat its results are dependent on the choice of seeds. ICA based approaches aredata-driven by mostly dividing fMRI observations into maximally independentspatial components [102]. At the intra-subject level, voxels that have been assigned21component weights higher than the threshold within the same spatial componentare grouped into subnetworks. However, this threshold is usually set in an ad hocmanner, which hinders statistical interpretation across the subjects [103]. We fo-cus on graph partitioning approaches since we use graph representations to studybrain networks. Recently, graphical approaches, such as distance-based clusteringtechniques [64], InfoMap [18] and the Potts model [104], have been applied to thewhole-brain data to extract human brain subnetworks.It is important to note that we use subnetworks to describe the subgroups de-rived from a larger brain network, namely, the large-scale whole brain system.Nonetheless, we use the term “brain network” to refer to the brain system in thisthesis so as to conform to the jargon used in the existing literature. Also, certainsubnetworks have been denoted in the literature using jargon which sometimesincludes the word “system”; however, in this thesis, they are referred to as “sub-networks”.1.4.2.5.1 Anatomical Subnetworks In one of the earliest subnetwork studies,based on anatomical data, Hagmann et al., when using spectral decomposition tomaximize modularity, discovered six communities: four (two pairs of bilaterallysymmetric) communities located in the frontal and temporoparietal cortex, and twopositioned over the precuneus and posterior cingulate cortex [105]. One interest-ing finding is that these modules are spatially contiguous; this may help conservephysical wiring costs. This study also indicates a close association between thestructural modules and functional domains of behaviour and cognition [106].Two recent studies explored the multiscale human anatomical subnetworks byusing a random walker moving at various timescales over the brain network [107],and by varying the resolution parameter in the multiresolution technique [108].Both studies discovered biologically meaningful modules at different scales. In-terestingly, anatomical subnetworks can be predictive of functional connectivity atsome specific scales / resolutions [107].1.4.2.5.2 Functional Subnetworks Numerous studies have shown that the hu-man brain can be grouped into modules of functionally interconnected regions that22Figure 1.13: Subnetwork structure based on functional connectivity. Sub-graphs from three thresholds are shown for the areal (spheres) andmodified voxel-wise graphs (surfaces). Subnetworks such as defaultmode subnetwork, frontoparietal executive control subnetwork, andattention subnetwork comprise of spatially distributed brain regions.Image courtesy of [18].are reproducible within subjects and consistent across subjects [109]. Power etal.. [18] extracted 25 subnetworks from rs-fcMRI data and mapped them to cog-nitive or behavioural functions based on task-based activations, Figure 1.13. Inthis study, they have discovered some functional brain subnetworks that are notgenerally present in anatomical brain networks. In terms of functional connectiv-ity, connections reflect statistical dependencies between brain regions rather thanphysical linkages in anatomical connectivity. Hence, functional connections do notcarry direct metabolic wiring cost, which imposes a penalty on strong long distanceconnections [106]. Therefore, most subnetworks obtained based on functional dataare spatially distributed. For instance, subnetworks such as the default mode, fron-toparietal executive control, and attention subnetworks are comprised of spatiallydistributed brain regions [18], Figure 1.13.Other than using rs-fcMRI, a study was able to generate a brain coactivationnetwork based on a meta-analytic approach, using the t-fcMRI [19]. Unlike thetraditional resting state FC, the foci information from task-based brain activationstaken from thousands of experiments was used to build a coactivation matrix. The23Figure 1.14: The functional coactivation network based on meta-analysis oftask-related fMRI studies has similar modularity to a functional con-nectivity network based on resting-state fMRI data. Co-activation andconnectivity networks plotted in anatomical space. The edges are de-fined by the minimum spanning tree for illustrative purposes. The sizeof the nodes is proportional to their strength, and their color corre-sponds to module membership. Image courtesy of [19].coactivation strength was estimated by the Jaccard index, which evaluates the simi-larities between activation patterns in paired regions over a large number of experi-ments found in 1,641 t-fcMRI or PET studies, derived from the BrainMap databaseand conducted between 1985 and 2010 [110]. Four subnetworks, which corre-spond to four domains, namely, perception, action, emotion, and mixed functions,have been identified by the modularity maximization approach, Figure 1.14. Thesubnetwork extraction performed on a parallel resting-state functional connectiv-ity network, employing an average group matrix of 27 volunteers, revealed similarsubnetworks [19], Figure 1.14. This result confirms that there is strong resem-blance, with great spatial overlap, between rest and task subnetworks [111, 112].Modular brain networks play important roles in conserving wiring costs andcreating specialized information and complex dynamics [106]. However, subnet-work extraction remains challenging due to the pronounced noise found in the neu-roimaging data. Few methods have exploited intrinsic properties of brain networksother than the modularity.241.4.3 Multimodal Fusion in Brain Connectivity AnalysisWe have introduced the graph theoretical approach to brain connectivity analysisfrom two perspectives, graph construction and analyzing the constructed graphs.The analysis depends on the accuracy of the graph construction derived from anatom-ical or functional connectivity estimations. However, the FC estimated from fMRImeasurements suffers from a significant number of false positives (paragraph 1.4.2.3.2),and the anatomical connectivity estimated from fiber tract strength, based on dMRImeasurements, suffers from a pronounced number of false negative connections(paragraph 1.4.2.3.1). These discrepancies may appear at first sight to be setbacks,but combining both sets of data may in fact help alleviate the problems linked toindividual modalities since functional and anatomical connectivity would containdifferent facets of information.Human brain studies ranging from healthy brains, neuronal development, andneurological disorders all yield examples of direct correspondence between struc-tural linkages and functional correlations. Such studies indicate that there exists ageneral structural core of functional networks, sustaining the notion of a global linkbetween anatomical and functional connectivity on a whole-brain scale [105, 113–115]. The underlying mechanism could be that the functional synchronizationbetween spatially distinct brain regions is enabled through neural fiber pathways[116]. Among these studies, statistical analysis demonstrates that the functionalconnectivity strength is positively correlated with that of anatomical connectivity[114]. However, functional connectivity is also observed between regions wherethere is little or no anatomical connectivity, which most likely indicates the exis-tence of functional correlations mediated by indirect structural connections, i.e.,via a third region [113]. In a sense, the brain structure acts as an anatomical skele-ton that condenses the dynamic brain function into a low-dimensional manifold orsubspace [117].The interrelationship between functional and anatomical connectivity revealedby the aforementioned studies indicates the potential benefits of the multimodalfusion of fMRI-dMRI data. Recently, multimodal fusion approaches for jointanatomical and functional connectivity estimation and analysis have been exploredto better understand the human brain network [85, 118–120]. It has been shown25that more robust parcellation can be produced based on multiple anatomical andfunctional criteria for the definition of brain regions and their boundaries [54, 121].Higher inter-subject reproducibility of the connectivity patterns can be achieved bymultimodal connectivity inference [85, 119, 120]. Great interest has been seen inmultimodal subnetwork extraction, which produces a more stable and biologicallymeaningful subnetwork structure [6, 85]. In terms of behaviour and populationstudies, multimodal approaches showed a higher discrimination power betweenthe clinical and control populations than unimodal methods [118]. The detailsof studies of multimodal neuroimaging data fusion methodologies and their ap-plications in cognitive and clinical neurosciences can be found in review papers[122, 123]. Promising results in these studies confirm the merits of multimodalfusion approaches, which can be applied to both constructing and analyzing thebrain network.1.4.4 Deep Learning in Brain Connectivity AnalysisDeep learning, part of a broader family of machine learning, has recently beenproven to be the state-of-the-art approach to enhancing performance in variousmedical applications [124]. The use of deep learning for neurological applicationshas only lately become more popular. It started with applying deep learning onstandard grid-like brain images for: segmentation [125], tumor or lesion detection[126], registration [127], and classification and prediction of disease stages [128–130]. Most of the work used either unsupervised Deep Auto-Encoder (DAE)/DeepBelief Network (DBN) along with Restricted Boltzmann Machines (RBM) fol-lowed by fine tuning in a discriminative setting; or variations of ConvolutionalNeural Networks (CNN), such as the widely used U-net [131], in a supervised man-ner. Recent endeavour has been made to discover the brain patterns using graphrepresentation for disease prediction [132–134], which was based on the work byShuman et al.[135] who generalized convolutions to graph structured domains us-ing multiplications in the graph spectral domain. Detailed review of applying deeplearning in brain image analysis can be found in [124] and [131].Very few work has been done in terms of applying deep learning to study thebasic components of graph construction and analysis of brain connectome. Re-26searches are still in early stages to discover the nature of brain connectivity basedon deep learning. Parcellation has recently been described as a semantic segmen-tation solved by CNN, but only based on raw sMRI image data [136] or regionalcell-body stained images [137]. Deep Auto-Encoder might be a direction to studythe brain subnetwork by using the learned weights to reveal the hierarchical re-lations among regional features [138]. However, the subnetwork results dependon a weight threshold and it is hard to show interpretable subnetworks (relations)[138]. Collectively, the few existing deep learning based brain connectivity studiesmostly use only raw brain imaging data, without capturing the topological informa-tion from the graph representation, nor utilizing multimodal fusion techniques tocombine information from different sources. The common challenge of the afore-mentioned deep learning work on brain connectivity analysis is the lack of groundtruth, which provides a significant amount of manually labeled data for training[124]. Although most of these work used unsupervised feature learning as the firststep, the final supervised fine tuning step, which finds the most relevant and essen-tial features for target tasks, may render learned features superficial and misrepre-sent the complexity of the nature of brain [124] (e.g., in application of Alzheimer’sdisease prediction, only discriminative patterns related to AD are extracted, whichare not necessarily related to healthy brains or other types of neurological disor-ders). Additional challenges for brain image analysis based on deep learning ingeneral, include:(1) The lack of large training data sets: Relatively small amount of availabledata for brain analysis application (∼1000 medical images vs. millions for naturalimages) leads to the related overfitting problems (when learning a large number ofparameters in the deep network model).(2) Data noise: Neuroimaging data suffer from pronounced noise and im-age resolution limitation. Recent research showed that current Deep Neural Net-works (DNNs) are vulnerable to noise, such as adversarial attacks [139], and lowdimension perturbation as small as one pixel attack can greatly affect the perfor-mance of the models [140].(3) Time consuming acquisition of relevant annotations/labeling [131]. Scarceand expensive medical expertise is needed for high quality annotation of medicalimaging data.27(4) Label noise: In computer vision, the noise in the labeling of images is typ-ically relatively low since the labeling is oftentimes intuitive. However, in medicalapplications, even when data are annotated by domain experts, the consensus issomewhat hard to reach, rendering label noise a significant limiting factor in de-veloping algorithms [131].(5) Class imbalance: In brain disorder classification and prediction applica-tions, images for the abnormal/disease class are always challenging to acquire.The typical strategy is applying specific data augmentation algorithms to increasethe size of the underrepresented class [131].(6) Deep learning is a black box [124]: It is hard to interpret the features learnedand link them to their biological intuition. There has been work trying to under-stand what intermediate layers of convolutional networks are responding to fornatural images [141]. Developing methods to explain the features in brain analysisarea is urgent and warrants the further research.(7) The choice of deep learning architecture/model. It is hard to make decisionswhen selecting a known architecture (transfer learning), devising a task-specificarchitecture to train from scratch, or fusing across architectures [131]. Reasonsneed to be justified to support the decisions.Due to the aforementioned challenges, it is essential and urgent to learn thebasics of the brain network to guide the deep architecture construction. Since agraph is a good representation of the brain, our contributions in brain networkanalysis based on graph theory can help to guide deep learning based approaches tospecific applications, such as disease classification and prediction. We will discusssome promising directions on future deep learning work for the purpose of brainconnectivity analysis in the Conclusion (Chapter 6).1.5 Current Challenges Addressed in This ThesisThe challenges of brain graph construction involve developing methods to definenodes and edges, which represent the adequate information for biological reality[16]. Detecting the boundaries between coherent regions on the basis of any one,or multiple numbers of, the existing criteria remains challenging [117]. On theother hand, the methods for node definition can often be unstable, with small mod-28ifications of the input data leading to large changes in the parcellation results [91].Devising a reliable method for parcellation is essential. At the same time, onepossible cause for this instability might be explained by the lack of explicit noisemodel [91]. Mainstream connectivity based parcellation depends greatly on thequality of the fMRI or dMRI data, both of which suffer from pronounced noiseand image resolution limitation. It is challenging to perform noise reduction andreduce the false positives and negatives in connectivity estimations.In terms of graph analysis, challenges exist particularly in understanding thesegregation and integration of the human brain based on numerous network mea-sures. Important issues currently under development involve improved and domain-appropriate approaches to module/subnetwork detection [106]. Few existing ap-proaches fully utilize informative graphical metric information in subnetwork ex-tractions. At the same time, significant attention should be paid to untangling thecontributions of spatial embedding and functional specialization to the definitionof network communities [106]. It has been shown that there is no clear one-to-one correspondence between network communities in anatomical and functionalconnectivity [142]. Hence, it is challenging to apply the promising multimodalfusion techniques without a clear guideline. In line with the complex relationshipbetween different brain connectivities, advanced models and algorithms, e.g., con-sidering higher order relations among network nodes, need to be developed for ajoint multimodal analysis of brain networks.1.6 Problem StatementWithin the framework of studying the human brain connectome as a network basedon graph theory, we try to tackle the challenges in the steps of node definition, edgebuilding, network measure estimation, and network analyses (particularly involv-ing subnetwork extraction). As the input of the brain graph, anatomical and func-tional connectivity analyses provide two views of brain connectomes. The aim ofthis thesis is to devise novel methods to jointly utilize multimodal information toanalyze human brain connectivity. The following Research Questions (RQ) raisedconcerning the challenges existing in the brain network analysis framework will beaddressed in this thesis.291.6.1 How Can We Improve Parcellation?• RQ1: Can we devise a more reliable/reproducible method for parcellation?• RQ2: Can we fuse connectivity information from different imaging modali-ties to better define the criteria for parcellation?• RQ3: How to validate improvements on parcellation?1.6.2 How Can We Achieve Noise Reduction When ConstructingEdges?• RQ4: Can we combat false negatives?• RQ5: Can we suppress false positives?1.6.3 How Can We Improve Subnetwork Extraction?• RQ6: Can we use brain graphical metrics to incorporate more domain-relatedinformation?• RQ7: Can we devise a model which would resemble the brain subnetwork’sbiological nature?1.6.4 How Can Multimodal Information Help with SubnetworkExtractions for Brain Connectivity Analysis?• RQ8: Can we fuse anatomical and functional connectivity to improve sub-network extraction?• RQ9: Can we devise a model for brain subnetwork extraction which consid-ers higher order relations among network nodes using multisource informa-tion?1.7 Thesis ContributionsIn this thesis, we propose novel approaches, including multimodal fusion tech-niques, to explore brain connectivity based on the graph theoretical framework.30By constructing the brain graph with a proper definition of nodes and edges, wehave achieved improved parcellation and applied noise reduction to perform reli-able brain connectivity estimation for the subsequent brain network analysis. Wethen utilized domain-appropriate graphical metrics to study brain subnetwork ex-traction. We further propose multimodal fusion approaches towards combininganatomical and functional connectivity for subnetwork extraction, and explore thehigh order features in brain networks.1.7.1 Brain Network Node DefinitionOur first contribution is towards improving parcellation, i.e., brain network nodedefinition, by utilizing neighbourhood information and multimodal integration tech-niques.Specifically, we embedded neighborhood connectivity information into the affin-ity matrix for the parcellation process to ameliorate the adverse effects of noise,achieving more reproducible parcellation. [P1]Further, we integrated the connectivity derived from anatomical and functionalmodalities based on adaptive weighting for improved parcellation. In order to val-idate these improvements, we designed a number of evaluation metrics includingreproducibility, functional homogeneity, leftout data likelihood, and overlaps withcytoarchitectonic areas. [P2-3]1.7.2 Brain Network Edge EstimationOur second contribution is to propose noise reduction techniques for brain edgeestimation.We proposed a matrix completion based technique to combat false negativesby recovering missing connections. We validated the effectiveness of this tech-nique using synthetic experiments which simulate the false negatives, and furtherindirectly verified our technique using an Intelligence Quotient (IQ) prediction ap-plication. [P4]We presented a local thresholding method which can address the regional biasissue when suppressing the false positives in connectivity estimates. We com-pared this local thresholding method against state-of-the-art thresholding methods31in brain graphs and confirmed the superiority of our approach. [P5]1.7.3 Graphical Metric Guided Subnetwork ExtractionOur third contribution is to improve brain subnetwork extraction by using multi-pronged graphical metric guided methods. We propose a connection-fingerprintbased modularity reinforcement model which reflects the putative modular struc-ture of a brain graph. We show that the subnetworks extracted using our modelmatched well with well-established brain systems. Compared with methods con-ducted without using the proposed graphical metric guided strategies, the resultsof using our approach manifest more biologically meaningful brain systems. [P5]We also propose a provincial hub guided feedback optimization model, whichresembles the brain subnetwork’s biological nature for more reproducible subnet-work extraction. The subnetworks derived from this model demonstrates greateroverlaps with well-established brain systems as compared to contrasted methods.[P6]1.7.4 Multimodal Fusion for Subnetwork ExtractionTo further improve subnetwork extraction, our fourth contribution is to developmultimodal techniques to integrate information from multiple sources. We firstpropose a provincial hub guided multimodal random walker based model to fuseanatomical and functional data by propagating the modular structural informationacross different modalities. [P6]We next integrate multi-task information into subnetwork extraction based onhypergraph to study the higher order relations among network nodes. [P7] We fur-ther propose a co-activated clique based overlapping subnetwork extraction method.[P8]321.8 Materials and Experimental Setup1.8.1 Overview of Currently Available Open Access NeuroimagingDatasetsModern advances in neuroimaging enable systematic exploration of the humanconnectome. The human connectome research requires massive datasets with min-imal variance in experimental procedures acquiring large samples in a reasonableamount of time. Efforts towards building benchmarks with expensive amount ofdata for datasharing have been seen in the recent years, such as the InternationalNeuroimaging Data-sharing Initiative (INDI) [143], 1000 Functional ConnectomesProject [144] and the Human Brain Project [145]. Among those dataset, HumanConnectome Project (HCP) is the most applicable dataset for this thesis due to thereasons that we summarize below in Section 1.8.2.1.8.2 Human Connectome ProjectHCP [146] stands out by undertaking a systematic effort to map macroscopic hu-man brain circuits and their relationship to behaviour in a large population ofhealthy adults using multiple imaging modalities. The HCP consortium provides adataset acquired from a cohort of 1,200 healthy adults that is made publicly avail-able. The HCP dataset has high spatial and temporal resolution. The typical ac-quisition resulotions are 2mm isotropic voxel size for dMRI, 4 mm isotropic andrepetition time (TR) at two-second for fMRI; however, HCP consortium managesto acquire higher resolution at 1.25 mm for dMRI, 2mm and 0.72 second for fMRIusing fast TR sampling. Due to the high quality and extensive scope of the open ac-cess HCP data, we have chosen to use HCP for our validation for brain connectivityanalysis in this Thesis.Data from the HCP Quarter three (Q3) were used in this thesis. The Q3 datasetrelease has data from 80 unrelated healthy subjects. Data from three of the sub-jects were excluded due to structural abnormalities in the original dataset. We onlyemployed the T1 sMRI, t-fcMRI, rs-fcMRI, and dMRI scans of these 77 subjects(36 males and 41 females, ages ranging from 22 to 35) who have no history ofneurological disease. The T1 images have an isotropic voxel size of 0.7 mm. The33rs-fcMRI data comprise two 30 min sessions, acquired at a TR of 0.72 second andan isotropic voxel size of 2 mm. The t-fcMRI data contain seven different tasks(working memory, gambling, motor, language, social cognition, relational process-ing, and emotional processing), each under 10 minutes at an isotropic voxel sizeof 2 mm. The dMRI data have an isotropic voxel size of 1.25 mm, three shells(b = 1000,2000,3000s/mm2), and 288 gradient directions with six b = 0 (B0) im-ages, acquired with right-to-left and left-to-right phase encoding polarities. Furtherdetails can be found in [146]. We used the volume parcellation provided in HCPdata package [147] to extract grey matter, white matter, and cerebral spinal fluidfor each subject. The T1 sMRI volumes were also used for drawing a correspon-dence between subjects’ native space and Montreal Neurological Institute (MNI)space templates. The subject-wise T1 images were registered to MNI space witha FMRIB’s Linear Image Registration Tool (FLIRT) 12 degree of freedom (DOF)affine and then a FMRIB’s nonlinear Image Registration Tool (FNIRT) nonlinearregistration, described in details in [147].1.8.3 Data PreprocessingIn order to study the brain connectivity, preprocessing of the raw MRI data, andsome further estimation need to be performed to obtain the Adjacency/Connectivitymatrix for both functional and anatomical connectivity estimations.1.8.3.1 Data Preprocessing for Functional ConnectivityPreprocessing already applied to the HCP fMRI data includes gradient distortioncorrection, motion correction, spatial normalization to MNI space with nonlin-ear registration based on a single spline interpolation, and intensity normalization[147]. It was suggested in the HCP preprocessing paper [147] that no slice timingcorrection needs to be employed, since the fast TR sampling reduces the need forslice timing correction as all slices in each volume are acquired much closer to-gether than in typical fMRI acquisitions (TR ∼ 2.5s). Additionally, we regressedout motion artifacts, mean white matter and cerebrospinal fluid confounds, andprincipal components of high variance voxels using compCor [148]. Next, we ap-plied a bandpass filter with cutoff frequencies of 0.01 and 0.1 Hz for rs-fcMRI data.34For t-fcMRI data, we performed similar temporal processing, except a high-passfilter at 1/128 Hz was used. The data were further demeaned and normalized bythe standard deviation.Let Z be a t×N matrix of preprocessed fMRI time courses, where t is thenumber of time points and N is the number of voxels of interest. We estimate theFC matrix using Pearson’s correlation: C = ZT Z/(t−1).1.8.3.2 Data Preprocessing for Anatomical ConnectivityPreprocessing already applied to the HCP dMRI data includes B0 intensity normal-ization, Echo-Planar Imaging (EPI) distortion correction, eddy current correction,gradient nonlinearity correction, and motion artifacts removal [147]. We used theMedical Imaging Interaction Toolkit (MITK) diffusion imaging package [149] forqBall reconstruction and global tractography. Specifically, for qBall reconstruc-tion, we calculated fiber ODF based on CSA [78]. CSA-ODF provides sharp ODFsand exploits multi-shell information, which enables multiple intra-voxel fiber ori-entations to be more easily resolved. We then performed whole-brain deterministicglobal tractography on the estimated ODFs using Gibbs tracking [81]. In contrastto the conventional streamline approach, global tractography jointly considers allfiber trajectories in determining the most plausible fiber configuration [81]. Thisstrategy alleviates error propagation along tracts and produces more known tractsmissed with conventional approaches.Fiber count between target brain regions normalized by volume voxels wasthen computed as an estimate of anatomical connectivity. To account for how thereare often too few fibers going through a single voxel, we adopted the strategy in[150] to incorporate information from each voxel’s neighbors, by using a Gaussiankernel at each endpoint and partitioning a tract across spatially proximal brain areasto model endpoint uncertainty. Both ODF estimation and global tractography weredone in the subjects’ native space. To draw a correspondence between the nativespace and the MNI space, we warped the target atlas onto the b= 0 volume of eachsubject before fiber counting.35Chapter 2Neighbourhood InformationEmbedding and MultimodalIntegration for ImprovedParcellationThis chapter focuses on improving parcellation, i.e., brain node definition, whichis based on papers [P1-3]. To generate a graph representation of the brain, properlydefining the brain nodes is critical. Parcellating the brain through subdivision of thebrain into sub-units that are internally homogeneous in certain criteria is a challeng-ing problem. Here, we propose to devise a reliable parcellation method and fusemultimodal information for parcellation. Specifically, we embed neighborhoodconnectivity information into the affinity matrix to ameliorate the adverse effectsof noise [P1]. Meanwhile, we integrate the connectivity derived from anatomicaland functional modalities based on adaptive weighting [P2-3].362.1 Neighborhood Connectivity Informed Parcellation2.1.1 Related WorkWith today’s wide availability of MRI scanners, most strategies are now designedaround using only MRI-based features for parcellation, especially connectivitybased parcellation [59, 66]. We here focus on unsupervised clustering methodsusing connectivity data, since there is no ground truth in parcellation applications.Such methods include ICA, Gaussian Mixture Model (GMM) [151], kmeans clus-tering and its fuzzy alternative [152], hierarchical clustering methods [153], regiongrowing [154], graph-based methods such as Normalized cuts (Ncuts) [155] andcommunity detection method [18], and boundary mapping [156]. The majority ofthese parcellation approaches rely on calculating an affinity matrix that encodes theconnectivity similarity between the image voxels. As such, defining an affinity ma-trix that accurately represents the underlying structure is central in clustering-basedparcellation algorithms.2.1.2 Neighborhood Information Embedded Affinity MatrixIn order to emphasize on discriminable features in the critical affinity matrix, wehave developed a novel approach by embedding neighborhood connectivity infor-mation into the affinity matrix. This approach serves the dual purpose of allowingself-adaptive adjustment of voxel affinity values and providing robustness againstnoise. Our rationale is that, being the sole input to most clustering methods, affinitymatrices should encapsulate as much relevant information as possible, rather thanjust average connectivity similarities. The advantages of our method are two-fold:First, the distinction between voxels at boundaries and those in interior regions,helps preclude pooling information from different parcels, which would result inambiguous affinity values. Second, such distinction enables emphasizing or de-emphasizing the affinity values based on putative cluster memberships of voxels,ameliorating the adverse effects of noise. The proposed affinity matrix can be usedwith any parcellation method that takes an affinity matrix as its input.372.1.2.1 Gaussian Kernel Affinity MatrixTraditionally, connectivity similarities or distances are mapped using a Gaussiankernel to produce an affinity matrix for clustering. Gaussian kernel function is atypical candidate in cases where the data points live in the Euclidean space whilekeeping the local information [157]. Thus, we transform the connectivity matrix Cusing a Gaussian kernel to better capture connectivity structure as below:Aij = exp(−d2ij2σ2), (2.1)where di j is the distance between voxels i and j, which we set to be 1−Cij, Cijis the estimate of the connectivity between voxel i and j, and σ is a parametercontrolling how rapidly Aij decays with increasing distance. Usually, σ is set tobe the average distance among neighboring voxels over the whole affinity matrix.However, the inherent assumption of fixed density distribution of affinity valuesrarely holds true in practice. To adaptively tune σ based on the local statistics ofvoxel neighborhoods, (2.1) has been modified as below in (2.2) as the multipledensity kernel, which has been proven to outperform the classical Gaussian kernel[158]:Aij = exp(−d2ijσiσj), (2.2)where σi and σj are local scaling parameters. Based on empirical observations,Zelnik et al.. suggested that setting σi to be the distance between voxel i and its 7thnearest neighbor outperforms the fixed density kernel [158].2.1.2.2 Proposed Affinity MatrixAlthough the multiple density kernel has been proven to outperform the classicalGaussian kernel, we argue that the choice of local scaling parameter σi is vulner-able to noise. Based on the multiple density affinity matrix, we propose a morerobust local density estimation approach and accentuate the difference betweenintra-cluster and inter-cluster affinity values. Essentially, each voxel in a clustercan be classified as either an Interior Point (IP) or a Boundary Point (BP). An il-lustration of IP and BP can be found in Figure 2.1. The neighbors of the latter havea heterogeneous cluster membership structure whereas all neighbors of the former38Figure 2.1: Illustration of IP (red dot) and BP (green and blue dots). Theneighbors of the latter have a heterogeneous cluster membership struc-ture whereas all neighbors of the former belong to the same cluster.belong to the same cluster.Hence, we propose a parameter K indicating voxel’s cluster membership by thelocal neighborhood information defined as below:Ki = Li−Si, (2.3)where we denote the distances between voxel i and its neighbors as {di}, the valuesbelow the 30th percentile of {di} as {di}S, and the values above the 70th percentileof {di} as {di}L, Li and Si are the median values of {di}L and {di}S. Intuitively, Kiis small when voxel i is an IP, and large when voxel i is a BP, since the spread of{di} will be larger if voxel i has neighbors from different clusters. With this defini-tion, which indirectly allows us to distinguish between IPs and BPs, we define ouraffinity matrix as Neighborhood-information-embedded Multiple Density (NMD)kernel:Aij = exp(−KiKjd2ijdidj), (2.4)where di is the average distance between voxel i and its neighbors in the samecluster, which is estimated as the one fourth of its nearest neighbors, replacing theσi in (2.2) referring to the distance between voxel i and its 7th nearest neighbor,which is prone to be erroneous in noisy neuroimaging data.This NMD kernel effectively modifies the Aij values based on the cluster mem-berships of voxels i and j. Specifically, it scales Aij up when i and j are in the39Figure 2.2: Theoretical illustration of NMD affinity matrix. Left is traditionalGaussian kernel, and the right is NMD. The affinity distribution, shownin the red bars, expands to a wider range when NMD is used.same cluster, and scales it down when they are not. If two neighboring voxels iand j belong to different clusters, they then must both be BPs. In this case, dijis large and large KiK j makes the numerator of the exponential function in (2.4)larger than in (2.2), scaling down Aij. In noisy cases, an adverse effect of noise isthat dij might decrease. However, Ki and K j are still large, which suppresses thenoise by keeping Aij relatively small. The opposite holds if i and j belong to thesame cluster. In this case, either both i and j are IPs, or one of them is a BP. Thiswill lead to a relatively small KiK j, scaling up Aij. In case that noise increases dij,KiK j mitigates the effect of noise by keeping relatively large Aij. The illustration ofhow NMD kernel modify the distribution of affinity values for clustering is shownin Figure 2.2.2.1.3 Experiments and ResultsTo evaluate our proposed approach, we apply Ncuts to both synthetic and real datafrom HCP dataset, comparing the self-tuning multiple density [158] in (2.2) withour proposed modified affinity matrix in (2.4). We opt to use Ncuts as the clusteringmethod since it has been shown to outperform GMM. Other clustering algorithmsare also applicable, but we have chosen Ncuts for its global optimality guaranteesso that our results are not prone to problems, such as instability to initializationin the case of kmeans clustering or obtaining only a local minimum in the caseof hierarchical Ward’s clustering. However, it is important to note that our pro-posed affinity matrix can be used in conjunction with any parcellation method thatoperates on affinity matrices.40For quantitative validation on synthetic data, we simulated a dataset compris-ing six horizontally connected clusters on a 3D grid of 5×5×30 voxels, with eachcluster comprising a cubic region of 5× 5× 5 voxels. Based on this configura-tion of voxels, we generated a binary N×N ground truth affinity matrix, whereN=750 is the total number of voxels. For each voxel, we simulated time coursesby randomly drawing samples from a multivariate normal distribution with zeromean and covariance structure given by the affinity matrix of the data. We addedGaussian noise to the time courses with the SNR set to -10 dB. This noise level wasdeliberately chosen to be low enough to assess the robustness of our method undersevere noise conditions. We repeated this process generating 50 noisy versions ofthe ground truth.We then compared the ground truth and the parcellations generated using ourapproach using Dice Similarity Coefficient (DSC) [159]. DSC is defined as (2|X ∩Y |)/(|X |+ |Y |) , where X is the index set of voxels in a given parcel from the firstparcellation (ground truth here), and Y is the index set of voxels in the matchedparcel from the second parcellation (the estimated parcellation here). X ∩Y is theset of voxels commonly assigned to the matched parcel pair. | · | is the cardinal-ity/size of the set. DSC lies between 0 and 1, with 1 indicating the parcel paircomprises exactly the same set of voxels. We perform parcel matching betweenthe two parcellations using Hungarian clustering [160] with DSC between eachpair of parcels as the similarity metric. All statistical comparisons are based on theWilcoxon signed rank test with significance declared at an α of 0.05 with Bonfer-roni correction.We show that higher clustering accuracy quantified by the average DSC can beattained with our modification of the multiple density affinity matrix, with statisti-cally significant improvement from 0.90 ± 0.038 to 0.94 ± 0.040.On the real data, cortical parcellation maps are challenging to validate due tothe lack of ground truth. However, assuming that there truly is a functional parcel-lation, it should presumably remain stable for each subject and consistent acrosssubjects [161]. Thus, we based our quantitative validation on intra-subject test-retest reliability (parcellations generated from two time courses of fMRI data ofthe same subject), and inter-subject test-retest reliability (parcellations from twodifferent subjects) of the resulting brain parcels, which we measured using DSC41Figure 2.3: Inter-subject test-retest reliability quantified using DSC in IPLand Cg for the HCP data. As reflected in the box plots, our methodconsistently outperforms the state-of-art method. The blue rectanglespans the first quartile to third quartile. The red line indicates the medianand the black whiskers indicate the minimum and maximum.after relabelling parcels using the Hungarian algorithm [162] to match labels. TheInferior Parietal Lobule (IPL) and Cingulate Cortex (Cg) have been used for vali-dation, since they possess great functional and anatomical heterogeneity. Here, weonly validated on fMRI data and set negative connectivity to zero before generatingthe affinity matrix, since there is a lack of anatomical evidence for negative corre-lations unlike positive correlations, mentioned in Section 1.8. We show that ourmethod exhibits statistically significantly higher intra-subject test-retest reliabilityfrom 0.90 ± 0.04 to 0.95 ± 0.03 for IPL and from 0.93 ± 0.02 to 0.98 ± 0.01for Cg. Our approach also achieved statistically significant improved inter-subjecttest-retest reliability with corresponding boxplot shown in Figure 2.3.Further, we qualitatively evaluate our method by visually comparing the groupparcellation map with those estimated from individual subjects. Our rationale hereis that the group map is expected to be more reliable since it is generated by pool-ing data across multiple subjects, effectively increasing the SNR. It can thus beused as pseudo ground truth to compare individual parcellation maps against. Forbrevity, we only present subjects showing the highest and lowest agreement with42(a) Inferior Parietal Lobule(b) Cingulate CortexFigure 2.4: Qualitative parcellation results for IPL and Cg using the HCPdata. In each area - Top: Using the multiple density in (2.2); Bottom:Using our proposed affinity matrix NMD in (2.4). The 1st column showsthe group maps while the 2nd and 3rd columns are the subject-specificmaps with the highest and lowest DSC with group maps, respectively.The significant differences between the subject-specific map having thelowest DSC (right) and the group map (left) are highlighted with arrows.the group map as measured by DSC in Figure 2.4. The group maps obtained using(2.2) and proposed affinity matrices using (2.4) did not exhibit major differences,implying that using multiple density [158] for parcellation suffices when there isenough data. However, the subject-specific map having the lowest DSC (top right)with the group map shows major differences compared to the group map (top left)when (2.2) is used. Specifically, both the number of parcels in the slice shown andthe parcel boundaries are significantly different. In contrast, using our proposedaffinity matrix results in much more consistent results (bottom left versus bottomright).2.1.4 DiscussionWe proposed a novel affinity matrix estimation for brain parcellation based on mul-tiple density kernel distribution. Our approach can be used in conjunction with anyparcellation method that takes an affinity matrix as its input. On synthetic data,43we demonstrated that our proposed affinity matrix (NMD) in equation (2.4) canbetter represent the data structure by capturing neighborhood connectivity leadingto more accurate results compared to the multiple density affinity matrix in equa-tion (2.2). On real data from HCP subjects, we demonstrated the superiority of ourmethod in terms of better intra-subject test-retest reliability and higher inter-subjecttest-retest reliability. Qualitatively, we showed that subject-specific parcellationmaps better resemble the group maps when parcellating using our affinity matrix.Our next step focuses on the extension to whole-brain parcellation and devising amultimodal integration approach.2.2 Multimodal Connectivity Fusion for Parcellation2.2.1 Related WorkThe majority of recent studies focuses on using fMRI data for parcellation [154,155, 163, 164]. A general limitation of these unimodal approaches is that they donot capture other brain attributes, such as fiber pathways (serving as the physicalsubstrate for functional interactions), which provide additional indication of howthe brain is organized. Incorporating this information would presumably providemore reliable parcellation than using fMRI data alone. Towards this end, consen-sus maps have been generated from overlaps among probabilistic parcellation mapswhich were derived from different modalities [165]. To the best of our knowledge,we are the first to propose integration of multimodal information for brain parcel-lation in 2015 [2], and we recently extended our strategy in [3]. Cortical surfaceparcellation approaches based on multimodal information have recently been pro-posed [54, 121]. We note that our multimodal volumetric parcellations could serveas excellent complements to these two recent multimodal cortical surface parcel-lation [54, 121], especially considering the importance of subcortical regions forclinical investigations, such as Parkinson’s disease studies.2.2.2 Proposed Multimodal Integrated Parcellation FrameworkMost existing parcellation methods use structural information (e.g., gyri and sulci),FC, or AC alone to group voxels into parcels. Each of these modalities has its own44Figure 2.5: Flowchart of proposed approach for multimodal brain parcella-tion. AC and FC estimates are generated for each voxel to derive voxel-by-voxel connectivity matrices followed by a distribution normaliza-tion. The resulting matrices are integrated to generate multimodal con-nectivity estimates. Ncuts is applied on the integrated similarity matri-ces, to generate whole-brain parcellation.inherent limitations. For instance, structural attributes might not relate to brainfunction, FC estimates are prone to false positives, and AC estimates are proneto false negatives. We thus examine whether combining information across thesebrain attributes could alleviate the limitations of each modality. To prevent biastowards any particular task, we use Resting State Functional Connectivity basedon MRI data, which captures intrinsic functional connectivity. At the same time,we use dMRI data to estimate AC. In order to incorporate structural informationon gyri and sulci, we present a region level extension of our approach.We propose an approach for integrating multimodal information, as summa-rized in the following and the corresponding workflow is shown in Figure 2.5.• Generate AC and FC estimates for each voxel to derive voxel-by-voxel con-nectivity matrices.• Map AC and FC estimates to a common value range using a distributionnormalization function.• Integrate these matrices based on adaptive weighting, where the weights arederived from the reliability of the AC and FC estimates.45• Apply standard clustering algorithms, such as Ncuts, on the integrated simi-larity matrices, to parcellate the brain.Our multimodal approach can be used for both subject and group level parcel-lation. For subject parcellation, we derive the integrated similarity matrix (derivedfrom the connectivity matrix) from each subject’s AC and FC estimates. For grouplevel parcellation, we average the AC and FC estimates across subjects prior toAC-FC integration.2.2.3 Anatomical and Functional Connectivity EstimationOur AC estimation is based on a fingerprint concept [65], that the voxels havingsimilar fiber connection fingerprints should have higher connectivity strength. Thistype of connectivity estimation can incorporate both node-to-node relationship andthe fingerprint pattern reflecting the relationship between one node and the remain-ing regions.Given the estimated fiber tracts (derived by following the steps in Section 1.8),we first define an AC fingerprint for each voxel as the number of tracts connectingthat voxel to a set of target regions. Target regions can be taken as every othervoxels or region of interest from existing atlases. We use the 112 regions in thewell-established HO atlas, because it consists of the highest number of subjectswith both manual and automatic labelling technique compared to other commonlyused anatomical atlases [166]. AC is then estimated as the cross-correlation be-tween the AC fingerprints of each voxel pair, the resulting matrix is denoted as CA.Using the HO atlas as target regions has multiple advantages. First, this strategyreduces the number of unknowns to be estimated, i.e., the number of elements ineach AC fingerprint is reduced from >100,000 to 112. Second, using ROIs insteadof voxels provides more tolerance for uncertainty in fiber endpoint location [13].Moreover, this strategy lowers computational cost.We note that we spatially normalize the HO atlas to the subject’s native spaceto avoid distorting the dMRI volumes, which impacts the qBall estimation. To con-strain the parcels to be spatially-contiguous, we keep only AC estimates of the 26neighbors for each voxel, which include voxels of one-ring neighbors in Cartesiancoordinates [164, 167]. The spatial contiguity within estimated parcels ensures46the representation of anatomically homogeneous regions, and hence preserve theinterpretability of the connectivity results [168]. Additionally, the spatial contigu-ity distinguish network nodes from large-scale networks of nodes, i.e., subnetworkextraction [168].On the other hand, the FC estimates, denoted as CF , are derived by followingthe steps in Section 1.8. Let Z be a t×N matrix of preprocessed rs-fcMRI timecourses, where t is the number of time points and N is the number of voxels ofinterest. We estimate the FC matrix using Pearson’s correlation: CF = ZT Z/(t−1). Also, only CF of the 26 neighbors of each voxel are retained to enforce spatialcontinuity.For our estimation of both the AC and FC, we set negative values in the bothconnectivity matrices to zero due to the difficulty in interpretingation of negativeconnectivity (there is a lack of anatomical evidence for negative correlations unlikepositive correlations) [93].2.2.4 Distribution NormalizationIt is important to note that naive combination of AC and FC values may not besuitable, since their estimates have different distributions and there is a lack ofstraightforward correspondence between these two estimates, Figure 2.6. Takevalue 0.5 for example, which is in the overlapping area in Figure 2.6, it refers tohigh correlation in the FC distribution, but low correlation in the AC distribution,leading to a misleading combined correlation value. This difference is importantto take into account to eliminate the risk of overweighting one modality over theother [2].To ensure no possible overweighting in generating combined connectivity es-timate occurs, we apply a nonlinear mapping based on distribution normalizationto gain a better correspondence between the two modalities. We use histogrammatching to match the distribution of AC values to that of FC [2].Let a1 ≤ a2, ...,≤ aM be the raw AC values, and f1 ≤ f2, ...,≤ fM be the FCvalues, where M is the number of voxel pairs, we compute the histograms of ACand FC values, as well as their cumulative distribution functions Fa(x) = P(X ≤x),x ∈ [a1,aM] and Ff (y) = P(Y ≤ y),y ∈ [ f1, fM]. P(·) is the probability that the47Figure 2.6: Histograms of FC and AC derived from the group data of 77 HCPsubjects. The difference between the two distributions could potentiallycause bias if the FC and AC are naively fused.random variable X takes on a value less than or equal to x. Next, we replace eachAC value ai with the FC value f j, which satisfies Fa(ai) = Ff ( f j) as:φ(a | Fa(a) = Ff ( f )) = f . (2.5)This distribution normalization procedure enables unbiased integration of ACand FC, where each mapped AC value is used in the multimodal connectivity esti-mation in the following section.2.2.5 Multimodal Connectivity Estimation using Adaptive WeightingTo overcome the inherent limitations associated with each of the two modalities,we propose combing connectivity estimated from two modalities using an adaptiveweighting scheme based on voxel-wise reliability:CMij = (1−wij)CFij +wijφ(CAij), (2.6)where CMij is the Multimodal Connectivity (MC) estimates between voxels i and j,CAij and CFij are AC and FC estimates, and wij is the voxel-wise adaptive weight.The adaptive weighting which enables us to integrate multimodal connectivityestimates, affectively weighs down the contribution of FC in voxels where fMRI48observations are deemed to be unreliable and AC estimates are reliable. We definethe integration weights based on a connection reliability ρ as:wij = (1−ρFij )ρAij , (2.7)where we construct a measure of connection reliability ρ based on the local voxelreliability γ of the FC and AC estimates defined as follows. For FC, we split thetime course of each voxel i into two halves and compute the Pearson’s correlationbetween its local fingerprints (i.e., similarity between a voxel and its 26 neighbors)derived from these two halves, which we take as the FC voxel reliability γ [2]. ForAC, we extract two subsets of dMRI data by downsampling the single qBall acqui-sitions by minimum distance pair matching (see Reproducibility in Section 2.2.7).We then generate local fingerprints for the two subsets and use their Pearson’s cor-relation as the AC voxel reliability. Note that we use the information from theimmediate 26 one-ring neighbors of a given voxel, as it has been shown that largerneighborhoods might make the analysis more prone to acquisition and registrationartifacts [114]. The reliability of an edge connection is implicitly bounded by oneof the two end voxels having relatively lower reliability. As such, we define ρij asthe minimum reliability of edge end voxels i and j as:ρij = min(γi,γj) (2.8)where γi is the reliability of voxel i, with γi derived from either FC or AC.2.2.6 Clustering and Region Level ExtensionWe apply Ncuts on A to generate brain parcellation. We opt to use Ncuts for itsglobal optimality guarantees to avoid instability to initialization and local mini-mum as mentioned in Section 2.1.3. With regard to the number of parcels, thereare indeed data-driven strategies to select the number of parcels, e.g. based on pre-dictability on leftout data [68] and stability over data splits [69]. However, thesecriteria tend to return different numbers of parcels depending on the number ofsamples (time samples/subjects) available. Also, the number of parcels largely de-pends on the application and ease of result interpretation. Hence, instead of finding49an “optimal” number, we adopt a multi-scale perspective by examining a range ofvalues 256, 512, 1024. 256 parcels is examined to compare with existing anatom-ical atlases [166], which have between 100 to 200 regions. 512 parcels is selectedbased on prior studies on architectonic [70] and functional [71] information thatsuggest the human brain comprises ∼500 regions. 1024 parcels is chosen sincefinely grained brain atlas e.g. with ∼1000 parcels has also been used in well-established studies [114].We further present a simple region level extension of our parcellation approachthat substantially reduces computational cost and simplifies the problem of parcelcorrespondence between subjects, while having a side-advantage of incorporatingstructural information on gyri and sulci. Specifically, we decompose the whole-brain parcellation problem into multiple ROI parcellation subproblems, by sepa-rately parcellating each ROI of the HO atlas (which was derived based on sMRIinformation). Exactly the same procedure used in whole-brain parcellation can beapplied at region level. The number of parcels within each ROI is set based on theproportion of voxels in the brain covered by the given ROI.2.2.7 Design of Evaluation Metrics2.2.7.1 ReproducibilityBased on the assumption that a parcellation exists, the parcellation should pre-sumably remain stable for each subject and consistent across subjects [161], weevaluate the parcellations on three types of reproducibility, namely intra-subjecttest-retest reliability, inter-subject test-retest reliability, and subject-group consis-tency.To measure intra-subject test-retest reliability, we first divide the rs-fcMRI dataand dMRI data into two splits and generate two subject-specific parcellations fromeach data split. For functional data, we use two rs-fcMRI scans accessible fromHCP dataset to assess reproducibility. In terms of diffusion data, we propose amethod for splitting a dMRI dataset into two subsets to assess reproducibility.Given a dMRI dataset from a uniformly distributed qBall sampling scheme, wefirst find all pairs of closest orientations based on minimum distance. We then use50local greedy search to assign each element i of an orientation pair to the subsetwith elements farthest away from element i. Iterating this procedure generates twoexclusive sets of approximately uniformly distributed orientations. We then com-pare the two parcellations using DSC, as defined in Section 2.1.3. The DSC ofall matched parcel pairs are then averaged and taken as the intra-subject test-retestreliability.For inter-subject test-retest reliability, exactly the same procedures are per-formed except we compare group parcellations generated from random half splitsof the subjects. 10 random splits are performed.Moreover, we assess the subject-group consistency with the same proceduresby comparing each subject level parcellation and the group level parcellation (gen-erated from all subjects’ data).Furthermore, we assess the stability of the group parcellations with respect tothe number of subjects, Ns, with Ns set between 5 and 75 at 5 subjects increment.The DSC between the group parcellations built with all subjects vs. Ns subjects isused to estimate stability.2.2.7.2 Functional HomogeneityTo assess functional homogeneity, we employ two metrics, namely connectivityhomogeneity [163] and temporal homogeneity [155]. The first metric evaluateshow much variance in the parcel FC patterns is captured by the largest principalcomponent (FC pc ratio). If all voxels within a parcel have similar FC patterns, thenthe largest principal component would capture a large proportion of the variance.The other metric (mean distance) evaluates the homogeneity of temporal patternsbased on the Euclidean distance between the fMRI time courses of each voxelwithin a parcel and the mean time courses across all voxels within that parcel. Ifall voxels have similar temporal patterns, then their Euclidean distance from themean should be small. For both metrics, the averages over parcels are reported.2.2.7.3 Leftout Data LikelihoodAccuracy of connectome estimation heavily depends on brain parcellation. Thus,another way to evaluate the parcellations is to assess the generalizability of con-51nectivity estimates derived from one scan to an unseen scan. For this, we employthe leftout log data likelihood [169]. Given a parcellation generated from the firstscan (using both rs-fcMRI and dMRI data for multimodal approach), we generateparcel time courses for each subject by averaging the voxel time courses withineach parcel. Assuming a multivariate Gaussian distribution, we then estimate theinverse covariance, K, from the first scan and the sample covariance, S, from thesecond scan. The leftout log data likelihood is given by:log(1n√2pidet(K)exp(− tr(SK)2)), (2.9)where det(·) is the determinant, tr(·) is the trace, and n is the number of samplesused to compute S. The role of the first and second scans were then switched, withthe average of the two likelihood estimates reported.2.2.7.4 Overlaps with CytoarchitectureWe examine the overlaps between the boundaries of our parcels and the arealborders of the Juelich probabilistic cytoarchitectonic mapping thresholded at 0.25[170]. Parcels with 70% of its volume lying within a given cytoarchitectonic areaare considered as part of that area. To avoid false boundary alignments arising fromthe structural constraints of the HO atlas, we assess overlaps between the cytoar-chitectonic areas and the multimodal group parcellation generated without usingthe region level extension.2.2.7.5 Subnetwork ExtractionTo attach a functional meaning to the parcels, we group the parcels into subnet-works and correspond the subnetworks to established brain systems [18]. Specif-ically, we first compute the Pearson’s correlation matrix between the rs-fcMRIparcel time courses and apply a threshold of 0.2, which helps reduce the influencefrom non-significant connections [171]. Ncuts is then applied to group the parcels.To set the number of subnetworks, we search from 5 to 45 at a step size of 1, andchoose the value that maximizes modularity. The optimal number of subnetworksfound with this scheme is 9. We also examine setting the number of subnetworks52to 25 following previous studies [18]. We further use the same procedures of inter-subject reproducibility calculation described in Section Reproducibility to assessconsistency of subnetworks generated from two exclusive datasets (two subsets ofQ3 from HCP). We apply Hungarian clustering to match the two sets of subnet-works with DSC between node labels subnetworks used as the similarity metric.2.2.8 Experiments and ResultsWe evaluated the proposed multimodal parcellation approach based on reproducibil-ity, functional homogeneity, leftout data likelihood, and cytoarchitectonic overlap.We further grouped the parcels into subnetworks and associated them to well estab-lished brain systems [18]. Multimodal parcellations at various resolutions derivedusing the whole-brain and region level schemes as well as parcels’ subnetwork la-bels are provided in the supplemental materials and (will be made) available online.Note that we empirically set wij to 0 when ρij derived from AC dropped below 0.5to only incorporate reliable AC. For comparisons against our multimodal approach(MC), we examined unimodal parcellations based on FC, AC, and physical dis-tance (DC), random parcellation (RM) with roughly equal-sized parcels [53], aswell as other existing atlases based on functional information [155, 163, 164] andanatomical information [57, 172] (HO and Automated Anatomical Labeling (AAL)atlases). Unless stated otherwise, all results are based on parcellations generatedusing the region level extension for both unimodal and multimodal strategies withthe number of parcels set to 512. The parcellation derived based on our multimodalapproach at the 512 scale at the group level has been shown in Figure 2.7 a. All sta-tistical comparisons are based on the Wilcoxon signed rank test with significancedeclared at an α of 0.05 with Bonferroni correction.2.2.8.1 ReproducibilityTo assess intra-subject test-retest reliability, we first generated two brain parcella-tions for each subject from the two rs-fcMRI scans and two subsets of the dMRIvolumes using our multimodal approach. The average DSC across parcels be-tween the two parcellations was used to estimate test-retest reliability. We com-pared multimodal parcellations against those generated solely based on fMRI data.53(a) Multimodalgroup parcellation(b) Multimodalsubject parcellation(c) Unimodal groupparcellation(d) Unimodal sub-ject parcellationFigure 2.7: Group-wise parcellation and Subject-group consistency. Both theparcellations based on the proposed multimodal approach (left two) andunimodal FC based parcellations (right two) are displayed at 512-parcelscale in 2D space. Second and fourth column: Subject level parcella-tion having the lowest DSC with the group parcellation. The proposedmultimodal parcellations attain higher subject-group consistency (a &b), compared to unimodal FC based parcellations (c & d), which wehighlight certain areas using blue circles.Multimodal parcellations achieved a DSC of 0.949 ± 0.009, which is significantlyhigher (p = 0.00475) than that generated with the unimodal FC approach (0.939 ±0.011). The same trend was observed with the number of parcels set to 256 (MC:0.980± 0.0078 against FC: 0.978± 0.0078 at p = 0.0017), and 1024 (MC: 0.9091± 0.01 against FC: 0.9040 ± 0.01 at p = 1.46e-7), Figure 2.8.We next estimated inter-subject test-retest reliability of the group parcellationsby first randomly splitting the 77 subjects into 2 halves in 10 different ways. Foreach random subject split, we computed average DSC across parcels between thepair of group parcellations. Multimodal group parcellations achieved a signifi-cantly higher DSC of 0.9507 ± 0.0049 (p = 0.0098) compared to that of FC basedparcellations (0.9419 ± 0.0053). The trend persisted with the number of parcelsset to 256 (MC: 0.9906 ± 0.0022 against FC: 0.9799 ± 0.0030 at p = 0.0020),and 1024 (MC: 0.9336 ± 0.0026 against FC: 0.9178 ± 0.0038 at p = 0.0019).We observed the difference of inter-subject test-retest reliability in different brainregions, Figure 2.9.54Unimodal Multimodal0.960.970.980.99DSC(a) M=256Unimodal Multimodal0.920.930.940.950.960.97DSC(b) M=512Unimodal Multimodal0.880.90.92DSC(c) M=1024Figure 2.8: Intra-subject test-retest reliability using the DSC as the evalua-tion criterion. The number of parcels M was set to 256, 512 and 1024,respectively. The blue rectangle spans the first quartile to third quar-tile. The red line indicates the median and the black whiskers indicatethe minimum and maximum. Multimodal parcellations achieved signif-icantly higher test-retest reliability than those generated based on fMRIdata alone.(a) X=88mm (b) X=72mm (c) X=56mmFigure 2.9: The inter-subject test-retest reliability (range from low red to highgreen) of each brain region of the Harvard-Oxford atlas (only shown onsagittal slices X=88mm, 72mm and 56mm as exemplars). Regions, suchas the Lingual Gyrus (LG), Lateral Occipital Cortex (LOC), PostcentralGyrus (PG), Frontal Pole (FP), and Superior Frontal Gyrus (SFG) showrelatively lower reproducibility.55(a) Frontal pole (right) illustrated as an insetwithin the whole brain (left)(b) Unimodalgroup parcellation(c) Unimodalsubject parcellation(d) Multimodalgroup parcellation(e) Multimodalsubject parcellationFigure 2.10: Subject-group consistency. Comparison between the group par-cellation and the subject parcellation of frontal pole having the lowestDSC with the group parcellations.We further evaluated the consistency between subject level and group levelparcellations, described in Reproducibility in Section 2.2.7. Quantitatively, mul-timodal approach significantly improved the subject-group consistency from 0.872± 0.0099 for the unimodal approach to 0.883 ± 0.0105 (p = 0.00138) at the scaleof 512 parcels. We observed similar trend with the number of parcels set to 256(MC: 0.8876 ± 0.0127 against FC: 0.8826 ± 0.0129 at p = 2.46 e-14), and 1024(MC: 0.5098 ± 0.011 against FC: 0.5069 ± 0.011 at p = 1.523 e-13).Qualitatively, we compared the subject parcellation having the lowest simi-larity with the group parcellation based on Dice coefficient (the worst case). Weobserved that multimodal parcellations have higher subject-group consistency thanits unimodal counterpart (Figure 2.7). To more clearly illustrate the differences, weplotted the parcellations of an exemplar ROI, namely the frontal pole of the righthemisphere in Figure 2.10. Major differences between the subject (Figure 2.10 c)and the group parcellations (Figure 2.10 b) were observed for the unimodal ap-560 10 20 30 40 50 60 70 800.90.920.940.960.981Inter-subject StabilityFigure 2.11: Stability with respect to the number of subjects in generatinggroup parcellations. We compared the multimodal group parcellationsgenerated from all 77 subjects against NS subjects with NS set between5 and 75 at interval of 5 subjects. Stability was estimated using theDice coefficient. The number of parcels was set to 512. The stabili-ties of the group parcellations as measured using the Dice coefficientplateaued at 0.96 after NS = 50.proach in both the number of parcels and the parcel boundaries, whereas substan-tially more consistent parcels were found with multimodal approach (Figure 2.10d & e).Moreover, we examined the stability of the group parcellations with respect tothe numbers of subjects. Specifically, we compared the multimodal group parcel-lations generated from all 77 subjects against using only NS subjects with NS setbetween 5 and 75 at interval of 5 subjects. The stabilities of the group parcellationsas measured using the DSC plateaued at 0.96 after NS = 50, Figure 2.11.2.2.8.2 HomogeneityWe assessed the functional homogeneity using the FC pc ratio and the mean dis-tance, defined in Section 2.2.7 Functional Homogeneity. We compared our multi-modal parcellations against unimodal parcellations derived from AC, FC, DC, andRM. Multimodal parcels achieved significantly higher functional homogeneity inboth FC pc ratio and mean distance (0.3751 and 67.2022) than those based on AC(0.3681 and 67.4321), DC (0.3695 and 67.6756), and RM (0.3461 and 68.1732),57and comparable to parcels based on FC (0.3750 and 67.2028), Figure 2.12. Sim-ilar trend was observed with the number of parcels set to 256 and 1024. Whenthe number of parcels was set to 256, multimodal parcels achieved significantlyhigher functional homogeneity in both FC pc ratio and mean distance (0.3493and 67.0495), than AC (0.3482 and 67.1005), DC (0.3393 and 67.4590), and RM(0.3335 and 67.4682) at p < 0.05 based on the Wilcoxon signed rank test, andcomparable to parcels based on FC (0.3493 and 67.0497). When the number ofparcels was set to 1024, multimodal parcels achieved significantly higher func-tional homogeneity in terms of both FC pc ratio and mean distance (0.3942 and66.7457) than FC (0.3938 and 66.7591), AC (0.3936 and 66.7536), DC (0.3937and 66.7883), and RM (0.3819 and 67.0490) at p < 0.05. To compare against ex-isting atlases [57, 155, 163, 164, 172], we also generated parcellations with com-parable number of parcels. Details on existing anatomical and functional atlas canbe found in Table 2.1 and Figure 2.13. We note that to compare against the AALand HO atlases, which has about a hundred parcels, we could not use the regionlevel strategy, i.e., dividing regions of the these two atlases will obviously resultin more regions than the original one. We thus instead used the whole-brain par-cellation approach without the region level strategy for comparison against AALand HO atlases. Multimodal parcellations achieved significantly higher functionalhomogeneity compared to existing atlases except for Gordon’s atlas [163] withp = 0.2372 (Figure 2.14), whose higher functional homogeneity is likely due tothe bias arising from parcellating only a thin shell of the cortex (see Figure 2.13).Specifically, grouping fewer voxels into the same number of parcels would result insmaller parcels, which inherently would have higher functional homogeneity thanlarger parcels (i.e., more voxels would introduce more variability).2.2.8.3 Leftout Data LikelihoodWe further evaluated the multimodal parcellations with leftout log-data likelihood.The mean leftout log data likelihood of multimodal parcellations was higher thanthat of FC and AC though not statistically significant (p = 0.53 and p = 0.37). RMand DC attained higher leftout log-data likelihood than AC, FC and MC. The trendmoderately changed with the number of parcels set to 256 and 1024. Specifically,58(a) Parcellation generated from subset one datawith its parcels’ functional homogeneity estimatedon subset one data.(b) Parcellation generated from subset one datawith its parcels’ functional homogeneity estimatedon subset two data.(c) Parcellation generated from subset two datawith its parcels’ functional homogeneity estimatedon subset two data.(d) Parcellation generated from subset two datawith its parcels’ functional homogeneity estimatedon subset one data.Figure 2.12: Functional homogeneity comparisons between multimodal andunimodal parcellations. In order to show consistent gain in each caseof comparisons, plots of MC-FC, MC-AC, MC-DC and MC-RM areprovided. Positive FC pc ratio and negative mean distance indicatehigher functional homogeneity gained from multimodal parcellations.Number of parcels were set to 512. Multimodal parcels achieved sig-nificantly higher functional homogeneity in both FC pc ratio and meandistance, see text for details.(a) HO (b) AAL (c) Gordon (d) Shen (e) CradockFigure 2.13: Existing anatomical and functional parcellations59HO Our AAL Our Gordon Our Shen Our Crad Our0.250.30.350.40.45FC pca %(a) FC pc ratioHO Our AAL Our Gordon Our Shen Our Crad Our6767.56868.569Mean Distance(b) Mean distanceFigure 2.14: Functional homogeneity comparison between multimodal par-cellations and existing atlases. Number of parcels for multimodal par-cellation was set to the exact number of parcels in the contrasted at-las. Contrasted atlases included HO (112 parcels), AAL (116 parcels),Gordon’s (333 parcels), Shen’s (268 parcels), and Craddock (190parcels). Multimodal approach achieved significantly higher homo-geneity against the contrasted atlases except for Gordon’s.60Table 2.1: Details on existing anatomical and functional parcellations.Name Reference # of parcels NotesHO [57] 112 Volumetric formatAAL [172] 116 Volumetric formatGordon [163] 333 Surface to volume mappingShen [155] 268 Volumetric formatCraddock [164] 190 Volumetric formatat the scale of 256 parcels, the likelihood of MC was higher than FC and DC, andsignificantly higher than AC. At the scale of 1024 parcels, the likelihood of MCwas significantly higher than FC, AC, and DC (Figure 2.15). Here, RM and DC atsome scales perform better due to the regular sampling of volume of interest fromparcels with roughly equal size [169].2.2.8.4 CytoarchitectureWe examined whether multimodal parcellations (without applying the region levelextension) match cytoarchitectures by comparing them to the areal borders of theJuelich probabilistic cytoarchitectonic atlas (thresholded at 0.25) [170], Figure 2.16.We opted to use parcellations generated without applying the region level strat-egy for this analysis since regional boundaries in the HO atlas might be inher-ently aligned to some of the cytoarchitectonic areas, hence positively biased. Forclearer visualization, we mapped the multimodal parcellations from MNI space tothe PALS midthickness surface using the Caret software [20]. Since the Juelichatlas does not cover the whole brain, we focused on well-established areas, namelythe primary somatosensory cortex including area 1, 2 and 3a and 3b; premotor cor-tex including area 6; visual cortex area including V1, V2, V3v, V4; and V5 andBA 44/45 area (Broca’s area).The areas in cytoarchitectonic atlas are bordered in black dotted lines, andparcels having at least 70% overlap with cytoarchitectonic areas are in solid col-ors. The parcel boundaries align well with the primary somatosensory area, thoughpart of the parcels extended into the posterior boundaries of the postcentral gyri,61(a) M=256 (b) M=512 (c) M=1024Figure 2.15: Leftout log-data likelihood comparison between multimodaland unimodal parcellation. Number of parcels was set to 256, 512and 1024, respectively. (a) The likelihood of MC was higher than FCand DC, and significantly higher than AC at the scale of 256 parcels.(b) The likelihood of MC was higher than FC and AC at the scale of512 parcels. (c) The likelihood of MC was significantly higher thanFC, AC and DC at the scale of 1024 parcels. RM performs better dueto the regular sampling of volume of interest from parcels with roughlyequal size.Figure 2.16 a. The Broca’s area is largely covered by the parcels, with part ofthe parcels extending into the frontal cortex, Figure 2.16 b. The parcels also con-formed well to the histological areas for the majority of the visual cortex, especiallythe primary visual cortex across the central visual field, Figure 2.16 c. For outervisual cortex, the parcels slightly extend beyond the histological areas. The parcelswell cover the histological areas of sensorimotor areas with each parcel approxi-mately corresponding to a functional subdivision of the motor map that representsa different part of the human body, the homunculus [173], Figure 2.16 d.62Figure 2.16: Comparison of the parcel boundaries to exemplar cytoarchitec-tonic areas. The areas in cytoarchitectonic map are bordered in blackdotted line, and the parcels having at least 70% overlap with the cy-toarchitectonic areas are displayed using solid color based on the ‘RG-BYR’ colormap in Caret [20].2.2.8.5 Subnetwork StructureTo annotate functions to the multimodal parcels, we grouped them into subnet-works using Ncuts (see Section 2.2.7 Subnetwork Extraction) and associatedthem to well established brain systems [18]. This way, the users can more eas-ily describe and interpret their results, e.g., parcels in the primary motor networkwere found active. Following [18], we grouped the parcels into 25 subnetworks,Figure 2.17 a. At this resolution, we found the Visual Network (two subnetworks inpurple and spring green), Auditory Network (grass green), “Hand” Somatosensory-Motor Network (yellow), “Mouth” Somatosensory-Motor Network (purple blueconnected to yellow), Dorsal Attention Network (orange), Frontoparietal Task Con-trol Network (blue), and Default Mode Network (red in frontal lateral and medialarea, posterior lateral area, and posterior medial area). We also found a number63(a) Parcels grouped into 25 subnetworks(b) Parcels grouped into 9 subnetworksFigure 2.17: Extracted subnetworks. Subnetworks, such as Visual Network(light green in Occipital Cortex), Ventral Motor and Sensory sys-tem and Auditory system (orange), Dorsal Motor and Sensory system(spring green), and the Default Mode network (dark red) have beenfound in (b).of less well-known systems (light purple) implicated in memory retrieval that havebeen detected in more recent studies [18, 69]. Furthermore, we tested the con-sistency of subnetwork structure generated from the two exclusive datasets fromHCP. The subnetwork overlap was 80.26%. Note that based on maximal modular-ity, the “optimal” number of subnetwork was found to be 9 for the datasets used,Figure 2.17 b.642.2.9 Discussion2.2.9.1 Purposes and Key Challenges of Multimodal ParcellationThrough evolution, the human brain has been structured in a certain way for ex-ecuting various functions. The cortical foldings, the white matter wiring, and theelectrical and chemical signaling mechanisms all pertain to this complex organi-zation. Thus, jointly exploiting all these brain attributes should theoretically gen-erate parcellations that better resemble the brain’s inherent division as comparedto examining each attribute in isolation. One of the key challenges to adoptinga multimodal approach is drawing the “right” balance between different brain at-tributes. We proposed basing this weighting on test-retest reliability of each at-tribute, which we empirically showed to work well in practice. More knowledgeof the relationship between different attributes warrants further research. Anotherkey challenge, which applies to brain parcellation in general, is the choice on thenumber of parcels. Choosing a criterion for finding the “ground truth” number ofparcels is nontrivial. In fact, whether such a number exist is debatable, i.e., thischoice is largely a matter of the spatial resolution that is suitable for one’s analy-sis. Our take on this issue is not to settle on any particular number, but to adopt amulti-scale strategy. Findings that are consistent across scales are less likely to befalse positives. We observed that the reproducibility Dice coefficients decreased asthe number of parcels increase (from 256, 512 to 1024), since the combinatorialincrease in possible voxel grouping decreases the probability of accurate assign-ments.2.2.9.2 Multimodal Parcellation Improves Reproducibility and DataLikelihood, and Maintains Functional HomogeneityOur results show that complementing functional connectivity with anatomical con-nectivity and structural information improves the reproducibility of the parcella-tions. Our reproducibility analysis also addresses the important question of par-cellation stability with respect to sample perturbations. The high inter-subjecttest-retest reliability obtained shows that our multimodal scheme produces con-sistent parcellations for different subject subsets. Although multimodal informa-65tion would presumably disrupt maximization of functional homogeneity, the func-tional homogeneity of our multimodal-based parcellations was on par with purefunctional parcellations, and higher than those based on anatomical connectivity,physical distance, randomly generated, and most of the existing atlases. The onlyexception is Gordon’s parcellation [163], but its higher functional homogeneity islikely due to the bias arising from parcellating only a thin shell of the cortex (seeFigure 2.13). Further, the multimodal-based parcellations achieved higher leftoutdata likelihood than parcellations based on AC and surprisingly FC as well eventhough this evaluation metric is solely based on functional data.2.2.9.3 Linking Parcels to Prior KnowledgeThe subnetworks extracted from our multimodal-based parcellations match wellwith a good number of established functional systems [18, 69], which enabled usto assign neuroanatomically-meaningful labels to the parcels. Currently, we usedonly the functional data for subnetwork extraction. An important next step (pre-sented in Chapter 5) would be to extend our multimodal scheme in grouping theparcels into subnetworks, and compare that with pure functional subnetwork ex-traction as well as existing multimodal methods. Multimodal parcel boundariesalso align with those of cytoarchitectonic areas, despite that we did not explicitlyoptimize for this overlap in multimodal parcellation scheme. We suspect this over-lap arose from the correlation between cytoarchitecture and brain function [174].As for the misalignments, a likely reason is inter-subject misregistration. Take theBA 44/45 area for example, the misregistration errors are in the order of 6–12 mmreported in previous studies [61]. Also, there exist transition regions between cy-toarchitecture areas [61] (not display in Figure 2.16). We further note that eachcytoarchitectonic area is covered by multiple parcels, as opposed to a single parcel,which is likely due to functional subdivisions within each cytoarchitectonic areathat are not apparent from the cytoarchitectonic attributes [163].2.2.9.4 Region Level vs. Whole-brain ParcellationWe presented a region level extension to our multimodal parcellation approach,which greatly reduces computational cost and provides the side-benefit of incorpo-66rating structural information on gyri and sulci. This extension, which uses whole-brain connectivity to separately parcellate each HO region, intrinsically boostsreproducibility since both the number of voxels and the number of parcels aresubstantially reduced, which decreases the possible grouping arrangements, sim-plifying the problem of parcel correspondence between subjects. Also, part ofthe boundaries for some parcels would necessarily be confined to the boundariesof the predefined regions. To ensure fair comparisons, we applied exactly thesame region level strategy for the contrasted unimodal parcellation methods andshowed that multimodal information indeed provides substantial gains. Also, thisregion level strategy reduced computational time from 30 minutes for simultaneouswhole-brain parcellation to 10 minutes with a 64-bit Windows machine on an IntelXeon E3-1240V2 3.40GHz processor with 32GB of RAM. The main caveat is thatbrain units spanning multiple predefined structural brain regions would be split.2.3 SummaryTowards improving the brain node definition, i.e., parcellation, we strive to tacklethe challenges mentioned in Section 1.5 from two aspects.In order to devise a more reliable method for parcellation, we embedded neigh-borhood connectivity information into the affinity matrix to ameliorate the adverseeffects of noise. Our proposed approach produces parcellations with better intra-subject test-retest reliability, higher inter-subject test-retest reliability and highersubject-group consistency.We investigated the important question of whether combining modalities canimprove brain parcellation. We proposed an approach for integrating anatomicaland functional connectivity information for brain parcellation. Applying our mul-timodal approach to the HCP data, we quantitatively showed the superiority ofmultimodal parcellation compared to unimodal parcellation and existing atlases interms of reproducibility, functional homogeneity and leftout data likelihood. Mul-timodal parcellations also overlap with known cytoarchitetonic areas and the ex-tracted subnetworks matched well with the established brain systems. Collectively,our results demonstrated that integrating multiple brain attributes that intrinsicallyreflect the brain’s organization does indeed improve brain parcellation. Our mul-67timodal volumetric parcellations could thus serve as excellent complements to therecent cortical multimodal surface parcellations, especially considering the impor-tance of subcortical regions for clinical investigations, such as Parkinson’s diseasestudies.68Chapter 3Noise Reduction for BrainNetwork Edge BuildingThis chapter focuses on improving brain edge building, which is based on paper[P4] and the first part of paper [P5]. The connectivity estimates depend greatlyon the quality of the fMRI or dMRI data, both of which suffer from pronouncednoise and image resolution limitations. To tackle the problems in reducing the falsenegatives and positives in connectivity estimates, we propose a matrix completionbased technique to combat false negatives by recovering missing connections, andpresent a local thresholding method which can address the regional bias issue whensuppressing false positives in connectivity estimates.3.1 Matrix Completion to Combat False Negatives3.1.1 Related WorkDue to the limited spatial resolutions, complexities of the underlying tissue and un-certainties of signal noise [83] (detailed in paragraph 1.4.2.3.1), Anatomical Con-nectivity estimates derived from the fiber tract strength suffer from both false pos-itive and negative connection problem, especially the false negatives. In particular,diffusion direction is ambiguous at the crossing fiber locations, causing pre-maturetermination of tracts with conventional streamline algorithms [13]. To better handle69crossing fibers, techniques for improving ODF estimation [175] and tractography[74, 79] have been proposed. However, even if the crossing fiber issue is resolved,the ambiguity in diffusion direction near the gray-white matter interface introducesgreat uncertainty to the fiber endpoint locations [13]. Heuristics for endpoint ex-trapolation have been explored with only modest improvement shown [150]. Byand large, due to crossing fibers and fiber endpoint uncertainty, AC estimates tendto be more prone to missing connections, i.e., false negatives [13].A fundamental limitation to the above strategies for improving AC estimationis the upper bound inherently imposed by attainable imaging resolutions. Endeav-ors for increasing the image spatial resolution include improving the acquisitionscheme [146, 176] and applying super-resolution techniques [177]. However, dueto the physical limitations of MRI technology, micro-meter resolution, i.e., thewidth of a fiber, is currently unattainable for whole-brain coverage. Even if suchfine resolution is attained, the required computational time for tractography wouldbe impractical.3.1.2 Low Rank Matrix Completion for Connectivity EstimationWe propose here a matrix completion based approach for recovering missing con-nections. The underlying assumption is that the missing connections are intrinsi-cally related to the observed entries of the AC matrix, hence can be recovered usingmatrix completion.In matrix completion problems, one attempts to fill missing entries of a par-tially observed matrix. The problem is underdetermined if one does not restrict thedegrees of freedom. The typical way to impose such restrictions is by minimizingthe rank:minX∈Rm×nrank(X), s.t. Xi,j = Mi,j,∀(i, j) ∈Ω (3.1)while constraining the matrix entries, Xi,j, to match the observed values, Mi,j. Onehas available s sampled entries {Mi,j : i, j ∈ Ω} where Ω is a random subset ofcardinality s. Since Equation 3.1 is Non-deterministic Polynomial-time (NP) hard,a convex relaxation is often employed [178]:minX∈Rm×n‖X‖∗, s.t. Xi,j = Mi,j,∀(i, j) ∈Ω (3.2)70where rank(X) is approximated by the nuclear norm (also known as the tracenorm), ‖X‖∗, i.e., sum of the singular values of X. One strategy for solving Equa-tion 3.2 that well suits the AC estimation problem is to find a low rank matrix,X = YZ, that minimizes ‖X−M‖2F [179], where Y ∈ Rm×r, r < m, and Z ∈ Rr×n,r < n. In the present context where M ∈RN×N is an AC matrix of N brain regions,X should factorize as YY> given the symmetry of the AC matrix, where eachcolumn i of Y ∈ RN×r can be interpreted as the membership weights of N brainregions belonging to subnetwork i. The rank, r, thus corresponds to the number ofsubnetworks identifiable from M. The detailed optimization process can be foundin [179]. To ensure symmetry, (X+X>)/2 is taken as our recovered AC estimates.We refine all AC connections by updating their values with the matrix completioncorrected values. The factorization algorithm [179] we used for minimizing thematrix completion objective was only one of many possible optimization schemes.This algorithm empirically gives us the best performance of missing entry recoverybased on our experiments in Section 3.1.5.2.3.1.3 Rank Range Search and AggregationIntuitively, AC matrix rank corresponds to the number of brain subnetworks. Am-ple studies suggested that the brain comprises only about a dozen large-scale sub-networks, which confirms the low rank assumption. Existing rank selection meth-ods are largely based on finding a transition point in the matrix eigenspectrum[180]. However, AC matrices typically do not display clear transition points (Fig-ure 3.1). To mitigate poor recovery due to choosing the “wrong” rank, we proposehere to aggregate recovered entries over a range of ranks. The intuition is thatsince matrix rank corresponds to the number of subnetworks, which arguably is amatter of resolution, aggregating over ranks in effect integrates modularity infor-mation across different subnetwork refinements. To select a rank range, we presentthe following automated strategy. Over a predefined range, we first find the rankthat provides the best recovery accuracy, a, based on removing a percentage of theobserved entries and assessing how well those entries are recovered. The range isthen defined as all ranks r ∈ [r1, . . . ,rk] with accuracy within [a− ε,a]. We thenaggregate the recovered AC matrices for this rank range by taking their median71Figure 3.1: Eigenspectrum of an exemplar ACvalue: X¯i, j = median(Xri, j,r ∈ [r1, ...,rk]), where Xri, j is the estimated Xi, j for rankr.3.1.4 Negative Entries Filling using Neighborhood InformationStandard matrix completion algorithms do not impose a non-negative constraint onmatrix entries [181]. However, as for the AC estimation, negative recovered entriesare not biologically interpretable, since fiber counts cannot be negative. Explicitlyimposing a non-negative constraint onto matrix completion is possible [181], butwe observe empirically that such constraint tends to decrease the recovery accuracy(Figure 3.2). The possible explanation could be the fiber count properties do notfit well in a non-negative low rank matrix completion setting. Instead, we devisea method to exploit those few entries which are estimated as negative based onneighborhood information. Under the hypothesis that a pair of highly interlinkedbrain regions are connected to similar brain areas, we first compute a Pearson’scorrelation matrix from X¯: C = X¯X¯>, for defining neighbors based on similarityin connection patterns X¯i,:. For each negative recovered entry, Xi, j, we searchfor h regions with the highest positive correlation to region j, H j = { j1, j2..., jh},where h is selected using cross-validation with recovery accuracy as the metric.The negative recovered entries are then interpolated by taking a weighted mean of72h positive recovered entries between region i and region j’s neighbors, H j:Xˆi, j =∑ht=1(X¯i, jt Ci, jt )∑ht=1(Ci, jt ), if X¯i, j < 0X¯i, j, otherwise(3.3)3.1.5 ExperimentsWe refer to our approach as MCmedFill (Matrix Completion with median aggrega-tion over candidate ranks and neighborhood information filling), which we validateon synthetic data using recovery accuracy, and apply to real data from the HCP[146] for IQ prediction.3.1.5.1 Materials3.1.5.1.1 Synthetic Data We generated 100 synthetic datasets that cover a vari-ety of network configurations. Each network comprised N=112 regions analogousto the HO atlas [57]. For each dataset, we set the number of subnetworks, M, to arandom value between 12 and 14, conforming to current literature [23]. The num-ber of regions in each subnetwork was set to (N/M) + q with q being a randomnumber between [−2,2]. With the resulting configuration, we created the corre-sponding adjacency matrix, Σ, and added Gaussian noise, Figure 3.2a. Negativematrix entries were set to zero. Lastly, we randomly set 20% of the ground truthconnections to 0 to model how AC estimates are prone to false negatives, Figure3.2b.3.1.5.1.2 Real Data We used the dMRI scans and fluid IQ scores of 77 healthysubjects from the HCP Q3 dataset [146], see details in Section 1.8. Given the pre-processed dMRI data [146], we applied global tractography based on CSA ODFand Gibbs tracking [81] using the MITK package [149]. To define brain regions,we employed the HO atlas [57], which has 112 regions. To compute the fiber countbetween regions, we warped the HO atlas onto the b= 0 volume of each subject us-ing affine registration. We further extrapolated the fiber endpoints using Gaussian73(a) Ground truth (b) Observed (c) MedFilter(d) NeighFill (e) OptSpace (f) GROUSE(g) MCNF (h) LMaFit (i) MCmedFillFigure 3.2: Matrix recovery on an exemplar synthetic dataset. MCmedFill(i) more accurately recovered the ground truth connections (in red) thanthe contrasted methods (c-h).74(a) MCmedFill (b) Y weightsFigure 3.3: Matrix recovery using MCmedFill (a) on an exemplar syntheticdataset. The weights in each column of the low rank matrix Y align withthe ground truth subnetwork assignment. (b) An example of thresholdedweights for MCmedFill shown.kernels [150]. The resulting fiber counts normalized by the region volume weretaken as our AC estimates.3.1.5.2 ResultsTo execute our approach, MCmedFill, we used Low-Rank Matrix Fitting (LMaFit)[179] for matrix completion due to the subnetwork analogy that it provides (Sec-tion 3.1.2) as well as its computational speed. Empirically, we set ε = 0.2a andpredefined rank range as [1, 30]. We first evaluated MCmedFill based on re-covery accuracy on synthetic and real data. Since brain connectivity presumablyrelates to IQ [182], we further evaluated MCmedFill based on IQ prediction onreal data. For comparison, we examined interpolating AC entries with zero val-ues by median filtering (MedFilter), with neighborhood information as describedin Section 3.1.4 (NeighFill), and three other widely-used rank-based matrix com-pletion algorithms: OptSpace [180], Grassmannian Rank-One Update SubspaceEstimation (GROUSE) [183], and Matrix Completion with Nonnegative Factoriza-75Figure 3.4: Rank selection method. Our criteria of using recovery accuracyon randomly removed AC entries displays clear NRMSE minima forrank selection.tion (MCNF) [181], which imposes a non-negative constraint. The vanilla LMaFitwas also tested. The rank for OptSpace, GROUSE, MCNF, and vanilla LMaFitwere selected based on highest recovery accuracy over a range of ranks (Fig-ure 3.4). All statistical comparisons were based on the Wilcoxon signed rank testwith MCmedFill as the reference. Significance was declared at an α of 0.05 withBonferroni correction.3.1.5.2.1 Recovery Accuracy We assessed recovery by first deleting 20% of theentries in AC matrices for both synthetic and real data. We then estimated therecovery accuracy using: normalized root-mean-squared-error (NRMSE).NRMSE =√∑(Mi, j− Xˆi, j)2/√∑M2i, j, where M is the ground truth and Xˆ isthe recovered matrix, and coefficient of determination: R2 = 1−SSres/SStot , whereSSres = ∑(Mi, j− Xˆi, j)2 and SStot = ∑(Mi, j− M¯i, j)2. Lower NRMSE and higherR2 indicate higher accuracy.On the 100 synthetic datasets, MCmedFill achieved significantly higher accu-racy than the contrasted methods, Figure 3.5a, 3.5b. For real data, we first gen-erated a group AC matrix by averaging the subject level AC estimates. We thenrandomly deleted 20% of the entries for 100 times. MCmedFill again achieved76significantly higher accuracy, Figure 3.6a, 3.6b. Note the reported accuraciesfor the four matrix completion algorithms were the best results across a rangeof ranks (Figure 3.4). Also, MCmedFill’s improvement over LMaFit might ap-pear small, but statistically significant since higher accuracies were consistentlyobserved across the random test cases.Examining the columns of the low rank matrix Y, we observed that the weights(thresholded at 50% of the maximum value) match the ground truth of subnetworkassignment for synthetic data (Figure 3.3b) using optimally selected rank, whichhappens to correspond to the ground truth number of subnetworks. Furthermore,some columns of Y were found to resemble known brain networks, e.g., Frontal-Parietal network (Figure 3.7b), suggesting the potential of using matrix completionfor subnetwork extraction, which we defer for future work.3.1.5.2.2 IQ Prediction Due to the absence of a ground truth human anatomicalconnectivity atlas, we further assessed MCmedFill based on IQ prediction withreal data. We used the lower triangular entries of the recovered subject level ACmatrices as predictors. Confounds including age and sex were regressed out fromthe predictors and the IQ scores prior to applying L2-regularized L2-loss SupportVector Regression (SVR) [184]. SVR solves the following primal problems:minw1/2wT w+Cl∑i=1(max(0, |yi−wT xi|− ε))2, (3.4)where yi is the real value of the IQ score, xi ∈Rn, C > 0 is a penalty parameter andε is a parameter to specify the sensitiveness of the loss. 100 random realizationsof 11-fold cross-validation were performed for estimating prediction accuracy andits variability. Prediction accuracy was defined as the Pearson’s correlation, R,between the 77 predicted and observed IQ scores for each cross-validation realiza-tion. Though indirect, we are very thrilled to show that our validation approachresulted in significant findings on real data that are neuroscientifically meaningful.Predicting IQ with the original AC estimates obtained an average predictioncorrelation of 0.1865, Figure 3.7a. Applying MedFilter, OptSpace, and GROUSEdegraded the prediction. NeighborFill and LMaFit performed slightly better than77(a) NRMSE on synthetic data(b) R2 on synthetic dataFigure 3.5: Matrix recovery on synthetic data. Mean values are indicated asblack diamonds and labeled at the bottomn of each subfigure. MCmed-Fill achieved significantly higher accuracy than contrasted methodsbased on Wilcoxon signed rank test.78(a) NRMSE on real data from HCP(b) R2 on real data from HCPFigure 3.6: Matrix recovery on real data. Mean values are indicated as blackdiamonds and labeled at the bottomn of each subfigure. MCmedFillachieved significantly higher accuracy than contrasted methods basedon Wilcoxon signed rank test.79AC MedFilter NeighFill OptSpace GROUSE LMaFit MCmedFill00.10.20.30.40.5IQ Prediction Correlation0.1865 0.126 0.2149 0.1381 0.1773 0.2151 0.4282(a) IQ prediction on real data from HCP(b) Y weights visualizationFigure 3.7: IQ prediction on real data. (a) MCmedFill achieved significantlyhigher correlation coefficient between observed and predicted IQ scoresthan contrasted methods. (b) Brain region subnetwork weights alongcolumns of Y (thresholded for clearer visualization) were found to re-semble known brain networks, e.g., Frontal-Parietal network.80original AC estimates, but not by a statistically significant amount. In contrast,MCmedFill achieved significantly higher prediction accuracy than the contrastedmethods, with an average prediction correlation of 0.4282 attained, which is higherthan values reported in most AC-based IQ prediction studies [182].To investigate the sources of prediction improvement, we contrasted the ACmatrices before and after applying MCmedFill with a focus on connections be-tween regions within the Default Mode Network (DMN) as well as within the Ex-ecutive Control Network (ECN) (both networks had been shown to relate to IQ[182]). Applying MCmedFill resulted in a 53.19% and a 28.41% increase in es-timated connectivity within DMN and ECN, respectively, which suggests that theimproved IQ prediction with our approach might be due to the recovery of relevantconnections that were missed with standard AC estimation.3.1.6 DiscussionWe proposed a matrix completion based approach for recovering missing connec-tions to improve the AC estimation. By aggregating recovered entries over ranksand interpolating negative entries with neighborhood information, MCmedFill at-tained higher accuracy in recovering deleted AC entries for both synthetic and realdata. Higher accuracy in IQ prediction was also shown. Our results thus demon-strated clear benefits of refining conventional AC estimates with our approach.The study on the capability of our approach to deal with the false positives thatalso exist in AC estimates [185] needs to be continued. The low rank assumptionhas a denoising effect, which could reduce the effect of false positives. On the otherside, we need to explore whether the proposed method could negatively affect thefalse discovery rate, which can impact the clinical feasibility of using anatomicalconnections to study some real life diseases. We will also work on more system-atic and detailed validation, such as direct comparison between publicly availablemacaque dMRI connections and tract tracing results [186, 187]. Tract tracing pro-vides a gold standard of brain connections, so the accuracy of our approach can beevaluated directly on macaque data.81Figure 3.8: Signal dropouts in the orbitofrontal cortex (OFC) caused by fieldinhomogeneities (at 1.5 T). Image courtesy of [21].3.2 Local Thresholding to Suppress False Positives3.2.1 Related WorkThe human brain regions and their pair-wise interactions constitute graph nodesand weighted edges, respectively. fMRI is one of the widely used modalities forconnectivity estimation, however, FC estimation is known to suffer especially fromfalse positives [90]. Moreover, confounds, such as region size bias [188, 189],effects of motion artifacts [190], and signal dropouts due to susceptibility arti-facts (especially in regions like the orbitofrontal cortex, Figure 3.8, and the inferiortemporal lobe) [21], introduce region-specific biases to the connectivity estimates.Such problems make applications such as subnetwork extraction very challenging,since brain network topology may be obscured by noisy connectivity estimates[16].The conventional way for dealing with noisy connectivity matrices is to applyGlobal Thresholding (GT) by either keeping only connections with values abovea certain threshold or keeping a certain graph density [16]. Due to region-specificconnectivity biases, e.g., brain regions in signal dropout locations tend to displaylower connectivity, certain regions that do belong to a subnetwork might not appearas such based on the fMRI measurements, especially after GT, which prunes weakedges. To mitigate this overlooked problem, a Local Thresholding (LT) methodbased on the MST-KNN has been proposed [22]. The idea in [22] was to build a82Figure 3.9: Schematic illustrating local thresholding (MST-KNN) and globalthresholding methods. Image courtesy of [22].83single connected graph using the minimum spinning tree and expand the tree byadding edges from each node to its nearest neighbors until a desired graph den-sity is reached, Figure 3.9. However, the key step of adding edges to all nodeswhen expanding the tree, based on the assumption of equal density/importance foreach node, lacks neuroscientific justifications. A few studies have explored spec-tral graph wavelet transform for graph denoising [191], but this approach does notexplicitly handle region-specific connectivity biases. In fact, most existing connec-tivity estimation [16] do not account for these biases.3.2.2 Local ThresholdingTo deal with noisy edges mostly with false positives and region-specific connectiv-ity biases, we propose a local thresholding scheme that normalizes the connectivitydistribution of each node prior to thresholding. Due to region-specific connectivitybiases, conventional GT might prune relevant connections with weak edge strength.To account for these biases, we present here a LT scheme. The idea is to first nor-malize the connectivity distribution of each node into a uniform interval to rectifythe biases. Subsequent Global Thresholding on this normalized graph would havethe effect of applying Local Thresholding on each node without enforcing the equalimportance of all nodes. Specifically, let C be an N×N connectivity matrix, whereN is the number of nodes in the brain graph. We normalize the connectivity dis-tribution by mapping each row of C from [min(Ci,:),max(Ci,:)] to [0, 1], whereCi,: denotes row i of C corresponding to the connectivity between brain region iand all other regions in the brain. A threshold is then applied to generate a binaryadjacency matrix, G, which we then symmetrize by taking the union of G and GT :A = Gi, j ∪G j,i, where GT is the transpose of G. This binary adjacency matrix Ais used to mask out the noisy edges from C: Cˆi,j = Ai,jCi,j, which is equivalentto applying a local threshold to Ci,: for all i. We note that in the event that noisynodes (signal dropout brain regions, Figure 3.8) are accidentally included, some ofthe connections to these noise nodes (that might not be kept by GT) would be keptby LT due to the normalization step.843.2.3 ExperimentsWe validated our LT scheme using subnetwork extraction application based on theassumption that we can achieve more accurate subnetwork structure using moreaccurate connectivity estimates.3.2.3.1 Materials3.2.3.1.1 Synthetic Data To illustrate our strategy, we synthesized a small-scalenetwork consisting of N = 13 nodes in Figure 3.10 a. These nodes are assigned totwo subnetworks with each subnetwork having a provincial hub (blue) and linkedby a connector hub (orange). The weights of edges in the network were set to 0.75.We then added Gaussian noise at a SNR of -5 dB to simulate the motion artifacts[190]. We further lowered the edge weights of node 2 and 12 by 20% to simulateregion-specific connectivity biases for smaller brain regions [188], Figure 3.10 b.We also generated synthetic data that cover 100 random network configurationswith N set to 100 nodes. For each network configuration, the number of subnet-works, M, was randomly selected from [10, 20]. The number of regions withineach subnetwork was set to round(N/M) + rn, where rn was randomly selectedfrom [-2, 2]. With the resulting configuration, we created the corresponding ad-jacency matrix, Σ, and drew time courses with 4,800 samples (analogous to realdata) from N(0,Σ). We then added Gaussian noise to the time courses with signal-to-noise ratio randomly set between [-6dB, -3dB]. Sample covariance was thenestimated from these time courses with correlation values associated with q% ofthe nodes reduced by z%, where q was randomly selected from [20%,30%] and zwas randomly selected from [30%,40%] to simulate region-specific connectivitybiases for smaller brain regions [188].3.2.3.1.2 Real Data We used the rs-fcMRI data of 77 healthy HCP subjects,details and preprocessing done on the data described in Section 1.8. We used theWill90fROI atlas [23] to define ROIs, which have 90 ROIs being assigned to 14well-established brain systems Figure 3.11. Voxel time courses within ROIs wereaveraged to generate region time courses. The region time courses were demeaned,85(a) Network structure(b) C (c) C¯ (d) CˆFigure 3.10: Schematic illustrating our proposed local thresholding using asmall scale example having two subnetworks with each subnetworkhaving a provincial hub (blue) and linked by a connector hub (orange),shown in (a). In (b), warmer color indicates higher connectivity andblack dots indicate the ground truth adjacency matrix. We denote C¯ asglobal thresholded, and Cˆ as local thresholded connectivity matrix. Ata graph density of 0.25, the GT generated isolated node 2 in (c), whileour LT preserved two edges linked to node 2 in (d).86Figure 3.11: Will90fROI atlas [23] with 90 ROIs being assigned to 14 well-established brain systems.normalized by the standard deviation, and concatenated across subjects for extract-ing group subnetworks. The Pearson’s correlation values between the region timecourses were taken as estimates of connectivity. Negative elements in the connec-tivity matrix were set to zero due to the currently unclear interpretation of negativeconnectivity [93].3.2.3.2 ResultsWe compared our LT strategy against GT and MST-KNN in [22]. Instead of usinga specific threshold, we examine a range of graph densities to test the robustnessof our proposed strategy. For synthetic data, evaluation was based on the accuracyof subnetwork extraction. To estimate accuracy, we matched the extracted subnet-works to the ground truth subnetworks using Hungarian clustering [162] with theDSC = 2 |X∩Y|/(|X|+ |Y|), where X is the set of regions of an extracted subnet-work and Y is the set of regions of a ground truth subnetwork. The average DSCover matched subnetworks was taken as accuracy.For real data, we assessed the overlap between the extracted subnetworks andfourteen well-established brain systems [23] and subnetwork reproducibility for arange of graph densities on HO atlas [57] using DSC.All statistical comparisons are based on the Wilcoxon signed rank test withsignificance declared at an α of 0.05 with Bonferroni correction.3.2.3.2.1 Synthetic Data In the example of small-scale network, with GT (Fig-ure 3.10 c), node 2 was isolated from subnetwork 1. In contrast, our LT scheme(Figure 3.10 d) was able to preserve node 2. Also, with our LT (Figure 3.10 d), one87Figure 3.12: Subnetwork extraction accuracy using synthetic dataset usingdifferent thresholding schemes. The bar chart indicates the averageDSC over 100 synthetic dataset across a graph density range of [0.005,0.5] at an interval of 0.01.of between-subnetwork edges (i.e., edges between nodes 6 and 7 & nodes 6 and 9)was pruned, which would help prevent the two subnetworks from being declaredas one, whereas none of between-network edges was pruned using GT (Figure 3.10c).On the 100 synthetic dataset with 100 nodes over a density range of [0.005,0.5] at an interval of 0.01, LT achieved statisticaly significantly higher accuracy(average DSC = 0.6537±0.0479 ) than GT (average DSC=0.6216±0.0417 ), andMST-KNN (average DSC=0.6327±0.0732 ), Figure 3.12.3.2.3.2.2 Real Data We first evaluated our strategy by examining the overlapbetween our extracted subnetworks and 14 well-established brain systems pre-sented in [23], which we used as ground truth. Our LT achieved an average DSCof 0.5936±0.0374, which was statistically significantly higher than GT (averageDSC=0.5384±0.0436), and MST-KNN (average DSC=0.4567±0.0653). We notethat although some node-wise variations in connectivity distribution might have aneuronal basis, we postulate that these variations would be overwhelmed by the88(a) Brain connections with global thresholding (b) Brain connections with local thresholding(c) Color labels of 14 well-established brain subnetworkFigure 3.13: Circular plots of the whole brain connections based on the 14subnetwork structure using different thresholding schemes. Exemplarresults shown here are thresholded at the graph density of 0.02. Thethickness of the connections indicate the strength of the connections.various confound-induced connectivity biases, as supported by how local thresh-olding outperforms global thresholding.Further, we qualitatively show how different thresholding schemes globally af-fect the connections across the whole brain. We observed that our LT has moreintra-subnetwork connections and reasonable inter-subnetwork connections com-pared to GT at different densities, as shown in Figure 3.13.893.2.4 DiscussionTo tackle the noise reduction problem in brain network edge building, we proposea Local Thresholding scheme to deal with false positives present mostly in theFC estimates. Our technique accounts for local variations in the estimate of theconnectivity strength, i.e., region bias across the connectome. Using an indirectevaluation, we attained higher accuracy in subnetwork extraction using our pro-posed Local Thresholding compared to conventional GT and state-of-the-art LocalThresholding (MST-KNN).We will focus on building a direct validation method for our Local Threshold-ing scheme, e.g., testing on a noise model built on the ground truth of the con-nectivity matrices. At the same time, we need to design validations for the falsepositive connections discovered in tractography-based AC estimates [185].3.3 SummaryTowards improving the brain edge building, we explored noise reduction directiondue to the lack of explicit noise model. We proposed two strategies to tackle thetwo most common estimation errors, i.e., false negatives and false positives.Specifically, to combat false negatives in the connectivity estimation, we pro-posed a matrix completion based technique by recovering missing connections.Based on the assumption that brain is comprised of a small number of subnet-works, we format the recovering missing connection problem as a low rank matrixcompletion problem. We effectively propose an information aggregation approachto tackle the problem that there is not a clear change point to estimate the rankdue to the noisy nature of neuroimaging data. Further, we advise a neighborhoodinformation filling strategy to solve the negative entry problem. Based on syntheticdata and real HCP data, we quantitatively demonstrated the superiority of our ma-trix completion based approach as compared to existing methods such as medianfilter or simple matrix completion approaches, in terms of the recovery accuracyand the IQ prediction. The refined AC estimation derived based on our approachcould benefit clinical investigations on anatomical connections on patient data withdiseases, such as glioma, multiple sclerosis, and amyotrophic lateral sclerosis.On the other hand, we present a local thresholding method to suppress false90positive connectivity estimates. Our method is able to tackle the regional biasproblem, which is caused by region size bias, motion artifacts, and signal dropoutsdue to susceptibility artifacts. Compared to widely used global thresholding andstate-of-the-art local thresholding method, our method achieved better performancein an application of subnetwork extraction in favor of a better denoising process.91Chapter 4Graphical Metric GuidedSubnetwork ExtractionThis chapter focuses on improving brain subnetwork extraction using graphicalmetrics, which is based on the second part of papers [P5] and the first part of [P6].The brain network can be abstracted as a graph. One can gain insights intothe fundamental architectures and functions of the brain from its modular structure[192], which can be extracted by clustering brain nodes into subnetworks (alsoreferred to communities) using community detection methods [106]. However, re-liable subnetwork extraction from either the AC or FC remains challenging, mostlysince the brain network topology may be obscured by noisy connectivity estimates.Even with the techniques we proposed in Chapter 3 for noise reduction, the subnet-work extraction is still not ideal. We have noticed that few subnetwork extractionmethods have exploited intrinsic properties of brain networks other than modu-larity. In this chapter, we investigate whether incorporating a greater number ofgraphical metrics, which was mentioned in Section 1.4.2.4, would give us moredomain-related information about the brain networks for better subnetwork extrac-tion. At the same time, we will explore if we can build a model that would resemblethe biological nature of brain subnetworks.924.1 Modularity Reinforcement Subnetwork Extraction4.1.1 Related WorkFunctional Magnetic Resonance Imaging is widely used for studying the functionalmodular structure of the brain. Due to false positives and negatives in connectiv-ity estimates and the region-specific biases, introduced in Section 3.2.1, existingsubnetwork extraction methods [106] (community detection methods, graphicalpartitioning methods, ICA based methods) still can not generate reliable subnet-work extraction results. Different from the existing work, we aim to utilize morerelated graphical information into the subnetwork extraction process. Combinedwith the local thresholding which we proposed in Section 3.2.2, we here propose amodularity reinforcement strategy for improving brain subnetwork extraction.4.1.2 Modularity Reinforcement ModelOur modularity reinforcement strategy is based on a similarity measure. The under-lying assumption is that node pairs belonging to the same subnetwork presumablyconnect to a similar set of brain regions, i.e., have similar connection fingerprints.We derive a node similarity measure from the thresholded graph by comparing theadjacency structure of each node pair. We note that we apply the local threshodingproposed in Section 3.2.2 to first reduce the noise and regional bias. Given thethresholded graph presented as the adjacency matrix A, where Ai,: is the connec-tion fingerprint of node i, we define the similarity between a pair of nodes (i, j)as the number of common adjacent nodes they share, normalized by the minimumnode degree of the node pair:Si,j =∑nk=1 Ai,kA j,kmin(di,dj), (4.1)where di =∑nk=1 Ai,k. We use the minimal degree for normalization, instead of e.g.,the average degree, so that connections associated with hub nodes (nodes with moreedges, defined in Section 1.4.2.4) will not be overly down-weighted. Since nodeswithin a subnetwork are expected to share more adjacent neighbors than nodesbelonging to different subnetworks, S boosts the within-subnetwork edges while93suppresses the between-subnetwork edges, which highlights the modular patterninherent in the local thresholded connectivity matrix Cˆ: Hence, we use S to refineCˆ to reinforce its modular structure:CˆSi,j = Si,jCˆi,j. (4.2)4.1.3 Subnetwork Extraction Based on Graph CutsFor subnetwork extraction, we employ Ncuts, chosen due to its wide use by thefMRI community. To set the number of subnetworks, M, we adopt an automatedtechnique based on the spectral properties of the graph Laplacian: L = D−W,where W is a distance matrix derived from the connectivity matrix C using Equa-tion 2.1, Dii = ∑nj=1 Wi,j. Specifically, an eigenvalue of 1 has been shown to cor-respond to the transition where single isolated nodes would no longer be declaredas a subnetwork [193]. We thus set M to the number of eigenvalues of L withvalues less than 1. Setting M based on the conventional approach of modularitymaximization [194] was also performed for comparison.4.1.4 Experiments4.1.4.1 MaterialsWe used the exact same synthetic and real data for local thresholding, described inSection 3.2.3.1 to validate our modularity reinforcement strategy. We additionallyperformed a reproducibility test using another atlas, HO atlas, which has 112 ROIswith a greater coverage of the brain than Will90fROI atlas.4.1.4.2 ResultsWe compared our strategy (LTMR-LT with modularity reinforcement) against GT,LT, GT with modularity reinforcement (GTMR) and MST-KNN in [22]. LT wasimplemented using our proposed scheme (Section 3.2.2). GTMR was implementedby deriving adjacency matrices with global thresholding, and subsequently execut-ing our proposed modularity reinforcement strategy (Section 4.1.2). Instead ofusing a specific threshold, we examined a range of graph densities to test the ro-94bustness of our proposed strategy. For synthetic data, evaluation was based on theaccuracy of subnetwork extraction. To estimate accuracy, we matched the extractedsubnetworks to the ground truth subnetworks using Hungarian clustering with theDSC, defined in Section 3.2.3.2. The average DSC over matched subnetworks wastaken as accuracy. For real data, we assessed the overlap between the extractedsubnetworks and fourteen well-established brain systems [23] and subnetwork re-producibility based on HO atlas for a range of graph densities. In the overlappingtest, DSC was calculated between the estimated subnetwork membership and thegiven one from the well-established brain systems [23] after matching the labelusing Hungarian assignment [160]. In the reproducibility test, DSC was calculatedbetween different subnetwork results on different graph densities.4.1.4.2.1 Synthetic Data An example of the various steps of our strategy isshown in Figure 4.1 c-f to demonstrate how our strategy highlights the modularstructure of the graph. With GT (Figure 4.1 c), node 2 was isolated from subnet-work 1. In contrast, our LT scheme (Figure 4.1 d) was able to preserve node 2.Also, with our LT (Figure 4.1 d), one of between-subnetwork edges (i.e., edges be-tween nodes 6 and 7 & nodes 6 and 9) was pruned, which would help prevent thetwo subnetworks from being declared as one, whereas none of between-networkedges was pruned using GT (Figure 4.1 c). Further, refining the graph (Figure 4.1c-d) with our similarity measure helped to highlight the modular pattern (Figure 4.1e-f), e.g., the between-network edges which were similar to or higher than somewithin-network edges (especially those edges between nodes 12 and 13, node 2and 1 & nodes 2 and 5) in Figure 4.1 c-d were surpressed by our similarity to bethe lowest values in Figure 4.1 e-f.On the 100 synthetic dataset with 100 nodes over a density range of [0.005, 0.5]at an interval of 0.01, LTMR achieved significantly higher accuracy (average DSC= 0.6735±0.0475) than GT (average DSC=0.6216±0.0417, p=7.56e-10), LT (av-erage DSC=0.6537±0.0479, p=2.89e-7), and MST-KNN (average DSC=0.6327±0.0732,p=7.38e-8) based on Wilcoxon signed rank test with Bonferroni correction, Fig-ure 4.2. LTMR also achieved higher DSC than GTMR (average DSC=0.6610±,p=0.34), though did not reach significance.95(a) Network structure (b) C(c) C¯ (d) Cˆ (e) C¯S (f) CˆSFigure 4.1: Schematic illustrating our method using small scale example hav-ing two subnetworks with each subnetwork having a provincial hub(blue) and linked by a connector hub (orange), shown in (a). In (b),warmer color indicates higher connectivity and black dots indicate theground truth adjacency matrix. We denote C¯ as global thresholded, andCˆ as local thresholded connectivity matrix. At a graph density of 0.25,the GT generated isolated node 2 in (c), while our LT preserved twoedges linked to node 2 in (d). Refining the graph (c) and (d) suppressedthe between-network edges (edges between nodes 6 and 7 & nodes 6and 9) to be the lowest connectivity in (e) and (f).4.1.4.2.2 Real Data We first evaluated our strategy by examining the overlapbetween our extracted subnetworks and 14 well-established brain systems pre-sented in [23], which we used as ground truth, Figure 4.4 a. For this assessment,we only considered connectivity matrices based on the Will90fROI atlas [23]. Ourproposed LTMR achieved an average DSC of 0.6222±0.0474, which was sig-nificantly higher than GT (average DSC=0.5384±0.0436, p=0.002), MST-KNN(average DSC=0.4567±0.0653, p=0.002), GTMR (average DSC=0.5422±0.0470,p=0.006) based on Wilcoxon signed rank test with Bonferroni correction, andhigher than LT (average DSC=0.5936±0.0374, p=0.063), as shown in Figure 4.3a. We note that an average M of 11 was estimated with the Laplace approach,96Figure 4.2: Subnetwork extraction on synthetic data at graph densities from0.005 to 0.5 at interval of 0.01. Our proposed strategy attained the high-est DSC overall.whereas an average M of 4 was estimated with modularity maximization. Thisresult shows the resolution limits of modularity maximization [100], i.e., it tendsto underestimate the number of subnetworks in favoring network partitions withgroups of modules combined into larger communities. This suggests the need toexplore alternative techniques for estimating the number of subnetworks.We next evaluated the subnetwork reproducibility over a range of graph densi-ties. We used connectivity matrices based on the HO atlas, which has larger braincoverage than the Will90fROI atlas but does not have subnetwork labels assigned tothe regions. We set subnetworks corresponding to an edge density of 0.2 as the ref-erence. Based on the Laplace approach, the optimal number of subnetworks M wasfound to be 11±5 over the range of graph density examined. Our proposed strat-egy achieved an average DSC of 0.7302±0.0575, which is significantly higher thanthat of GT (DSC=0.6121±0.0620, p=0.004), LT (DSC=0.6677±0.0599, p=0.027),MST-KNN (DSC=0.5737±0.0754, p=0.003) based on Wilcoxon signed rank testwith Bonferroni correction, and higher than GTMR (DSC=0.7004±0.0923, p=0.262),Figure 4.3 b. The results hold with other densities used as reference.Qualitatively, based on the subnetwork extraction resulting using the HO atlas,with GT (Figure 4.4 b), we observed two subnetworks comprising only isolated97(a) Overlap with established subnetworks (b) Reproducibility over density rangeFigure 4.3: Subnetwork extraction on real data at graph densities from 0.05to 0.5 at interval of 0.05. Our proposed strategy attained the highestDSC overall.nodes in the left and right Pallidum (yellow and light grey node in the blue circle).We also observed that a region in the right premotor area was falsely grouped intothe auditory subsystem (the light green region with a red arrow). With GTMR, twosubnetworks comprising single nodes were found as well. As for LT (Figure 4.4 c),we observed the left and right insular cortex as well as the right Frontal OperculumCortices (orange nodes with red arrows) were falsely grouped with Dorsal De-fault Mode regions and the left paracingulate gyrus was excluded. In contrast, ourproposed strategy LTMR correctly identified known Dorsal Default Mode regions,such as paracingulate gyrus, anterior division of cingulate gyrus, and Accumbens,as a single subnetwork. Further, LT excluded the left Cuneal Cortex in the visualsystem (blue arrow in Figure 4.4 c). Other found subnetworks with our strategy,such as left and right executive control subnetworks (red and yellow), Figure 4.4d, also conform well to known brain systems as was quantitatively demonstrated inFigure 4.4 a.4.1.5 DiscussionWe proposed a modularity reinforcement strategy for improving brain subnetworkextraction. By applying local thresholding in combination with modularity rein-forcement based on connection fingerprint similarity, we attained higher accuracyin subnetwork extraction compared to subnetwork extraction approach with con-98(a) Will90fROI subnetwork (b) Global thresholding(c) Local thresholding (d) Proposed LTMRFigure 4.4: Subnetwork visualization. 11 subnetworks were extracted fromgraphs with a density of 0.2. (a) Well-established brain systems [23].(b) Two subnetwork formed by isolated nodes and false inclusionof premotor-related regions into auditory system was observed usingglobal thresholding. (c) Local thresholding failed to detect one regionof known visual systems and falsely detected four unrelated regions intodorsal default mode system. (d) Our strategy LTMR correctly detectmost of the subnetworks found in [23].99ventional global thresholding, the state-of-art local thresholding, and our own localthresholding. Higher overlap with established brain systems and higher subnet-work reproducibility were also shown on the real data. Our results thus demon-strated clear benefits of incorporating a greater number of graphical informationwith our strategy for subnetwork extraction. In fact, our strategy can be extendedto applications beyond subnetwork extraction. The rescaled correlation values arenot intended to be directly used for analysis, but we can derive features based onthe extracted subnetworks, e.g., within-subnetwork connectivity computed fromthe original connectivity estimates, and use those features for group analysis andbehavioural association studies. Also, we can use, e.g., mutual information or Jac-card index, to compare the cluster labels of two groups and use permutation toassess significance of group differences.4.2 Provincial Hub Guided Random Walker BasedSubnetwork ExtractionFollowing the section above, we are to explore if incorporating a greater number ofgraphical metrics would give us better subnetwork extraction. Also, we will builda model inspired by the brain subnetwork’s biological nature.4.2.1 Related WorkExisting brain subnetwork extraction methods typically use modularity, introducedin Section 1.4.2.5 as a fitness measure to optimize a graph partitioning [106]. Mod-ularity as defined by the Q value below reflects the intra and inter subnetwork con-nection structure of a network.Q =14m∑ij(Ai, j− kik j2m )(sis j +1) =14m∑ij(Ai, j− kik j2m )sis j. (4.3)Asides from the limitations such as resolution limit, such subnetwork extractionremains challenging due to the pronounced noise in neuroimaging data. Besides,few methods have exploited intrinsic properties of brain networks other than mod-ularity.Informative network metrics such as hubs, within-module degree scores, and100participant coefficients can be estimated given a graph [17]. However, these net-work metrics have not been used to guide the subnetwork extraction process. Pre-vious studies on anatomical [105, 106] and functional networks [195, 196] sug-gest the presence of the “provincial hubs” [197], hubs that are highly connected tonodes within a subnetwork. These provincial hubs are thought to be responsiblefor the formation and stability of the subnetworks [197, 198]. Given the criticalrole of provincial hubs, we argue that incorporating provincial hubs to guide thesubnetwork extraction process would be beneficial.4.2.2 Resemblance Between Provincial Hubs and the SeedsWe here propose a Random Walker (RW) [199] based approach which utilizesbrain network properties to guide the brain subnetwork extraction. We hypothesizethat the manner in which the nodes are clustered into groups, based on the proba-bilities of walking to seeds in the RW, closely resembles the mechanism wherebybrain regions within a subnetwork are inter-linked via provincial hubs. Throughan iterative optimization process, we update the RW model architecture based onthe feedback from the brain properties estimated from the previous iteration. Thefeedback mechanism enables the incorporation of network information within iter-ations to efficiently identify improved subnetwork assignments. RW [199] capturesthe probability transition along the pathway from a node to a seed, rather than justthe distance between nodes and seeds in other seeded based methods such as re-gion growing and kmeans clustering. The RW with prior model [200] is in fact wellsuited to incorporate a feedback from network properties via prior edges connect-ing nodes to augmented seeds. We thus deploy a feedback informed optimizationmodel based on RW with prior model. Further, most clustering methods producehard subnetwork assignments, which forces all nodes to be assigned into singlecertain subnetworks. By using the RW, we can infer the probability of a nodebeing assigned to subnetworks, and further investigate the significance of nodesbelonging to the subnetworks.1014.2.3 Overview of Random Walker ModelGiven a weighted graph G, a set of weighted edges wi j ∈ E, and a set of N nodes V ,comprising labeled nodes (i.e., seeds) VS , and unlabeled nodes, VU , such that VS∪VU =V , we wish to assign each node vi ∈VU with a label from set {1,2, . . . ,M}. Wedefine the set MS = m1,m2, . . . ,mM as subnetwork structure, in which each subsetmk,k ∈ 1∼M of MS is a set of nodes within subnetwork k. The RW approach [199]assigns to each unlabeled node the probability, xki , that a random walker startingfrom that node i first reaches a seed assigned to label k. Each unlabeled node is thenassigned to the label for which it has the highest probability, i.e., yi = maxkxki . Theminimization of xk>Lxk yields the probability xk, where L(N×N) is the Laplacianmatrix of the graph defined as:Lviv j =di if i = j,−wi j if vi and v j are adjacent nodes,0 otherwise.(4.4)where di = ∑jwi j. By partitioning the Laplacian matrix into labeled LS(M×M)and unlabeled LU(N−M×N−M) blocks:L =[LS BB> LU](4.5)and denoting an |VS|×1 indicator vector as f ki =1 if yi = k0 if yi 6= k , the minimizationof xk>Lxk with respect to xkU is given by the linear equation as:LU xkU =−B f k. (4.6)102Figure 4.5: Augmented graph model for our augmented RW with priormodel. The use of prior is equivalent to using M labeled “floating” aug-mented nodes (dash-line circles in black) that correspond to each labeland are connected to each blue node with black dash lines.4.2.4 Feedback Informed Optimization Model4.2.4.1 Relating to RW with Prior ModelGiven the important role of provincial hubs in forming and stabilizing the subnet-works [198], we propose an iterative optimization RW model by introducing anprior edge based on the RW posterior probability to reflect the affinity betweena node and a seed. This affinity resembles how nodes are connected to provin-cial hubs. We deploy the RW with prior model [200] with an exact closed-formposterior probability update as below:(LU + γM∑r=1ΛkU)xkU = λkU +B fk, (4.7)103where λ ki is a prior, that represents the probability of node i belonging to subnet-work k, and Λk is a diagonal matrix with the values of λ k. According to [200],the incorporation of priors in Equation 4.7 yields the same solution as would beobtained for the RW probabilities on an augmented graph shown in Figure 4.5. Inthis augmented graph, we denote the set H = {h1,h2, . . . ,hM} as the augmentedseeds (black “floating” seeds indicated as dotted circles in black), which are gen-erated by the subnetwork extraction result from the previous iteration. The blueedges in Figure 4.5 are node-to-node edges wi j, and we define the black dash lines,λ ki ,k ∈ 1∼M as the “prior edges” connecting the nodes to the floating augmentedseeds. Here γ is used to modulate the degree of the incorporation of the prior. Thecorresponding Laplacian matrix in our proposed model is:L =[LS BB> LU + γI], (4.8)where Bi,k = −γλ ki , I is an identity matrix, and the element in the right cornerunlabeled block is derived using the following equation:diLU +M∑k=1−Bi,k = diLU + γM∑k=1λ ki = diLU + γ, (4.9)where ∑Mk=1λ ki = 1, as the probability of a node walking to all seeds sum to unity.4.2.4.2 Multi-seed ModelBased on the augmented graph model (one floating augmented seed per subnet-work), we extend our model to contain multiple augmented seeds per subnetwork.The reason for using multiple seeds is the possible over-parcellation, which nat-urally splits a provincial hub into multiple seeds [201]. Further, using multipleseeds can increase robustness of the model against the problems caused by noisyconnections, which could affect the power of single seed by creating strong “false”walks (e.g., through the connector hubs). By using the multi-seed strategy, we canpreserve several reliable connections to battle against false positive connections(see an illustration in Figure 4.6). We empirically observed the outperformance ofusing multiple seeds compared to single seed.104Figure 4.6: Illustration of robustness of using multiple seeds by mitigatingthe problems caused by noisy connections between subnetworks. Noisyconnections between subnetworks could affect the power of single seedby creating strong “false” walks (thick purple line connecting the blueseed to a node which should be in the red subnetwork). By using multi-seeds (the smaller floating dots around the big dot in the center), we canpreserve several reliable connections (the dash lines connecting thosesmaller dots to the nodes) to battle against possible false positive con-nections.We denote a new set of seeds, HM = {H1,H2, . . . ,HM}, where each subsetHk = {hk1,hk2, . . . ,hknk} represents the set of multiple seeds that belong to subnet-work k,k ∈ {1,2, . . . ,M}. We define the prior edge connecting a node i to one ofthe augmented seeds hkj in the subnetwork k as the split version of the prior edgeconnecting a node to a single augmented seed in (Equation 4.10):µhkji = wi,hkjλki , j ∈ 1∼ nk, (4.10)accordingly, the corresponding B will be updated as Bi,hkj =−γµhkji . Then we gen-erate the subnetwork assignment probability by summing up the probabilities of anode walking to each of the multiple seeds.4.2.4.3 Subnetwork Assignment Confidence Based on Posterior ProbabilityOne major gain in using RW model is that it offers a confidence value that a givennode belongs to a particular subnetwork (as represented by the probability). Thusthe posterior probability derived from the RW model provides users with options105to discard those non-significant nodes with low probabilities belonging to any sub-networks.4.2.4.4 Framework of the Feedback Informed Optimization ModelWe first initialize the process by conducting a pre-partitioning of the brain graph.In the iterations, we update the seeds based on the subnetwork assignments fromthe previous iteration, and the corresponding prior edges using the probabilities ofa node belonging to a certain subnetwork. We automatically set seeds based on thenetwork properties of provincial hubs, which have high within-module degree Zscore and low participation coefficient P score [17], defined as:Zi =di(mi)− d¯(mi)σd(mi);Pi = 1− ∑m∈MS(di(m)di)2, (4.11)where mi is the subnetwork containing node i, di(mi) is the within-module degreeof i, i.e., the number of links between i and all the other nodes in mi, and d¯(mi) andσd(mi) are the corresponding mean and standard deviation of the within-module midegree distribution. We stress the prior edges by γ when relatively reasonable graphpartitioning has been achieved from the previous iteration, based on relatively highQ value and vice versa.The stopping criteria of the optimization on the seeds and corresponding prioredges is the convergence of the subnetwork extraction results, which is measuredby the Normalized Mutual Information (NMI) [202] between the subnetwork as-signments from two successive iterations.The initial seeds are set from a pre-partitioned subnetwork structure usingNcuts [171]. We have chosen Ncuts due to its wide use in brain graph studycommunity and its global optimality guarantees, making Ncuts outperform otherpartitioning methods mentioned in Section 2.1.3. We derive the prior edges inthe first iteration using the average weights between a node and all the remainingnodes within a particular subnetwork, followed by a normalization to guarantee∑Mk=1λ ki = 1. The average weights have been chosen to increase robustness tonoise.We note that we set negative values to zero in the connectivity matrix C, given106the currently unclear interpretation of negative connectivity [93]. Further, we mapthe connectivity matrix C to the graph weights w using a Gaussian kernel, where σis the decaying parameter which is estimated by averaging non-zero values of theconnectivity distance:wij = exp(−(1−Cij)22σ2). (4.12)4.2.5 ExperimentsWe evaluated our proposed provincial hub guided RW based approach by compar-ing our approach against conventional subnetwork extraction method which doesnot utilize multi-pronged graphical metrics nor the biological intuition. We havechosen Ncuts due to its wide use in the subnetwork extraction studies and its mer-its which were mentioned in Section 2.1.3. We applied the subnetwork extractionapproaches on the resting state function connectivity data.4.2.5.1 Materials4.2.5.1.1 Synthetic Data We generated 200 synthetic datasets that cover a widevariety of network configurations to simulate FC graphs using the technique from[203]. Each dataset comprised N = 500 regions and 4 scans of t = 1200 timepoints as in the real data. We set the number of subnetworks, M, to a random valuebetween 10 and 20 in each dataset. The number of ROIs in each subnetwork wasset to dN/Me+q with q being a random number between -2 and 2, and ROIs wererandomly assigned to subnetworks. We generated a N×N adjacency matrix, Σ,which was taken as the ground truth based on the network configurations. Next,we built a ΣF matrix, by randomly setting p% of the values in Σ to 1 to model howFC estimates are prone to false positives. p was randomly chosen from [0, 20].Time series were then generated by drawing random samples from N(0, ΣF). ThenGaussian noise was added to the time series with SNR randomly set between -6 and-3 dB. Finally, FC matrices were simulated by computing the Pearson’s correlationof these noisy time series.1074.2.5.1.2 Real Data We used the Resting State Functional Connectivity basedon MRI scans of 77 unrelated healthy subjects from the HCP dataset [146]. Twosessions of rs-fcMRI with 30 minutes for each session were available. Preprocess-ing applied was described in Section 1.8.3.1. We then used the Willard atlas [204]which has 499 ROIs to define the brain nodes. We chose Willard atlas since ithas subnetwork labels for 142 significant nodes belonging to 14 well-establishedsubnetworks [23], which we used as subnetwork label pseudo ground truth. Theremaining 357 nodes can be studied to verify if our RW based approach can detectthe nodes with low probabilities being assigned to any subnetworks. Voxel timecourses within ROIs were averaged to generate region time courses. The regiontime courses were demeaned, normalized by the standard deviation, and concate-nated across subjects. The Pearson’s correlation values between the region timecourses were taken as estimates of FC matrices.4.2.5.1.3 Parameter Setting We examined brain graphs at different graph den-sities from the range [0.005, 0.5] at an interval of 0.005 to test the robustness ofour approach and the local thresholding proposed in Section 3.2.2 was used. Thenumber of seeds needed within each subnetwork is dependent on the graph density.Denser graphs require more seeds to tackle the noisy connection problem. Empiri-cally, we set nk to 15 for density [0.005, 0.1], 20 for density [0.105, 0.3], and 25 fordensity [0.305, 0.5]. For the optimization stopping criteria, we empirically set thethreshold for the NMI of subnetwork assignments between successive iterations to0.99. All the results in our experiments have reached convergence.4.2.5.2 ResultsAll statistical comparisons are based on the Wilcoxon signed rank test with signif-icance declared at an α of 0.05 with Bonferroni correction.4.2.5.2.1 Synthetic Data We assessed our method and contrasted method bycomputing the DSC between the ground truth and the estimated subnetwork labels.Our proposed provincial hub guided RW based approach achieved statistically sig-nificantly higher DSC at 0.9868 ± 0.060 than the contrasted method at 0.9703 ±1080.046.4.2.5.2.2 Real Data We evaluated our approach by examining the overlap be-tween our extracted subnetworks and well-established brain systems comprising14 subnetworks [23] which we took as the pseudo ground truth. We show that ourproposed approach achieved statistically significantly higher average DSC (0.5542± 0.056) over a range of graph densities as compared to the contrasted method(0.5125 ± 0.054).We further verified that the posterior probability derived from our RW basedapproach match with the assignment of significant and non-significant nodes in theWillard atlas. Based on an exemplar brain graph which achieved the highest sub-network extraction DSC=0.6738 to the established brain systems [204]. Amongstthe 94 matched nodes out of the 142 significant nodes in the Willard atlas, theaverage posterior probabilities of a node belonging to the assigned subnetwork is0.6411, and an average value at 0.4864 probability has been found in the remaining357 non-significant nodes.4.2.6 DiscussionWe explored utilizing intrinsic properties of brain networks to incorporate a greaternumber of graphical metrics for improved subnetwork extraction. We proposed aprovincial hub guided random walker based optimization approach. We have cho-sen RW based approach since the manner in which the nodes are clustered intogroups based on the probabilities of walking to seeds, closely resembles the mech-anism whereby brain regions within a subnetwork are inter-linked via provincialhubs. Moreover, the posterior probability of a node belonging to subnetworks pro-vides a way to study the significance of a node.We have explained the outperformance of using multiple seeds over a singleseed within each subnetwork. It still remains as a challenge to determine the num-ber of seeds automatically. For simplicity, we set the number of the seeds to bethe same across different subnetworks based on graph densities; however, an ad-justable number should be determined based on the size of the provincial hubs andthe subnetwork size. Our future work on this open question will be identifying the109fully connected subgraphs which are derived from a provincial hub. Specifically,the major provincial hub will be detected based on the brain network properties,and a clique extraction based on the major hub will generate the nodes in the fullyconnected subgraphs as the multiple seeds.4.3 SummaryIn order to better extract the brain subnetwork structure, we have exploited intrin-sic properties of brain networks and incorporated a greater number of graphicalmetrics other than modularity into the process. At the same time, we built a modelinspired by the brain subnetwork’s biological nature.Specifically, we proposed a modularity reinforcement strategy based on a con-nection fingerprint similarity concept. We attained higher accuracy in subnetworkextraction on synthetic data, higher overlap to well-established brain systems andhigher reproducibility on the real HCP data compared to conventional state-of-the-art community detection methods.We also proposed a provincial hub guided random walker based model inspiredby the similarity between how the nodes are clustered into groups based on theprobabilities of walking to seeds and the mechanism of brain regions within a sub-network are inter-linked via provincial hubs. We devised a multi-seed strategy totackle the noisy connection problem. We fully utilized informative module-relatednetwork metrics such as hubs, within-module degree scores, and participant coef-ficients to guide the brain subnetwork extraction. We have demonstrated that in-corporating domain-related information and building a biological intuition inspiredmodel result in better subnetwork extraction in terms of subnetwork extraction ac-curacy on the synthetic data and the overlaps to well-established brain systems onthe real data.110Chapter 5Multimodal/Multisource BrainSubnetwork ExtractionThis chapter focuses on further improving brain subnetwork extraction based onmultimodal/multisource fusion technologies, which is based on the second part of[P6], [P7] and [P8].First, along the line of subnetwork extraction methods in the previous chapter,we further explore multimodal fusion techniques to improve subnetwork extrac-tion. We propose multimodal provincial hub guided approach to fuse AC and FC.Moreover, we propose a high order relation informed approach based on hyper-graph to combine information from rs-fcMRI and t-fcMRI for subnetwork extrac-tion. We further study multisource approach for overlapping brain subnetworkextraction using canonical network components, i.e., cliques, which we defined asco-activated node groups across multiple tasks. Based on the overlapping subnet-work extraction results derived from co-activated cliques, we study the subnetworkoverlaps and their relationships to network measures such as hubs.1115.1 Multimodal Random Walker based SubnetworkExtraction5.1.1 Related WorkCurrently, fMRI and dMRI are the most widely used modalities to estimate FCand AC for the brain subnetwork extraction purpose. We argue that combining FCand AC could help improve subnetwork extraction, as mentioned in Section 1.4.3.Most existing multimodal subnetwork extraction methods aimed to fuse the con-nectivity matrices from different modalities to estimate common patterns, such asrepresentative MultiView Spectral Clustering (MVSC) [85], and related co-trainingmethods [205–207]. MVSC achieves the agreement between two views by project-ing the affinity matrix of one view to the eigenspace of the other view [85]. Othermodels added in techniques such as regularization: CO-training with REGulariza-tion (COREG) [208] or overlapping subnetwork assumption: Coupled Stable Over-lapping Replicator Dynamics (CSORD) [203]. COREG enforces the view-specificeigenvectors to look similar by regularizing them towards a common consensus(centroid based co-regularization), and optimizes for individual clusterings as wellas the consensus using a joint cost function [208]. CSORD is a multimodal inte-gration technique based on a sex-differentiated formulation of replicator dynamics[203].Although studies have shown the close correspondence between anatomicaland functional connectivities and subnetworks [105, 113, 114], network analyseshave so far failed to demonstrate a clear one-to-one correspondence between net-work communities in AC and FC [142]. Despite that, recent studies suggest thathighly connected brain regions are greatly involved in establishing network-widecommunication, acting as the focal points in large-scale anatomical and functionalnetworks [142]. A computational model of the large-scale structure of the cerebralcortex suggested a partial correspondence between anatomical and functional hubseven at very short time scales (msecs. to secs.) [209].Here we intend to facilitate multimodal integration, but not by trying to findthe common patterns in the connectivity matrices as the existing methods, insteadby using important hubs to guide the subnetwork extraction process. Meanwhile,112the limitations of existing multimodal methods include the lack of exploitation ofbrain properties other than modularity and the mostly hard/crisp subnetwork as-signments. Based on our model, we incorporate a greater number of brain proper-ties from the graphical metrics and provide the probabilistic membership of a nodebelonging to subnetworks, rather than just a crisp assignment.5.1.2 Multimodal Provincial Hub Guided Random Walker ModelIn Section 4.2, we proposed an provincial hub guided RW based model by intro-ducing an prior edge based on the RW posterior probability, which resembles themechanism whereby brain regions within a subnetwork are inter-linked via provin-cial hubs. In the model, we update the seeds based on the subnetwork assignmentsfrom the previous iteration, and the corresponding prior edges using the probabili-ties of a node belonging to subnetworks using Equation 5.1 and Equation 5.2.(LU + γM∑r=1ΛkU)xkU = λkU +B fk, (5.1)L =[LS BB> LU + γI], (5.2)Inspired by the findings of close relationship between hubs in anatomical and func-tional networks (Section 5.1.1), we argue that the provincial hubs should be usedto integrate the information flow between the two modalities. We extend the pro-posed feedback informed iterative optimization model in Section 4.2 to enable mul-timodal integration based on the provincial hub guided RW model (henceforth re-ferred to as multi-modal Random Walker (mmRW)). The underlying assumptionis that subnetworks captured via multiple imaging modalities might share infor-mation through provincial hubs. By alternating the connectivity modalities for thenode-to-node edges within the iterative model in Equation 5.1, provincial hubs canbe used to guide the feedback information propagation across modalities until thesubnetwork assignments from different modalities converge, as measured by NMI.As illustrated in Figure 5.1, when the graph weights in the previous iteration t−1are defined by FC shown in blue edges, the weights in the current iteration t aredefined by AC shown in red edges. Similarly to the unimodal RW framework (Sec-113Figure 5.1: Schematic illustration of mmRW, where the graph weights in theprevious iteration t− 1 are defined by FC edges in blue and the seedsare found using the provincial hubs derived from AC (shown as red dashcircles), then the weights in the current iteration t are defined by ACshown in red edges and the seeds are found using the provincial hubsderived from FC (shown as blue dash circles).tion 4.2.4.4), we stress the prior edges by γ when relatively reasonable graph par-titioning has been achieved from the previous iteration, based on relatively higherQ value and vice versa.We base our connectivity estimation on a fingerprint formulation [65], wherewe derive the connectivity matrices C by estimating the cross-correlation betweenthe fingerprint profiles of each brain node pair. The AC fingerprint profile foreach brain node is defined as the fiber connection strength to the remaining brainnodes, namely, the number of tracts connecting two nodes normalized by the nodesizes. The FC fingerprint profile is defined as the cross-correlation between thetime courses of a particular node and all the remaining nodes in the brain. We alsoset negative values in C to zero before mapping C to the graph weights w using aGaussian kernel as in Equation 4.12.1145.1.3 Subnetwork Assignment Confidence and OverlappingSubnetwork Exploration Based on Posterior ProbabilitySame as Section 4.2.4.3, multimodal RW based subnetwork extraction approachprovides a confidence value that a given node belongs to a particular subnetwork(as represented by the probability). Thus the derived posterior probability can bestudied to find significant nodes (detailed results in paragraph 5.1.4.2.2), or thepotential overlapping subnetwork structure (detailed results in Section 5.3.3.4).5.1.4 Experiments5.1.4.1 Materials5.1.4.1.1 Synthetic Data Similarly as in Section 4.2.5.1, we generated 200 syn-thetic datasets that cover a wide variety of network configurations to simulate ACand FC graphs using the technique from [203]. Each dataset comprised N = 500regions and 4 scans of t = 1200 time points as in the real data. We set the number ofsubnetworks, M, to a random value between 10 and 20 in each dataset. The numberof ROIs in each subnetwork was set to dN/Me+q with q being a random numberbetween -2 and 2, and ROIs were randomly assigned to subnetworks. We gener-ated an N×N adjacency matrix, Σ, which was taken as the ground truth based onthe network configurations. Next, we built an AC matrix, ΣA, by randomly settingp1% of the values in Σ to 0 to model how AC estimates contain false negatives, andbuilt a FC matrix, ΣF, by randomly setting p2% of the values in Σ to 1 to modelhow FC estimates are prone to false positives. p1 and p2 were randomly chosenfrom [0, 20]. Two sets of time series were then generated by drawing random sam-ples from N(0, ΣA) and N(0, ΣF). Then Gaussian noise was added to the time serieswith SNR randomly set between -6 and -3 dB. Finally, AC and FC matrices weresimulated by computing the Pearson’s correlation of these noisy time series.5.1.4.1.2 Real Data We used the rs-fcMRI and dMRI scans of 77 unrelatedhealthy subjects from the HCP dataset [146]. Two sessions of rs-fcMRI with 30minutes for each session, and one session dMRI data were available for multimodal115integration. Preprocessing applied was described in Section 1.8.3.1. We then usedthe same Willard atlas [204] which has 499 ROIs to define the brain nodes. Voxeltime courses within ROIs were averaged to generate region time courses. Theregion time courses were demeaned, normalized by the standard deviation, andconcatenated across subjects. The Pearson’s correlation values of time coursesbetween one region and the remaining regions were taken as estimates of the FCfingerprint profiles. To compute the fiber counts between brain nodes, which weretaken as the AC fingerprint profile estimates, we warped the Willard atlas onto theb = 0 volume of each subject. Subject-wise AC fingerprint profiles were concate-nated for a group-level fingerprint profiles.We examined a range of graph densities of [0.005, 0.5] at an interval of 0.005using local thresholding in Section 3.2.2 to test the robustness of our approach. Thenumber of seeds within each subnetwork nk was set to 15 for density [0.005, 0.1],20 for density [0.105, 0.3], and 25 for density [0.305, 0.5], same as the previousSection 4.2.5.1.For the stopping criteria for convergence, we empirically set the threshold forNMI (from successive iterations) between inter-modality subnetwork assignmentsto 0.8. All the results in our experiments have reached convergence.5.1.4.2 Results5.1.4.2.1 Synthetic Data We compared our mmRW against our unimodal RWusing Equation 4.7, unimodal Ncuts, multimodal methods MVSC [85], COREG[208], and CSORD [203]. For unimodal techniques, we reported their performanceusing both AC and FC estimates. On 200 synthetic datasets, we assessed eachmethod by computing the DSC between the ground truth and estimated subnetworklabels as: DSC = 2|Lest⋂Lgnd |/(Lest +Lgnd), where Lgnd is the set of ROIs in theground truth subnetwork, and Lest is the set of ROIs in the estimated subnetworkmatched to Lgnd using Hungarian assignment [160]. mmRW achieved significantlyhigher DSC than each contrasted method at p = 10−5 based on Wilcoxon signedrank test with Bonferroni correction (Figure 5.2). We note that the numbers ofsubnetworks were set to the ground truth subnetwork numbers for all methods forsimplification.116Figure 5.2: Subnetwork extraction accuracy on synthetic data. Our mmRWapproach achieved significantly higher DSC than unimodal and existingmultimodal methods.5.1.4.2.2 Real Data Due to the lack of the ground truth, we first evaluated ourapproach by examining the overlap between our extracted subnetworks and a set ofwell-established brain systems [23] comprising 14 subnetworks which we took asthe pseudo ground truth. We derived the connectivity matrix based on the Willardatlas [204]. Both the atlas and corresponding subnetwork assignment were man-ually inspected and edited by neurologists [204], which included intensive userselection. 142 out of 499 ROIs in the Willard atlas were classified as significantbrain nodes, we thus compared the DSC on these significant nodes between theestimated and the established subnetwork assignments using our approach againstunimodal and representative existing multimodal approaches. We set the numberof subnetworks to 14 as the number of the established brain systems.mmRW approach achieved significantly higher average DSC over a range ofgraph densities compared to each contrasted method at p= 10−7 based on Wilcoxonsigned rank test with Bonferroni correction, as shown in Figure 5.3. We observedthat the results of using unimodal RW iterative model based on FC data can attainbetter results compared to AC based results. We assume that the false positives inthe FC estimation can be mitigated by using our multi-seed strategy, while the falsenegatives in the AC estimation is harder to tackle with. The other reason might bethe well-established brain systems were derived mostly by functional data. Butadding AC in improved the accuracy, which confirms the benefit of multimodal in-tegration. We note that the low DSC from CSORD could be caused by the implicit117Figure 5.3: Subnetwork extraction overlap to well-established brain system[23] on real data from HCP. Our mmRW approach achieved signifi-cantly higher DSC than contrasted methods.overlapping subnetwork assumption embedded in the approach while the pseudoground truth has a non-overlapping subnetwork setting.We next evaluated the inter-subject subnetwork reproducibility using the re-peatability of the group subnetwork extraction results with respect to the numbersof subjects. Specifically, we compared the group subnetwork assignments gen-erated from all 77 subjects against using only NS subjects with NS set between5 and 75 at interval of 5 subjects. The average repeatability over different num-bers of subjects as measured using DSC derived from our mmRW approach seemslower than our unimodal RW approach, but the difference is negligible and notstatistically significant at p = 0.5614 for RW based on AC and p = 0.1205 forRW using FC based on Wilcoxon signed rank test. However, the reproducibil-ity of our approach is significantly higher than contrasted multimodal method atp = 10−2 based on Wilcoxon signed rank test (Figure 5.4). We note that mmRWapproach only achieved comparable reproducibility compared to our unimodal RWapproach since the stability was not reinforced when information was pooled fromtwo different modalities. However, we still achieved higher reproducibility thanother multimodal approaches, confirming the superior stability of mmRW amongstmultimodal strategies.In order to utilize the posterior probability derived from our approach, we se-lected to show the subnetwork extraction results of an exemplar brain graph whichachieved the highest DSC=0.7682 to the established brain systems [204] using118Figure 5.4: Inter-subject reproducibility on real data from HCP. MultimodalRW approach achieved comparable DSC to unimodal RW approach, butsignificantly higher DSC than existing multimodal methods.mmRW. Amongst the 103 matched nodes out of the 142 significant nodes in theWillard atlas, the average posterior probabilities of a node belonging to the as-signed subnetwork is 0.6473 with a minimum value at 0.3118. On the other hand,an average value at 0.4689 and a minimum value at 0.1156 of probabilities havebeen found in the remaining 357 non-significant nodes. This result further con-firms that the nodes with higher probabilities derived from our approach matchedwell with those significant nodes within the 14 established brain systems. We fur-ther visualized the posterior probabilities by the size of the nodes in Figure 5.5 a-b.Here users have the option to focus on the bigger nodes by using desired thresh-olds of probability, or study the roles of nodes with lower probabilities, either themresiding in overlapping subnetworks or simply being insignificant.mmRW identified all commonly found subnetworks in the literature [204]. Theextracted dorsal default mode subnetwork (dDMN), visuospatial subnetwork andLeft Executive Control Network (LECN) are shown in Figure 5.5 c-e as exemplarresults.5.1.5 DiscussionWe proposed a provincial hub guided random walker based optimization approachfor brain subnetwork extraction that exploits brain properties and facilitates multi-modal integration of fMRI and dMRI. Our approach helps tackle the pronouncednoise in the neuroimaging data. On synthetic data, we showed that our approachachieved statistically significantly higher subnetwork identification accuracy than119(a) Right hemisphere (b) Left hemisphere(c) dDMN (d) visuospatial network (e) LECNFigure 5.5: Subnetwork extraction results on an exemplar brain graph. (a-b)Probabilities of a node being assigned to a given subnetwork (color-coded in 14 subnetworks), the larger size of a node indicates a higherprobability. (c-e) Exemplar subnetworks identified by mmRW.a number of state-of-the-art approaches. On real data, we demonstrated that ourestimated subnetworks matched well with established brain systems and attainedcomparable or higher inter-subject reproducibility. The majority of current ap-proaches typically generate hard subnetwork assignments, while our probabilisticapproach can provide users with options to examine the uncertainty of the subnet-work assignments. Our future work will focus on studying the roles of nodes withlow probabilities and their possible biological meaning, e.g., them residing withinoverlapping subnetworks as connector hubs, or simply being insignificant to beincluded into any specific functional subgroups.120Our proposed multimodal integration based on RW with prior model is essen-tially an implicit combined loss function in terms of Q = Qspatial+βQaspatial,where AC and FC are associated to one of the energy items respectively. Definingan explicit combined loss function is challenging since that there is no clear one-to-one correspondence between subnetworks from the two modalities. Instead, weused the provincial hubs to guide the integration in iterations, where the hubs havebeen shown, in recent studies, to reside at the focal points in both functional andstructural subnetworks [142]. Our approach has been proven to outperform a di-rectly combined loss function in the COREG method.We used the data from 77 healthy subjects from HCP dataset Q3 release. Wedid not use a dataset with much bigger size since we discovered the stability at60 subjects by examining the stability of the subnetwork assignments at the grouplevel with respect to the numbers of subjects. Specifically, we compared the sub-network assignments generated from all 77 subjects against using only NS subjectswith NS being set between 5 and 75 at interval of 5 subjects. The stabilities of thesubnetwork generated at the group level as measured using the DSC plateaued at0.65 after NS = 60.One observation from our results was that the improvement of using our ap-proach based on the synthetic data is slight, even though statistically significant.The reason could be that the noise added was not challenging enough. Gaussiannoise was added to the time series with SNR randomly set between -6 to -3 dB,corresponding to the typical levels seen in task-based fMRI data, i.e., between 0.2and 0.5. However, the approach was applied on resting state fMRI data, whoseSNR is hard to determine. Our future work will be adding experiments of morenoisy cases with lower SNR and more realistic percentage of missing connectionsbased on tract tracing gold standard from the macaque data, which is similar to thehumans.Our approach does not include an automatic strategy to determine the numberof the subnetworks, which is quite a challenging problem per se. Existing methodsare largely based on finding a transition point in the matrix eigenspectrum. How-ever, connectivity matrices typically do not display clear transition points as shownin Section 3.1. Our future work might include the multiscale version to study thehierarchy of the brain subnetworks.1215.2 Fusion of Task and Resting State FunctionalConnectivity for Subnetwork Extraction Based onHypergraphMost existing functional subnetwork extraction methods focus on resting statefunction connectivity data [171, 201], using functional homogeneity clustering,ICA, or graph community detection. However, resting state functional connectiv-ity is inherently with low SNR and prone to false positive and negative correla-tions [90]. Such noisy resting state functional connectivity information leads tounreliable subnetwork extraction results. Given the resemblance between restingstate and task functional subnetworks [111] and high order nodal relations reflectedfrom multi-task data, we here aim to incorporate information from task data intothe subnetwork extraction based on multilayer network. We explore if this integra-tion can improve the subnetwork extraction by exploiting the mechanism of howgroups of nodes collaborate together to execute a function and how these groupscommunicate with each other.5.2.1 Related Work - Relationship between Task and RestingFunctional ConnectivityRecent studies indicate that resting state functional activity actually persists duringtask performance [210], and similar network architecture is present across task andrest, which is supported by the existence of similar multi-task FC and resting-stateFC matrices that were averaged across subjects [211]. Studies have also shownthat there is a strong resemblance between rest and task subnetworks [106, 111].The spatial overlap between resting-state functional subnetworks and task-evokedactivities has been discovered [112, 212].Based on the close relationship between the two, resting state data have beenused to predict the task activities, by using group ICA to discover repertories ofcanonical network components that will be recruited in tasks [213]; by applyingthe graphical connectional topology of brain regions at rest to predict functionalactivity of them during task [212]; or based on a voxel-matched regression methodto estimate the magnitude of task-induced activity [214].On the other hand, aggregating brain imaging data from thousands of task re-122lated studies allowed the construction of ‘co-activation networks’, whose majorcomponents and overall network topology strongly resembled functional subnet-works derived from resting-state recordings [19, 46, 215].It has been suggested that networks involved in cognition are a subset of net-works embedded in spontaneous activity [111, 216], and a number of canonicalnetwork components in the pre-existing repertoires of intrinsic subnetworks areselectively and dynamically recruited for various cognitions [213, 217].5.2.2 Related Work - Multilayer Brain Network AnalysisMultilayer network has recently been used to model and analyze complex high or-der data, such as multivariate and multiscale information within the human brain[218]. Different layers can represent relationships across different temporal vari-ations [219], reflect different imaging modalities (such as task and rest) [218], ordifferent frequency bands [220], etc. Hypergraph is a type of multilayer graphs,in which edges can link any number of nodes [221]. Hypergraphs have been usedto identify non-random structure in structural connectivity of the cortical microcir-cuits [222], identify high order brain connectome biomarkers for disease prediction[223], and study relationships between functional and structural connectome data[224].5.2.3 High Order Relation Informed Subnetwork ExtractionOur assumption is that task data can be beneficial for subnetwork extraction sincethe repeatedly activated nodes in different tasks could be the canonical networkcomponents in the spontaneous resting state subnetworks. At the same time, themultilayer structure of repeatedly activated nodes across multi-task can be ele-gantly presented as a hypergraph. We propose a high order relation informed sub-network extraction model, which (1) facilitates multisource integration of task andrest data for subnetwork extraction, (2) utilizes information from the relationshipbetween groups of activated nodes across different tasks, and (3) enables the studyon higher order relations among brain network nodes.1235.2.3.1 FrameworkWe propose a high order relation informed approach based on hypergraph to inte-grate both resting state and task information for brain subnetwork extraction. Wefirstly construct a brain graph based on a certain parcellation atlas. Secondly, wedetect activation of brain nodes from task data to define the nodes for multiplelayers in the hypergraph, and define the connection strength between nodes us-ing task-induced connectivity. Thirdly, we construct the multitask hypergraph andincorporate resting state FC strength information when setting the weights of hy-peredges. Fourthly, we fuse task and rest FC using weighted combination modelbefore performing graphcut on the constructed graph.5.2.3.2 Notation Overview of Hypergraph5.2.3.2.1 Notations We here follow most of the notations presented in [221].Let V denote a set of nodes, and E denote a family of subsets e of V such that∪e ∈ E = V . Then we define G = (V ;E) a hypergraph with the vertex set V andthe hyperedge set E. A hyperedge containing just two nodes is a simple graphedge. A hyperedge e is said to be incident with a node v when v ∈ e. Two nodesare connected if they both belong to the same hyperedge. Two hyperedges are con-nected if the intersection of them is not an empty set, ei∩e j 6= /0. Given an arbitraryset X , let |X | denote the cardinality of X . A hypergraph G can be represented by a|V |×|E| incidence matrix H with entries h(v,e) = 1 if v∈ e and 0 otherwise, see anexample in Figure 5.6. A weighted hypergraph, G= (V ;E;w), is a hypergraph thathas a positive number w(e) associated with each hyperedge e, called the weight ofhyperedge e. Next, we define four important measures of hypergraph properties.For a hyperedge e ∈ E:1. We follow [221] to define its degree as d h(e) = δ (e) := |e|, which countsthe number of nodes that exist in the hyperedge. If one uses the incidence ma-trix, δ (e) := ∑{v∈V} h(v,e). Let De denote the diagonal matrices containing thehyperedge degrees. Take Figure 5.6 as an example, δ (e1) = 3, and δ (e2) = 2.2. We further define the hyperdegree of a hyperedge as the number of hy-peredges connected to it, denoted as d hH(e) := ∑{ei∈E,ei 6=e} e∩ ei. For example,124(a) Toy example of a hypergraph (b) Simple graph (c) Incidencematrix HFigure 5.6: Hypergraph and its corresponding simple graph and incidencematrix. Left: an hyperedge set E = {e1,e2,e3,e4} and a node setV = {v1,v2,v3,v4,v5,v6,v7}. Middle: the corresponding simple graph.Right: the incidence matrix H of the hypergraph on the left, with theentry (vi,e j) being set to 1 if vi is in e j, and 0 otherwise.d hH(e1) = 3, d hH(e3) = 2, and d hH(e4) = 0 in Figure 5.6.For a node v ∈V :3. We follow [221] to define its degree by d(v) = ∑{e∈E|v∈e}w(e). If oneuses the incidence matrix, d(v) = ∑{e∈E}w(e)h(v,e). When all w(e) = 1, d(v)counts the number of hyperedges which include this node: d(v) = ∑{e∈E|v∈e} 1,or d(v) = ∑{e∈E} h(v,e). Let Dv denote the diagonal matrices containing the nodedegrees.4. We then define the hyperdegree of a node as d H(v) := ∑{v∈e|e∈E} δ (e),which counts the number of nodes connected to a particular node across all hyper-edges. For example, d H(v2) = 5, d H(v3) = 6, d H(v5) = 3 in Figure 5.6. Itsweighted version will be estimating the strength between the connected node pairs.Next, let W denote the diagonal matrix containing the weights w(e) of hyper-edges. Correspondingly, the adjacency matrix A of hypergraph G is defined as:A = HWHT −Dv, (5.3)where HT is the transpose of H.1255.2.3.2.2 Graphcut of the Hypergraph One can group the nodes into subsetsusing graph partitioning methods, i.e., graphcut. The intuition is to find a parti-tion of the graph such that the edges within a subset have high weights (strongintra-class connections), and the edges between different subsets have low weights(weak inter-class connections). Let S ∈ V denote a subset of nodes and Sc de-note the complement of S. Follow the notations in [157], the adjacency matrixA(X ,Y ) := ∑i∈X , j∈Y ai j. For a given number M of subnets, the Mincut approach[225] implements the graphcut by generating a partition S1, . . . ,SM which mini-mizescut(S1, . . . ,SM) :=12M∑i=1A(Si,Sci ). (5.4)To solve the problem of separating individual nodes as a subset in Mincut,RatioCut [226] and Ncuts [227] have been proposed to encode the information ofthe size of a subset.RatioCut(S1, . . . ,SM) :=12M∑i=1A(Si,Sci )|Si| =M∑i=1cut(Si,Sci )|Si| , (5.5)where |S| measures the number of nodes in S.Ncut(S1, . . . ,SM) :=12M∑i=1A(Si,Sci )vol(Si)=M∑i=1cut(Si,Sci )vol(Si), (5.6)where vol(S) measures the volume of S by summing over the weights of all edgesattached to the nodes as vol(S) :=∑v∈S ds(v), and node strength ds(v) is the weightedversion of node degree d(v).Ncuts has been widely used in image segmentation and brain study commu-nity, since it utilizes the weight information. In the following, we show that Ncutsapproach can be generalized from simple graphs to hypergraphs, which has beenproven in [221].For a hypergraph G = (V ;E;w), a cut is a partition of V into two parts S andSc. A hypergraph e is cut when it is incident with the nodes in S and Sc at thesame time. The hyperedge boundary of S is defined as ∂S := {e ∈ E|e∩S 6= /0,e ∈E|e∩ Sc 6= /0}, which is a hyperedge set consisting of the hyperedges which are126cut [221]. The definition of the volume in a hypergraph vol(S) is the sum of thedegrees of the nodes in S, vol(S) := ∑v∈S d(v). Each hyperedge is essentially afully connected subgraph, then the edges in a subgraph is called subedges, beingassigned with the same weight w(e)/δ (e). When a hyperdege e is cut, there are|e∩S||e∩Sc| subedges are cut. Hence, the volume of ∂S is defined byvol(∂S) := ∑e∈∂Sw(e)|e∩S||e∩Sc|δ (e), (5.7)which is the sum of weights over the subedges being cut. By this definition, wehave vol(∂S) = vol(∂Sc). Similar to the simple graphs, Normalized hypergraphcut is to keep the high intra-class connection and low inter-class connection with apartition S1, . . . ,SM by minimizing the cut as below:argmin/0 6=S1,...,SM⊂VM∑i=1vol(∂Si)vol(Si). (5.8)5.2.3.3 Task Activation Detection - Node Definition in the HypergraphIn order to construct the multiple layers in the hypergraph, we apply the activationdetection technique on the task data to define the nodes that are contained in differ-ent hyperedges. The standard way of activation detection is to use a General LinearModel (GLM) where statistics, such as t-values, reflect the degree of the similaritybetween the stimulus and voxel time courses. The estimated statistics produce anactivation statistics map (t-map), followed by a thresholding of the map to identifythe activated voxels [228]. Due to the pronounced noise in the fMRI data, activa-tion detection at the individual level could be inaccurate [229]. In order to derivemore reliable task-induced activation, we have chosen a group activation detectionover the individual based approach. First, to compute the intra-subject activationpatterns, a standard GLM is applied as below [228]:Yi = Xiβ i+Ei, (5.9)where Yi is a t×N matrix of the task-induced fMRI time courses of N brain regionsfrom subject i, β i is a d×N activation matrix to be estimated, Ei is a t×N residual127matrix, and Xi = [Xtask|Xiconfounds] is a t × d matrix. Xtask is the task regressorsand Xiconfounds is the confound regressors. Next, we combine the activation resultsacross subjects to assemble a group activation map, which is used to define nodesfor each layer of the hypergraph. Specifically, we apply a max-t permutation test[230] on β i aggregated from all the subjects, which implicitly accounts for multiplecomparisons and control over false detections [86]. Group activation is declared ata p-value threshold of 0.05.5.2.3.4 Strength Informed Weighted Multi-task HypergraphIn the beginning of Section 5.2.3, we argued that multi-task information can be pre-sented as a hypergraph, with the hyperedges being different tasks, and the nodesin each hyperedge being the brain regions activated in a certain task. In the tra-ditional definition of hypergraph, nodes are connected to each other binarily, i.e.,the edge weights between a node pair are 1 if they are connected, or 0 otherwise.We here propose a strength informed weighted hypergraph model by incorporatingthe strength information from the connections between nodes. We further deter-mine the hyperedge weight w(e) using the graphical measures defined in para-graph 5.2.3.2.1.5.2.3.4.1 Pairwise Nodal Connection Strength Estimation In order to estimatethe strength of the connections between two nodes, we usually use the Pearson’scorrelations between time courses from pairs of brain regions. The computationfor resting state connectivity matrix Crest has been described in Section 1.8.3.1.To produce the task-induced connectivity matrix Ctask, we use the task-inducedtime course information. We follow the strategy in [211] to remove all inter-blockrest periods from all regions’ time courses, before computing the pairwise Pear-son’s correlations across all concatenated block/event duration time courses withina task. To keep the consistency when combining information from the nodes acrossdifferent layers, we keep all the Ctask having the same dimension of N×N as theCrest, then set the rows and columns of non-activated nodes to zero.1285.2.3.4.2 Proposed Strength Informed Weighted Hypergraph We present a mod-ified hypergraph cut criteria formulation based on Equation 5.7 to incorporate pair-wise nodal connection strength information from C as below in Equation 5.10. Thesymbol .˜ indicates the usage of strength information.˜vol(∂S) := ∑e∈∂Sw˜(e)∑i∈{e∩S}, j∈{e∩Sc}Cei jδ (e), (5.10)where ˜vol(∂S) is a strength informed version of vol(∂S) in Equation 5.7, Ce is theconnectivity matrix derived from the task corresponding to the layer e, and w˜(e)is the modified weight item in the hypergraph. We propose here to incorporatestrength information from the connectivity matrix and utilize the four hypergraphmeasures defined in paragraph 5.2.3.2.1 to determine w˜(e), whose nature is theimportance of the hyperedge in the hypergraph. Based on the definition of the fourhypergraph measures, we exploit their corresponding biological meanings to set˜vol(∂S) and w˜(e) as below:1. The degree of a hyperedge δ (e) counts the number of brain regions thatare activated in a task. To avoid the bias of the hyperedge size, ˜vol(∂S) should benormalized by δ (e).2. The hyperdegree of a hyperedge is defined as the number of hyperedges thatare connected to it. Higher value indicates that more frequently activated patternsin the brain activities exist in this hyperedge. Thus, w˜(e) should be proportional tod hH(e), i.e., w˜(e) ∝ d hH(e).3. The degree of a node counts the number of hyperedges that contain this node,and the biological equivalence is the number of different tasks in which one node isactivated. A node with a higher degree is similar to the definition of the connectorhubs residing within different subnetworks. Hence, w˜(e) should be proportionalto some statistics derived from d(v) of the nodes in a hyperedge e. We denote thestatistics computation method as stat here and it can be widely used statistics suchas average value (mean), median value (median) and maximum value (max). Thus,w˜(e) ∝ stat(d(v)).4. The hyperdegree of a node reflects the number of all other nodes that areconnected to it across all layers, which equals the number of connections from129other co-activated nodes to it across multiple tasks. The biological meaning of anode with a high value coincides with the definition of hubs. Hence, w˜(e) shouldbe proportional to some statistics derived from d H(v) of the nodes in a hyperedgee, i.e., w˜(e)∝ stat(d H(v)). Here, in order to incorporate strength information, weapply the weighted version of d H(v), the strength of the node d Hs(v) as definedin Equation 5.11, i.e., w˜(e) ∝ stat(d Hs(v)).d Hs(v) := ∑{v∈e|e∈E}∑u∈eCeuv, (5.11)where Ce is the task-induced connectivity matrix for the eth task.In order to utilize strength information and hypergraph measures, we proposethe w˜(e) formulation as below:w˜(e) := w1 ·d hH(e)+w2 · stat(d(v))+w3 · stat(d Hs(v)), (5.12)where w1,w2,w3 are free parameters to control the contributions of each measureto the hyperedge.5.2.3.5 Multisource Integration of Rest and Task fMRIGiven the close correspondence between task and rest connectivity architectureand subnetworks, we further extend the multi-task hypergraph model to integraters-fcMRI information. To do that, we use Crest for the pairwise nodal connectionstrength computation in Equation 5.11 as below:d Hs(v) := ∑{v∈e|e∈E}∑u∈eCrestuv , (5.13)Furthermore, we explicitly combine the two sources of task and rest data forsubnetwork extraction. We firstly fuse the multiple layers of the multi-task hyper-graph into one single layer, and secondly combine it with a resting state connectiv-ity layer. Given that the hypergraph cut criterion (Equation 5.7) is to evaluate theaggregated sum of the cuts across all the pairwise subedges (nodal connections)in the hypergragh, we propose to aggregate the strength information between nodepairs across all the layers. To do that, we transform the multiple pairwise nodal130connections across task layers (Equation 5.10) into one single nodal connection asbelow:C¯taski j =1TT∑k=1w˜(ek)δekCeki j , (5.14)where the subscript k = 1, . . . ,T is the indicator for tasks, T is the total numberof tasks available, and ek is the hyperedge in the kth layer of the hypergraph. Cekis the connectivity matrix derived using the time courses in the task k using theprocedure described in paragraph 5.2.3.4.1.We next explicitly combine the two sources by a linear weighted combinationbetween the aggregated multi-task connectivity matrix from above (Equation 5.14)and the resting state connectivity matrix in Equation 5.15 as below:Ct-r := γC¯task+(1− γ)Crest, (5.15)where γ a free parameter, which can be optimized by cross-validation, or deter-mined by the number of the tasks available. Our linear model for combining twosources, which are both derived from functional modality, was motivated by thestudy indicating a largely linear superposition of task-evoked signal and restingstate modulations in the brain [210]. We also explore combining the two by apply-ing a multislice community detection approach [231], which extends modularityquality function based on the stability of communities under Laplacian dynamicswith a coupling parameter ω to control over interslice correspondence of commu-nities.5.2.4 ExperimentsWe first investigated the similarity of connectivity between resting state and task-general and task-specific connectivity. To evaluate our proposed approaches, weassessed the graphical metric modularity Q value, the inter-subject reproducibil-ity and examined the biological meaning of subnetwork assignments. We appliedsubnetwork extraction on (1) resting state FC alone, (2) task-induced FC alone, (3)multi-task hypergraph, (4) multi-task hypergraph integrated with resting state con-nectivity strength, (5) weighted combination of (4) and resting state FC, (6) com-bination of (4) and resting state FC using multislice community detection method131[231].5.2.4.1 MaterialsWe used the resting state fMRI and task fMRI scans of 77 unrelated healthy sub-jects from the HCP dataset [146]. Two sessions of resting state fMRI with 30minutes for each session, and 7 sessions of task fMRI data were available for mul-tisource integration. The seven tasks are working memory (total time: 10:02),gambling (6:24), motor (7:08), language (7:54), social cognition (6:54), relationalprocessing (5:52) and emotion processing (4:32). Preprocessing applied was de-scribed in Section 1.8.3.1. We then used the HO atlas [57], which has 112 ROIs,to define the brain region nodes. We chose the well-established HO atlas becauseit sampled from every major brain system, and consists of the highest number ofsubjects with both manual and automatic labelling technique compared to othercommonly used anatomical atlases. We didn’t use the parcellation we generated inChapter 2 to avoid potential circularity or biases by using an independently iden-tified node community partition. We also didn’t use Willard atlas as the previoussection Section 5.1, since HO has bigger coverage of the brain with comprehensiveannotation of each region while Willard atlas doesn’t include the annotation of theanatomical areas of each region and only 142 out of 499 nodes are assigned withinthe functional systems. Voxel time courses within ROIs were averaged to generateregion time courses. The region time courses were demeaned, normalized by thestandard deviation. Group level time courses were generated by concatenating thetime courses across subjects. The Pearson’s correlation values between the regiontime courses were taken as estimates of FC matrices. Negative elements in all con-nectivity matrices were set to zero due to the currently unclear interpretation ofnegative connectivity [93]. For task activation, we applied the activation detectionon the seven tasks available following the steps described in Section 5.2.3.3.We summarize here the annotation of the graphs for six methods being evalu-ated for subnetwork extraction. (1) Resting state FC matrix Crest is used. (2) Thetask general FC Ctask was generated by concatenating the time courses across alltasks before the Pearson’s correlation. In (3), we use task-specific FC in Equa-tion 5.11 and Equation 5.12 for each hyperedge, denoted as Chyper-task. We imple-132ment (4) by using resting state FC in Equation 5.11 and Equation 5.12 as describedin Section 5.2.3.5, denoted as Chyper-t-r. For (5), we first generate C¯taski j by usingtask-specific FC as Ceki j , and resting state Crest to compute w˜(ek) based on Equa-tion 5.11 and Equation 5.12. We next applied our proposed local thresholding [5]on resting state FC Crest to match with the graph density of C¯task at 0.2765, whichlies within the normal range of thresholding before subnetwork extraction between[0.2, 0.3] [171]. We then estimate Ct-r using Equation 5.15. We empirically set freeparameters w1,w2,w3 to one, and the stat to median value. For (6), we generatedthe C¯taski j and thresholded Crest as the same way as in (5), then the multisource inte-gration is implemented using a multislice approach [231], denoted as Ct-r-multislice.We set the weighting for multisource integration γ or coupling parameter ω from0.01 to 1 at an interval of 0.01. In order to perform fair comparison, Crest in method(1) and Ctask in method (2) have also been local thresholded at the graph densityof 0.2765. Method (1) to (5) used Ncuts and (6) used generalized Louvain as thegraph partitioning approach. The number of subnetworks was set to seven giventhat there are seven tasks available to examine if subnetwork assignments can berelated to tasks. We note that setting the number of subnetworks is non-trivial asdiscussed in the previous section that we leave as future work. All statistical com-parisons are based on the Wilcoxon signed rank test with significance declared atan α of 0.05 with Bonferroni correction.5.2.4.2 Similarity of FC Between Resting State and Task DataWe observed a similarity at DSC = 0.7845 between resting state FC and task gen-eral FC, which was generated by concatenating the time courses across all differenttasks. For seven specific tasks, the corresponding DSC between task-specific FCand task general FC are 0.8971 for emotion processing, 0.8557 gambling, 0.8676for language, 0.9043 for motor, 0.8594 for relational processing, 0.8307 for socialcognition, and 0.8751 for working memory. This high similarities confirms thefindings in [211] that a set of small but consistent changes common across taskssuggests the existence of a task-general network architecture distinguishing taskstates from rest.When resting state FC is compared to task-specific FC, the DSC are 0.7193133for emotion processing, 0.7689 for gambling, 0.7390 for language, 0.7067 for mo-tor, 0.7533 for relational processing, 0.7659 for social cognition and 0.7118 forworking memory, respectively. The variation of similarities between task-specificand resting state FC around a relatively high average level further confirms that thebrain’s functional network architecture during task is configured primarily by anintrinsic network architecture which can be present during rest, and secondarily bychanges in evoked task-general (common across tasks) and task-specific network[211].These findings confirms the close relationship between task and rest, and thesupport for integrating multitask information into resting state based subnetworkextraction.5.2.4.3 Modularity Q ValueModularity Q value, defined in Equation 4.3, has been used to assess a graph parti-tioning through reflecting the intra- and inter- subnetwork connection structure ofa network [106]. We observe that Q values of group level subnetwork extractionfor method (1)-(6) are 0.1401, 0.1282, 0.1624, 0.1711, 0.2290 and 0.1905 when γand ω were selected at the highest inter-subject reproducibility.At the subject-wise level, the modularity Q values estimated from the subnet-work extraction using method (1)-(6) are 0.1397±0.0142, 0.1234±0.0159, 0.2072± 0.0199, 0.2094±0.0189, 0.2183±0.0192, and 0.2089±0.0165 respectively, Fig-ure 5.7.We show that the modularity estimated from subnetworks extracted based onsimply concatenating task time courses is lower than using resting state data. Usinghypergraph framework (3) Chyper-task and (4) Chyper-t-r achieves statistically highermodularity values than using either resting state data or simple concatenation oftask data. Moreover, incorporating resting state information into the hypergraphframework (5) Ct-r can increase modularity compared to hypergraph method. Mul-tislice integration (6) Ct-r-slice results in a lower modularity than (5) the linearmodel; however, it still outperforms all the other uni-source methods. Overall,incorporating resting state information explicitly using a weighted combinationstrategy, i.e., method (5) gives a statistically higher modularity than all contrasted134(1)Crest (2)Ctask (3)Chyper-task (4)Chyper-t-r (5)Ct-r (6)Ct-r-multislice00.050.10.150.20.250.30.35Figure 5.7: Subject-wise level modularity Q values using Method (1)-(6). Formethod (5) and (6), parameter γ andω were selected at the highest inter-subject reproducibility.methods at p< 10−4 based on Wilcoxon signed rank test. We note that the Q valuesderived here are around 0.2, when the number of the subnetworks was set to seven,i.e., the number of tasks. It is relatively low due to the inherent resolution limitof Q, i.e., Q decreases when the number of subnetworks increases. We exploredthis direction by achieving the similar level of Q values around 0.3-0.4 when thenumber of subnetworks decreases to 4 as in [19].5.2.4.4 Inter-subject Reproducibility of Subnetwork ExtractionWe assessed the inter-subject reproducibility by comparing the subnetwork ex-traction results using subject-wise data against the group level data. The aver-age DSC between subject-wise and group level subnetworks across 77 subjectsbased on methods (1)-(6) are 0.6362±0.0828, 0.5704±0.0872, 0.7083±0.1094,0.7258±0.1201, 0.7561±0.1199, and 0.7406±0.0725, Figure 5.8. We noticed thatthe reproducibility using resting state FC Crest is higher than simple concatena-tion of task time courses data Ctask. It could be that there exist great differencesin reaction to stimuli from different subjects, and simple concatenation is hard todiscover the higher order relationship between canonical network components. Onthe other hand, analyzing multi-task information using hypergraph (3) Chyper-task135(1)Crest (2)Ctask (3)Chyper-task (4)Chyper-t-r (5)Ct-r (6)Ct-r-multislice0.30.40.50.60.70.80.91Figure 5.8: Subject-wise level inter-subject reproducibility of subnetwork ex-traction using Method (1)-(6). For method (5) and (6), parameter γ andω were selected at the highest inter-subject reproducibility.achieved much higher stability in subnetwork extraction, and incorporating rest-ing information implicitly within the hypergraph (4) Chyper-t-r, or explicit weightedcombination (5) Ct-r can even further enhance reproducibility. We note that theweighted combination outperforms multislice integration (6) Ct-r-slice, which is stillbetter than all the other uni-source methods. The reason could be that a simplelinear model suffices the fusion of task and rest data. Overall, the inter-subject re-producibility derived by (5) Ct-r is statistically higher than all contrasted methodsat p < 10−4 based on Wilcoxon signed rank test.5.2.4.5 Biological MeaningWe next examined the biological meaning of the subnetworks extracted from method(1) - (6), where γ was set to 0.5 to report the results when resting state and hyper-graph based multitask information are equally combined as an example. Sevensubnetworks were extracted based on the number of tasks available. Method (1)detects most of the traditional resting state subnetworks with several false positiveand negative detection. The results of method (2) oftentimes combined some im-portant regions from different subnetworks, which lacks biological justifications.Method (3) and (4) generate similar results and both improve the results of method136(2) greatly when bringing task dynamics into the subnetwork extraction. Over-all, method (5) detects brain regions, which are more biologically meaningful, bycombining the intrinsic network architecture from resting state data and the taskdynamics based on high-order hypergraph. We report our findings in details as thefollowing and the visualization of subnetwork extraction results can be found inFigure 5.9.Using method (1) based on resting state FC alone, subnetwork 1 and 6 are de-tected as left and right side of a combination of ECN and frontoparietal network,which include superior frontal gyrus, middle frontal gyrus, inferior frontal gyrus,posterior supramarginal gyrus, angular gyrus, frontal orbital cortex, and frontal op-erculum cortex. Method (1) mistakenly classified left inferior lateral occipital cor-tex and left anterior supramarginal gyrus into the LECN. Anterior supramarginalgyrus is part of the somatosensory association cortex, which interprets tactile sen-sory data and is involved in perception of space and limbs location or languageprocessing, thus it should be included in DMN instead of ECN [232]. On the otherhand, our proposed method (5) detects both the left and right sides of most of theanterior portion of ECN and posterior supramarginal gyri for subnetwork 1. Usingmethod (5), the left inferior lateral occipital cortex was not include in ECN, whichis more accurate. Besides, method (5) clustered anterior supramarginal gyrus sym-metrically into subnetwork 6, which includes both sides of Posterior CingulateCortex (PCC), precuneus, and angular gyrus, comprising most of the posterior por-tion of DMN defined in [232]. As for method (2), the simple concatenation ofmultitask time courses, subnetwork 1 consists of frontal medial cortex and only theleft side of frontal orbital cortex, and subnetwork 6 consists of most of the ante-rior portion of ECN, angular gyrus and only the left posterior supramarginal gyrus,which should be symmetrically included in DMN. Besides, there are two otherROIs, left subcallosal cortex and left caudate, included in subnetwork 6, whichlacks biological meaning. Subnetwork 1 derived from method (3) and (4) bothconsist of most of the anterior portion of ECN, except that method (3) has twomore one-sided frontal areas, which makes (4) more biological meaningful (withsymmetric results). Subnetwork 6 of method (3) and (4) both consist of one iso-late area: left anterior parahippocampal gyrus, which further indicates that thereis need to incorporate resting state information into the multitask based on hyper-137(a) Method (1) Crest (b) Method (2) Ctask(c) Method (3) Chyper-task (d) Method (4) Chyper-t-r(e) Method (5) Ct-r (f) Method (6) Ct-r-sliceFigure 5.9: Visualization of subnetworks extraction using methods (1)-(6).The brain is visualized in the axial view. The mass center of each ROIis plotted in the MNI space and colorcoded by the membership of sevensubnetworks.138graph framework.Subnetwork 2 of method (1) includes both sides of Anterior Cingulate Cortex(ACC), caudate, thalamus, putamen and accumbens. Method (5) includes all thesame brain regions as method (1) plus one other region, the insula. This subnetworkshould be related to the gambling task and emotional processing, which expect toactivate ACC [233, 234], ventral striatum (such as thalamus [234] and accumbens[235]), and insula [236]. Usually insula is part of the salience network and has beenfound to play key roles in emotional processing [237]. However, using method (1),the insula was clustered into subnetwork 5 (mostly motor system). Method (2)included right ACC and both sides of PCC, precuneous, left side of supracalcarinecortex, and accumbens inside subnetwork 2, which seems like a mixture of part ofDMN, one-sided region from motor system, and one region from gambling system.As for method (3) and (4), they both extracted similar regions for subnetwork 2 asusing method (5), except that they missed thalamus and falsely included left frontalmedial cortex.Subnetwork 3 derived from method (1) includes superior lateral occipital cor-tex, frontal medial cortex, left subcallosal cortex, PCC, precuneous, parahippocam-pal gyrus, temporal fusiform cortex, brain stem, hippocampus and amygdala. Thisassignment does not make too much sense by clustering regions from visual, au-ditory, emotion circuit and frontal system together. Meanwhile, the results usingmethod (5) consists mostly of emotion circuit and social processing, which in-cludes brain stem [238], hippocampus and parahippocampal gyrus [239], amyg-dala [240], and subcallosal cortex [241]. Method (5) also detected regions relatedto auditory functions such as temporal pole, which is reasonable since the negativeemotion was induced by listening to stories. Subnetwork 3 detected by method (2)includes right anterior parahippocampal gyrus, temporal fusiform cortex and brainstem, which still lacks important brain regions in the emotion circuit. Method (3)detects more biologically meaningful regions than (2), such as hippocampus andamygdala. Using method (4) can even detect more related regions than method (3),such as frontal orbital cortex [242].Method (1) and (5) detected almost the same brain regions for subnetwork4, which is the visual system, except that method (5) detected one more regionof left inferior lateral occipital cortex, making the results more symmetric. This139subnetwork includes inferior lateral occipital cortex, intracalcarine cortex, cunealcortex, lingual gyrus, occipital fusiform gyrus, temporal occipital fusiform cortex,occipital pole, and supracalcarine cortex. Method (2) detected most of the visualregions except for cuneal cortex and the right supracalcarine cortex. Method (3)and (4) detected extra regions in right ECN and auditory system besides all theregions found using (5) in the visual system.Subnetwork 5 derived from method (1) comprises of the motor system, includ-ing precentral gyrus, postcentral gyrus, only the right side of anterior supramarginalgyrus, juxtapositional lobule cortex; and the frontoparietal network including leftcentral opercular cortex, superior parietal lobule, and parietal operculum cortex.Method (5) generated similar results as method (1), only that the results are moresymmetric, which include both sides of anterior supramarginal gyrus (part of so-matosensory association cortex); and more accurate in terms of frontoparietal net-work, which includes frontal operculum cortex instead of central opercular cortex.Both method (3) and (4) generated similar regions for subnetwork 5 as well, whichincludes motor system and frontoparietal network, except that they both includedbrain stem into this subnetwork. However, method (2) mis-classified insula, puta-men and thalamus into the motor and frontal parietal networks. We note that themotor system and frontoparietal network are clustered together, it could be that theworking memory tasks recruited both the motor system and frontoparietal network.As for the subnetwork 7, both method (1) and (5) detected brain regions cor-responding to language task and related auditory regions, such as anterior superiortemporal gyrus, planum temporale, planum polare, and Heschls gyrus (includesH1 and H2) [243]. Different from method (1), method (5) included central opercu-lar cortex, which can be explained by how fronto-opercular is related to language[244]. Method (2) detected some false positive brain regions in the language sys-tem such as parahippocampal gyrus, hippocampus and amygdala. Method (3) and(4) correctly clustered all the brain regions into the language network as method(5).Method (6) generated similar results compared to method (5), only a coupleregions in subnetworks 2 and 5 were switched, a couple regions in subnetwork 6and 7 were switched, and a couple regions in 1 and 6 were switched. Overall, Thesubnetwork results derived by method (5) Ct-r have more biological meaning than140contrasted methods.5.2.5 Discussion5.2.5.1 Hypergraph Encodes Higher Order Nodal RelationshipSubnetwork results derived from methods based on hypergraph achieved highermodularity, higher inter-subject reproducibility, and more reasonable biologicalmeaning than traditional connectivity analysis of pairwise correlation between nodes.These results indicate that hypergraph, which is a natural presentation of multi-task activation, can be explored to study higher order relations among the networknodes. The proposed strength informed version of automatic weight setting of thehyperedge incorporates connectivity information to reveal more accurate higherorder relationship among nodes rather than just using binary information.5.2.5.2 Multisource Integration Improves Subnetwork ExtractionWe have proved that multisource integration of task and rest information can im-prove subnetwork extraction compared to using a single source in terms of graph-ical metrics, inter-subject reproducibility, along with biologically meaningful sub-network assignments. We note that the implicit integration of rest information intomultitask hypergraph achieved less improvements as the explicit integration basedon the linear combination. The reason could be that the limited number of tasksavailable restricts the comprehensive representation of the brain using the hyper-graph. Thus, by integrating rest data to compensate possible missing informationresulted in overall better outcomes. Another observation is that the linear combi-nation outperforms the multislice community detection, which still performs betterthan uni-source approaches. Our assumption is that rest and task FC are both de-rived from a single functional modality, which complements each other by reveal-ing the two sides of FC, i.e., the resting intrinsic side and the activated evoked side.Thus, a simple linear weighted combination would suffice this situation, whichoutperforms other alternative combination approach in practice.1415.2.5.3 Limitations and Future DirectionsThere are several limitations in our present work. First, our study investigated onlyseven available tasks with high quality data and decent amount of data per task.This sample of seven tasks is not enough. A possible solution is to have access toboth task and rest data from previous task studies or co-activation studies, whichcover much wider variety of tasks. At the same time, with much more informationfrom a greater amount of task data, we can devise a reliable automatic manner todetermine the integration weighting parameter γ . The underlying rationale is thatwith more tasks available, we can rely more on the hypergraph based multitasksource, hence the higher γ .Secondly, we set the number of the subnetworks to be seven, which corre-sponds to the number of tasks available. The reason is simply to see if we can as-sociate the subnetwork results to different tasks and gain insights from the findingsbased on task-induced functions. We note that setting the number of subnetworksis non-trivial as discussed in the previous chapters. Our future work will focus onexploring a finer scale of subnetwork extraction using multi-scale hierarchical ap-proach, such as recently proposed multislice approach [211], which would improvethe interpretation of the findings.Moreover, we have explored multimodal integration of resting state FC and ACin the previous section, and we are to investigate if combining FC along with taskinformation with AC information will further improve the multimodal subnetworkextraction. We have conducted preliminary studies on applying Ct-r and AC as twomodalities using our proposed RW based approach. However, more systematicvalidation approaches and related neuroscience study evidences are required forcomparing three sources with two sources.5.3 Clique Based Multisource Overlapping BrainSubnetwork Extraction5.3.1 Related Work - Overlapping Brain Subnetwork ExtractionThe mainstream of brain subnetwork extraction and standard definition of modular-ity focus on nonoverlapping definition. However, studies have shown evidences of142the existence of overlapping brain subnetworks, hence the methods for nonover-lapping subnetwork extraction are limited by neglecting inclusive relationships[99]. There are emerging approaches for discovering overlapping modular networkstructure, which implies that single nodes may belong in more than one specificmodule. We here summarize some representative approaches used in brain sub-network extraction application, and detailed information can be found in a reviewpaper on general overlapping community detection [245] .The Clique Percolation Method (CPM) is one of the earliest methods for over-lapping community detection [246]. It is based on the assumption that commu-nities tend to be comprised of overlapping sets of cliques, i.e., fully connectedsubgraphs. It identifies overlapping communities by searching connected cliques.First, all cliques of a fixed size k must be detected, and a clique adjacency ma-trix is constructed by taking each clique as a vertex in a new graph. Two cliquesare considered connected if they share k-1 nodes. Communities are detected cor-responding to the connected components of the clique adjacency matrix. Since avertex can be in multiple cliques simultaneously, mapping the communities fromthe clique level back to the node level may result in nodes being assigned to multi-ple communities [106, 245]. The limitation of CPM is that it operates on binarizedgraph edges, thus cannot handle weighted graphs [86].A new definition of modularity has been proposed to discover the overlappingsubnetwork based on unbiased cluster coefficients using resting state connectivity[99]. However, methods based on the modularity function Q suffer from degeneratepartitions and resolution limit [106].Another line of studies is to transform a network into its corresponding linegraph, where the nodes represent the connections in the original network. Thus, thenonoverlapping community detection (modularity maximization used in [247] andagglomerative hierarchical clustering used in [248]) on the line graph will result inoverlapping subnetworks in the original network. There exist inherent limitationsin the nonoverlapping community detection used for the line graph (resolution limitfor modularity maximization and local sub-optimum for hierarchical clustering).Fuzzy community detection algorithms quantify the strength of association be-tween all pairs of communities and nodes [245]. Fuzzy k-means clustering [249]and fuzzy affinity propagation [250] have been applied to detect overlapping brain143subnetwork extraction. However, one has to use an ad hoc threshold for extractinginteracting nodes or independent nodes from the membership vector.Local expansion and optimization algorithms grow a natural community [251]or a partial community based on local benefit functions [245]. One example isConnected Iterative Scan (CIS), which has been explored for brain subnetworkextraction [252]. Taking each node as a partial subnetwork, CIS expands the sub-network by determining if any other nodes belong to this existing subnetwork usinga local function to form a densely connected group of nodes. Its limitation is thesensitivity to a density factor that controls subnetwork size [253]. Another goodexample is the Order Statistics Local Optimization Method (OSLOM) [254], whichuses statistical significance of a subnetwork when tested against a global randomlygenerated null model during community expansion. OSLOM has been shown tooutperform many state-of-the-art community detection techniques.In a previous work from our lab, the Replicator Dynamics (RD) concept fromtheoretical biology for modeling the evolution of interacting and self-replicatingentities was used to identify subnetworks. Further, the RD formulation was ex-tended to enable overlaps between subnetworks by incorporating a graph aug-mentation strategy [255]. This approach, Stable Overlapping Replicator Dynam-ics (SORD) [253], has demonstrated its superiority over many commonly usedoverlapping subnetwork extraction methods, including OSLOM.Most of the algorithms aforementioned are based on one single source, such asresting state functional connectivity. CSORD, the multimodal version of SORD,is one of the few overlapping methods which considers multi-source information.CSORD is based on survival probabilities of different genders in evolution andgraph augmentation [255]. However, its theoretical background for overlappingassumption based on graph augmentation has relatively indirect neuroscientific jus-tifications. We here explore the direction of integrating multisource information forthe overlapping subnetwork extraction by using the straightforward clique concept.5.3.2 Co-activated Clique Based Multisource OverlappingSubnetwork ExtractionRecent study has indicated that repeatedly activated nodes in different tasks couldbe canonical network components in the pre-existing repertoires of intrinsic sub-144Figure 5.10: The schematic illustration of multisource clique based overlap-ping subnetwork extraction approach.networks [213], we argue that the clique concept closely resembles groups of nodeswhich are the canonical network components. Based on the basic observation thattypical communities consist of several cliques that tend to share many of theirnodes [246], clique-based approach would be a straightforward way to find over-lapping brain subnetworks. However, the existing clique-based subnetwork ex-traction approach CPM (kclique) [246] has three major limitations that it can onlyhandle binary graphs, but not weighted graphs; the size k of cliques is fixed, whichneeds to be adjusted for different types of networks; and it only uses uni-sourceinformation. In order to tackle the aforementioned limitations, we here propose amultisource subnetwork extraction approach based on co-activated clique, which(1) uses task co-activation and task connectivity strength information for cliqueidentification, (2) automatically detects cliques with different sizes having moreneuroscientific justifications, and (3) shares the subnetwork membership, derivedfrom multisource hypergraph based approach we proposed in the previous section,among nodes within a clique for overlapping subnetwork extraction. The schematicillustration of our approach is shown in Figure 5.10.We first detect co-activated groups of brain nodes across different tasks basedon an activation fingerprint idea, and then identify densely connected cliques based145on task-induced weighted connectivity. Core cliques are further detected usingclique properties we defined. The nodes within a clique should belong to thesame subnetworks due to the close relationship between nodes in a fully connectedclique, we thus share the subnetwork membership of nodes within a clique to fa-cilitate overlapping subnetwork assignment. The initial subnetwork membershipfor each node is derived from non-overlapping subnetwork extraction technique,which is based on the fusion of resting state connectivity and task information em-bedded with high order relations using hypergraph.The difference of our approach from the traditional uni-source kclique methodis the utilization of both the task co-activation information and the connectivityweights (only the binarized connectivity is used in kclique method). The co-activated cliques derived using our approach have flexible clique sizes, whichhas more neuroscientific justifications than the fixed k. Besides, we explore ifour proposed clique node subnetwork membership sharing idea can generate morestraightforward and biologically meaningful results than the existing multisourcemethod CSORD.5.3.2.1 Clique Identification Based on Task Co-activationWe define cliques as co-activated groups of brain nodes that are densely connectedin our approach. We first identify the co-activated groups of brain nodes (coarsecliques) using an activation fingerprint idea. Then we refine coarse cliques intocliques, within which nodes are densely connected to each other based on task-induced connectivity information. We denote the clique set as CS, and the coarseclique set as CCS. Given T different tasks, one can construct a hypergraph withan N×T incidence matrix H, where N is the number of brain regions and T is thenumber of tasks (hyperedge e). h(v,e) = 1 when the brain region node v is activatedin the task corresponding to hyperedge e. The task-induced connectivity matrixCtask is generated by removing all inter-block rest periods from all regions’ timecourses and computing pairwise Pearson’s correlations of time courses which wereconcatenated through block/event durations across all the tasks. The underlyingassumption for our clique identification is that nodes in the same clique should beco-activated across tasks at times from t = 1 . . .T , where t indicates the number146of tasks, in which the nodes are co-activated. There are two steps involved in ourclique identification, which (1) pre-selects sets of coarse cliques in all T layers, (2)and refines the coarse cliques into cliques.The approach starts with a pre-selection of coarse cliques CCS, which mightinclude loosely connected nodes that are co-activated. Take each row from theincidence matrix H as a activation fingerprint vector f corresponding to the taskactivation pattern of a node. For example, if one node is activated in the 1st, 3rdand 6th out of the seven tasks, the corrsponding f = [1010010]. We next operatebit-wise and between the fingerprints from a node pair {i, j}, which gives us anoutput fingerprint vector of co-activation patterns SFi j:SFi j = fi ∧ f j, (5.16)where fi and f j are the activation fingerprint vectors of node pair i and j, and SFis the matrix containing the co-activation fingerprint vectors between the nodesin each node pair. We then define a matrix NT which counts the number of co-activated tasks between two nodes:NTi j =T∑t=1SFi j(t), (5.17)where SFi j is the co-activation fingerprint vector of length T . Next, we define thenode set PS=t which contains nodes that are co-activated together for t times as:PS=t =⋃∀i, j s.t. NTi j=t{i, j}, (5.18)and define the node set PS>t which contains nodes that are co-activated togetherfor greater than t times as:PS>t =⋃∀i, j s.t. NTi j>t{i, j}. (5.19)Based on the definition above, we follow the four steps as below to identify thecoarse cliques.Step 1 We extract Mt pre-selected sets of co-activated coarse cliques from the147nodes in PS=t . We identify {CCS=t1 ,CCS=t2 , . . . ,CCS=tMt} by ensuring all the nodepairs within a certain set share the same co-activation fingerprint vector in SF:CCS=tm = {pm1 , pm2 , . . . , pmNm | ∃ pmi , pmj ∈ PS=t , s.t. SFpmi pmj = SFpm1 pm2 }. (5.20)where m = 1, . . . ,Mt . The minimal rank of CCS=tm is 2, being only one node pairwithin a coarse clique. The nodes identified in a coarse clique are fully connectedto each other defined by sharing the same co-activation pattern.Step 2 Similarly, we extract Mt extended sets of co-activation coarse cliques,{CCS>t1 ,CCS>t2 , . . . ,CCS>tMt}, from the nodes in PS>t , based on the co-activationpatterns between nodes in CCS=tm and PS>t :CCS>tm =⋃∀ i∈CCS=tm , ∃ j∈PS>t , s.t. SFi j∧SFpm1 pm2 =SFpm1 pm2{ j}. (5.21)We do not consider the coarse clique set selection for the nodes which only exist inthe node set PS>t for the tth layer, since those will be selected in the pre-selectedsets in t+1 layer.Step 3 We then generate Mt coarse clique sets by merging the pre-selected andextended sets together as:CCSt = {CCSt1,CCSt2, . . . ,CCStMt}CCStm =CCS=tm ∪CCS>tm , m = 1, . . . ,Mt(5.22)Step 4 Extract the coarse clique set CCS across layers in the order from T to 1:CCS =⋃t=T,...,1CCSt . (5.23)The second part of clique identification is to refine the coarse cliques intocliques. When we extract CCS= {CCS1,CCS2, . . . ,CCSM}, there still exist looselyconnected nodes in the coarse cliques, mostly from lower layers when t is small, es-pecially when t = 1. Hence, we subsequently extract cliques based on the strengthinformation from task-induced connectivity matrix Ctask and hypergraph proper-ties. We formulate a coarse clique set, CCSk = {pk1, pk2, . . . , pkMk} where there are148Mk nodes within, as an Mk×Mk simple graph with the weights between nodes be-ing the task-induced connectivity pairwise edge strength. We next apply a localthresholding [5] on the Mk×Mk connectivity matrix Ctask-k to find out the mostclosely connected nodes to each node, and binarize the thresholded matrix to gen-erate an adjacency matrix Atask-k. We then transform Atask-k into its hypergraphHtask-k using Equation 5.3: A = HWHT −Dv, where the locations with 1 in eachhyperedge correspond to the nodes that comprise a fully connected subgraph, i.e.,cliques CSc. We extract Nc cliques:CS = {CS1,CS2, . . . ,CSNc}. (5.24)5.3.2.2 Clique Property ComputationWe present three properties that can be derived to study the cliques for furthernetwork analysis.(1) Co-activation times NT c of a clique CSc, i.e., the number of ones in theclique co-activation fingerprint:CSFc =∧∀ i∈CScfi, (5.25)then the co-activation times:NCOAc =T∑t=1CSFc(t). (5.26)(2) Activation times in a clique:NAc =1|CSc| ∑∀ p∈CSc ∑t=1,...,Tfp(t). (5.27)(3) Clique overlap ratio - the times of a clique overlaps with other cliques di-vided by the size of a clique, i.e., the number of nodes within a clique. We firstdefine the set of cliques which node i belongs to as a label set:LCi = {ci1,ci2, . . . ,ciNi},cik ∈ 1, . . . ,Nc, (5.28)149where LCi is an empty set when node i does not belong to any cliques. We thendefine the clique overlap ratio as:RCOc =1|CSc| |⋃∀ p∈CScLCp|. (5.29)5.3.2.3 Core Clique IdentificationBased on the clique properties, we further identify core cliques out of clique sets forthe future overlapping subnetwork extraction. We argue that core cliques shouldhave relatively high co-activation times, high activation times, and high cliqueoverlap ratio. We then devise a core clique selection criterion based on the combi-nation of the clique properties. We normalize all the property values into the rangeof [0, 1] by dividing individual values by the maximum across all the cliques. Thecriterion is set as below:ρ =median∀ i∈CS{NCOAi}max∀ i∈CS{NCOAi} +median∀ i∈CS{NAi}max∀ i∈CS{NAi} +median∀ i∈CS{ROCi}max∀ i∈CS{ROCi} .(5.30)For any clique c which satisfies the criterion:NCOAcmax∀ i∈CS{NCOAi} +NAcmax∀ i∈CS{NCOAi} +ROCcmax∀ i∈CS{ROCi} > ρ, (5.31)it is selected into the core clique set.5.3.2.4 Clique Based Overlapping Subnetwork ExtractionBased on the identified core cliques, we further deploy a subnetwork membershipsharing technique to identify overlapping subnetworks. The underlying rationaleis that the nodes residing within the same clique behave very similarly to performsome basic functions in tasks, thus, they should be within the same subnetworks.In a brain graph with N nodes, let Crest be an N×N resting state connectivitymatrix, and we have already labeled the non-overlapping subnetwork membershipfor each node using Crest. We have also defined the clique membership of a nodei as LCi in Equation 5.28. We then share the subnetwork membership of the nodes150within a clique to facilitate overlapping subnetwork assignment.First, Ms subnetworks are extracted using non-overlapping community detec-tion approach applied on Crest. We define the subnetwork membership of a node ias:label(i) = s, i ∈ 1, . . . ,N, s ∈ 1, . . . ,Ms. (5.32)Next, we deploy a sharing scheme of the subnetwork membership label from label(i)of a node i, with the label set of the remaining nodes in the clique where node ibelongs to:LS(i) =⋃∀ c∈LCi⋃∀ p∈CSclabel(p), (5.33)andlabel(i) = label(i)∪LS(i). (5.34)We have also explored replacing the resting state connectivity matrix Crest withthe multisource connectivity matrix Ct-r defined in Equation 5.15. We argue that weshould further integrate the activation information from task data with high orderrelation information presented by hypergraph and the rest data when identifyingthe non-overlapping subnetwork membership.5.3.3 ExperimentsWe first compare our multisource clique based approach against the uni-sourcekclique method [246], which is the closest straightforward way to identify over-lapping subnetworks. Next we compare against SORD, which has been proven tooutperform the state-of-the-art techniques such as OSLOM, and CSORD (the mul-tisource version of SORD) [203] to see if our proposed approach have more directbiological intuition for the overlapping subnetwork extraction. We also examinethe nodes within subnetwork overlaps derived by our approach by assessing theprobability of a node belonging to subnetworks using our recently proposed mul-timodal RW approach [6], to verify that our overlapping subnetwork assignmentscorrespond with the posterior probability.1515.3.3.1 MaterialsSimilarly as the previous section, we used the resting state fMRI and task fMRIscans of 77 unrelated healthy subjects from the HCP dataset [146]. The same pre-processings as in the previous section has been applied on resting state and taskdata. The resting state, task connectivity matrices, and the task activation weregenerated using the same procedures described in the previous section. Same asbefore, we applied our proposed local thresholding [5] on resting state FC Crestto match with the graph density of C¯task at 0.2765, which lies within the normalrange between [0.2, 0.3] for thresholding before subnetwork extraction [171]. Thenon-overlapping subnetworks were derived from the resting state connectivity ma-trix Crest or multisource connectivity matrix Ct-r using Ncuts, when the number ofsubnetworks was set to 7, same as in the previous section.We further applied local thresholding [5] on C¯task by setting graph density to be0.1 to generate the hypergraph when we identified cliques from the coarse cliqueset. We selected a relatively strict threshold to only select those most closely con-nected nodes to form cliques. 0.1 has been chosen based on the cross-validation oninter-subject reproducibility within the range between 0.03 to 0.2 at the interval of0.01.We compared the overlapping subnetwork extraction using our proposed Mul-tisource Clique-based Subnetwork Extraction (MCSE) approach with Ct-r, or Crestagainst the uni-source kclique approach [246], SORD [253], which has been demon-strated to outperform state-of-the-art overlapping community detection methods in-cluding OSLOM when applied to brain subnetwork extraction, and CSORD [203],the multisource extension of SORD. Two uni-source approaches extract overlap-ping subnetworks using resting state data. The parameters for kclique were setusing the cross-validation on the clique size k based on inter-subject reproducibil-ity from the suggested range [3, . . . ,6] [246] and reasonable graph densities from0.03 to 0.2 at the interval of 0.01. SORD and CSORD applied 100 bootstraps bysampling with replacement as suggested in [253]. We also evaluated the proba-bility of a node being assigned to a subnetwork using our recently proposed RWbased approach [6] to examine the proposed clique-based overlapping subnetworkidentification. All statistical comparisons are based on the Wilcoxon signed rank152Figure 5.11: Group-level Subnetwork Extraction reproducibility based ondata from two different sessions. MCSE outperforms all other con-trasted approaches.test with significance declared at an α of 0.05 with Bonferroni correction.5.3.3.2 Comparison with Existing Overlapping Subnetwork ExtractionMethodsWe quantitatively evaluated the contrasted approaches based on test-retest reliabil-ity and inter-subject reproducibility, since ground truth subnetworks are unknownfor the real data of human brain.5.3.3.2.1 Group-level Subnetwork Extraction Reproducibility We first assessedthe test-retest reliability based on group level subnetworks extracted separatelyfrom two sessions of rest and task data (each of the seven tasks includes two ses-sions of fMRI data) using DSC. The subnetworks extracted from the first session’sdata are taken as the “ground truth”, against which the subnetworks from the sec-ond session are compared. We found that our proposed MCSE outperforms allother contrasted approaches, by achieving a DSC between subnetworks extractedfrom two sessions of data at 0.8917 with Ct-r and 0.8865 with Crest, against kcliqueat 0.7514, SORD at 0.8378, and CSORD at 0.8514, see Figure 5.11.153MCSE t-r MCSE rest kclique SORD CSORD00.20.40.60.81Figure 5.12: Subject-wise level inter-subject reproducibility of subnetworkextraction. Our proposed MCSE approach outperforms existing state-of-the-art overlapping community detection methods.5.3.3.2.2 Subject-wise Subnetwork Extraction Reproducibility We assessed theinter-subject reproducibility by comparing the subnetwork extraction results us-ing subject-wise data against the group level data, Figure 5.12. The average DSCbetween subject-wise and group level subnetworks across 77 subjects based on fiveapproaches are MCSE with Ct-r at 0.7024±0.0722, MCSE with Crest at 0.6281±0.0583,kclique at 0.4967±0.0430, SORD at 0.5129±0.0774, and CSORD at 0.5952±0.0901,respectively. MCSE with both Ct-r and Crest are found to achieve statistically higherinter-subject reproducibility than constrasted approaches based on the Wilcoxonsigned rank test at p< 10−10 and p< 0.005. respectively. Further, MCSE withCt-r outperforms Crest at p<0.00001, which confirms the benefit of incorporat-ing the task information embedded with higher order relations in assigning non-overlapping subnetwork membership.5.3.3.3 Biological Meaning - Analyzing Function IntegrationWe further examined the biological meaning of the overlapping subnetworks foundusing all five methods, i.e., our proposed MCSE with Ct-r, MCSE with Crest,kclique, SORD and CSORD, Figure 5.13. We first measured the overlapping ratioby dividing the number of nodes residing in the subnetwork overlaps by the total154(a) Task activation (b) MCSE with Ct-r(c) MCSE with Crest (d) kclique(e) SORD (f) CSORDFigure 5.13: Visualization of Task activation and overlapping subnetworksextracted from our proposed approach and contrasted three other meth-ods. The brain is visualized in the axial view. Our proposed MCSEapproach outperforms existing state-of-the-art overlapping communitydetection methods by detecting well-known hubs which reside withinsubnetwork overlaps.155number of brain regions detected in subnetworks. The ratio of five methods are0.3482, 0.3482, 0.4444, 0.4328 and 0.2885. Our proposed approach can gener-ate the similar ratio of interacting nodes which reside within subnetwork overlapsto the existing overlapping methods. We note that CSORD generated relativelysmaller number of interacting nodes, the possible reason is that the strict stabilityselection resulted in exclusion of some meaningful nodes, which were taken asfalse detected nodes arising from noise [203].By examining the locations of those interacting nodes, we found that our pro-posed MCSE with Ct-r approach identified subnetwork overlaps within pre- andpostcentral gyri, medial superior frontal cortex, inferior frontal gyrus, superiorparietal lobule, precuneous, lateral occipital cortex, occipital pole and frontal or-bital cortex; which match well with functional hubs previously identified by graph-theoretical analysis based on the degree of the voxels [256]. Besides, brain regionsof insula, putamen, thalamus, supramarginal gyrus have been found within sub-network overlaps, which match well with the connector hubs identified using thecentrality measures [257]. The results of using MCSE with Crest is very similar toCt-r, only that precuneous cortex was missed, and the temporal pole was misclas-sified into the subnetwork overlaps. This result confirms the benefit of integratingthe information from both task and rest data. Both MCSE methods also identifiedlingual gyrus and fusiform cortex around as interacting nodes. Lingual gyrus wasidentified as a hub based on cortical thickness correlation [258] and the fusiformcortex within occipitotemporal cortex has been found to be intermediary “hub”linking visual and higher linguistic representations [259].As for the traditional kclique approach, biologically meaningful subnetworkoverlaps were found within inferior frontal gyrus, superior and middle temporalgyri [256], supramarginal gyrus, insula [257], inferior temporal gyrus [260], andoccipitotemporal cortex [259]. kclique failed to identify all the other aforemen-tioned (connector) hubs which were found using our proposed methods. Instead,regions normally were not considered to reside in subnetwork overlaps were found,such as temporal fusiform cortex, central opercular cortex, and parietal operculumcortex. On the other hand, this kclique approach detected angular gyrus (function-ing as a semantic hub) within subnetwork overlaps.SORD was able to find subnetwork overlaps within inferior, superior and mid-156dle temporal gyri, superior parietal lobule, lateral occipital cortex, occipital poleand lingual gyrus that match well with functional hubs, but failed to find other hubregions identified by MCSE. Instead, SORD detected many regions as interactingnodes, which normally are not considered as hubs, such as intracalcarine cortexand cuneal cortex in the visual system, and regions in language related system, in-cluding central opercular cortex, parietal operculum cortex, planum polare, planumtemporale, heschls gyrus, and supracalcarine cortex.With relatively lower number of overlapping ratio, CSORD identified biolog-ical meaningful subnetwork overlaps within regions such as pre- and postcentralgyri, middle temporal gyrus, angular gyrus and lateral occipital cortex, while failedto find any other hubs. Similar to SORD, it included some regions in language re-lated system to the subnetwork overlaps, such as central opercular cortex, parietaloperculum cortex, and planum temporale. We did not discover the single subnet-work constituting the visual corticostriatal loop, striatothalamo-cortical loop, andcerebello-thalamo-cortical loop, which was found in [203]. The reason could bethis connection was reflected in AC, instead of task functional connectivity.Collectively, our proposed MCSE approach is able to identify subnetwork over-laps which constitute more biologically meaningful brain regions, such as hubs,compared against contrasted methods.5.3.3.4 Comparison Between the Subnetwork Overlaps and the RWPosterior ProbabilityWe also examined the overlapping subnetworks derived from our approach MCSEwith Ct-r by assessing the probability of a node belonging to subnetworks usingour own recently proposed multimodal RW approach [6], to verify that our over-lapping subnetwork assignments correspond with the posterior probability. Theunderlying rationale is that for an interacting node, which resides within the sub-network overlaps, its probability of belonging to a subnetwork will be distributedacross the subnetworks it resides in. On the other hand, an individual node, whichdoes not reside within subnetwork overlaps, would have higher chances to possessa dominant probability of being assigned to a particular subnetwork. Hence the dif-ference of probabilities of a node being assigned to the first two subnetworks withthe first two highest probabilities indicates the possibility of a node residing within157subnetwork overlaps. Interacting nodes tend to have a smaller value of differenceof first two highest probabilities.We here define the degree of overlapping confidence as the subtraction fromone of the difference between the first two highest probabilities of a node beingassigned to subnetworks. The nodes identified within the subnetwork overlaps(interacting nodes) and outside of the overlaps (individual nodes) are consideredas two populations. For each population, the average overlapping confidence isdefined as below in Equation 5.35:overCon f =1|S|∑i∈S(1− (pmaxi − psmaxi )), (5.35)where S is a set of nodes, either nodes residing within or outside the subnetworkoverlaps, and pmax is the maximal probability of a node belonging to subnetworks,and psmax is the second maximal probability. Thus, the interacting node populationis expected to have higher overCon f compared to individual nodes.We first derived the probabilities of each node being assigned into all possiblesubnetworks using our recently proposed multimodal RW approach [6], where twosources of connectivity matrices are Crest and C¯task, matching with how Ct-r wasgenerated in our approach. The number of seeds within each subnetwork nk wasset to [2, . . . ,9], where 9 is 75% of 12, the minimal number of nodes which areincluded in non-overlapping subnetwork extraction using Ct-r.We found that the overlapping confidence of the interacting nodes with an av-erage overCon f of 0.6884 are statistically higher than the individual nodes with anaverage overCon f of 0.6338 based on the Wilcoxon signed rank test at p=0.006,see Figure 5.14. This finding confirms that the overlapping subnetwork assign-ments based on our proposed MCSE match with the probability derived indepen-dently from our RW based approach.158Figure 5.14: Overlapping confidence of interacting nodes in blue versus in-divual nodes in red derived by MCSE with Ct-r. The probability of anode being assigned into subnetworks was derived by the RW basedapproach [6].5.3.4 Discussion5.3.4.1 Benefits of Clique Identification Based on Task Co-activationThe traditional definition of clique is the fully connected subgraphs identified bythe connections between brain regions mostly on resting state connectivity. In ourapproach, we present a novel way to identify cliques based on the similarity ofactivation patterns between nodes. We argue that the clique concept closely resem-ble the canonical network components that are recruited selectively and repeatedlyin different task-induced activities [213]. Different from the traditional kcliquemethod [246], our clique-based approach is able to utilize both the task activa-tion information and the task-induced connectivity strength rather than only thebinarized connectivity information used in kclique method. Besides, the cliquesderived using our approach have flexible clique size, which was determined auto-matically, having more neuro-scientific justifications than the fixed clique size in[246]. Moreover, we estimate the properties from cliques to indicate the impor-tance of cliques, which gives us a better control over falsely including some fakecliques due to noise. We did find cliques within brain areas that well match withhubs, which indicates that our approach can identify subnetwork overlaps withmore biological meaning than the traditional kclique method.1595.3.4.2 Multisource Information Integration Improves the OverlappingSubnetwork ExtractionCompared to the widely used overlapping community detection methods, our ap-proach integrates information from multiple sources. We used both task informa-tion (including task activation and connectivity strength) for clique identificationand resting state connectivity information for subnetwork membership sharing.The results from reproducibility and biological meaning indicate that our multi-source approach, especially MCSE with Ct-r, outperforms uni-source methods suchas kclique and SORD, which has been proven to give better overlapping brain sub-network extraction results compared to state of the art techniques such as OSLOM.We note that our multisource approach further outperformed the multisource ver-sion of SORD, CSORD. The reason could be that clique based idea and the sharingof the node subnetwork membership is more straightforward and have more directbiological intuition than relying on survival probabilities of different genders inevolution, which is used in CSORD.5.3.4.3 Overlapping Subnetwork Identification Corresponds with the RWPosterior ProbabilityWe have identified subnetwork overlaps within brain regions that well match withhubs defined using functional, structural and anatomical information. The resultsenable us to study the interaction and integration between subnetworks and how in-teracting nodes (or important hubs) play their roles in the information flow acrossdifferent subnetworks. We further demonstrated that the assignments of interact-ing/individual nodes using our proposed MCSE correspond with the posterior prob-ability derived independently from our previously proposed RW based approach[6]. The finding of more distinguishable overlapping confidence between two pop-ulations of nodes when the number of seeds was set within a range of [6, 8] con-firms the merit of using multiple seeds within a reasonable range (not includingconnector hubs) in the RW based approach.1605.3.4.4 Other ConsiderationsWe have also discovered that the uni-source traditional kclique approach has highcomputational complexity when the graph density increases, where there existlarge number of fully connected subgraphs. The computational complexity of bothSORD and CSORD increases when the bootstrap sampling increases [86]. How-ever, the computation time of our proposed MCSE is quite reasonable and notsensitive to the graph densities.In terms of the coverage of the brain area from the subnetwork extraction re-sults, SORD and CSORD neglected some brain regions which are not selected assignificant nodes by the stability selection. While these two approaches offered thisextra feature, they sometimes falsely missed important nodes and failed to coverthe whole brain for analysis.5.3.4.5 Limitations and Future WorkIn this work, we presented an approach to identify cliques based on task informa-tion and extract overlapping subnetworks using both task and rest data. However,the ideal multimodal framework would be able to integrate AC into the fusionfor detecting overlapping subnetworks. The challenge is to discover the relation-ship between AC and task activation, which enables the clique identification toincorporate anatomical information. Our future work will focus on integrating ACinto task-activation based clique identification, or into the multimodal subnetworkmembership assignment, e.g., using the multimodal RW approach or the multisliceapproach [231].5.4 SummaryIn order to further improve brain subnetwork extraction, we explored multimodal/-multisource fusion technologies and the higher order relations among networknodes.First, we extended the RW based subnetwork extraction methods to a multi-modal approach for the fusion of AC and FC using provincial hubs to propagatethe modular structure information across different modalities. Second, we pro-posed a high order relation informed model based on hypergraph to integrate both161rest and task information for non-overlapping brain subnetwork extraction. Next,we further proposed multisource overlapping brain subnetwork extraction usingcanonical network components, i.e., cliques, which we defined based on task co-activation. Based on the clique concept, we investigated overlapping subnetworksbased on a label sharing scheme which incorporates the rest data information andtask data embedded with higher order relations.We have demonstrated that integrating multimodal/multisource informationand using high order relations result in better subnetwork extraction in terms of theoverlaps to well-established brain systems, test-retest repeatability, inter-subjectreproducibility and biological meaning.162Chapter 6ConclusionsIn this thesis, we presented our contributions towards improving brain connectiv-ity analysis based on the graph theory representation of the human brain network.We proposed novel multimodal techniques to study brain connectomics througha network-based inquiry into the brain’s structure, function and connectivity. Weconclude in this chapter the methods we employed within different stages in graph-ical analysis framework, how they tackled the RQs raised in Section 1.6, and thepossible directions we would suggest for future investigations.6.1 Our Contributions in Graphical Framework forHuman Brain Connectivity AnalysisIn connectome data graphical representations, human brain connectivity analysisis usually conducted in terms of brain node identification, brain edge building andbrain network property analysis. We have made contributions towards each corre-sponding step in the framework.6.1.1 Definition of Nodes: Parcellation-based Brain ConnectomeOur first contribution is towards improving parcellation, i.e., the brain networknode definition. In Chapter 2, we first devised a more reliable method for parcella-tion (RQ1) by embedding neighborhood connectivity information into the affinitymatrix to ameliorate the adverse effects of noise in the neuroimaging data. We were163able to distinguish between voxels at the boundaries and those found in the interiorregions of clusters, and to better estimate the affinity values based on the voxels’membership in putative clusters. We have demonstrated that our proposed ap-proach produces parcellations with better intra-subject test-retest reliability, higherinter-subject test-retest reliability and higher subject-group consistency.In order to investigate whether combining the modalities can actually improvebrain parcellation (RQ2), we proposed to integrate the connectivity from bothanatomical and functional modalities based on adaptive weighting using voxel-wise test-retest reliability. Our rationale is to overcome the inherent limitationsassociated with each of the two modalities, by fusing different facets of informa-tion from the AC and FC. We further incorporated structural information on thegyri and sulci when performing a regional level extension.We then designed a number of evaluation metrics to validate our improvementson parcellation (RQ3). We quantitatively demonstrated the superiority of multi-modal parcellation as compared with unimodal parcellation, and existing, widelyused functional and anatomical atlases in terms of reproducibility, functional ho-mogeneity, and leftout data likelihood. Multimodal parcellations also overlap withknown cytoarchitetonic areas, and the extracted subnetworks matched well withthe established brain systems.Our results demonstrated that incorporating neighborhood information and in-tegrating multiple brain attributes that reflect different aspects of the brain’s orga-nization do indeed improve brain parcellation.6.1.2 Definition of Edges: Estimating Connections Between BrainRegionsIn Chapter 3, we proposed the employment of noise reduction techniques in thebrain edge building step. The edge weights are derived from the connectivity esti-mates, which depend greatly on the quality of fMRI or dMRI data. Both modalitiessuffer from image resolution limitations and pronounced noise, which might ob-scure the brain network topology.First, to combat false negatives in connectivity estimation (RQ4), we proposeda matrix completion based technique, which (i) formats the recovering missingconnection problem as a low rank matrix completion problem, (ii) tackles the non-164trivial rank estimation using the information aggregation approach, and (iii) ex-ploits neighborhood information to solve the negative entry problem. Based onsynthetic data and real HCP data, we quantitatively demonstrated the superiorityof our approach as compared with the existing state-of-the-art methods in terms ofrecovery accuracy and IQ prediction.Next, we proposed a local thresholding method to suppress the false positiveconnectivity estimates (RQ5), which tackled the regional bias problem in connec-tivity estimation. We showed that our local thresholding method achieved betterperformance results in brain subnetwork extraction application in terms of its ac-curacy and reproducibility.6.1.3 Network Measures: Graphical Metrics for Brain ConnectomicsOur third contribution was to improve brain network property analyses, partic-ularly the subnetwork extraction, by using multi-pronged graphical metric guidedmethods. In Chapter 4, we investigated whether incorporating more domain-relatedgraphical metrics improves subnetwork extractions (RQ6), and explored if a modelthat resembles the brain subnetwork’s biological nature improves subnetwork ex-tractions (RQ7). Specifically, we first proposed a modularity reinforcement modelbased on connection fingerprints which highlight the putative modular structure ofbrain graphs. The underlying assumption is that node pairs belonging to the samesubnetwork presumably connect to a similar set of brain regions, i.e., have similarconnection fingerprints. By applying modularity reinforcement, we demonstratedthat our approach attained higher accuracy in subnetwork extraction as comparedto conventional community detection methods on synthetic data. A higher degreeof overlap with established brain systems and a higher subnetwork reproducibilitywere also shown in the real data.In order to incorporate a greater number of graphical metrics, we have furtherpresented a random walker based provincial hub guided model, which utilizes in-formative, module-related network metrics including hubs, within-module degreescores, and participant coefficients. At the same time, we argued that the mannerin which the nodes are clustered into groups, based on the probabilities of walk-ing to seeds in the Random Walker, closely resembles the mechanism whereby165brain regions within a subnetwork are inter-linked via provincial hubs. We havefurther devised a multi-seed strategy to reinforce robustness against noisy connec-tivity connections. The results on synthetic data from the subnetwork extractionaccuracy and the overlaps to well-known brain systems in the real data have bothdemonstrated the clear benefits gained in applying multi-pronged graphical metricsand devising a model based on biological intuition for subnetwork extraction.6.1.4 Multimodal/Multisource Fusion to Improve Brain ConnectivityAnalysisOur fourth contribution was to develop multimodal/multisource fusion techniquesto further improve brain subnetwork extraction. In Chapter 5, we investigatedthe feasibility and benefits of applying the multimodal fusion (RQ8) and multi-source integration approaches towards studying higher order relations among net-work nodes (RQ9).We first demonstrated that anatomical and functional data can be fused togetherby the provincial hubs to propagate modular structural information across differentmodalities. Our proposed random walker based approach facilitates multimodalintegration by updating the augmented prior edge linking nodes to those provincialhubs connecting the anatomical and functional subnetworks. Moreover, we wereable to infer the probability of a node being assigned to a subnetwork for a furtherinvestigation of the overlapping subnetworks. We showed that our multimodalapproach achieved a significantly higher subnetwork identification accuracy thana number of state-of-the-art approaches on synthetic data. We also demonstratedthat our estimated subnetworks matched well with established brain systems andattained comparable or higher inter-subject reproducibility when applied to realdata.Secondly, we proposed a high order relation informed approach based on hy-pergraph to combine the information from multi-task data and resting state datato improve subnetwork extraction. Our assumption is that task data can be bene-ficial for the subnetwork extraction process, since the repeatedly activated nodesinvolved in diverse tasks might be the canonical network components which com-prise pre-existing repertoires of resting state subnetworks [213]. Our proposed highorder relation informed subnetwork extraction are based on a strength information166embedded hypergraph. Our approach (1) facilitates the multisource integrationfor subnetwork extraction, (2) utilizes information on relationships and changesbetween the nodes across different tasks, and (3) enables the study on higher or-der relations among brain network nodes. We demonstrated that fusing task ac-tivation, task-induced connectivity and resting state functional connectivity basedon hypergraphs improves subnetwork extraction compared to employing a singlesource from either rest or task data in terms of subnetwork modularity measure,inter-subject reproducibility, along with more biologically meaningful subnetworkassignments.We further studied the overlapping brain subnetwork extraction using cliques,which we defined as co-activated node groups performing multiple tasks. We pro-posed a multisource subnetwork extraction approach based on the co-activatedclique, which (1) uses task co-activation and task connectivity strength informa-tion for clique identification, (2) automatically detects cliques of different sizeshaving more neuroscientific justifications, and (3) shares the subnetwork member-ship, derived from a fusion of rest and task data, among the nodes within a cliquefor overlapping subnetwork extraction. Compared to the commonly used over-lapping community detection techniques, we showed that our approach improvedsubnetwork extraction in terms of group-level and subject-wise reproducibility. Wealso showed that our multisource approach identified subnetwork overlaps withinbrain regions that matched well with hubs defined using functional and anatomicalinformation, which enables us to study the interactions between the subnetworksand how hubs play their role in information flow across different subnetworks. Wefurther demonstrated that the assignments of interacting/individual nodes using ourapproach correspond with the posterior probability derived independently from ourmultimodal random walker based approach.Our results collectively indicate that the multimodal fusion of resting FC andAC information and the multisource integration of task and rest functional dataoutperform classical unimodal brain connectivity analysis methods.1676.2 Future WorkIn this work, we mostly focus on studying human brain connectivity from multiplesources of anatomical and functional connectivity using graph theory. In Chapter 2,we fused the AC and FC for improved parcellation based on an adaptive weight-ing scheme. We have proved that our approach provides improved parcellationin practice; however, understanding the exact nature of the relationship betweenAC and FC warrants further research. With the acquisition of a greater degree ofknowledge on this subject, studies on fusing the two modalities based on advancedtheories, and/or applying deep learning methods and reformulating parcellation asa voxel-wise classification problem might give us new insight into multimodal par-cellation.In Chapter 3, we focused on denoising connectivity strength which is used asgraph edge weights. Future improvements in the quality of data collection with ad-vanced MRI scans and more reasonable protocols will greatly benefit the accuracyof connectivity estimation. Throughout the thesis, we used the most commonlyemployed Pearson’s correlation coefficient between regional activity time courses.Alternative measures such as partial correlations, mutual information and coher-ence might lead to some new discoveries or confirm our current findings. Also, ourwork employed the AC and FC to study undirected brain graphs. Another promis-ing direction is to incorporate the effective connectivity when estimating graphedges using functional data with a higher temporal resolution, such as EEG data,which enables more accurate EC estimations using spectral coherence or Grangercausality measures [75]. This direction will include one more aspect for multi-modal fusion which could be based on multi-layer graph, while simultaneouslyallows for studies on directed graphs.In Chapter 4 and Chapter 5, we made contributions towards improved brainsubnetwork extraction by applying multi-pronged graphical metrics and the mul-timodal/multisource fusion approach. We have linked image and graphical theoryanalysis tools to biological intuition to closely model the brain’s nature. Knowl-edge of the exact nature of the relationship between all the multimodal/multisourceinformation requires further research. One additional direction would be to con-sider higher order relations among the network nodes by pursuing further applica-168tions in multilayer graphs in brain connectivity analysis, where dynamic connec-tivity [261] can be incorporated into the multimodal framework.In this thesis, we have studied brain connectivity to understand the basic ar-chitecture of the healthy human brain. The resulting methods might provide uswith important new insights into cognitive and clinical neuroscience, which bringsus one step closer to understand the relationships of brain architecture to diseaseswhose alternations may be described as dysconnectivity syndromes. One promis-ing future research direction is to explore how brain network features can be linked,e.g., to gender, aging, genetic factors, and the stages of disease severity, possiblybased on methods such as deep learning approaches.For example, we think it is feasible to build a deep learning framework forneurological disorder classification or disease stage prediction. Ideally, this modelwill incorporate both information from the original image data and topological in-formation extracted from the brain connectome, possibly through a multistreamstrategy. Firstly, the results from our work in this thesis (brain parcellation andedge building) can be used to construct the brain connectome graph for topologicalinformation extraction. Secondly, we can design a deep neural network architec-ture to learn the features by training an unsupervised deep learning scheme basedon a big number of healthy controls. Importantly, the results from our current re-search (such as parcellation, subnetwork extraction, clique identification and hubs)can be utilized to guide the design of the architecture, e.g., using the (hierarchical)connectivity information to design the hidden layers. Incorporating model priorinto the framework might improve the performance of deep learning architectures[262, 263], with which it becomes possible to reflect or involve the domain-specificknowledge [124]. Moreover, a good knowledge of the brain connectivity from ourwork can help explain the features learned from the deep black box, e.g., hiddenlayers from low to high can be related to attributes derived from nodal featuresbased on parcellation, edge-to-edge relationships, clique-wise node group interac-tion, subnetwork structure, to hub features or brain network-wide integration fea-tures. One other interesting direction is to incorporate clinical prior features [131],such as patient history, age, demographics, behavioural scores, genetic phenotypesand others. Finally, we can fine tune the parameters in the deep learning modelusing the labeled disease data for specific applications.169We have also considered some directions to handle important steps in the modeltraining. First, the prior knowledge learned from this thesis can be used for weightinitialization (e.g., how nodes are closely connected to a subnetwork it belongs to,can be related to how strong the weights connecting units across layers). Secondly,in order to tackle the local minima problem, we suggest to deploy a multimodalperturbation strategy. Thirdly, we suggest to use regularization or ensemble learn-ing (e.g., majority voting from a number of deep network models) to tackle theoverfitting problem.Other directions to improve brain connectivity analysis using machine learningor deep learning approaches include (1) big data collection and label acquisitionusing methods such as multiple-instance or active learning approaches [131]; (2)designing methods to overcome class imbalance, such as modifying loss function[264]; (3) devising methods suitable for graph representation of brain connectomedata [265]; and (4) developing algorithmic techniques such as transfer learning tomore efficiently adopt networks trained from the computer vision area and handlecross-institution difference and cross-modality generalization.170Bibliography[1] Chendi Wang, Burak Yoldemir, and Rafeef Abugharbieh. Improvedfunctional cortical parcellation using aneighborhood-information-embedded affinity matrix. In IEEEInternational Symposium on Biomedical Imaging, pages 1340–1343. IEEEPress, 2015.[2] Chendi Wang, Burak Yoldemir, and Rafeef Abugharbieh. Multimodalcortical parcellation based on anatomical and functional brain connectivity.In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 21–28. Springer, 2015.[3] Chendi Wang, Bernard Ng, and Rafeef Abugharbieh. Multimodal wholebrain parcellation. submitted, xx(x):xxxx–xxxx, 2018.[4] Chendi Wang, Bernard Ng, Alborz Amir-Khalili, and Rafeef Abugharbieh.Recovering missing connections in diffusion weighted mri using matrixcompletion. In International Conference on Medical Image Computing andComputer-Assisted Intervention Workshop on Computational DiffusionMRI. Springer, 2017.[5] Chendi Wang, Bernard Ng, and Rafeef Abugharbieh. Modularityreinforcement for improving brain subnetwork extraction. In InternationalConference on Medical Image Computing and Computer-AssistedIntervention, pages 132–139. Springer, 2016.[6] Chendi Wang, Bernard Ng, and Rafeef Abugharbieh. Multimodal brainsubnetwork extraction using provincial hub guided random walks. InInternational Conference on Information Processing in Medical Imaging,pages 287–298. Springer, Cham, 2017.[7] Chendi Wang and Rafeef Abugharbieh. Fusion of task and resting statefunctional connectivity for subnetwork extraction based on hypergraph.submitted, xx(x):xxxx–xxxx, 2018.171[8] Chendi Wang and Rafeef Abugharbieh. Co-activated clique basedmultisource overlapping subnetwork extraction. submitted,xx(x):xxxx–xxxx, 2018.[9] GBD 2015 DALYs and HALE Collaborators. Global, regional, andnational disability-adjusted life-years (dalys) for 315 diseases and injuriesand healthy life expectancy (hale), 1990–2015: a systematic analysis forthe global burden of disease study 2015. Lancet, 388(October8):1603–1658, 2016.[10] GBD 2015 Disease, Injury Incidence, and Prevalence Collaborators.Global, regional, and national incidence, prevalence, and years lived withdisability for 310 diseases and injuries, 1990–2015: a systematic analysisfor the global burden of disease study 2015. Lancet, 388(October8):1545–1602, 2016.[11] Terrence J Sejnowski, Patricia S Churchland, and J Anthony Movshon.Putting big data to good use in neuroscience. Nature neuroscience,17(11):1440–1441, 2014.[12] Rolf Ko¨tter. Online retrieval, processing, and visualization of primateconnectivity data from the cocomac database. Neuroinformatics,2(2):127–144, 2004.[13] Saad Jbabdi and Heidi Johansen-Berg. Tractography: where do we go fromhere? Brain connectivity, 1(3):169–183, 2011.[14] R Cameron Craddock, Saad Jbabdi, Chao-Gan Yan, Joshua T Vogelstein,F Xavier Castellanos, Adriana Di Martino, Clare Kelly, Keith Heberlein,Stan Colcombe, and Michael P Milham. Imaging human connectomes atthe macroscale. Nature methods, 10(6):524–539, 2013.[15] Marco Catani. From hodology to function. Brain, 130(3):602–605, 2007.[16] Alex Fornito, Andrew Zalesky, and Michael Breakspear. Graph analysis ofthe human connectome: promise, progress, and pitfalls. Neuroimage,80:426–444, 2013.[17] Mikail Rubinov and Olaf Sporns. Complex network measures of brainconnectivity: uses and interpretations. Neuroimage, 52(3):1059–1069,2010.172[18] Jonathan D Power, Alexander L Cohen, Steven M Nelson, Gagan S Wig,Kelly Anne Barnes, Jessica A Church, Alecia C Vogel, Timothy OLaumann, Fran M Miezin, Bradley L Schlaggar, et al. Functional networkorganization of the human brain. Neuron, 72(4):665–678, 2011.[19] Nicolas A Crossley, Andrea Mechelli, Petra E Ve´rtes, Toby TWinton-Brown, Ameera X Patel, Cedric E Ginestet, Philip McGuire, andEdward T Bullmore. Cognitive relevance of the community structure of thehuman brain functional coactivation network. Proceedings of the NationalAcademy of Sciences, 110(28):11583–11588, 2013.[20] David C Van Essen, Matthew F Glasser, Donna L Dierker, John Harwell,and Timothy Coalson. Parcellations and hemispheric asymmetries ofhuman cerebral cortex analyzed on surface-based atlases. Cerebral cortex,22(10):2241–2262, 2011.[21] Nikolaus Weiskopf, Chloe Hutton, Oliver Josephs, Robert Turner, and RalfDeichmann. Optimized epi for fmri studies of the orbitofrontal cortex:compensation of susceptibility-induced gradients in the readout direction.Magnetic Resonance Materials in Physics, Biology and Medicine,20(1):39–49, 2007.[22] Aaron F Alexander-Bloch, Nitin Gogtay, David Meunier, Rasmus Birn, LivClasen, Francois Lalonde, Rhoshel Lenroot, Jay Giedd, and Edward TBullmore. Disrupted modularity and local connectivity of brain functionalnetworks in childhood-onset schizophrenia. Frontiers in systemsneuroscience, 4:147, 2010.[23] WR Shirer, S Ryali, E Rykhlevskaia, V Menon, and MD Greicius.Decoding subject-driven cognitive states with whole-brain connectivitypatterns. Cerebral cortex, 22(1):158–165, 2012.[24] Public Health Agency of Canada. Mapping connections: An understandingof neurological conditions in canada. The National Population HealthStudy of Neurological Conditions, 2014.[25] Cornelis J Stam. Modern network science of neurological disorders.Nature Reviews Neuroscience, 15(10):683–695, 2014.[26] Olaf Sporns, Giulio Tononi, and Rolf Ko¨tter. The human connectome: astructural description of the human brain. PLoS computational biology,1(4):e42, 2005.173[27] MJ Rosa, J Daunizeau, and KJ Friston. Eeg-fmri integration: a criticalreview of biophysical modeling and data analysis approaches. Journal ofintegrative neuroscience, 9(04):453–476, 2010.[28] Michael E Phelps. PET: molecular imaging and its biological applications.Springer Science & Business Media, 2004.[29] Seiji Ogawa, Tso-Ming Lee, Alan R Kay, and David W Tank. Brainmagnetic resonance imaging with contrast dependent on bloodoxygenation. Proceedings of the National Academy of Sciences,87(24):9868–9872, 1990.[30] Scott A Huettel, Allen W Song, and Gregory McCarthy. Functionalmagnetic resonance imaging, volume 1. Sinauer Associates Sunderland,2004.[31] Robert Turner and Terry Jones. Techniques for imaging neuroscience.British medical bulletin, 65(1):3–20, 2003.[32] Michael C Stevens. The contributions of resting state and task-basedfunctional connectivity studies to our understanding of adolescent brainnetwork maturation. Neuroscience & Biobehavioral Reviews, 70:13–32,2016.[33] Yong He, Zhang J Chen, and Alan C Evans. Small-world anatomicalnetworks in the human brain revealed by cortical thickness from mri.Cerebral cortex, 17(10):2407–2419, 2007.[34] Peter J Basser, James Mattiello, and Denis LeBihan. Estimation of theeffective self-diffusion tensor from the nmr spin echo. Journal of MagneticResonance, Series B, 103(3):247–254, 1994.[35] Haz-Edine Assemlal, David Tschumperle´, Luc Brun, and Kaleem Siddiqi.Recent advances in diffusion mri modeling: Angular and radialreconstruction. Medical image analysis, 15(4):369–396, 2011.[36] Martijn P Van Den Heuvel and Hilleke E Hulshoff Pol. Exploring the brainnetwork: a review on resting-state fmri functional connectivity. EuropeanNeuropsychopharmacology, 20(8):519–534, 2010.[37] Tianzi Jiang. Brainnetome: a new-ome to understand the brain and itsdisorders. Neuroimage, 80:263–272, 2013.174[38] Marwan N Baliki, Paul Y Geha, A Vania Apkarian, and Dante R Chialvo.Beyond feeling: chronic pain hurts the brain, disrupting the default-modenetwork dynamics. The Journal of Neuroscience, 28(6):1398–1403, 2008.[39] Zhenyu Zhou, Yonghong Chen, Mingzhou Ding, Paul Wright, Zuhong Lu,and Yijun Liu. Analyzing brain networks with pca and conditional grangercausality. Human brain mapping, 30(7):2197–2206, 2009.[40] M De Luca, CF Beckmann, N De Stefano, PM Matthews, and Stephen MSmith. fmri resting state networks define distinct modes of long-distanceinteractions in the human brain. Neuroimage, 29(4):1359–1367, 2006.[41] Chris HQ Ding, Xiaofeng He, and Horst D Simon. On the equivalence ofnonnegative matrix factorization and spectral clustering. In SDM,volume 5, pages 606–610. SIAM, 2005.[42] Franco Cauda, Federico D’Agata, Katiuscia Sacco, Sergio Duca, GiulianoGeminiani, and Alessandro Vercelli. Functional connectivity of the insulain the resting brain. Neuroimage, 55(1):8–23, 2011.[43] Raymond Salvador, John Suckling, Christian Schwarzbauer, andEd Bullmore. Undirected graphs of frequency-dependent functionalconnectivity in whole brain networks. Philosophical Transactions of theRoyal Society B: Biological Sciences, 360(1457):937–946, 2005.[44] Bertrand Thirion, Silke Dodel, and Jean-Baptiste Poline. Detection ofsignal synchronizations in resting-state fmri datasets. Neuroimage,29(1):321–327, 2006.[45] CJ Stam, W De Haan, A Daffertshofer, BF Jones, I Manshanden,AM Van Cappellen Van Walsum, T Montez, JPA Verbunt, JC De Munck,BW Van Dijk, et al. Graph theoretical analysis ofmagnetoencephalographic functional connectivity in alzheimer’s disease.Brain, 132(1):213–224, 2009.[46] Danielle S Bassett and Olaf Sporns. Network neuroscience. Natureneuroscience, 20(3):353, 2017.[47] Alex Fornito and Edward T Bullmore. Connectomics: a new paradigm forunderstanding brain disease. European Neuropsychopharmacology,25(5):733–748, 2015.175[48] JG White, E Southgate, JN Thomson, and S Brenner. The structure of thenervous system of the nematode caenorhabditis elegans: the mind of aworm. Phil. Trans. R. Soc. Lond, 314:1–340, 1986.[49] Ravi D Mill, Takuya Ito, and Michael W Cole. From connectome tocognition: The search for mechanism in human functional brain networks.NeuroImage, 2017.[50] Xi-Nian Zuo, Ye He, Richard F Betzel, Stan Colcombe, Olaf Sporns, andMichael P Milham. Human connectomics across the life span. Trends incognitive sciences, 21(1):32–45, 2017.[51] NA Crossley, PT Fox, and ET Bullmore. Meta-connectomics: human brainnetwork and connectivity meta-analyses. Psychological medicine,46(5):897–907, 2016.[52] Cornelis J Stam. Modern network science of neurological disorders.Nature Reviews Neuroscience, 15(10):683–695, 2014.[53] Andrew Zalesky, Alex Fornito, Ian H Harding, Luca Cocchi, Murat Yu¨cel,Christos Pantelis, and Edward T Bullmore. Whole-brain anatomicalnetworks: does the choice of nodes matter? Neuroimage, 50(3):970–983,2010.[54] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl DHacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson,Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellationof human cerebral cortex. Nature, 536(7615):171–178, 2016.[55] Matthew L Stanley, Malaak N Moussa, Brielle M Paolini, Robert G Lyday,Jonathan H Burdette, and Paul J Laurienti. Defining nodes in complexbrain networks. Frontiers in computational neuroscience, 7, 2013.[56] Marcel A de Reus and Martijn P Van den Heuvel. The parcellation-basedconnectome: limitations and extensions. Neuroimage, 80:397–404, 2013.[57] Rahul S Desikan, Florent Se´gonne, Bruce Fischl, Brian T Quinn,Bradford C Dickerson, Deborah Blacker, Randy L Buckner, Anders MDale, R Paul Maguire, Bradley T Hyman, et al. An automated labelingsystem for subdividing the human cerebral cortex on mri scans into gyralbased regions of interest. Neuroimage, 31(3):968–980, 2006.[58] Laurence J Garey. Brodmann’s ‘localisation in the cerebral cortex’. WorldScientific, 1994.176[59] Matthew F Glasser and David C Van Essen. Mapping human cortical areasin vivo based on myelin content as revealed by t1-and t2-weighted mri.Journal of Neuroscience, 31(32):11597–11616, 2011.[60] Svenja Caspers, Axel Schleicher, Mareike Bacha-Trams, NicolaPalomero-Gallagher, Katrin Amunts, and Karl Zilles. Organization of thehuman inferior parietal lobule based on receptor architectonics. CerebralCortex, 23(3):615–628, 2012.[61] Bruce Fischl, Niranjini Rajendran, Evelina Busa, Jean Augustinack, OliverHinds, BT Thomas Yeo, Hartmut Mohlberg, Katrin Amunts, and KarlZilles. Cortical folding patterns and predicting cytoarchitecture. Cerebralcortex, 18(8):1973–1980, 2007.[62] Brian A Wandell, Serge O Dumoulin, and Alyssa A Brewer. Visual fieldmaps in human cortex. Neuron, 56(2):366–383, 2007.[63] Feng Shi, Yong Liu, Tianzi Jiang, Yuan Zhou, Wanlin Zhu, Jiefeng Jiang,Haihong Liu, and Zhening Liu. Regional homogeneity and anatomicalparcellation for fmri image classification: application to schizophrenia andnormal controls. Medical Image Computing and Computer-AssistedIntervention–MICCAI 2007, pages 136–143, 2007.[64] Yong Yang, Lingzhong Fan, Congying Chu, Junjie Zhuo, Jiaojian Wang,Peter T Fox, Simon B Eickhoff, and Tianzi Jiang. Identifying functionalsubdivisions in the human brain using meta-analytic activationmodeling-based parcellation. Neuroimage, 124:300–309, 2016.[65] Heidi Johansen-Berg, TEJ Behrens, MD Robson, I Drobnjak, MFSRushworth, JM Brady, SM Smith, DJ Higham, and PM Matthews. Changesin connectivity profiles define functionally distinct regions in humanmedial frontal cortex. Proceedings of the National Academy of Sciences ofthe United States of America, 101(36):13335–13340, 2004.[66] Sandrine Lefranc, Pauline Roca, Matthieu Perrot, Cyril Poupon, DenisLe Bihan, Jean-Franc¸ois Mangin, and Denis Rivie`re. Groupwiseconnectivity-based parcellation of the whole human cortical surface usingwatershed-driven dimension reduction. Medical image analysis, 30:11–29,2016.[67] Guillermo Gallardo, William Wells, Rachid Deriche, and DemianWassermann. Groupwise structural parcellation of the whole cortex: alogistic random effects model based approach. NeuroImage, 2017.177[68] Gae¨l Varoquaux, Alexandre Gramfort, Fabian Pedregosa, Vincent Michel,and Bertrand Thirion. Multi-subject dictionary learning to segment an atlasof brain spontaneous activity. In Biennial International Conference onInformation Processing in Medical Imaging, pages 562–573. Springer,2011.[69] BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu,Danial Lashkari, Marisa Hollinshead, Joshua L Roffman, Jordan WSmoller, Lilla Zo¨llei, Jonathan R Polimeni, et al. The organization of thehuman cerebral cortex estimated by intrinsic functional connectivity.Journal of neurophysiology, 106(3):1125–1165, 2011.[70] David C Van Essen and Kamil Ugurbil. The future of the humanconnectome. Neuroimage, 62(2):1299–1310, 2012.[71] Alan Tucholka, Bertrand Thirion, Matthieu Perrot, Philippe Pinel,Jean-Franc¸ois Mangin, and Jean-Baptiste Poline. Probabilisticanatomo-functional parcellation of the cortex: how many regions? InInternational Conference on Medical Image Computing andComputer-Assisted Intervention, pages 399–406. Springer, 2008.[72] Viktor Jirsa, Olaf Sporns, Michael Breakspear, Gustavo Deco, andAnthony Randal McIntosh. Towards the virtual brain: network modeling ofthe intact and the damaged brain. Archives italiennes de biologie,148(3):189–205, 2010.[73] Annette M Molinaro, Richard Simon, and Ruth M Pfeiffer. Prediction errorestimation: a comparison of resampling methods. Bioinformatics,21(15):3301–3307, 2005.[74] J-F Mangin, Pierre Fillard, Yann Cointepas, Denis Le Bihan, VincentFrouin, and Cyril Poupon. Toward global tractography. NeuroImage,80:290–296, 2013.[75] Linda J Larson-Prior, Robert Oostenveld, Stefania Della Penna,G Michalareas, F Prior, Abbas Babajani-Feremi, J-M Schoffelen, LauraMarzetti, Francesco de Pasquale, F Di Pompeo, et al. Adding dynamics tothe human connectome project with meg. Neuroimage, 80:190–201, 2013.[76] Peter J Basser, Sinisa Pajevic, Carlo Pierpaoli, Jeffrey Duda, and AkramAldroubi. In vivo fiber tractography using dt-mri data. Magnetic resonancein medicine, 44(4):625–632, 2000.178[77] David S Tuch. Q-ball imaging. Magnetic resonance in medicine,52(6):1358–1372, 2004.[78] Iman Aganj, Christophe Lenglet, Guillermo Sapiro, Essa Yacoub, KamilUgurbil, and Noam Harel. Reconstruction of the orientation distributionfunction in single-and multiple-shell q-ball imaging within constant solidangle. Magnetic Resonance in Medicine, 64(2):554–566, 2010.[79] Timothy EJ Behrens, H Johansen Berg, Saad Jbabdi, Matthew FSRushworth, and Mark W Woolrich. Probabilistic diffusion tractographywith multiple fibre orientations: What can we gain? Neuroimage,34(1):144–155, 2007.[80] Marco Reisert, Irina Mader, Constantin Anastasopoulos, Matthias Weigel,Susanne Schnell, and Valerij Kiselev. Global fiber reconstruction becomespractical. Neuroimage, 54(2):955–962, 2011.[81] BW Kreher, I Mader, and VG Kiselev. Gibbs tracking: a novel approachfor the reconstruction of neuronal pathways. Magnetic Resonance inMedicine, 60(4):953–963, 2008.[82] Thorsten Kahnt, Luke J Chang, Soyoung Q Park, Jakob Heinzle, andJohn-Dylan Haynes. Connectivity-based parcellation of the humanorbitofrontal cortex. Journal of Neuroscience, 32(18):6240–6250, 2012.[83] Rachid Deriche. Computational brain connectivity mapping: A core healthand scientific challenge, 2016.[84] Jun Yi Wang, Herve´ Abdi, Khamid Bakhadirov, Ramon Diaz-Arrastia, andMichael D Devous. A comprehensive reliability assessment of quantitativediffusion tensor tractography. Neuroimage, 60(2):1127–1138, 2012.[85] Hanbo Chen, Kaiming Li, Dajiang Zhu, Xi Jiang, Yixuan Yuan, Peili Lv,Tuo Zhang, Lei Guo, Dinggang Shen, and Tianming Liu. Inferringgroup-wise consistent multimodal brain networks via multi-view spectralclustering. IEEE transactions on medical imaging, 32(9):1576–1586, 2013.[86] Burak Yoldemir. Multimodal fusion for assessing functional segregationand integration in the human brain. PhD thesis, University of BritishColumbia, 2016.[87] Andrew L Alexander, Khader M Hasan, Mariana Lazar, Jay S Tsuruda, andDennis L Parker. Analysis of partial volume effects in diffusion-tensor mri.Magnetic Resonance in Medicine, 45(5):770–780, 2001.179[88] Stephen M Smith, Karla L Miller, Gholamreza Salimi-Khorshidi, MatthewWebster, Christian F Beckmann, Thomas E Nichols, Joseph D Ramsey, andMark W Woolrich. Network modelling methods for fmri. Neuroimage,54(2):875–891, 2011.[89] Mark Fiecas, Hernando Ombao, Dan van Lunen, Richard Baumgartner,Alexandre Coimbra, and Dai Feng. Quantifying temporal correlations: atest–retest evaluation of functional connectivity in resting-state fmri.NeuroImage, 65:231–241, 2013.[90] Kevin Murphy, Rasmus M Birn, and Peter A Bandettini. Resting-state fmriconfounds and cleanup. Neuroimage, 80:349–359, 2013.[91] Gael Varoquaux and Bertrand Thirion. How machine learning is shapingcognitive neuroimaging. GigaScience, 3(1):28, 2014.[92] Mikail Rubinov and Olaf Sporns. Weight-conserving characterization ofcomplex functional brain networks. Neuroimage, 56(4):2068–2079, 2011.[93] Pawel Skudlarski, Kanchana Jagannathan, Vince D Calhoun, MichelleHampson, Beata A Skudlarska, and Godfrey Pearlson. Measuring brainconnectivity: diffusion tensor imaging validates resting state temporalcorrelations. Neuroimage, 43(3):554–561, 2008.[94] Heidi Johansen-Berg. Human connectomics-what will the future demand?NeuroImage, 80:541–544, 2013.[95] Giulio Tononi, Olaf Sporns, and Gerald M Edelman. A measure for braincomplexity: relating functional segregation and integration in the nervoussystem. Proceedings of the National Academy of Sciences,91(11):5033–5037, 1994.[96] Michelle Girvan and Mark EJ Newman. Community structure in social andbiological networks. Proceedings of the National Academy of Sciences,99(12):7821–7826, 2002.[97] Mark EJ Newman. Fast algorithm for detecting community structure innetworks. Physical review E, 69(6):066133, 2004.[98] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and EtienneLefebvre. Fast unfolding of communities in large networks. Journal ofStatistical Mechanics: Theory and Experiment, 2008(10):P10008, 2008.180[99] Luca Ferrarini, Ilya M Veer, Evelinda Baerends, Marie-Jose´ van Tol,Remco J Renken, Nic JA van der Wee, Dirk Veltman, Andre´ Aleman,Frans G Zitman, Brenda WJH Penninx, et al. Hierarchical functionalmodularity in the resting-state human brain. Human brain mapping,30(7):2220–2231, 2009.[100] Santo Fortunato and Marc Barthe´lemy. Resolution limit in communitydetection. Proceedings of the National Academy of Sciences,104(1):36–41, 2007.[101] Kaiming Li, Lei Guo, Jingxin Nie, Gang Li, and Tianming Liu. Review ofmethods for functional brain connectivity detection using fmri.Computerized Medical Imaging and Graphics, 33(2):131–139, 2009.[102] Anthony R McIntosh and Bratislav Misˇic´. Multivariate statistical analysesfor neuroimaging data. Annual review of psychology, 64:499–525, 2013.[103] Martin J McKeown, Scott Makeig, Greg G Brown, Tzyy-Ping Jung,Sandra S Kindermann, Anthony J Bell, and Terrence J Sejnowski. Analysisof fmri data by blind separation into independent spatial components.Technical report, NAVAL HEALTH RESEARCH CENTER SAN DIEGOCA, 1997.[104] VA Traag and Jeroen Bruggeman. Community detection in networks withpositive and negative links. Physical Review E, 80(3):036115, 2009.[105] Patric Hagmann, Leila Cammoun, Xavier Gigandet, Reto Meuli,Christopher J Honey, Van J Wedeen, and Olaf Sporns. Mapping thestructural core of human cerebral cortex. PLoS biology, 6(7):e159, 2008.[106] Olaf Sporns and Richard F Betzel. Modular brain networks. Annual reviewof psychology, 67:613–640, 2016.[107] Richard F Betzel, Alessandra Griffa, Andrea Avena-Koenigsberger,Joaquı´n Gon˜i, Jean-Philippe Thiran, Patric Hagmann, and Olaf Sporns.Multi-scale community organization of the human structural connectomeand its relationship with resting-state functional connectivity. NetworkScience, 1(3):353–373, 2013.[108] Christian Lohse, Danielle S Bassett, Kelvin O Lim, and Jean M Carlson.Resolving anatomical and functional structure in human brain organization:identifying mesoscale organization in weighted network representations.PLoS computational biology, 10(10):e1003712, 2014.181[109] Timothy O Laumann, Evan M Gordon, Babatunde Adeyemo, Abraham ZSnyder, Sung Jun Joo, Mei-Yen Chen, Adrian W Gilmore, Kathleen BMcDermott, Steven M Nelson, Nico UF Dosenbach, et al. Functionalsystem and areal organization of a highly sampled individual human brain.Neuron, 87(3):657–670, 2015.[110] Angela R Laird, Jack J Lancaster, and Peter T Fox. Brainmap.Neuroinformatics, 3(1):65–77, 2005.[111] Stephen M Smith, Peter T Fox, Karla L Miller, David C Glahn, P MickleFox, Clare E Mackay, Nicola Filippini, Kate E Watkins, Roberto Toro,Angela R Laird, et al. Correspondence of the brain’s functionalarchitecture during activation and rest. Proceedings of the NationalAcademy of Sciences, 106(31):13040–13045, 2009.[112] I Tavor, O Parker Jones, RB Mars, SM Smith, TE Behrens, and S Jbabdi.Task-free mri predicts individual differences in brain activity during taskperformance. Science, 352(6282):216–220, 2016.[113] Jessica S Damoiseaux and Michael D Greicius. Greater than the sum of itsparts: a review of studies combining structural connectivity andresting-state functional connectivity. Brain Structure and Function,213(6):525–533, 2009.[114] Christopher J Honey, O Sporns, Leila Cammoun, Xavier Gigandet,Jean-Philippe Thiran, Reto Meuli, and Patric Hagmann. Predicting humanresting-state functional connectivity from structural connectivity.Proceedings of the National Academy of Sciences, 106(6):2035–2040,2009.[115] Jing Sui, Rene Huster, Qingbao Yu, Judith M Segall, and Vince D Calhoun.Function–structure associations of the brain: evidence from multimodalconnectivity and covariance studies. Neuroimage, 102:11–23, 2014.[116] Christopher J Honey, Jean-Philippe Thivierge, and Olaf Sporns. Canstructure predict function in the human brain? Neuroimage,52(3):766–776, 2010.[117] Olaf Sporns. The human connectome: origins and challenges. Neuroimage,80:53–61, 2013.[118] Archana Venkataraman, Yogesh Rathi, Marek Kubicki, Carl-FredrikWestin, and Polina Golland. Joint modeling of anatomical and functional182connectivity for population studies. Medical Imaging, IEEE Transactionson, 31(2):164–182, 2012.[119] F DuBois Bowman, Lijun Zhang, Gordana Derado, and Shuo Chen.Determining functional connectivity using fmri data with diffusion-basedanatomical weighting. NeuroImage, 62(3):1769–1779, 2012.[120] Farras Abdelnour, Henning U Voss, and Ashish Raj. Network diffusionaccurately models the relationship between structural and functional brainconnectivity networks. Neuroimage, 90:335–347, 2014.[121] Lingzhong Fan, Hai Li, Junjie Zhuo, Yu Zhang, Jiaojian Wang, LiangfuChen, Zhengyi Yang, Congying Chu, Sangma Xie, Angela R Laird, et al.The human brainnetome atlas: a new brain atlas based on connectionalarchitecture. Cerebral Cortex, 26(8):3508–3526, 2016.[122] Dajiang Zhu, Tuo Zhang, Xi Jiang, Xintao Hu, Hanbo Chen, Ning Yang,Jinglei Lv, Junwei Han, Lei Guo, and Tianming Liu. Fusing dti and fmridata: a survey of methods and applications. NeuroImage, 102:184–191,2014.[123] Vince D Calhoun and Jing Sui. Multimodal fusion of brain imaging data:A key to finding the missing link (s) in complex mental illness. Biologicalpsychiatry: cognitive neuroscience and neuroimaging, 1(3):230–244, 2016.[124] Dinggang Shen, Guorong Wu, and Heung-Il Suk. Deep learning in medicalimage analysis. Annual Review of Biomedical Engineering, 2017.[125] Youngjin Yoo, Tom Brosch, Anthony Traboulsee, David KB Li, and RogerTam. Deep learning of image features from unlabeled data for multiplesclerosis lesion segmentation. In International Workshop on MachineLearning in Medical Imaging, pages 117–124. Springer, 2014.[126] Pavel Dvorak and Bjoern H Menze. Local structure prediction withconvolutional neural networks for multimodal brain tumor segmentation.In MCV@ MICCAI, pages 59–71, 2015.[127] Shaoyu Wang, Minjeong Kim, Guorong Wu, and Dinggang Shen. Scalablehigh performance image registration framework by unsupervised deepfeature representations learning. In Deep Learning for Medical ImageAnalysis, pages 245–269. Elsevier, 2017.183[128] Siqi Liu, Sidong Liu, Weidong Cai, Sonia Pujol, Ron Kikinis, and DaganFeng. Early diagnosis of alzheimer’s disease with deep learning. InBiomedical Imaging (ISBI), 2014 IEEE 11th International Symposium on,pages 1015–1018. IEEE, 2014.[129] Heung-Il Suk, Seong-Whan Lee, Dinggang Shen, Alzheimer’sDisease Neuroimaging Initiative, et al. Latent feature representation withstacked auto-encoder for ad/mci diagnosis. Brain Structure and Function,220(2):841–859, 2015.[130] Jun Shi, Xiao Zheng, Yan Li, Qi Zhang, and Shihui Ying. Multimodalneuroimaging feature learning with multimodal stacked deep polynomialnetworks for diagnosis of alzheimer’s disease. IEEE journal of biomedicaland health informatics, 2017.[131] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, ArnaudArindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian,Jeroen AWM van der Laak, Bram van Ginneken, and Clara I Sa´nchez. Asurvey on deep learning in medical image analysis. arXiv preprintarXiv:1702.05747, 2017.[132] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutionalnetworks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.[133] Jeremy Kawahara, Colin J Brown, Steven P Miller, Brian G Booth, VannChau, Ruth E Grunau, Jill G Zwicker, and Ghassan Hamarneh.Brainnetcnn: Convolutional neural networks for brain networks; towardspredicting neurodevelopment. NeuroImage, 146:1038–1049, 2017.[134] Sarah Parisot, Sofia Ira Ktena, Enzo Ferrante, Matthew Lee,Ricardo Guerrerro Moreno, Ben Glocker, and Daniel Rueckert. Spectralgraph convolutions on population graphs for disease prediction. arXivpreprint arXiv:1703.03020, 2017.[135] David I Shuman, Benjamin Ricaud, and Pierre Vandergheynst. Awindowed graph fourier transform. In Statistical Signal ProcessingWorkshop (SSP), 2012 IEEE, pages 133–136. Ieee, 2012.[136] Wenqi Li, Guotai Wang, Lucas Fidon, Sebastien Ourselin, M JorgeCardoso, and Tom Vercauteren. On the compactness, efficiency, andrepresentation of 3d convolutional networks: Brain parcellation as a pretexttask. In International Conference on Information Processing in MedicalImaging, pages 348–360. Springer, 2017.184[137] Hannah Spitzer, Katrin Amunts, Stefan Harmeling, and Timo Dickscheid.Parcellation of visual cortex on high-resolution histological brain sectionsusing convolutional neural networks. arXiv preprint arXiv:1705.10545,2017.[138] Heung-Il Suk, Chong-Yaw Wee, Seong-Whan Lee, and Dinggang Shen.State-space model with deep learning for functional dynamics estimation inresting-state fmri. NeuroImage, 129:292–307, 2016.[139] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining andharnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.[140] Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi. One pixelattack for fooling deep neural networks. arXiv preprint arXiv:1710.08864,2017.[141] Matthew D Zeiler and Rob Fergus. Visualizing and understandingconvolutional networks. In European conference on computer vision, pages818–833. Springer, 2014.[142] Bratislav Misˇic´, Richard F Betzel, Marcel A De Reus, Martijn P VanDen Heuvel, Marc G Berman, Anthony R McIntosh, and Olaf Sporns.Network-level structure-function relationships in human neocortex.Cerebral Cortex, 26(7):3285–3296, 2016.[143] Maarten Mennes, Bharat B Biswal, F Xavier Castellanos, and Michael PMilham. Making data sharing work: The fcp/indi experience. Neuroimage,82:683–691, 2013.[144] Chao-Gan Yan, R Cameron Craddock, Xi-Nian Zuo, Yu-Feng Zang, andMichael P Milham. Standardizing the intrinsic brain: towards robustmeasurement of inter-individual variation in 1000 functional connectomes.Neuroimage, 80:246–262, 2013.[145] Yves Fre´gnac and Gilles Laurent. Neuroscience: Where is the brain in thehuman brain project? Nature, 513:27–29, 2014.[146] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJBehrens, Essa Yacoub, Kamil Ugurbil, WU-Minn HCP Consortium, et al.The wu-minn human connectome project: an overview. Neuroimage,80:62–79, 2013.185[147] Matthew F Glasser, Stamatios N Sotiropoulos, J Anthony Wilson,Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, SaadJbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimalpreprocessing pipelines for the human connectome project. Neuroimage,80:105–124, 2013.[148] Yashar Behzadi, Khaled Restom, Joy Liau, and Thomas T Liu. Acomponent based noise correction method (compcor) for bold andperfusion based fmri. Neuroimage, 37(1):90–101, 2007.[149] Klaus H Fritzsche, Peter F Neher, Ignaz Reicht, Thomas van Bruggen,Caspar Goch, Marco Reisert, Marco Nolden, Sascha Zelzer, Hans-PeterMeinzer, Bram Stieltjes, et al. Mitk diffusion imaging. Methods ofinformation in medicine, 51(5):441, 2012.[150] Bernard Ng, Gae¨l Varoquaux, Jean Baptiste Poline, and Bertrand Thirion.Implications of inconsistencies between fmri and dmri on multimodalconnectivity estimation. In International Conference on Medical ImageComputing and Computer-Assisted Intervention, pages 652–659. Springer,2013.[151] Xiaoying Tang, Shoko Yoshida, John Hsu, Thierry AGM Huisman,Andreia V Faria, Kenichi Oishi, Kwame Kutten, Andrea Poretti, Yue Li,Michael I Miller, et al. Multi-contrast multi-atlas parcellation of diffusiontensor imaging of the human brain. PloS one, 9(5):e96985, 2014.[152] Xiangyu Long, Dominique Goltz, Daniel S Margulies, Till Nierhaus, andArno Villringer. Functional connectivity-based parcellation of the humansensorimotor cortex. European Journal of Neuroscience, 39(8):1332–1342,2014.[153] David Moreno-Dominguez, Alfred Anwander, and Thomas R Kno¨sche. Ahierarchical method for whole-brain connectivity-based parcellation.Human brain mapping, 35(10):5000–5025, 2014.[154] Thomas Blumensath, Saad Jbabdi, Matthew F Glasser, David C Van Essen,Kamil Ugurbil, Timothy EJ Behrens, and Stephen M Smith. Spatiallyconstrained hierarchical parcellation of the brain with resting-state fmri.Neuroimage, 76:313–324, 2013.[155] Xilin Shen, Fuyuze Tokoglu, Xenios Papademetris, and R Todd Constable.Groupwise whole-brain parcellation from resting-state fmri data fornetwork node identification. Neuroimage, 82:403–415, 2013.186[156] Gagan S Wig, Timothy O Laumann, Alexander L Cohen, Jonathan DPower, Steven M Nelson, Matthew F Glasser, Francis M Miezin,Abraham Z Snyder, Bradley L Schlaggar, and Steven E Petersen.Parcellating an individual subject’s cortical and subcortical brain structuresusing snowball sampling of resting-state correlations. Cerebral cortex,page bht056, 2013.[157] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics andcomputing, 17(4):395–416, 2007.[158] Lihi Zelnik-Manor and Pietro Perona. Self-tuning spectral clustering. InAdvances in neural information processing systems, pages 1601–1608,2004.[159] Lee R Dice. Measures of the amount of ecologic association betweenspecies. Ecology, 26(3):297–302, 1945.[160] Harold W Kuhn. The hungarian method for the assignment problem. NavalResearch Logistics (NRL), 2(1-2):83–97, 1955.[161] Xilin Shen, Xenophon Papademetris, and R Todd Constable. Graph-theorybased parcellation of functional subunits in the brain from resting-state fmridata. Neuroimage, 50(3):1027–1035, 2010.[162] James Munkres. Algorithms for the assignment and transportationproblems. Journal of the Society for Industrial & Applied Mathematics,5(1):32–38, 1957.[163] Evan M Gordon, Timothy O Laumann, Babatunde Adeyemo, Jeremy FHuckins, William M Kelley, and Steven E Petersen. Generation andevaluation of a cortical area parcellation from resting-state correlations.Cerebral cortex, 26(1):288–303, 2014.[164] R Cameron Craddock, G Andrew James, Paul E Holtzheimer, Xiaoping PHu, and Helen S Mayberg. A whole brain fmri atlas generated via spatiallyconstrained spectral clustering. Human brain mapping, 33(8):1914–1928,2012.[165] Jiaojian Wang, Yong Yang, Lingzhong Fan, Jinping Xu, Changhai Li, YongLiu, Peter T Fox, Simon B Eickhoff, Chunshui Yu, and Tianzi Jiang.Convergent functional architecture of the superior parietal lobule unraveledwith multimodal neuroimaging approaches. Human brain mapping,36(1):238–257, 2015.187[166] Jason W Bohland, Hemant Bokil, Cara B Allen, and Partha P Mitra. Thebrain atlas concordance problem: quantitative comparison of anatomicalparcellations. PloS one, 4(9):e7200, 2009.[167] Qianjun Xu, Marie Desjardins, and Kiri Wagstaff. Constrained spectralclustering under a local proximity structure assumption. In In FLAIRSConference. Citeseer, 2005.[168] Bertrand Thirion, Guillaume Flandin, Philippe Pinel, Alexis Roche,Philippe Ciuciu, and Jean-Baptiste Poline. Dealing with the shortcomingsof spatial normalization: Multi-subject parcellation of fmri datasets.Human brain mapping, 27(8):678–693, 2006.[169] Bertrand Thirion, Gae¨l Varoquaux, Elvis Dohmatob, and Jean-BaptistePoline. Which fmri clustering gives good brain parcellations? Frontiers inneuroscience, 8, 2014.[170] Simon B Eickhoff, Klaas E Stephan, Hartmut Mohlberg, Christian Grefkes,Gereon R Fink, Katrin Amunts, and Karl Zilles. A new spm toolbox forcombining probabilistic cytoarchitectonic maps and functional imagingdata. Neuroimage, 25(4):1325–1335, 2005.[171] Martijn Van Den Heuvel, Rene Mandl, and Hilleke Hulshoff Pol.Normalized cut group clustering of resting-state fmri data. PloS one,3(4):e2001, 2008.[172] Nathalie Tzourio-Mazoyer, Brigitte Landeau, Dimitri Papathanassiou,Fabrice Crivello, Olivier Etard, Nicolas Delcroix, Bernard Mazoyer, andMarc Joliot. Automated anatomical labeling of activations in spm using amacroscopic anatomical parcellation of the mni mri single-subject brain.Neuroimage, 15(1):273–289, 2002.[173] John Wiley and Sons. Motor map, 2011.[174] Sebastian Bludau, Simon B Eickhoff, Hartmut Mohlberg, Svenja Caspers,Angela R Laird, Peter T Fox, Axel Schleicher, Karl Zilles, and KatrinAmunts. Cytoarchitecture, probability maps and functions of the humanfrontal pole. Neuroimage, 93:260–275, 2014.[175] Maxime Descoteaux, Rachid Deriche, Thomas R Knosche, and AlfredAnwander. Deterministic and probabilistic tractography based on complexfibre orientation distributions. IEEE transactions on medical imaging,28(2):269–286, 2009.188[176] Emmanuel Caruyer, Christophe Lenglet, Guillermo Sapiro, and RachidDeriche. Design of multishell sampling schemes with uniform coverage indiffusion mri. Magnetic resonance in medicine, 69(6):1534–1540, 2013.[177] Lipeng Ning, Kawin Setsompop, Oleg Michailovich, Nikos Makris,Martha E Shenton, Carl-Fredrik Westin, and Yogesh Rathi. A jointcompressed-sensing and super-resolution approach for very high-resolutiondiffusion imaging. NeuroImage, 125:386–400, 2016.[178] Emmanuel J Candes and Benjamin Recht. Exact low-rank matrixcompletion via convex optimization. In Communication, Control, andComputing, 2008 46th Annual Allerton Conference on, pages 806–812.IEEE, 2008.[179] Zaiwen Wen, Wotao Yin, and Yin Zhang. Solving a low-rank factorizationmodel for matrix completion by a nonlinear successive over-relaxationalgorithm. Mathematical Programming Computation, 4(4):333–361, 2012.[180] Raghunandan H Keshavan, Andrea Montanari, and Sewoong Oh. Matrixcompletion from a few entries. IEEE Transactions on Information Theory,56(6):2980–2998, 2010.[181] Yangyang Xu, Wotao Yin, Zaiwen Wen, and Yin Zhang. An alternatingdirection algorithm for matrix completion with nonnegative factors.Frontiers of Mathematics in China, 7(2):365–384, 2012.[182] Yonghui Li, Yong Liu, Jun Li, Wen Qin, Kuncheng Li, Chunshui Yu, andTianzi Jiang. Brain anatomical network and intelligence. PLoS ComputBiol, 5(5):e1000395, 2009.[183] Laura Balzano, Robert Nowak, and Benjamin Recht. Online identificationand tracking of subspaces from highly incomplete information. InCommunication, Control, and Computing (Allerton), 2010 48th AnnualAllerton Conference on, pages 704–711. IEEE, 2010.[184] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, andChih-Jen Lin. Liblinear: A library for large linear classification. Journal ofmachine learning research, 9(Aug):1871–1874, 2008.[185] Klaus Maier-Hein, Peter Neher, Jean-Christophe Houde, Marc-AlexandreCote, Eleftherios Garyfallidis, Jidan Zhong, Maxime Chamberland,Fang-Cheng Yeh, Ying Chia Lin, Qing Ji, et al. Tractography-basedconnectomes are dominated by false-positive connections. bioRxiv, page084137, 2016.189[186] Hojjatollah Azadbakht, Laura M Parkes, Hamied A Haroon, Mark Augath,Nikos K Logothetis, Alex de Crespigny, Helen E D’Arceuil, andGeoffrey JM Parker. Validation of high-resolution tractography against invivo tracing in the macaque visual cortex. Cerebral Cortex,25(11):4299–4309, 2015.[187] Chad J Donahue, Stamatios N Sotiropoulos, Saad Jbabdi, MoisesHernandez-Fernandez, Timothy E Behrens, Tim B Dyrby, TimothyCoalson, Henry Kennedy, Kenneth Knoblauch, David C Van Essen, et al.Using diffusion tractography to predict cortical connection strength anddistance: a quantitative comparison with tracers in the monkey. Journal ofNeuroscience, 36(25):6758–6770, 2016.[188] Sophie Achard, Jean-Franc¸ois Coeurjolly, Romain Marcillaud, and JonasRichiardi. fmri functional connectivity estimators robust to region size bias.In Statistical Signal Processing Workshop, pages 813–816. IEEE, 2011.[189] Alex Fornito, Andrew Zalesky, and Edward T Bullmore. Network scalingeffects in graph analytic studies of human resting-state fmri data. Frontiersin systems neuroscience, 4, 2010.[190] Tama´s Spisa´k, Andra´s Jakab, Sa´ndor A Kis, Ga´bor Opposits, CsabaAranyi, Ervin Bere´nyi, and Miklo´s Emri. Voxel-wise motion artifacts inpopulation-level whole-brain connectivity analysis of resting-state fmri.PloS one, 9(9):e104947, 2014.[191] David K Hammond, Pierre Vandergheynst, and Re´mi Gribonval. Waveletson graphs via spectral graph theory. Applied and Computational HarmonicAnalysis, 30(2):129–150, 2011.[192] Ed Bullmore and Olaf Sporns. Complex brain networks: graph theoreticalanalysis of structural and functional systems. Nature reviews.Neuroscience, 10(3):186, 2009.[193] Jianwei Niu, Jinyang Fan, and Ivan Stojmenovic. Jlmc: A clusteringmethod based on jordan-form of laplacian-matrix. In PerformanceComputing and Communications Conference, pages 1–8. IEEE, 2014.[194] Santo Fortunato and Marc Barthelemy. Resolution limit in communitydetection. Proceedings of the National Academy of Sciences,104(1):36–41, 2007.190[195] David Meunier, Renaud Lambiotte, Alex Fornito, Karen D Ersche, andEdward T Bullmore. Hierarchical modularity in human brain functionalnetworks. Frontiers in neuroinformatics, 3, 2009.[196] David Meunier, Renaud Lambiotte, and Edward T Bullmore. Modular andhierarchically modular organization of brain networks. Frontiers inneuroscience, 4, 2010.[197] Roger Guimera and Luis A Nunes Amaral. Functional cartography ofcomplex metabolic networks. nature, 433(7028):895, 2005.[198] Carlo Nicolini, Ce´cile Bordier, and Angelo Bifone. Community detectionin weighted brain connectivity networks beyond the resolution limit.Neuroimage, 146:28–39, 2017.[199] Leo Grady. Random walks for image segmentation. IEEE transactions onpattern analysis and machine intelligence, 28(11):1768–1783, 2006.[200] Leo Grady. Multilabel random walker image segmentation using priormodels. In Computer Vision and Pattern Recognition, 2005. CVPR 2005.IEEE Computer Society Conference on, volume 1, pages 763–770. IEEE,2005.[201] Carlo Nicolini and Angelo Bifone. Modular structure of brain functionalnetworks: breaking the resolution limit by surprise. Scientific reports, 6,2016.[202] Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. DataMining: Practical machine learning tools and techniques. MorganKaufmann, 2016.[203] Burak Yoldemir, Bernard Ng, and Rafeef Abugharbieh. Coupled stableoverlapping replicator dynamics for multimodal brain subnetworkidentification. In International Conference on Information Processing inMedical Imaging, pages 770–781. Springer, 2015.[204] Jonas Richiardi, Andre Altmann, Anna-Clare Milazzo, Catie Chang,M Mallar Chakravarty, Tobias Banaschewski, Gareth J Barker, Arun LWBokde, Uli Bromberg, Christian Bu¨chel, et al. Correlated gene expressionsupports synchronous activity in brain networks. Science,348(6240):1241–1244, 2015.191[205] Luca Dodero, Alessandro Gozzi, Adam Liska, Vittorio Murino, and DiegoSona. Group-wise functional community detection through joint laplaciandiagonalization. In MICCAI (2), pages 708–715, 2014.[206] Michael An, Hon Pong Ho, Lawrence Staib, Kevin Pelphrey, and JamesDuncan. Multimodal mri analysis of brain subnetworks in autism usingmulti-view em. In Signals, Systems and Computers (ASILOMAR), 2010Conference Record of the Forty Fourth Asilomar Conference on, pages786–789. IEEE, 2010.[207] Bo Wang, Jiayan Jiang, Wei Wang, Zhi-Hua Zhou, and Zhuowen Tu.Unsupervised metric fusion by cross diffusion. In Computer Vision andPattern Recognition (CVPR), 2012 IEEE Conference on, pages 2997–3004.IEEE, 2012.[208] Abhishek Kumar, Piyush Rai, and Hal Daume. Co-regularized multi-viewspectral clustering. In Advances in neural information processing systems,pages 1413–1421, 2011.[209] Christopher J Honey, Rolf Ko¨tter, Michael Breakspear, and Olaf Sporns.Network structure of cerebral cortex shapes functional connectivity onmultiple time scales. Proceedings of the National Academy of Sciences,104(24):10240–10245, 2007.[210] Michael D Fox, Abraham Z Snyder, Justin L Vincent, and Marcus ERaichle. Intrinsic fluctuations within cortical systems account for intertrialvariability in human behavior. Neuron, 56(1):171–184, 2007.[211] Michael W Cole, Danielle S Bassett, Jonathan D Power, Todd S Braver,and Steven E Petersen. Intrinsic and task-evoked network architectures ofthe human brain. Neuron, 83(1):238–251, 2014.[212] Micaela Y Chan, Fahd H Alhazmi, Denise C Park, Neil K Savalia, andGagan S Wig. Resting-state network topology differentiates task signalsacross the adult life span. Journal of Neuroscience, 37(10):2734–2745,2017.[213] Bumhee Park, Dae-Shik Kim, and Hae-Jeong Park. Graph independentcomponent analysis reveals repertoires of intrinsic network components inthe human brain. PloS one, 9(1):e82873, 2014.[214] Maarten Mennes, Clare Kelly, Xi-Nian Zuo, Adriana Di Martino, Bharat BBiswal, F Xavier Castellanos, and Michael P Milham. Inter-individual192differences in resting-state functional connectivity predict task-inducedbold activity. Neuroimage, 50(4):1690–1701, 2010.[215] Maxwell A Bertolero, BT Thomas Yeo, and Mark D’Esposito. Themodular and integrative functional architecture of the human brain.Proceedings of the National Academy of Sciences, 112(49):E6798–E6807,2015.[216] Angela R Laird, P Mickle Fox, Simon B Eickhoff, Jessica A Turner,Kimberly L Ray, D Reese McKay, David C Glahn, Christian F Beckmann,Stephen M Smith, and Peter T Fox. Behavioral interpretations of intrinsicconnectivity networks. Journal of cognitive neuroscience,23(12):4022–4037, 2011.[217] M-Marsel Mesulam. From sensation to cognition. Brain: a journal ofneurology, 121(6):1013–1052, 1998.[218] Manlio De Domenico. Multilayer modeling and analysis of human brainnetworks. Giga Science, 6(5):1–8, 2017.[219] Sarah Feldt Muldoon and Danielle S Bassett. Network and multilayernetwork approaches to understanding human brain dynamics. Philosophyof Science, 83(5):710–720, 2016.[220] Manlio De Domenico, Shuntaro Sasai, and Alex Arenas. Mappingmultiplex hubs in human functional brain networks. Frontiers inneuroscience, 10, 2016.[221] Denny Zhou, Jiayuan Huang, and Bernhard Scho¨lkopf. Learning withhypergraphs: Clustering, classification, and embedding. In Advances inneural information processing systems, pages 1601–1608, 2007.[222] Pawe Dotko, Kathryn Hess, Ran Levi, Max Nolte, Michael Reimann,Martina Scolamiero, Katharine Turner, Eilif Muller, and Henry Markram.Topological analysis of the connectome of digital reconstructions of neuralmicrocircuits. arXiv preprint arXiv:1601.01580, 2016.[223] Chen Zu, Yue Gao, Brent Munsell, Minjeong Kim, Ziwen Peng, YingyingZhu, Wei Gao, Daoqiang Zhang, Dinggang Shen, and Guorong Wu.Identifying high order brain connectome biomarkers via learning onhypergraph. In International Workshop on Machine Learning in MedicalImaging, pages 1–9. Springer, 2016.193[224] Brent C Munsell, Guorong Wu, Yue Gao, Nicholas Desisto, and MartinStyner. Identifying relationships in functional and structural connectomedata using a hypergraph learning method. In International Conference onMedical Image Computing and Computer-Assisted Intervention, pages9–17. Springer, 2016.[225] Mechthild Stoer and Frank Wagner. A simple min-cut algorithm. Journalof the ACM (JACM), 44(4):585–591, 1997.[226] Lars Hagen and Andrew B Kahng. New spectral methods for ratio cutpartitioning and clustering. IEEE transactions on computer-aided design ofintegrated circuits and systems, 11(9):1074–1085, 1992.[227] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation.IEEE Transactions on pattern analysis and machine intelligence,22(8):888–905, 2000.[228] Karl J Friston, Andrew P Holmes, Keith J Worsley, J-P Poline, Chris DFrith, and Richard SJ Frackowiak. Statistical parametric maps in functionalimaging: a general linear approach. Human brain mapping, 2(4):189–210,1994.[229] Bernard Ng, Ghassan Hamarneh, and Rafeef Abugharbieh. Modeling brainactivation in fmri using group mrf. IEEE transactions on medical imaging,31(5):1113–1123, 2012.[230] Thomas Nichols and Satoru Hayasaka. Controlling the familywise errorrate in functional neuroimaging: a comparative review. Statistical methodsin medical research, 12(5):419–446, 2003.[231] Peter J Mucha, Thomas Richardson, Kevin Macon, Mason A Porter, andJukka-Pekka Onnela. Community structure in time-dependent, multiscale,and multiplex networks. science, 328(5980):876–878, 2010.[232] Jarmo Heinonen, Jussi Numminen, Yevhen Hlushchuk, Henrik Antell, VesaTaatila, and Jyrki Suomala. Default mode and executive networks areas:Association with the serial order in divergent thinking. PloS one,11(9):e0162234, 2016.[233] Caroline J Charpentier, Benedetto De Martino, Alena L Sim, Tali Sharot,and Jonathan P Roiser. Emotion-induced loss aversion andstriatal-amygdala coupling in low-anxious individuals. Social cognitiveand affective neuroscience, 11(4):569–579, 2015.194[234] Saskia Koehler, Smadar Ovadia-Caro, Elke van der Meer, Arno Villringer,Andreas Heinz, Nina Romanczuk-Seiferth, and Daniel S Margulies.Increased functional connectivity between prefrontal cortex and rewardsystem in pathological gambling. PLoS One, 8(12):e84565, 2013.[235] Eve Helen Limbrick-Oldfield, I Mick, RE Cocks, John McGonigle,SP Sharman, Anthony P Goldstone, PRA Stokes, Adam Waldman,D Erritzoe, H Bowden-Jones, et al. Neural substrates of cue reactivity andcraving in gambling disorder. Translational psychiatry, 7(1):e992, 2017.[236] Josiah K Leong, Franco Pestilli, Charlene C Wu, Gregory RSamanez-Larkin, and Brian Knutson. White-matter tract connectinganterior insula to nucleus accumbens correlates with reduced preference forpositively skewed gambles. Neuron, 89(1):63–69, 2016.[237] Franco Cauda, Federico D’agata, Katiuscia Sacco, Sergio Duca, GiulianoGeminiani, and Alessandro Vercelli. Functional connectivity of the insulain the resting brain. Neuroimage, 55(1):8–23, 2011.[238] Anand Venkatraman, Brian L Edlow, and Mary Helen Immordino-Yang.The brainstem in emotion: A review. Frontiers in neuroanatomy, 11, 2017.[239] Yu Ohmura, Takeshi Izumi, Taku Yamaguchi, Iku Tsutsui-Kimura,Takayuki Yoshida, and Mitsuhiro Yoshioka. The serotonergic projectionfrom the median raphe nucleus to the ventral hippocampus is involved inthe retrieval of fear memory through the corticotropin-releasing factor type2 receptor. Neuropsychopharmacology, 35(6):1271, 2010.[240] David H Zald. The human amygdala and the emotional evaluation ofsensory stimuli. Brain Research Reviews, 41(1):88–123, 2003.[241] Adrian W Laxton, Joseph S Neimat, Karen D Davis, Thilo Womelsdorf,William D Hutchison, Jonathan O Dostrovsky, Clement Hamani, Helen SMayberg, and Andres M Lozano. Neuronal coding of implicit emotioncategories in the subcallosal cortex in patients with depression. Biologicalpsychiatry, 74(10):714–719, 2013.[242] Sara M Levens, Orrin Devinsky, and Elizabeth A Phelps. Role of the leftamygdala and right orbital frontal cortex in emotional interferenceresolution facilitation in working memory. Neuropsychologia,49(12):3201–3212, 2011.195[243] To¨mme Noesselt, Nadim Jon Shah, and Lutz Ja¨ncke. Top-down andbottom-up modulation of language related areas–an fmri study. BMCneuroscience, 4(1):13, 2003.[244] Martin Meyer, Kai Alter, and Angela Friederici. Functional mr imagingexposes differential brain responses to syntax and prosody during auditorysentence comprehension. Journal of Neurolinguistics, 16(4):277–300,2003.[245] Jierui Xie, Stephen Kelley, and Boleslaw K Szymanski. Overlappingcommunity detection in networks: The state-of-the-art and comparativestudy. Acm computing surveys (csur), 45(4):43, 2013.[246] Gergely Palla, Imre Dere´nyi, Ille´s Farkas, and Tama´s Vicsek. Uncoveringthe overlapping community structure of complex networks in nature andsociety. arXiv preprint physics/0506133, 2005.[247] TS Evans and Renaud Lambiotte. Line graphs, link partitions, andoverlapping communities. Physical Review E, 80(1):016105, 2009.[248] Yong-Yeol Ahn, James P Bagrow, and Sune Lehmann. Link communitiesreveal multiscale complexity in networks. arXiv preprint arXiv:0903.3178,2009.[249] Shihua Zhang, Rui-Sheng Wang, and Xiang-Sun Zhang. Identification ofoverlapping community structure in complex networks using fuzzyc-means clustering. Physica A: Statistical Mechanics and its Applications,374(1):483–490, 2007.[250] Fan Ding, Zhigang Luo, Jinlong Shi, and Xiaoyong Fang. Overlappingcommunity detection by kernel-based fuzzy affinity propagation. InIntelligent Systems and Applications (ISA), 2010 2nd InternationalWorkshop on, pages 1–4. IEEE, 2010.[251] Andrea Lancichinetti, Santo Fortunato, and Ja´nos Kerte´sz. Detecting theoverlapping and hierarchical community structure in complex networks.New Journal of Physics, 11(3):033015, 2009.[252] Xiaodan Yan, Stephen Kelley, Mark Goldberg, and Bharat B Biswal.Detecting overlapped functional clusters in resting state fmri withconnected iterative scan: a graph theory based clustering algorithm.Journal of neuroscience methods, 199(1):108–118, 2011.196[253] Burak Yoldemir, Bernard Ng, and Rafeef Abugharbieh. Stable overlappingreplicator dynamics for brain community detection. IEEE transactions onmedical imaging, 35(2):529–538, 2016.[254] Andrea Lancichinetti, Filippo Radicchi, Jose´ J Ramasco, and SantoFortunato. Finding statistically significant communities in networks. PloSone, 6(4):e18961, 2011.[255] Andrea Torsello, Samuel Rota Bulo, and Marcello Pelillo. Beyondpartitions: Allowing overlapping groups in pairwise clustering. In PatternRecognition, 2008. ICPR 2008. 19th International Conference on, pages1–4. IEEE, 2008.[256] Randy L Buckner, Jorge Sepulcre, Tanveer Talukdar, Fenna M Krienen,Hesheng Liu, Trey Hedden, Jessica R Andrews-Hanna, Reisa A Sperling,and Keith A Johnson. Cortical hubs revealed by intrinsic functionalconnectivity: mapping, assessment of stability, and relation to alzheimer’sdisease. Journal of Neuroscience, 29(6):1860–1873, 2009.[257] R GeethaRamani and K Sivaselvi. Human brain hubs (provincial andconnector) identification using centrality measures. In Recent Trends inInformation Technology (ICRTIT), 2014 International Conference on,pages 1–6. IEEE, 2014.[258] Yong He, Zhang Chen, and Alan Evans. Structural insights into aberranttopological patterns of large-scale cortical networks in alzheimer’s disease.Journal of Neuroscience, 28(18):4756–4766, 2008.[259] Quintino R Mano, Colin Humphries, Rutvik H Desai, Mark S Seidenberg,David C Osmon, Ben C Stengel, and Jeffrey R Binder. The role of leftoccipitotemporal cortex in reading: reconciling stimulus, task, andlexicality effects. Cerebral Cortex, 23(4):988–1001, 2012.[260] Je-Yeon Yun, Jae-Chang Kim, Jeonghun Ku, Jung-Eun Shin, Jae-Jin Kim,and Soo-Hee Choi. The left middle temporal gyrus in the middle of animpaired social-affective communication network in social anxietydisorder. Journal of Affective Disorders, 214:53–59, 2017.[261] Joana Cabral, Morten Kringelbach, and Gustavo Deco. Functionalconnectivity dynamically evolves on multiple time-scales over a staticstructural connectome: Models and mechanisms. NeuroImage, 2017.197[262] Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. Training afeedback loop for hand pose estimation. In Proceedings of the IEEEInternational Conference on Computer Vision, pages 3316–3324, 2015.[263] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routingbetween capsules. In Advances in Neural Information Processing Systems,pages 3857–3867, 2017.[264] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, andM Jorge Cardoso. Generalised dice overlap as a deep learning loss functionfor highly unbalanced segmentations. In Deep Learning in Medical ImageAnalysis and Multimodal Learning for Clinical Decision Support, pages240–248. Springer, 2017.[265] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and PierreVandergheynst. Geometric deep learning: going beyond euclidean data.arXiv preprint arXiv:1611.08097, 2016.198
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Multimodal human brain connectivity analysis based...
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Multimodal human brain connectivity analysis based on graph theory Wang, Chendi 2018
pdf
Page Metadata
Item Metadata
Title | Multimodal human brain connectivity analysis based on graph theory |
Creator |
Wang, Chendi |
Publisher | University of British Columbia |
Date Issued | 2018 |
Description | Billions of people worldwide are affected by neurological disorders. Recent studies indicate that many neurological disorders can be described as dysconnectivity syndromes, and associated with changes in the brain networks prior to the development of clinical symptoms. This thesis presents contributions towards improving brain connectivity analysis based on graph theory representation of the human brain network. We propose novel multimodal techniques to analyze brain imaging data to better understand its structure, function and connectivity, i.e., brain connectomics. Our first contribution is towards improving parcellation, \ie brain network node definition, in terms of reproducibility, functional homogeneity, leftout data likelihood and overlaps with cytoarchitecture, by utilizing the neighbourhood information and multi-modality integration techniques. Specifically, we embed neighborhood connectivity information into the affinity matrix for parcellation to ameliorate the adverse effects of noise. We further integrate the connectivity information from both anatomical and functional modalities based on adaptive weighting for an improved parcellation. Our second contribution is to propose noise reduction techniques for brain network edge definition. We propose a matrix completion based technique to combat false negatives by recovering missing connections. We also present a local thresholding method which can address the regional bias issue when suppressing the false positives in connectivity estimates. Our third contribution is to improve the brain subnetwork extraction by using multi-pronged graphical metric guided methods. We propose a connection-fingerprint based modularity reinforcement model which reflects the putative modular structure of a brain graph. Inspired by the brain subnetwork's biological nature, we propose a provincial hub guided feedback optimization model for more reproducible subnetwork extraction. Our fourth contribution is to develop multimodal integration techniques to further improve brain subnetwork extraction. We propose a provincial hub guided subnetwork extraction model to fuse anatomical and functional data by propagating the modular structure information across different modalities. We further propose to fuse the task and rest functional data based on hypergraphs for non-overlapping and overlapping subnetwork extraction. Our results collectively indicate that combing multimodal information and applying graphical metric guided strategies outperform classical unimodal brain connectivity analysis methods. The resulting methods could provide important insights into cognitive and clinical neuroscience. |
Genre |
Thesis/Dissertation |
Type |
Text |
Language | eng |
Date Available | 2018-04-24 |
Provider | Vancouver : University of British Columbia Library |
Rights | Attribution-NonCommercial-NoDerivatives 4.0 International |
DOI | 10.14288/1.0365983 |
URI | http://hdl.handle.net/2429/65623 |
Degree |
Doctor of Philosophy - PhD |
Program |
Electrical and Computer Engineering |
Affiliation |
Applied Science, Faculty of Electrical and Computer Engineering, Department of |
Degree Grantor | University of British Columbia |
GraduationDate | 2018-05 |
Campus |
UBCV |
Scholarly Level | Graduate |
Rights URI | http://creativecommons.org/licenses/by-nc-nd/4.0/ |
AggregatedSourceRepository | DSpace |
Download
- Media
- 24-ubc_2018_may_wang_chendi.pdf [ 19.15MB ]
- Metadata
- JSON: 24-1.0365983.json
- JSON-LD: 24-1.0365983-ld.json
- RDF/XML (Pretty): 24-1.0365983-rdf.xml
- RDF/JSON: 24-1.0365983-rdf.json
- Turtle: 24-1.0365983-turtle.txt
- N-Triples: 24-1.0365983-rdf-ntriples.txt
- Original Record: 24-1.0365983-source.json
- Full Text
- 24-1.0365983-fulltext.txt
- Citation
- 24-1.0365983.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.24.1-0365983/manifest