Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Jiang, Minyang
Master of Applied Science - MASc
University of British Columbia
Crowd analysis has been widely used in everyday life. Among different crowd analysis tasks, crowd counting is the most basic but essential one that measures the number of people in a particular area. Such counting and crowd density information is crucial to determine the maximum occupancy of a room or public area to address safety concerns. Counting by hand can yield an accurate number, but it is a tedious job and may not fulfill the time requirement for analysis. Therefore, automated crowd counting for accurately estimating the number of people in crowded scenes is needed. Deep learning has shown beyond human-level accuracy in many computer vision tasks. Recently, researchers have shown that deep learning methods can achieve the state-of-the-art performance in the task of crowd counting in images. The majority of these methods are based on density regression, where a density map is predicted and its integral is further calculated to obtain the final count. However, this learned density map is uninterpretable and could deviate largely from the actual person distribution even if the final count is accurate. Besides, existing crowd counting methods are mainly based on cumbersome feature extraction networks and can not be deployed in edge devices with limited computational power. In this thesis work, we proposed two models that tackle the above interpretability and computational efficiency concerns respectively: the joint crowd counting and localization model extends traditional counting only methods and provides precise localization results without additional model complexity; the ShuffleCount is a computationally efficient and accurate model that is trained through the specially designed knowledge distillation process and can satisfy the real-time crowd analysis requirement on edge devices. We evaluated both models by training and testing on public crowd counting benchmark datasets. Both quantitative and qualitative results are obtained and compared with existing state-of-the-art methods. Our superior results show the potential of deploying the proposed methods to real-life applications for efficient crowd analysis. The methods proposed in both JCCL and ShuffleCount can be generalized to other ideas to improve the interpretability and computational efficiency of the models.
Deep Learning Approaches for Crowd Counting in ImagesbyMinyang JiangB. Software Engineering, University of Waterloo, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Electrical and Computer Engineering)The University of British Columbia(Vancouver)April 2021© Minyang Jiang, 2021The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Deep Learning Approaches for Crowd Counting in Imagessubmitted by Minyang Jiang in partial fulfillment of the requirements for the de-gree of MASTER OF APPLIED SCIENCE in Electrical and Computer Engi-neering.Examining Committee:Z. Jane Wang, Electrical and Computer Engineering, UBCSupervisorRabab Ward, Electrical and Computer Engineering, UBCCommittee MemberPurang Abolmaesumi, Electrical and Computer Engineering, UBCChairiiAbstractCrowd analysis has been widely used in everyday life. Among different crowdanalysis tasks, crowd counting is the most basic but essential one that measuresthe number of people in a particular area. Such counting and crowd density infor-mation is crucial to determine the maximum occupancy of a room or public areato address safety concerns. Counting by hand can yield an accurate number, butit is a tedious job and may not fulfill the time requirement for analysis. There-fore, automated crowd counting for accurately estimating the number of people incrowded scenes is needed. Deep learning has shown beyond human-level accu-racy in many computer vision tasks. Recently, researchers have shown that deeplearning methods can achieve the state-of-the-art performance in the task of crowdcounting in images. The majority of these methods are based on density regression,where a density map is predicted and its integral is further calculated to obtain thefinal count. However, this learned density map is uninterpretable and could devi-ate largely from the actual person distribution even if the final count is accurate.Besides, existing crowd counting methods are mainly based on cumbersome fea-ture extraction networks and can not be deployed in edge devices with limitedcomputational power. In this thesis work, we proposed two models that tackle theabove interpretability and computational efficiency concerns respectively: the jointcrowd counting and localization model extends traditional counting only meth-ods and provides precise localization results without additional model complexity;the ShuffleCount is a computationally efficient and accurate model that is trainedthrough the specially designed knowledge distillation process and can satisfy thereal-time crowd analysis requirement on edge devices. We evaluated both modelsby training and testing on public crowd counting benchmark datasets. Both quan-iiititative and qualitative results are obtained and compared with existing state-of-the-art methods. Our superior results show the potential of deploying the proposedmethods to real-life applications for efficient crowd analysis. The methods pro-posed in both JCCL and ShuffleCount can be generalized to other ideas to improvethe interpretability and computational efficiency of the models.ivLay SummaryAutomated crowd counting models that can accurately estimate the number of peo-ple in crowded scene images are absolutely necessary for conducting crowd anal-ysis. For instance, it helps control the number of people in dense areas and thushelps address safety concerns. Deep learning has achieved great success in manyimages and video-related tasks, including crowd counting. However, people cur-rently build cumbersome deep learning models to achieve higher counting accu-racy without considering the interpretability of the models and the feasibility ofdeploying such models in real life. In this thesis, to address the above concerns, wepropose and evaluate two crowd counting models. First, we design a model that canaccurately count the number of people in crowded scenes by providing precise headlocalization of each person. With precise localization results, the counting resultis more persuasive, and additional detailed crowd analysis can be conducted usingthe localization results. Secondly, we build a very compact yet accurate modelthat can be executed quickly on edge devices without requiring high computationpower. This model is desirable for the deployment of crowd counting applicationsin real life. Both methods yield superior results over existing comparison crowdcounting methods.vPrefaceThis thesis research was jointly initiated by Dr. Z. Jane Wang, Dr. Jianzhe Lin, andthe thesis author. The majority work of the research was conducted by the authorof this thesis with valuable suggestions from Dr. Z. Jane Wang and Dr. JianzheLin. Chapter 2 is part of submitted work with the title of âA Smartly Simple Wayfor Joint Crowd Counting and Localizationâ; and Chapter 3 is part of submittedwork with the title of âShuffleCount: task-specific knowledge distillation for crowdcounting.âviTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Introduction to machine learning . . . . . . . . . . . . . . . . . . 11.1.1 Neural Network and Deep Learning . . . . . . . . . . . . 11.1.2 Neural Network . . . . . . . . . . . . . . . . . . . . . . . 41.1.3 Convolutional Neural Network . . . . . . . . . . . . . . . 71.2 Introduction to Crowd Counting . . . . . . . . . . . . . . . . . . 91.2.1 Density regression based crowd counting . . . . . . . . . 101.3 Thesis Objective . . . . . . . . . . . . . . . . . . . . . . . . . . 131.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16vii2 Crowd Counting and Localization in Images . . . . . . . . . . . . . 182.1 Human Localization . . . . . . . . . . . . . . . . . . . . . . . . . 182.2 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 192.2.1 End-to-End Framework Overview . . . . . . . . . . . . . 192.2.2 Deep Integrated Module . . . . . . . . . . . . . . . . . . 202.2.3 Scale Adaptive Module . . . . . . . . . . . . . . . . . . . 222.2.4 Interval Aware Module . . . . . . . . . . . . . . . . . . . 242.2.5 Pedestrian-Aware Density Map . . . . . . . . . . . . . . 252.2.6 Model Loss . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.7 Non-maximum suppression . . . . . . . . . . . . . . . . 282.2.8 Confidence map Regression . . . . . . . . . . . . . . . . 282.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.3.2 Implementation Details . . . . . . . . . . . . . . . . . . . 302.3.3 Training Details . . . . . . . . . . . . . . . . . . . . . . . 312.3.4 Evaluation on Counting . . . . . . . . . . . . . . . . . . 322.3.5 Evaluation on Localization . . . . . . . . . . . . . . . . . 352.3.6 Qualitative Results . . . . . . . . . . . . . . . . . . . . . 362.3.7 Complexity Comparison . . . . . . . . . . . . . . . . . . 393 Crowd Counting With Knowledge Distillation . . . . . . . . . . . . 403.1 Knowledge Distillation . . . . . . . . . . . . . . . . . . . . . . . 403.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.2.1 Optimized ShuffleNet . . . . . . . . . . . . . . . . . . . 433.2.2 General Knowledge Distillation Loss Definitions . . . . . 463.2.3 Crowd Counting Loss . . . . . . . . . . . . . . . . . . . 473.2.4 Hierarchic Feature Regulation . . . . . . . . . . . . . . . 483.2.5 Task-Specific Knowledge Distillation . . . . . . . . . . . 503.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.3.1 Training Details . . . . . . . . . . . . . . . . . . . . . . . 513.3.2 Effect of Knowledge Distillation . . . . . . . . . . . . . . 523.3.3 Comparison with State-of-the-art . . . . . . . . . . . . . 54viii4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564.1 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61ixList of TablesTable 2.1 Crowd counting task: Summary of counting accuracy compar-isons on 4 crowd counting benchmark datasets. . . . . . . . . . 32Table 2.2 Summary of localization precision performance comparisonson 4 crowd counting benchmark datasets. The standard APfrom MS-COCO are used. mAP is computed using the aver-age from AP.50 to AP.95 with a stride of 0.05. . . . . . . . . . 33Table 2.3 Summary of localization recall performance comparisons on 4crowd counting benchmark datasets. The standard AR metricsfrom MS-COCO are used. mAR is computed using the averagefrom AR.50 to AR.95 with a stride of 0.05. . . . . . . . . . . . 34Table 2.4 Computation complexity comparisons. . . . . . . . . . . . . . 39Table 3.1 Accuracy and computational complexity comparisons betweenthe proposed ShuffleCount network and the teacher network.The computation complexity metric GMacs is computed using1920Ã1080 input images. . . . . . . . . . . . . . . . . . . . 54Table 3.2 Accuracy and computational complexity comparisons with thepopular large network and state-of-the-art compact counting net-works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54xList of FiguresFigure 1.1 An illustrative example of linear regression . . . . . . . . . . 3Figure 1.2 Sample diagram of a single perceptron . . . . . . . . . . . . . 3Figure 1.3 Firing rate versus input voltage plot in the LIF model . . . . . 5Figure 1.4 An example of a neural network with 2 hidden layers [40] . . 6Figure 1.5 General structure of a convolution neural network [41] . . . . 8Figure 1.6 Challenges in crowd counting . . . . . . . . . . . . . . . . . 10Figure 1.7 multi-column design from MCNN [70] . . . . . . . . . . . . 11Figure 1.8 Architecture of PGCNet [61] . . . . . . . . . . . . . . . . . . 12Figure 1.9 Illustration of density map problems in regression-based crowdcounting methods. (a) the congested scenario; (b) the esti-mated person density by a representative method; (c) the trueperson density map; (d) comparison of the true person distri-bution against the estimated distribution by a representativemethod; (e) an example scenario with large scale variationcaused by different depths of field; (f) the estimated persondensity by a representative method; and (g) the true persondensity map. . . . . . . . . . . . . . . . . . . . . . . . . . . 14Figure 1.10 Roadmap for the crowd counting research in this thesis. . . . . 16Figure 2.1 The flowchart of the proposed Joint Crowd Counting and Lo-calization (JCCL) model. The model is simple, with three pro-posed modules (i.e., Deep Integrated Module (DIM), Scale Adap-tive Module (SAM), Interval Aware Module (IAM)) and a novelPedestrian-Aware Density Map (PADM). . . . . . . . . . . . . 20xiFigure 2.2 Illustration of the effect of the proposed Interval Aware Mod-ule. (a) the congested scene; (b) the ground truth confidencemap generated from the image in (a); (c) Zoomed-in by a fac-tor of 8 and upsampled to the same resolution as the originalimage; (d) the ground truth confidence map generated from theimage in (c). . . . . . . . . . . . . . . . . . . . . . . . . . . 23Figure 2.3 Illustration of differences between the traditional density mapand the proposed PADM. (a) the congested scene with a se-lected dense region; (b) the traditional density map by sum-ming up normalized Gaussian kernels; and (c) the pedestrianaware density map based on the confidence distribution. . . . 25Figure 2.4 localization results on sample images with normal density. . . 37Figure 2.5 More sample localization results on extremely dense crowds. . 38Figure 3.1 Diagram for general knowledge distillation procedure. . . . . 41Figure 3.2 Architectures of ShuffleCount: teacher network DMCount (left),student network Optimized ShuffleNet (right) and knowledgedistillation modules (middle) . . . . . . . . . . . . . . . . . . 44Figure 3.3 Regular Residual Block (left) vs. Inverted Residual Block (right) 45Figure 3.4 Feature loss (left) vs. Counting loss (right) during training time 51Figure 3.5 Density outputs from DMCount and ShuffleCount. . . . . . . 53xiiGlossaryADC Adaptive Dilated ConvolutionANN Artificial Neural NetworkAP Average PrecisionAR Average RecallCNN Convolutional Neural NetworkDIM Deep Integrated ModuleDSC Depthwise Separable ConvolutionFCN Fully Convolutional NetworkIAM Interval Aware ModuleIRB Inverted Residual BlockxiiiJCCL Joint Crowd Counting and LocalizationKD Knowledge DistillationLIF Leaky Integrate-and-FireMAE Mean Absolute ErrorMAC multiplierâaccumulatorMSE Mean Squared ErrorNMS Non-maximum suppressionOSN Optimized ShuffleNetPADM Pedestrian-Aware Density MapSAM Scale Adaptive ModulexivAcknowledgmentsI would like to express my deepest appreciation to my supervisor, Dr. Z. Jane Wangand lab mate Dr. Jianzhe Lin for their patient guidance, enthusiastic encouragementand valuable suggestions of this research work. I want to thank my friends and labmates for their help and company throughout my master study.In addition, I wish to thank my parents and other family members for theirsupport and understanding.xvChapter 1Introduction1.1 Introduction to machine learningRecently, machine learning, especially deep learning, has gained increasing popu-larity and achieved great success in many areas, including medical diagnosis, au-tonomous vehicle, language translation, and so on. As a major branch of ArtificialIntelligence, machine learning is the study where computer algorithms can auto-matically learn and improve themselves through using a large amount of data.Machine learning is mainly used for solving complex problems that humanscan only give general goals without explicit instructions. In general, most machinelearning algorithms solve these problems by modeling data distributions in highdimensional spaces using statistical and probability theory. Among machine learn-ing methods, deep learning using Artificial Neural Network (ANN) is probably themost successful one in recent years, partially due to the evolution in computationalpower. In the following section, we will give a background introduction to artificialneural networks and deep learning concepts.1.1.1 Neural Network and Deep LearningTo introduce the concept of Neural Network, we start by introducing the basicmodels of linear regression and perception, and we then build up neural networksusing ideas from these basic models. Lastly, we introduce one specific type ofneural networks that has a dominant position in the area of computer vision.1Linear regression is the most basic statistical model that models linear relation-ships between the inputs and outputs. Given a dataset of size N {(xi,yi);1 â¤ i â¤N,xi â Rk,yi â Rd}, where xi is k-dimensional vector representing input observa-tion and yi is d-dimensional vector representing output response. The goal of linearregression is to find a linear function that maps from observation to response. Inother words, we want to find a linear function that takes in observations and min-imizes the difference between predicted output and true response. The simplestexample for linear regression is shown below in Figure 1.1 where we have a setof points (x,y) on a 2-D plane, and the target linear function is the straight linepassing through the points.The linear function f : Rkâ Rd can be defined as follows:yi = f (xi) = xiW +b (1.1)where W is a kÃd weight matrix and b is a d-dimensional bias or intercept term.To find the optimal model parameters, people usually minimize the L2 distanceor equivalently the Mean Squared Error (MSE) between the predictions and trueresponses. The objective function that needs to be minimized can be written as:E =12Nâi=1âyiâ xiWâ22 =12âY âXWâ22 (1.2)By finding the global minima of this quadratic objective function, the optimal pa-rameter W has the closed form solutionW = (X>X)â1X>Y (1.3)Besides regression problems producing continuous outputs, classification is an-other branch of problems that machine learning mainly deal with, which gives dis-crete class outputs. Among all classification algorithms, perceptron is the mostbasic but important one that builds up neural networks.Unlike linear regression, where inputs and outputs are real numbers, perceptrontakes in several binary inputs x1,x2, . . . ,xn and produces a single binary output. Thesample diagram of a single artificial neuron called perceptron is shown in Figure2Figure 1.1: An illustrative example of linear regressionFigure 1.2: Sample diagram of a single perceptron31.2. There is a real number weight parameter wi that corresponds to each input xi,measuring the importance of the input. The overall input to the perception neuronis calculated by summing up the weighted sum of binary inputs. And the outputis determined by whether the weighted sum is greater than the neuron thresholdparameter b, which acts similar to the bias term in regression. The formulation ofperception is as follow:output =ï£±ï£²ï£³0, if âi xiW âbâ¤ 01, otherwise (1.4)It is easy to find that perceptron has the same computation formula as linearregression. In practice, the input of perceptron can be extended to real numbersfor factors that have continuous values. The line corresponds to the linear functionf (x) = xW â b can be viewed as a decision boundary in the classification case.When we look back on the linear regression sample plot Figure 1.1, the pointsabove the straight line are classified as positive classes while points below the lineare classified as negative classes.With the basic linear regression and perception model in mind, we can buildartificial neural networks with some modifications.1.1.2 Neural NetworkArtificial neural network (ANN) is inspired by biological neural networks in thebrain, and the intuition behind ANN is to simulate how biological neurons work.Hodgkin-Huxley model [18] uses a 4-D nonlinear system to model action poten-tial in neurons, and later work Leaky Integrate-and-Fire (LIF) model [14] furthersimplified the system only considering the existence of voltage spike due to actionpotential. Without knowing the details of the model, we can get rough ideas fromFigure 1.3. The x-axis is the input voltage to the neuron, and the y-axis correspondsto the output spiking rate. We can see, the activity of neurons is very low when theinput is low, and the activity suddenly goes up while the input voltage increases.This behavior can be simulated via activation function in artificial neural networks.Going back to linear regression and perceptrons, they are useful in solvingmany problems, but they are incapable of modeling complicated non-linear rela-4Figure 1.3: Firing rate versus input voltage plot in the LIF modeltionships. Using the neuron analogy, model inputs x1w1,x2w2, . . . ,xnwn are incom-ing connections to the neuron, and the neuron needs to decide whether to activatebased on incoming signals. In order to stimulate the activation behavior modeledin LIF, non-linear activation functions are added to the weight sum of incominginputs, and thus the output from a single layer of neurons can be expressed as:h1 = Ï(xW1 +b1) (1.5)where h1 is d-dimensional output and Ï is the non-linear activation function ap-plied elementally to each vector element. Commonly used activation include tanh,sigmoid and Rectified Linear Unit (ReLU).sigmoid =11+ eâx(1.6)5Figure 1.4: An example of a neural network with 2 hidden layers [40]ReLU =ï£±ï£²ï£³x if xâ¥ 00 otherwise (1.7)Due to the vanishing gradient problem caused by sigmoid and tanh during op-timization time, it is recommended to use ReLU as the default activation function.Besides, ReLU has the most similar shape as the LIF firing rate versus the inputvoltage plot shown above.To further add complexity to the model, we can stack more than one level ofperceptions with nonlinear activation together, and this results in the well-knownartificial neural network. Like the neurons in our brain with more than one level ofstructure, multi-layer perceptron significantly boosts the representation power ofneural networks. We name these intermediate layers as hidden layers. An examplewith two hidden layers can be found in Figure 1.4.Based on the Universal approximation theorem [9], an ANN with one hiddenlayer that has arbitrary width can be used to approximate any continuous functionf : Rnâ Rd . Using W1 with dimension of nÃm and W2 with dimension of mÃd6neural network defined byfË (x) = Ï(xW1 +b1)W2 +b2 (1.8)can be arbitrarily close to the target function f when the hidden layer side d is arbi-trarily large. The theorem also applies if the number of hidden layers in ANN canbe arbitrary. In practice, researchers have found that deep narrow neural networksgenerally produce better results compared with shallower but wider networks [27].Thus, deeper and deeper networks have appeared recently to address more complexproblems.1.1.3 Convolutional Neural NetworkIn the previous section, the neural networks we discussed are fully connected neu-ral networks where among two adjacent layers, each output dimension dependson each input dimension. Assuming the two layers have dimensions of n and mrespectively, the weight matrix used to map between these two layers has a di-mensionality of nÃm, which can be a substantial number when both n and m arelarge. The overwhelming parameters in fully connected networks cause signifi-cantly slower computation when neural networks go deeper. For computer visiontasks where inputs are image or video data, this phenomenon is further exagger-ated. For example, considering a standard image from ImageNet [12] with thesize of 224Ã224Ã3, the input layer has 224Ã224Ã3 = 150,528 values in total,and any layer that fully connects to such an input layer will introduce millions ofweight parameters. After stacking multiple of this layer, the number of parametersadds up very quickly. Other than increasing computation, overfitting can happendue to an increasing number of parameters. Overfitting is the problem where thenumber of parameters in the model is too large to justify the data. The model fitsa particular set of training data too closely, losing generality for additional unseendata.Obviously, too many connections in the image case are wasteful because dis-tant pixels are not correlated to each other in most cases. Thus Convolutional Neu-ral Network (CNN) [29] is introduced to resolve the problem. Instead of learningpixel-wise correlation, each neuron in convolutional layers only connects to a local7Figure 1.5: General structure of a convolution neural network [41]region of the input by taking advantage of prior knowledge of spatial correlationin images and videos. Filters in convolutional layers are applied systematicallyfrom left to right, top to bottom, to detect specific types of features in the inputswhile preserving spatial information of the input images. The convolutional ker-nels can be reformulated as a single matrix multiplication operation for optimizedcomputation during computation time. Assuming we have an input layer with sizeC1ÃH1ÃW1 and output layer with size C2ÃH2ÃW2, if we use fully connectedlayer here, we will have C1ÃH1ÃW1ÃC2ÃH2ÃW2 +C2ÃH2ÃW2 weight pa-rameters. But if we use a convolutional layer with a kernel size of K, we end upwith C1ÃC2ÃKÃK +C2 parameters, which are significantly less than the fullyconnected one. Conventionally, kernels are applied to each filter-size region onthe input, but recently, dilated convolution [64] and deformable convolution [10]further extends traditional convolutions by giving it more degrees of freedom.Another special layer in CNN is the pooling layer used to downsample images,eliminating useless information while preserving important ones. Max pooling andAverage pooling are the most commonly used ones. The prior take the maximumvalue in kÃk patch, and the latter takes the average of all value in the kÃk patch.In original CNN, fully connected layers are also used as the last few layersfor final decision making after extracting high dimensional features using convolu-tional layers and pooling layers. The overall CNN structure can be found in Figure1.5.More recently, Fully Convolutional Network (FCN) [47] is proposed for taskssuch as semantic segmentation, which requires pixel-level classification. In fully8convolutional networks, transposed convolution layers are added at the end to up-sample extracted features. It is used to preserve output resolution, which is criticalfor semantic segmentation tasks. The fully connected layers are replaced by convo-lutional layers with 1Ã1 kernel to produce the final pixel-level classification result.In this thesis work, we use fully convolutional networks to tackle the problems ofcrowd counting.1.2 Introduction to Crowd CountingWith the wide deployments of surveillance cameras around cities, analysis of crowddata collected from such cameras has been increasingly critical. As an essentialtechnique for crowd data analysis, crowd counting is widely used in video mon-itoring, public security, and city management. Especially in the era of Covid 19,controlling the number of people is essential to preventing the spread of viruses.Crowd counting has been an active research topic. There are several inevitablechallenges that crowd counting models need to face for getting accurate countingresults. Among all of the challenges, we listed few major ones in Figure 1.6. Incrowded scenes, occlusion is the first challenge that significantly increases the dif-ficulty of counting. With increasing people density in input images, the percentageof the occluded body or head increases. Counting models need to infer whether thepartially occluded object is part of the human body with extremely limited featureinformation. Second, a complex background is another big challenge for accuratecounting models. Besides humans, the environment can involve many confusingobjects, including photos on advertisements, reflections from mirrors, and otherthings with a similar appearance as humans. It is critical for crowd counting mod-els to identify these disturbing factors by learning robust features of the humanbody and considering the information from surrounding environments. The nextchallenge is scale variation, and it is the primary problem all existing methods tryto solve. For people close to the camera, their heads can almost conquer up tohalf of the entire image, while heads of people far away from the camera can beas small as only a few pixels. Although heads in these two cases have substantialsize differences, they both contribute 1 to the overall count. Thus, scale varia-tion is exceptionally challenging because crowd counting methods must have large9Figure 1.6: Challenges in crowd countingenough receptive fields for large ones that are close by and detect very fine detailsfor small ones that are far away at the same time. There are also some other minorchallenges, including camera distortion, rotation of viewpoints, and illuminationvariations. These problems can be either fixed by additional camera parametersor simulated by some data augmentation method such as affine transformation andcolor modification in various color spaces. In most cases, variation due to thesechallenges can be learned by models via training through augmented data and thuswill not have a significant impact on model feature extraction.Recently, density estimation algorithms based on deep learning have signifi-cantly improved crowd counting accuracy.1.2.1 Density regression based crowd countingDensity Regression-based models have become the major trend for crowd count-ing. Traditional regression-based crowd counting methods provide the number of10Figure 1.7: multi-column design from MCNN [70]people directly [6, 7, 44], generally ignoring critical spatial information. To bet-ter exploit the correlation between neighboring pixels, recent works [5, 11, 34, 51,56, 66, 68, 70] utilize the density map, which is generated based on point annota-tion from peopleâs heads. Such a density map provides an effective intermediaterepresentation of people inside the scene.From a model structure perspective, most of the existing density regression-based crowd counting methods can be divided into two categories: multi-columnand single column. Methods from each category use their specific ways to dealwith challenges mentioned in the previous section.Many methods, including [4, 37, 45, 51, 60, 62, 66, 70], employ the multi-column approach to address the challenging scale variation problem in crowd count-ing. In multi-column approaches, each branch serves the purpose of capturing in-formation corresponding to different receptive fields. The MCNN [70] is the firstwork that proposed multi-column architecture, shown in Figure 1.7. It utilizesthe 3-column architecture to employ different sets of kernel sizes (large, medium,small) on each branch and assemble results focusing on different receptive fields to-gether at the end. Instead of fusing, the Switch-CNN [45] trains a few independentdensity regressors and uses a well-trained classification sub-network to determinethe most suitable regressor for the given input image. Both MCNN and Switch-11Figure 1.8: Architecture of PGCNet [61]CNN tackle the problem by assembling several similarly structured sub-networkstargeting different receptive fields. Different from these two, CP-CNN [51] usesdifferent branches to encode local context, global context, and density information.Thus the model can yield a better quality density map. Recent McML [8] uses astatistical network to estimate the mutual information among different branches. Italso proposes a mutual learning schema that alternatively optimizes each columnwhile keeping the other columns fixed on each mini-batch data. With help from theadditional statistical network, the accuracy of previous multi-column models suchas MCNN is further boosted.The other category of methods uses single column architecture where one sin-gle deeper network is used instead of multiple shallower networks. Since layerson different depths can encode information of receptive fields with different sizes,the SaNet [5] combines feature maps from different backbone layers to achieve thesame purpose of multi-column design. CSRNet [31] has a single column designand uses the multiple dilated convolution layers to enlarge the receptive field whilemaintaining the resolution of feature layers. As a further improvement, the AD-SCNet [2] utilizes adaptive dilated convolution, allowing different dilation rates atdifferent locations. Such a design can further reduce the problem of variance ofscale.In terms of methodology, perspective maps and attention are the most effectiveways to reduce errors due to scale variance. Many works handle scale variation byincorporating additional perspective map information since scale variation directlyresults from perspective distortion. [7, 13, 48, 61, 63] take advantage of this prior12knowledge and train a dedicated network that estimates perspective maps from in-put images. [13, 48, 61] fuse estimated perspective map with features from themain crowd counting network to help mitigate the impact of scale variance in dif-ferent depth. As shown in Figure 1.8, PGCNet uses estimated perspective mapsto guide feature extraction in the crowd counting branch. [63] removes the effectof perspective distortion at the very beginning by reconstructing the input imageusing the estimated perspective map.Some other researchers tackle the scale variation issue using attention zoom-ing. The RAZNet [33] has a specific region proposal branch to identify extremelyhighly dense locations and recurrently zoom in to proposed regions. This attentionoperation dramatically reduces the variation on scales. [24] takes one step fur-ther and uses a density attention network to build attention masks and an attentionscaling network to estimate corresponding density maps.1.3 Thesis ObjectiveAccurately estimating the number of people in a crowd scene is a challenging andpractically important task with increasing demands in controlling people densityin some specific areas, especially during the era of Covid-19. Deep learning-basedcrowd counting models have achieved great success and are very beneficial in pub-lic security and city management applications.In existing crowd counting methods, the dominant approach represents peo-ple in crowd scenes through Gaussian blobs, which can be integrated to obtain thefinal count. Although the number of people in crowd scenes can be accuratelyestimated using the Gaussian blob modeling method, the count-only informationprovided by these deep learning models is indeed very limited for crowd analysis.More detailed crowd analysis tasks (e.g., including pedestrian density estimationand crowd anomaly detection) require more detailed information on the crowd.The density map calculated from current crowd counting methods could largelydeviate from the true pedestrian distribution. An example is given in Figure 1.9. InFigure 1.9 (a)-(d), we illustrate that the Gaussian blobs for crowd counting overlapin dense regions, meaning that the density value of one pixel can be accumulatedfrom its neighboring Gaussian blobs. In Figure 1.9 (e)-(g), we show the scale vari-13Figure 1.9: Illustration of density map problems in regression-based crowdcounting methods. (a) the congested scenario; (b) the estimated persondensity by a representative method; (c) the true person density map; (d)comparison of the true person distribution against the estimated distri-bution by a representative method; (e) an example scenario with largescale variation caused by different depths of field; (f) the estimated per-son density by a representative method; and (g) the true person densitymap.14ation problem for such a density map. Since people with different depths of field inthe picture would have different density distributions, existing works cannot givea consistent density map estimation even for sub-regions with the same number ofpeople. To further increase the explainability of crowd estimation, it is desirablethat crowd counting models can provide both counting and localization predictionsimultaneously with minimal increment in model complexity. The existing workRAZNet [33] specifically designs a localization branch for estimating people lo-cations, but the redundant localization branch greatly increases the computationaltime of the model. Some other methods require box-level annotation of peopleâsheads, but this could be infeasible in most cases of having over one thousand peoplein the same crowded scene.It is essential to develop a method that can address the above challenges, wherecounting results are based on precise localization while the network architectureis kept simple. In the first half of this thesis, we propose a simple framework totackle the joint crowd counting and localization problem efficiently. The proposedframework includes three primary modules: Deep Integrated Module, Scale Adap-tive Module, and Interval Aware Module. Deep Integrated Module can get a betterunderstanding of the contextual information of the input scene. At the same time,Scale Adaptive Module and Interval Aware Module can deal with locally denseareas with significant head overlaps. In addition to designing a smartly simplecrowd counting framework, we re-model the target crowd counting problem usinga unique pedestrian-aware density map. Based on this map, each person can beboth counted and accurately localized. Compared with the traditional density map,the proposed Pedestrian-Aware Density Map is more interpretable and can betterdeal with dense local areas.While dealing with the first challenge, we further consider the practical require-ments of crowd counting methods. The majority of crowd counting applicationsshould be real-time and likely deployed on edge devices such as surveillance cam-eras. However, existing well-performed deep-learning-based crowd counting net-works are mainly based on cumbersome classification backbones such as VGG16and Resnet50. Although they are accurate, these state-of-the-art crowd countingmethods cannot be used directly to serve in edge devices where computation re-sources are extremely limited. On the other hand, some shallower networks have15Figure 1.10: Roadmap for the crowd counting research in this thesis.relatively low computational costs, but their counting accuracy is not satisfying inpractice.Finding a good trade-off between counting accuracy and computational effi-ciency is another challenging problem we plan to address. In the second halfof this thesis, we try to create a compact yet accurate model for solving crowdcounting problems. To achieve this goal, we thoroughly investigated KnowledgeDistillation (KD) for crowd counting. We proposed a hierarchic feature regulationmethod to selectively transfer knowledge from the cumbersome teacher network tothe small target (student) network. Besides, we designed an optimized ShuffleNetto strike the best trade-off between accuracy and computational efficiency.1.4 Thesis OutlineIn Chapter 2 (Crowd Counting and Localization in Images), we introduce a crowdcounting model that can provide both the accurate counting result and precise lo-16calization coordinate of human position. Experiments are conducted on variousbenchmark datasets. Both quantitative and qualitative results are reported and dis-cussed in this chapter.In Chapter 3 (Crowd Counting using Knowledge Distillation), we propose a ac-curate and computationally efficient crowd counting model trained through knowl-edge distillation, aiming for widening the deployment of crowd counting modelson mobile devices. Model accuracy and complexity are assessed using one selectedbenchmark dataset.In Chapter 4, we summarize the contributions of this thesis work and discussfuture work suggestions.17Chapter 2Crowd Counting andLocalization in ImagesHuman localization should never be a task that is independent of crowd counting.Indeed, these two tasks are tightly coupled and can complement each other. Inother words, localization models can get extra density and scale information withthe help of crowd counting results. Meanwhile, the crowd counting model can nat-urally give the counting result if it can correctly localize all the people in crowdedscenes. In this section, we demonstrate our work that jointly handles crowd count-ing and localization problems.2.1 Human LocalizationHuman localization is a practical task in our everyday life. Traditional detectionframeworks, such as Faster R-CNN [42], can be applied but often have limitedaccuracy on crowd scenes. Unlike conventional detection tasks where there are fewobjects in each scene to be detected, the localization task we are trying to solve is oncrowd scenes that consist of hundreds of people. A representative work for humanlocalization is Tinyface [20] which is designed to detect tiny objects in the image.It has better detection accuracy on crowded scenes than the traditional detectionframeworks. PSDDN [35] tries to build the box level ground truth based on pointannotation of the crowd data and apply the Faster R-CNN with minimal change.18The DISAM [26] combines R-CNN with the scale map to generate scale-awareproposals and then classifies each proposal with the fine-tuned head classifier. [1]uses a similar approach to build the ground truth by assigning each pixel one of theclasses with predefined box sizes and then does pixel-level classification. The non-maximum suppression is then used to eliminate overlappings. However, humanlocalization methods cannot provide good counting accuracy in dense scenes whencompared with regression-based counting methods.A recent method RAZNet [33] combines the crowd counting with the local-ization framework. In RAZNet, different from the box-level human localizationframeworks mentioned above, the localization branch uses point-level annotationto directly generate the ground truth density map. It detects peopleâs heads byfinding high-confidence peaks during the estimation. However, their joint crowdcounting and localization tasks are completed separately by two branches. Theframework is not sufficiently efficient in terms of model complexity and computa-tional cost. The accuracy for both tasks could be improved.In this thesis work, we, for the first time, propose a highly efficient end-to-endframework for joint crowd counting and localization. The computational efficiencyand accuracy of the proposed framework are verified in the experimental part.2.2 Model Architecture2.2.1 End-to-End Framework OverviewIn the proposed simple end-to-end Joint Crowd Counting and Localization (JCCL)framework, we aim to simultaneously find the number of people and precise loca-tions of peopleâs heads. The framework includes three major modules and a noveldensity map, named the Pedestrian-Aware Density Map (PADM). We will elaborateon each component of the framework in detail.We first introduce the Deep Integrated Module (DIM) where information fromdifferent scales is fused to extract rich features efficiently. We use the Scale Adap-tive Module (SAM) to handle the scale variance of heads in complex crowd scenes.An Interval Aware Module (IAM) is proposed to mitigate the problem where peopleare too close to each other due to limited image resolution.19Figure 2.1: The flowchart of the proposed JCCL model. The model is simple,with three proposed modules (i.e., DIM, SAM, IAM) and a novel PADM.In addition to these modules, we re-formulate the problem and convert thetraditional heatmap to the PADM which provides better interpretability and moretolerance to inconsistent human labeling. Focalloss [32] is also used to address theforeground-background class imbalance problem where background pixels domi-nate. Last but not least, a simple regression decoder is added on top of the PADMto further improve the counting accuracy in extremely dense regions.The overall structure of our proposed model is illustrated in Figure 2.1. It onlyhas a single branch where modules are sequentially connected. The input of ourmodel is a set of crowd scenes represented by I = {in}Nn=1, where N is the numberof images in the dataset; and the predicted head locations are represented withL = {{(xmn ,ymn )}cnm=1}Nn=1, where (x,y) represents head coordinate and cn representsthe counting number on the nth input image in.2.2.2 Deep Integrated ModuleIn our proposed framework, to better explore the contextual information of theinput scene from different levels, DIM integrates features from layers with differ-ent depth, scale, and resolutions together. The module is mainly used for feature20extraction, and it consists of upsampling and downsampling parts. For the upsam-pling part, unlike the multi-column approaches [51, 66, 70] which acquire differentscale features through specified design branches and fuse at the end, the DIM ef-ficiently integrates the multi-scale feature extraction and fusion process [65] in ahierarchical manner. The feature maps have the size of 1/2,1/4,1/8,1/16,1/32of the original input. The way that DIM fuse these features is that it first upsamplethe 1/32 feature map by a factor of 2, one projection layer is used to project theupsampled feature into the same latent space of the original 1/16 feature map, anda new 1/16 feature layer is formed by fusing these two. The same operations areagain applied on both the original 1/16 feature layer and the fused one, resultingin 3 feature layers with the size of 1/8. By repeating the same operations, we endup with four feature maps with the size 1/4. One of these contains rich multi-scale features integrated hierarchically from all previous feature maps, and thus itis used for prediction. For the upsampling part, we adopted extra skip connectionsbetween low-level features and high-level features to integrate different layers offeatures, similar to [72]. Such skip connections allow updating the gradients toflow towards each level more quickly, resulting in efficient learning.Compared with other commonly used backbones for feature extraction, suchas VGG16 [49] and Resnet50 [16], the proposed module is a more compact choicewithout sacrificing any feature extraction power. Such a module can allow us toinput much higher resolution images under the same hardware configuration. It iscritical in crowd counting as higher resolution input images can avoid center col-lisions in the output by separating heads well in dense scenes. Denoting the inputimage with height H, and width W, the resulting feature map from the DIM has sizeCÃH f eatureÃWf eature where H f eature = H/4 and Wf eature = W/4. Further upsam-pling feature maps to higher resolution only bring limited accuracy improvement.Meanwhile, it will result in a significant increase in computational cost. Based onobservations from multiple crowd counting datasets, almost all head centers aremore than 4 pixels away from each other. Using a feature map with a stride of 4 issufficient to avoid center collision problems.212.2.3 Scale Adaptive ModuleScale variation is a major concern that all crowd counting models encounter. It isthe primary challenge that all models try to address in the first step. The size ofpeopleâs heads can vary from a few pixels for a faraway person to almost half of theentire image for a nearby person. The size of the receptive field is crucial becauseof this large scale variation. Besides integrate features from different depths usingDIM module, we propose the SAM to address this concern further. Specifically,SAM is used to control pooling and upsampling in the network. This module ismainly based on the Adaptive Dilated Convolution (ADC) [2]. Using a 3Ã3 kernelas an example, defining the grid G as:G = {(â1,â1),(â1,0), . . . ,(0,1),(1,1)} (2.1)ADC can be formulated as follows: for each output pixel location p0, the outputfeature Fo(p0) is calculated as,Fo(p0) = âpnâGFi(p0 + pnà dË)w(pn) (2.2)where pn enumerates the grid G, w(pn) denotes the corresponding kernel param-eter and dË is the adaptive dilation rate learnt through training. Different from thetraditional dilated convolution, the ADC allows dilation rate to be adaptive andlocation specific receptive fields to be learnt.To give more levels of freedom compared with the ADC which only makesthe dilation rate adaptive, we further replace the dilated convolution in ADC withthe deformable convolution [10] in the upsampling stage and our proposed SAM isformulated as:Fo(p0) = âpnâGFi(p0 + pn +âpn)w(pn) (2.3)The dilation rate dË shared by the entire kernel is replaced by the grid-specific offsetâpn . The grid-specific offset âpn is adjusted and optimized through training. Withsuch a deformable convolution, the kernel can deform to better fit any head andbody shapes (including unsymmetrical ones due to occlusion in dense scenes). Theadded flexibility greatly increases the effectiveness of learning. Previously com-22Figure 2.2: Illustration of the effect of the proposed Interval Aware Module.(a) the congested scene; (b) the ground truth confidence map generatedfrom the image in (a); (c) Zoomed-in by a factor of 8 and upsampledto the same resolution as the original image; (d) the ground truth confi-dence map generated from the image in (c).plex patterns in large regions requiring multiple levels of regular square-shapedconvolution kernels to extract can be obtained by one single deformable kernel.Small spatial deviations caused by the grid-level offset can be negligible since thedeviation in ground truth labeling is much greater. Point labels have different po-sitions on peopleâs heads due to the impreciseness of human labeling. In the caseswhere symmetrical features are optimal, the deformable convolution in SAM candegenerate to the regular dilated convolution with the same offset value sharedacross the entire kernel.We also want to point out that deformable convolution has been widely usedand achieved great success in other computer vision tasks such as object detectionand semantic segmentation. Here we take advantage of its flexibility in crowdcounting tasks and use it in our proposed SAM.232.2.4 Interval Aware ModuleA very practical problem in crowd counting is overlapping. When two targets aretoo close, it is not easy to localize/count the two targets, not only because of limitedfeatures but also because the target head centers are too close to each other to beidentified. We propose a simple yet effective IAM to solve this problem. IAMis an operation that brings a considerable accuracy boost without adding an extracomponent to the model itself. The general idea of IAM can be illustrated in Figure2.2, which is to enlarge the resolution of input images. Inside of red region on thetop left corner of the image, many heads are close to each other. As we can see in(b), when we generate a ground truth density map from such region, the gaussiankernel centered at annotation has obvious overlap since all heads in that regionare only a few pixels away. With the downsample operations in the model, theheads center on the top left in the predicted density map will be closer or collide.However, if we crop the red region and zooming in by a factor of 8, we achieveimage (c) with the same resolution as the original image (a). The ground truthdensity (d) is generated from zoomed-in (c). We can see that distance between topleft head centers are now eight times larger than the original ground truth densitymap, and the Gaussian kernels centered at those head annotation separates verywell. Using (c) as input into crowd counting models can significantly simplify thetask and boost counting accuracy in this sub-region. As mentioned in the previoussection, it is much easier to get accurate counting for images with higher resolutionwhen the scenes are extremely dense. Higher resolution means easier separationfor head features with a much more clear boundary in-between. Most importantly,a higher resolution input can dramatically reduce center collisions on both groundtruth and predicted heatmaps.To mitigate the center collision problem, IAM is an operation to zoom in on theoriginal image according to the estimated count number. To be more specific, weinitialize the prediction on the original image first. If the estimated count is greaterthan a predetermined threshold T (we set T as 200 in our experiments), which isclose to the average number of people in each dataset based on validation, the im-age will be divided along the long edge into two parts, and the divided half imagewill be upsampled to the original resolution. A new prediction, summing two sub-24Figure 2.3: Illustration of differences between the traditional density map andthe proposed PADM. (a) the congested scene with a selected dense re-gion; (b) the traditional density map by summing up normalized Gaus-sian kernels; and (c) the pedestrian aware density map based on theconfidence distribution.image counts, replaces the former estimation on the original image. For example,assuming the predicted count of one input image is 300, which is considered denseand will trigger the dividing operations. After passing upsampled left and right por-tion of the input image into our model, we get counts of 150 and 160 respectively.The final counting result for this image will be 150 + 160 = 310 since we trustmore on the result from zoomed due to the less likelihood of miss counting causedby center collisions. We recursively apply this dividing and zooming process untila certain recursion depth is reached, which is set as 2-3 empirically. More zoomingoperations will cause degradation in the counting accuracy due to reduced recep-tive field and loss of contextual information. We also want to point out that thetraining images are augmented with random cropping and zooming during modeltraining to better deal with different zoomed sub-images in the testing phase.2.2.5 Pedestrian-Aware Density MapThe public crowd counting datasets provide the count numbers of people and anno-tations for visible human heads. Such annotation information allows us to generatevarious forms of ground truth that could better suit our problems. Traditional den-25sity map is one type of annotation, where normalized Gaussian kernel is used togenerate Gaussian blobs centered at each annotation. To be more specific, eachdensity map value M(p0) is calculated as follow:M1(p0) =Nâi=1N (p0|u = pi,Ï = 0.2b¯i) (2.4)whereN means the Gaussian distribution, pi enumerates N ground truth head lo-cations, and b¯i is its average distance from the nearest four neighbors. All valuesin the density map are integrated to achieve the final count. As mentioned in Sec-tion 1.3, such a density map is uninterpretable because of overlapping and scalevariation.Unlike the traditional density map, We want to model the ground truth differ-ently by our proposed PADM. PADM is generally a confidence heatmap, in whicheach pixel value represents the likelihood of a pixel being the center of the head.With unnormalized Gaussians and adaptive kernel variance, each confidence mapvalue can be formulated as:M2(p0) = max{ZiN (p0|u = pi,Ï = 0.2b¯i) : i = 1 . . .N} (2.5)where Zi is the normalization factor for the pi centered Gaussian kernel. In thisequation, we take the element-wise maximum confidence value as the pixel valuewhen overlapping occurs. By modeling the ground truth with PADM, we can trans-form the crowd counting task to a pixel-level classification task and achieve betterinterpretability, where the meaning of every single value on the ground truth mapis self-contained. The annotated point is at the peak of Gaussian and has the confi-dence of 1. The confidence gradually decreases with increasing distance to anno-tated head centers, reducing the negative impact from inconsistent labeling. FromFigure 3.5, we can find the difference between the traditional density map and ourproposed PADM. On traditional density map (b), normalized Gaussian kernels areused, and density values are the sum of multiple values when gaussian blobs over-lap with each other. On the zoomed-in view of the center image, there is a hotspot in the middle when two annotated heads are too close, and the middle of twoheads get the highest value on the density map without actually being a person. It26is hard for people to interpret the peak values when looking at this density map.By contrast, maximum confidence is taken for each pixel on PADM. Better separa-tions of Gaussian blobs are achieved from a visual perspective. Most importantly,lower confidence means further away from annotation, and peak values are foundon the center of heads. Thus, with the proposed PADM, we can also obtain thelocalization.2.2.6 Model LossWith PADM, Crowd Counting now can be modeled as a pixel-wise binary clas-sification problem, that is, to predict the likelihood of a pixel being the center ofthe head. As introduced in the previous section, pixels that are close to the pointannotation are assigned high confidence values. Since we are using adaptive kernelsize, a larger head will be represented by a smoother Gaussian. If only the peakvalue one is viewed as a positive class, human label inconsistency will be amplifiedin this case where a large area close to the center can be view as positive. Thus weset 0.95 as a positive class threshold to relax the dependence of perfect annotation.Class imbalance issues arise with this configuration where the majority of pix-els are negative since there are at most 5% positive pixels even on the most densecrowd image. To mitigate the heavily imbalance issue, we take advantage of focalloss.L =â 1Nâxyï£±ï£²ï£³(1â YËxy)Î± log(YËxy), if Y (xy)>= 0.95(YËxy)Î± log(1â YËxy), otherwise (2.6)Where N is the total number of pixels, x and y enumerate all image locations, Yxyis the ground truth confidence value, and YËxy is the predicted confidence value. Byadjusting hyper-parameter α , we can adjust the weight on easy examples, thus re-solving the imbalance problem. Different from regular binary classification, whereclass labels are 0 or 1, we have continuous confidence values for both classes. Espe-cially on the negative class side, there are many pixels with high confidence scoresthat are greater than 0.5 but less than the 0.95 threshold. To further improve thefocal loss on this specific case, we add another coefficient (1âYxy)Î² to the nega-tive class loss to reduce the penalty on those not so wrong predictions. β is anotherhyper-parameter introduced here which controls weight on those high confidence27value negative pixels. Finally, we have our modified focal loss as follow:L =â 1Nâxyï£±ï£²ï£³(1â YËxy)Î± log(YËxy), if Y (xy)>= 0.95(1âYxy)Î² (YËxy)Î± log(1â YËxy), otherwise (2.7)2.2.7 Non-maximum suppressionTo compute the final count and head location, we need to find high confidencepeaks from the heatmap. The Non-maximum suppression (NMS) can be efficientlyachieved via 3Ã 3 max pooling. Then pixel-wise comparisons are then used be-tween the predicted confidence map before and after NMS operation to find peaks.To further remove noisy peaks, we filter out all peaks with a confidence lower thana certain threshold. The threshold is empirically set to 0.35, determined throughmultiple validation experiments. The higher the threshold number is, the fewerfalse positives will be kept, but at the same time, more true negatives can be intro-duced. After applying the threshold, we convert the predicted confidence heatmapinto a 0 1 binary map where each 1 represents one predicted center of a personâshead. The total count will be the number of ones on the predicted binary map.Since the predicted map has a resolution of H/4ÃW/4, the precise location of thepredicted heads on the original image can be computed easily via simple transfor-mation.2.2.8 Confidence map RegressionIn the previous section, we achieved the target counting number and location es-timation by identifying high confidence peaks on predicted PADM. But it can beinaccurate when the constant threshold value is not suitable in some specific sam-ples. For example, in extreme cases, a dense area has lower than average predictionconfidence, and the constant threshold will filter out too many true heads. To solvethis problem, we decide to view the PADM as a highly representative encoding ofthe original image and use a separate simple network to decode the number ofpeople from the confidence heatmap.Again, the approach is straightforward, where we decide to have a regression28model that directly predicts the global count from the predicted PADM. UsingF : R2 â R to represent the regression network and YË as predicted PADM, wehave the regression loss as followL =1Nâ(CâF(YË ))2 (2.8)where C is the number of people on the image. The decoding network has a veryconcise structure where only two ResNet basic blocks, one max-pooling layer, andfew fully connected layers are used. Since the PADM is already a representativeencoding of the original crowd image, the additional regression network does nothave to have complicated feature extraction layers. Such a simple regression sub-network is used to better estimate the number of people, especially on density mapswith low confidence crowding regions. In other words, besides counting high con-fidence peaks and provide precise locations as NMS and thresholding method fromthe previous section, the regression network is used as a supplementary to provideaccurate counting only prediction in the case of images with low confidence crowdregions.2.3 ExperimentsIn this section, we test on challenging crowd counting benchmark datasets, includ-ing ShanghaiTech A, Shanghai Tech B [71], UCF-QNRF [23], and JHU-CROWD[53]. Both qualitative and quantitative comparisons with state-of-the-art methodsare conducted.2.3.1 DatasetShanghaiTech [71]: The ShanghaiTech dataset consists of 1198 images with a totalnumber of 330,165 people annotated. It splits into two parts. Part A consistsof 482 random images, with an average people density of 501.4 each image. Incomparison, Part B has 716 images with a much sparser average density of 123.6per image.UCF-QNRF [23]: UCF-QNRF is one of the most challenging datasets in crowdcounting. It consists of 1535 high-quality images with more than 1 million annota-29tions. The dataset is highly diverse with different viewpoints, densities, and light-ing conditions. The number of people varies from 49 to 12865 on each image. Theaverage people density in this dataset is 815.JHU-CROWD [53]: JHU-CROWD is the most recent crowd counting dataset,and it is the largest dataset we tested on, in which 4250 high-resolution images with1.11 million annotations are provided. It contains images with diverse scenariosand environmental conditions. Many challenging conditions, including illumina-tion variation and distractors, are included in this dataset.2.3.2 Implementation DetailsEvaluation ProtocolFor the crowd counting task, the goal is to minimize the difference between thepredicted and ground truth counts. We employ the standard Mean Absolute Er-ror (MAE) and Mean Squared Error (MSE) as accuracy metrics. Suppose the test-ing dataset has N images, for the ith image, the ground truth count is Ci and thepredicted count is CËi. The MAE and MSE are as follows:MAE =1NNâi=1|CËiâCi| (2.9)MSE =â1NNâi=1(CËiâCi)2 (2.10)Note MAE and MSE can vary largely for dataset with different people density.Models generally have higher MAE and MSE on higher density datasets whichconsist of more challenging scenarios.For the localization task, we adopted the evaluation method used in [33]. Thegeneral routine is as follows. We first sort the predicted head locations by confi-dence in the predicted PADM from high to low. For each predicted point, we findthe nearest unmatched ground truth head annotation and match the prediction withit if their distance is under a threshold, which is defined by the Gaussian standarddeviation Ï and precision/recall threshold. True positive is assigned to matchedprediction and false positive otherwise. Finally, the remaining unmatched true30heads are considered as false negatives.By converting the localization problem to a keypoint matching problem, wecan easily compute standard Average Precision (AP) and Average Recall (AR) asperformance metrics. For each image, with the same notation, we have:AP =1NNâi=1T PiCËi(2.11)AR =1NNâi=1T PiCi(2.12)where T Pi represent number of true positives on ith image.Data AugmentationSince most crowd counting datasets are relatively small, we applied various dataaugmentation techniques to improve the accuracy and prevent overfitting. First, tosolve the problem of scale variance and maintain the accuracy on zoom-in images,we perform random cropping and resizing to augment the training set with differentscales. Second, to reduce the effect of lighting variance, we augment images onthe HSV color space to better mimic the brightness and color changes in differentenvironments and camera configurations. We apply affine transformations to enrichthe training dataset further, thus prevent overfitting. Unlike most previous methodsusing the pre-generated density map as ground truth, we generate the PADM onthe fly efficiently from point annotations during sampling time, and it dramaticallyincreases our flexibility in data augmentation.2.3.3 Training DetailsWe initialize the backbone network with the pre-trained DIM from ImageNet, andthe rest layers are initialized using Gaussian distribution with zero mean and stan-dard deviation of 0.01. We use Adam to optimize our models. The algorithm isimplemented using PyTorch and experimented on a single GTX 2080Ti GPU ma-chine. The learning rate is initialized to 1e-5 and halved after the validation lossgets stuck for an extended period.31MethodShanghaiTech A ShanghaiTech B UCF-QNRF JHU-CROWDMAE MSE MAE MSE MAE MSE MAE MSEZhang et al[67] 181.8 277.7 32.0 49.8 - - - -MCNN[70] 110.2 173.2 26.4 41.3 277 426 109 291Huang et al[21] - - 20.2 35.6 - - - -Switch-CNN[45] 90.4 135.0 12.6 33.4 228 445 99 255CP-CNN[51] 73.6 106.4 20.1 30.1 - - 94 239CSRNet [31] 68.2 115.0 10.6 16.0 - - 78 243SFCN[58] 64.8 107.5 7.6 13.0 102 171 - -RAZ localization+[33] 71.6 120.1 9.9 15.6 118 198 - -RAZ fusion[33] 65.1 106.7 8.4 14.1 116 195 - -LSC-CNN[1] 66.4 117.0 8.1 12.7 120 218 - -JCCL 67.9 122.8 7.1 11.4 109 213 71 279JCCL+Reg 63.9 99.7 8.2 14.8 100 169 70 273Table 2.1: Crowd counting task: Summary of counting accuracy comparisonson 4 crowd counting benchmark datasets.2.3.4 Evaluation on CountingWe first compare the proposed model with the state-of-the-art ones for the count-ing task. Table 2.1 reports the quantitative results. Among all comparison meth-ods, RAZ-localization+ [33] and LSC-CNN [1] are the two state-of-the-art jointworks on crowd counting and head localization. The proposed JCCL is the resultby counting the number of high-intensity peaks on predicted PADM. And the pro-posed âJCCL+Regâ is the result obtained with the trained regression decoder usingthe predicted PADM as the input.Based on the results, we can see that our proposed classification-based methodoutperforms other methods in most cases. Especially on Shanghai B datasets, JCCLyields the most accurate counting result. The extended regression decoding net-work is a supplementary method on top of our classification-based method, andâJCCL+Regâ indeed provides accuracy improvements in dense cases. For Shang-hai A and UCF-QNRF datasets, âJCCL+Regâ dramatically improves the countingaccuracy, which means that the decoding regression network can accurately esti-mate the number of counts on extremely dense parts of images where the classifi-cation network fails to give clear decision because of inference from surroundingobjects. Also, on the Shanghai B dataset, compared with JCCL, âJCCL+Regâ does32Dataset Method Ï AP.50 AP.75 mAPShanghaiTech ARAZ localization+ [33]40 0.745 0.699 0.69120 0.667 0.601 0.5845 0.360 0.205 0.197JCCL40 0.838 0.838 0.83620 0.836 0.829 0.8135 0.668 0.431 0.427ShanghaiTech BRAZ localization+ [33]40 0.753 0.716 0.71020 0.687 0.649 0.6345 0.456 0.281 0.280JCCL40 0.905 0.904 0.90120 0.903 0.890 0.8685 0.671 0.418 0.420UCF-QNRFRAZ localization+ [33]40 0.573 0.481 0.46220 0.414 0.287 0.2845 0.079 0.031 0.037JCCL40 0.829 0.795 0.77520 0.759 0.660 0.6355 0.332 0.188 0.197JHU-CROWD JCCL40 0.851 0.835 0.82620 0.820 0.779 0.7595 0.553 0.370 0.373Table 2.2: Summary of localization precision performance comparisons on 4crowd counting benchmark datasets. The standard AP from MS-COCOare used. mAP is computed using the average from AP.50 to AP.95 witha stride of 0.05.not improve the counting accuracy. We believe it is because regression does nothave the advantage of sparse confidence maps where the predicted results alreadyhave very high confidence and can be accurately counted without any estimation.It is worth noting that, although our classification-based method provides bet-ter MAE results than comparison methods, its MSE is only compatible with thosemethods. The reason is that the classification-based method works better in gen-eral cases where people are sparse or reasonably dense. Still, it cannot give correctpredictions in extreme scenes (e.g., more than a few thousand people). In such33Dataset Method Ï AR.50 AR.75 mARShanghaiTech ARAZ localization+ [33]40 0.847 0.820 0.81220 0.799 0.753 0.7415 0.409 0.579 0.422JCCL40 0.827 0.827 0.82520 0.825 0.819 0.8045 0.662 0.423 0.420ShanghaiTech BRAZ localization+ [33]40 0.857 0.834 0.83120 0.820 0.794 0.7845 0.661 0.516 0.492JCCL40 0.883 0.882 0.87920 0.881 0.868 0.8465 0.651 0.403 0.406UCF-QNRFRAZ localization+ [33]40 0.719 0.652 0.63620 0.602 0.497 0.4835 0.242 0.143 0.148JCCL40 0.753 0.722 0.70320 0.690 0.599 0.5765 0.297 0.165 0.174JHU-CROWD JCCL40 0.682 0.670 0.66320 0.659 0.626 0.6095 0.439 0.291 0.295Table 2.3: Summary of localization recall performance comparisons on 4crowd counting benchmark datasets. The standard AR metrics from MS-COCO are used. mAR is computed using the average from AR.50 toAR.95 with a stride of 0.05.extreme cases, people are too close to each other. Thus interference from sur-roundings will be more significant, and it is tough to find the exact location of eachperson confidently, resulting in potential under-counting. This type of inaccuracyis expected due to the nature of our classification-based method, and it will causea larger squared error.342.3.5 Evaluation on LocalizationFor localization evaluation, there are only a few crowd counting methods in theliterature that can also provide precise localization results. Based on [59], RAZ-localization+ (RAZNet) [33] is the best crowd counting method that also providesprecise localization results, so we adopt the same evaluation protocol as in [33] andcompare our localization accuracy with it on three benchmark datasets. Table 2.2and Table 2.3 report quantitative localization results. For each ground truth headlocation, an unnormalized Gaussian function with Ï is used. The predicted head isonly considered a match with that specific ground truth location when its evaluationfrom the Gaussian function is greater than a certain threshold (e.g., 0.50 for AP.50and AR.50, 0.75 for AP.75 and AR.75). With Ï and the confidence threshold, wecan specify a maximum acceptable distance for matching. The configuration onthe table covers the distance from 3.8 pixels to 47.1 pixels. Thus the table showsthe performances at different accuracy levels.The proposed JCCL significantly outperforms the RAZNet [33] at almost allaccuracy levels based on the table. JCCL can maintain very high precision on allthree datasets when the equivalent acceptance distance is over 20 pixels. There isalmost no precision degradation on JCCL when Ï drops from 40 to 20, indicatingthat JCCL has very accurate location predictions, i.e., most predicted heads arewithin the range from the ground truth ones. The UCF-QNRF dataset is especiallychallenging for localization due to its higher resolution. Good precision can onlybe achieved when the predicted head locations are very close to the ground truthones from the visual perspective. The proposed method performs very well in suchcases, with over 60% mAP for the configuration of Ï = 20.For recalls, the proposed method also outperforms the comparison method inmost cases. Recall can be viewed as a more strict evaluation of the counting resultsince it is the percentage of ground truth ones matched in predictions. From the ta-ble, we can see that the recall only drops slightly from AR.50 to AR.75, indicatingthe highly precise localization prediction from the proposed model.Note that the localization precision and recall for both the proposed JCCL andthe comparison method degrades significantly when Ï is 5. In this case, the equiv-alent acceptance distance is from 3.7 pixels to 5.7 pixels, which can be smaller35than the distance between two eyes in high-resolution images. Inconsistent humanlabeling has a significant impact on this accuracy level. Especially on the high-resolution UCF-QNRF dataset, such an effect is further exaggerated.Overall, our proposed method takes the precise localization of dense crowds tothe next level with high precision and good recall.2.3.6 Qualitative ResultsFor qualitative results, we illustrate the prediction results from our proposed methodon various crowds and perspectives. Figure 2.4 and Figure 2.5 show several ex-amples of our results. Input colored images are converted to grayscale for bettervisualization, and the predicted head centers are annotated using red dots.Images on the top two rows of Figure 2.4 are from regular street surveillancecamera views on the Shanghai B dataset. In these images, people have significantsize variation from front to back. Unlike traditional regression methods, whichspread the weight over the large head and then sum all density values to get theestimated counting, our method gives a precise prediction on the center of the head,as shown in the figure. The prediction of our method is relatively consistent, beingrobust to the effect of head sizes. The bottom two rows of Figure 2.4 are fromShanghai A dataset. Among these images, the camera viewing angles are verydifferent, but our model can localize all the heads precisely.For the images in Figure 2.5, we show some sample predictions from ShanghaiA, UCF-QNRF, and JHU-CROWD, which contain more dense crowds. The densityof these images is much greater than previous images. More challenges, includingdiverse capture angle, varying head size due to perspective, and head overlapping,greatly increase the difficulty of counting and localizing people accurately in theseimages. Especially on the cases from the bottom rows, each head only has fewpixels related to it, and the counting network can extract very limited features. Wecan see that our proposed method handles these extreme cases very well, providingaccurate counting results and precise head location predictions.36Figure 2.4: localization results on sample images with normal density.37Figure 2.5: More sample localization results on extremely dense crowds.38Method Computational ComplexityCSRNet[31] 508.35 GMACSFCN[58] 760.45 GMACLSC-CNN[1] 1436.77 GMACJCCL 98.84 GMACOur Regression Decoder 5.18 GMACTable 2.4: Computation complexity comparisons.2.3.7 Complexity ComparisonIn this section, we conduct a complexity study of our proposed method and otheropen-sourced comparison methods. We use the hardware operation unit multi-plierâaccumulator (MAC) as the unit of computational complexity, and the prefix Gpresents the unit prefix Giga. To make a fair comparison, we use images with thesame resolution as inputs for different methods. The results are listed in Table 2.4.Our proposed method has significantly lower computation complexity. The pro-posed method can achieve comparable counting accuracy to that of [31, 58] withrequiring only 1/5 computation cost and provide precise localization results. [1]can also provide localization results, but it requires 14 times computational costcompared with our method. In addition to the main network, the supplementary re-gression decoder (not included in the JCCL) has extremely low complexity, whichcan be negligible in the overall computational complexity.39Chapter 3Crowd Counting WithKnowledge Distillation3.1 Knowledge DistillationCrowd counting models estimate the number of people in a congested scene image[6]. State-of-the-art frameworks (which are mainly regression-based) for crowdcounting generally care more about the counting accuracy. Previous work on CNNsuggested that a deeper network for crowd counting can extract more abstract se-mantic information. As the neural network becomes deeper, its representation abil-ity is getting better and thus results in higher accuracy [3, 5, 36, 52, 54, 70]. Mean-while, a primary concern for these crowd counting models is high computationalcost, which makes them infeasible to be applied on mobile or embedded systemsto perform the real-time surveillance task [57].To improve the computational efficiency of crowd counting, lightweight mod-els are preferred, though likely it is at the expense of accuracy [28]. KD methods formodel compression are proposed to strike a trade-off between accuracy and com-putational efficiency, [17, 22, 73]. However, they are primarily for classificationand rarely studied for regression tasks, and therefore cannot be directly applied tocrowd counting. To our knowledge, there is limited research in KD-based crowdcounting, and we only found [57], where the focus is designing a compact modeland authors only briefly mention the KD concept by using teacher network density40Figure 3.1: Diagram for general knowledge distillation procedure.map outputs directly as the only additional guidance for student network training.The general procedure for knowledge distillation is as shown in Figure 3.1.There are mainly three components in the general knowledge distillation proce-dure: teacher model, student model, and loss function. The procedure is as follow:same data are feed into both cumbersome teacher network and small student net-work. The well-trained teacher network generates a soft target containing richknowledge representation learned by high capacity teacher network. Loss is usu-ally formulated as a combination of label information, soft target from teachernetwork, and student network predictions. The gradient of loss function then getsbackpropagated to the student model for updating parameters in student networks.In this diagram, the only difference from regular neural network training is theextra path that generates soft target using an addition teacher network. It is the keyto successful knowledge distillation. A cumbersome teacher network with higherknowledge capacity can achieve good performance on target tasks. However, thelarge knowledge capacity of the teacher network is usually not fully utilized wheremany neurons in the network are always inactive, which causes waste in compu-tation. In knowledge distillation, we use the soft target output from a well-trainedteacher network to supervise the training of a small student network, which canlearn concise knowledge representation directly from the teacher network discard-41ing those inactive units. Thereby, the student network can achieve accuracy closeto the teacher network with a higher utilization rate of its smaller knowledge ca-pacity. Thus student networks are low in computational cost and can be employedon mobile devices with limited computational resources for real-time inference.To achieve a better knowledge distillation, research has been conducted onhow knowledge from teacher networks can be better utilized to guide student net-works. In TAKD [39], an additional intermediated-size teacher assistant moduleis introduced in the middle of teacher and student networks. The module acts asa teaching assistant to further translates complex knowledge in a more acceptableway for students to absorb it better. In ResKD [30], authors try to use multiplelight-weighted res-students to learn the residual error between teacher and studentto make up significant gaps between heavy teacher and single lightweight student.However, it is worth highlighting that there is still a large performance gap be-tween the teacher network and the student network [25, 39] even with help fromthese modules. In addition to the knowledge gap problem previous methods tryto resolve, another major reason for the performance gap is that the quality of in-formation transferred is not good enough; or has a negative influence [15] on thestudent. Since the prediction of the teacher network is not always correct, if theinformation from incorrect output is transferred from the teacher network, and thestudent network makes a decision totally regarding the teacherâs decisive result, thepoor performance of student networks is expected.In this work, we try to address the above concern by focusing on the prob-lem of what to distill from teachers. To realize a student network with extremelylow computational complexity, we propose the task-specific knowledge distillationframework for crowd counting. Its major novelties are as follows. First, it is thefirst attempt that thoroughly investigates knowledge distillation for crowd counting.Second, we selectively transfer knowledge from the teacher network by using theproposed hierarchic feature regulation. Third, the student network can have its owntask-specific judgment for the crowd counting task through a teacher-diminishingway. Last but not least, an optimized ShuffleNet [38] is proposed to act as thestudent network. We, therefore, refer to the proposed framework as ShuffleCount.423.2 MethodIn this section, we present our framework of KD-based crowd counting. First, weexplain the intuition behind network selection and modification. Then we introducethe losses used to transfer knowledge from the teacher to the student network. Lastbut not least, we present the task-specific knowledge distillation training that canachieve optimal results.3.2.1 Optimized ShuffleNetThe major goal of this work is to build a computationally efficient crowd count-ing network that can give accurate people count on crowded scenes. Among allcompact network architectures, ShuffleNet V2 [38] shows its supremacy in termsof computational efficiency and accuracy in various classification tasks. We, there-fore, employ ShuffleNet V2 as our starting point for the crowd counting task.The high computational efficiency of ShuffleNet mainly comes from the usageof Depthwise Separable Convolution (DSC) [19], Inverted Residual Block (IRB)[46] and channel shuffle operation [69]. DSC is the building block for InvertedResidual Block, which reduces the computational cost of traditional convolutionby a factor of 8 to 9 with only a small reduction in accuracy. To be more specific,assuming the input has M channels and output has N channels, instead of applyingN filters to M times to each input channels, DSC applies M filters to the correspond-ing input channel once and then a 1Ã1 pointwise convolution can be used to mapM channel feature into expected N channels if necessary. Although one additionalconvolutional layer is added, depthwise convolution reduces the computation by afactor of N, thus saves the overall computational cost.Inverted Residual Block, opposite to regular residual block which takes a wideâ narrowâwide channel arrangement, takes advantage of depthwise convolutionand uses narrowâ wideâ narrow to save computational cost as well as increasememory efficiency. The comparison of a regular residual block and an invertedresidual block can be found in Figure 3.3.Finally, the channel shuffle operations mix information from different chan-nels, thus enhancing the compact networkâs representation power. The operation isdone by splitting channels into two groups and applying different Inverted Residual43Figure 3.2: Architectures of ShuffleCount: teacher network DMCount (left),student network Optimized ShuffleNet (right) and knowledge distilla-tion modules (middle)44Figure 3.3: Regular Residual Block (left) vs. Inverted Residual Block (right)Block to each group. The results from two branches are concatenated and shuffledaround to allow information to flow through different channels to enhance repre-sentation.Unlike classification and detection tasks which Shuffnet was initially designedfor, the crowd counting task has higher requirements on image details due to ex-treme crowdedness in crowded scenes. Resolution is critical for counting accu-racy because likely only a few pixels are related to each person in many crowdedscenes. In the original ShuffleNet V2, input images are downsampled by a factor of4 directly at the very beginning for reducing the computational cost, but it greatlysacrifices the resolution and thus results in a degraded accuracy in crowd count-ing tasks. To achieve a balance between accuracy and computational complexity,we propose the pooling-free ShuffleNet, in which one upsampling layer betweendensity regression layer and feature extraction layers is added. This ShuffleNetdesigned explicitly for crowd counting is named Optimized ShuffleNet (OSN).We also design the corresponding teacher network considering the OSN. Dif-ferent from the previous crowd counting works, which mainly use either particularpost-processing layers or multi-branch architectures that deviate a lot from the cho-45sen student network, we hold the view that to minimize the knowledge gap betweenthem, a well-performed teacher network must have a structure similar to the studentnetwork. Therefore, to make the knowledge transferable, we choose DMCount[55] with the basic VGG19 structure [50] as the teacher. Using the DistributionMatching loss proposed in DMCount, the simply structured teacher network canachieve very competitive accuracy on crowd counting benchmark datasets. Also,the VGG19 structure is close to our proposed OSN architecture with five featurelayers and thus makes the knowledge distillation much easier. The detailed layerstructure of both DMCount and Optimized ShuffleNet can be found in Figure 3.2where a 512Ã512 image is used as a sample input. Both networks are fully con-volutional and are capable of taking input images with arbitrary resolutions.3.2.2 General Knowledge Distillation Loss DefinitionsIn classification settings, knowledge from the teacher network is concentrated inthe logits. The objective loss used to acquire rich information from teachersâ logitoutputs while training the student network can be formulated as follows:LKD = Î±t2LTâS(Ï(St),Ï(Tt))+(1âÎ±)LCE(Ï(S),y) (3.1)where y is the class label, Ï is the softmax function, T and S correspond to the logitoutput from teacher and student network respectively. LTâS is the loss function thatmeasures the distance between teacher and student outputs with temperature t, andLCE is the regular cross entropy loss used in classification. Hyper-parameter α isused to balance the weight between learning from teacher soft targets and learn-ing from ground truth labels. Greater α value makes student networks rely moreon the knowledge distillation loss, replicating what the teacher networks predict.However, the same mistakes as teachers can be made if the loss relies too much onthe KD-term.Since we aim for a compact network that produces accurate counting estima-tion, the conventional density-regression approach is more suitable, neglecting ad-ditional localization requirements. Different from classification settings, the outputfrom the density-regression problem is a density map with real value without any46intermediate logit output as a soft target. Knowledge from the teacher networkmust be distilled from other layers such as feature layers and output layers. Ourproposed loss contains two major parts, including losses for crowd counting andknowledge distillation.3.2.3 Crowd Counting LossThe overall crowd counting loss includes three loss items. We use the Distribu-tion Matching loss proposed in DMCount as our first loss for crowd counting sinceit provides a tighter error bound compared with Gaussian smoothed methods andBayesian loss. This loss itself performs very well even with simple network struc-tures. The key idea of the Distribution Matching loss is the use of optimal trans-port. Optimal transport is the optimal cost to transfer one probability distributionto the other. To convert crowd counting into a distribution matching problem, thetwo-dimensional ground truth annotation and prediction are converted into one-dimensional vectors, where z â Rn+ denotes the binary dot-annotation and zË â Rn+represents the predicted density map from the neural network. Both z and zË arenormalized by dividing their corresponding norms. The Optimal Transport (OT)Loss is defined as follows:LOT (z, zË) =W(zâzâ1 ,zËâzËâ1)(3.2)Here â · â1 is the L1 norm of a vector, and W (.) is the Monge-KantorovichâsOptimal Transport cost function, which is defined in the original DMCount paper[55].The second item of the crowd counting loss is the total variation loss, whichis used to mitigate poor approximation of the OT loss in low-density areas. Thisis due to limited iterations used when approximating the optimal transport costsolution. The loss is defined as:LTV (z, zË) =12â¥â¥â¥â¥ zâzâ1 â zËâzËâ1â¥â¥â¥â¥1(3.3)Finally, the counting loss is defined as the absolute counting difference between47the predicted density map and the ground truth. It is defined as:LC(z, zË) = |âzâ1ââzËâ1| (3.4)The overall crowd counting loss can be defined as:LCC = LC(z, zË)+Î»1LOT (z, zË)+Î»2âzâLTV (z, zË) (3.5)where λ1,λ2 are hyperparameters and âzâ is multiplied to the total variance loss toensure that it has the same scale as the counting loss.3.2.4 Hierarchic Feature RegulationThe knowledge distillation loss includes two loss items: feature loss and outputloss. In this section, we mainly discuss the hierarchic feature regulation realizedby the constraint of the feature loss.We first use the convolution regressors [43] to match up the number of channelsfor different feature extraction layers between the teacher and the student networks.With r denoting the regressor, we have teacher features F t0 ,F t1 , . . . ,F t4 and thecorresponding student features r1(Fs1),r2(Fs2), . . . ,r4(Fs4). We want the studentnetwork to imitate what the teacher network learns by minimizing the differencebetween the features. In [43], L2 distance is used between features whereLHT (F t ,Fs) =12âF t â r(Fs)â2 (3.6)Applying such a L2 loss puts an extremely strong constraint to the student net-work, i.e., every single pixel must match exactly with the teacher network. In com-puter vision problems, pixel-level regularization is too strong, while regularizingchannel-level information is more reasonable. The student network should mimicthe teacher network features in a channel-based fashion because the same amountof information can be represented by various patterns. The cth channel feature F¯cis computed as follow:F¯c =1HÃW âFc (3.7)where Fc is the cth channel of the feature map with size HÃW . The channel-wise48feature loss is now defined as:Lchannel(F ti ,Fsi ,ri) =12âFÂ¯ tiâ ¯ri(Fsi)â2 (3.8)We propose the hierarchic feature regulation to give another degree of guidanceto the feature loss. Deeper feature layers are more important than shallower fea-tures since they are closer to the output. Considering an extreme condition wherethe student can perfectly repeat the last feature layer from the teacher network, allprevious feature layers become trivial in that case. In reality, the guidance from allfeature layers is necessary and helpful because of the infeasibility of perfect repli-cation due to limited representation power in the student network. To consolidatesuch an idea, in the loss, we give deeper feature layers with larger weights by afactor of ki, where k â¥ 1 is the factor and i represents the ith feature layer. Theoverall feature loss is defined as follows:L f eature =4âi=0kiÃLchannel(F ti ,Fsi) (3.9)With this hierarchical feature regulation, we can guide the student network to havea preference for learning deeper features. The student network can better replicatethe teacher network logic on deeper layers with a growing number of parametersinvolved.In addition to the intermediate feature loss, we minimize the difference betweendensity map outputs from teacher and student network by the output loss, which isdefined as:Lout put =12âzËt â zËsâ (3.10)where zËt and zËs correspond to the outputs from teacher and student network respec-tively. Just like using the logit layer as a soft target in classification settings, theteacher network output in regression also acts as a similar role where additionalinformation such as the correlation between pixels densities is encoded. In an idealsituation, replicating the output exactly can make the student network do at least aswell as the teacher network.Overall, our training loss consists of 3 terms with α,β ,γ as weight hyperpa-49rameters can be written as:L = αLCC +βL f eature + γLout put (3.11)3.2.5 Task-Specific Knowledge DistillationLearning from a well-trained teacher is beneficial initially, where imitating teacherfeatures and outputs can speed up the converging process of the student network.However, at the same time, completely learning from teacher features will bringadverse effects, as the student network will lose its own task-specific decision-making for crowd counting. Especially during later stages, the goal of featureconstraints will deviate from the crowd counting target. Figure 3.4 shows a pair ofsample plots of feature loss and counting loss over time during training. Althoughintuitively that imitate features from teacher network and the overall counting targetshould have the same direction, they indeed conflict with each other as the trainingiteration increases. The feature loss drops rapidly at the very beginning, indicatingit complies with the overall counting target. Later, the feature distance betweenstudent and teacher starts to increase while the counting loss still decreases. Itis because we manually set higher weight on counting loss since it is the mainobjective. Assuming greater weight is given to the feature loss, counting loss willinevitably increase in a later stage. Some may claim that we can simply give featureloss much lower weight at the beginning to prevent its impact on counting results.But doing so defeats the purpose of knowledge distillation because the feature losshas a positive impact on model convergence in the early stages.To solve this problem and relax the feature constraint, we would diminish thefeature loss gradually, which in turn increases the effects of the crowd countingloss. To be more specific, we diminish the effect of learning from the teachernetwork by adding a soft decay factor [73] to the feature loss. Denoting the decayfactor as d, the number of current epochs as e and decaying every T epochs, wehave the final loss defined as:L = αLCC +de/TβL f eature + γLout put (3.12)In other words, learning from the teacher network features at the beginning50Figure 3.4: Feature loss (left) vs. Counting loss (right) during training timesets a good initialization point for the student network, and the guidance fromteacher network features gradually diminishes as the number of epochs increases.Eventually, the feature loss will vanish, and the student learns purely from thecrowd counting and the output losses.3.3 ExperimentsIn this section, we use one popular crowd counting benchmark dataset, Shang-haiTech A [71], to evaluate the proposed ShuffleCount. This dataset contains 482random online images with variant angle, resolution, and sparsity. The averagepeople density in this dataset is 501.4. Both accuracy and computational complex-ity are compared with the state-of-the-art compact methods. The MAE and MSE areused as accuracy measures; MAC is used as the metric for computational complex-ity.3.3.1 Training DetailsTraining of ShuffleCount is a multi-stage process. During the first stage, we usethe standalone counting loss introduced in 3.2.3 to train both teacher and studentnetworks independently. Based on the evaluation result on the validation set, well-performing teacher parameters are saved and used as feature guidance for studentnetwork training in the next stage. The student network trained with standalonecounting loss is used for comparison purposes to verify the effectiveness of cho-sen knowledge distillation architecture and hyper-parameters. The second stage is51training the student network using both hierarchic feature regulation and objectivecounting loss. The student network parameters are initialized using Gaussian dis-tribution with zero mean and standard deviation of 0.01. We use Adam to optimizeour models. The learning rate is initialized to 1e-4 and halved after 100, 500, and1000 epochs. Random cropping and horizon flipping are used to add diversity tothe dataset as well as to keep consistent image resolution for batch operation dur-ing training. Additional augmentation in HSV color space is also used to simulateillumination variation, adding variance to the dataset for overfitting prevention.Ground truth density maps are generated on the fly after data augmentation formaximum flexibility.3.3.2 Effect of Knowledge DistillationAs discussed in Section 3.2, we choose DMCount as our teacher network due toits simplicity in structure. In Table 3.1, we reported the number of parametersand computational complexity of the teacher network and our proposed Shuffle-Count. ShuffleCount contains only 6% parameters of its teacher network. In termsof computational cost, the proposed ShuffleCount uses less than 5% arithmeticoperation compared with DMCount. Training the ShuffleCount using the stan-dalone crowd counting loss yields the MAE of 88.7, which shows a significantaccuracy gap from 63.5 of the teacher network, indicating limited learning abilityof the student network without teacher guidance. After applying the task-specificknowledge distillation, the accuracy is dramatically boosted to 70.9. Based on theaccuracy improvement, we can see the effectiveness of crowd counting specificknowledge distillation. Rich knowledge learned by the teacher network gets con-densed and transferred successfully, allowing the ShuffleCount network to achieveits potential fully.Figure 3.5 shows three sample density map outputs with different levels ofcrowdedness. Besides producing similar quality density map outputs as the teacher,ShuffleCount can yield the same level of counting accuracy.52Figure 3.5: Density outputs from DMCount and ShuffleCount.53#Params (M) GMacsShanghai AMAEâ MSEâDMCount 21.5 854.58 63.5 96.6Ours 1.31 37.17 88.7 143.1Ours (KD) 1.31 37.17 70.9 117.0Table 3.1: Accuracy and computational complexity comparisons between theproposed ShuffleCount network and the teacher network. The compu-tation complexity metric GMacs is computed using 1920Ã 1080 inputimages.#Params (M) GMacsShanghai AMAEâ MSEâMCNN [70] 0.13 56.2 110.2 173.2CMTL [52] 2.46 243.8 101.3 152.4CSRNet [31] 16.26 857.8 68.2 115.0SANet [5] 1.39 182.3 67.0 104.5MCount [57] 3.40 16.5 89.4 146.0MCount (*1.25) 5.47 34.1 82.9 137.9MCount (*2) 13.39 63.0 81.4 133.3Ours (KD) 1.31 37.2 70.9 117.0Table 3.2: Accuracy and computational complexity comparisons with thepopular large network and state-of-the-art compact counting networks.3.3.3 Comparison with State-of-the-artWe compare our proposed ShuffleCount (KD) with other popular compact meth-ods, including the knowledge distillation-based MobileCount with three differentsizes. Among all comparison models in Table 3.2, our ShuffleCount with knowl-edge distillation achieves top-tier accuracy and meanwhile remains low computa-tional cost. All models here have a relatively small model size with less than 20Mparameters. With the least amount of parameters, the accuracy of MCNN is lim-ited. SANet has the best accuracy metric in this table, but its computational costis almost six times higher than ShuffleCount. Although MobileCount has the leastcomputational cost, there is room for accuracy improvement. With increasing the54number of parameters, MobileCount with size 1.25 or 2 can achieve higher accu-racy, with the price of the increased computational cost. In terms of computationcomplexity, MobileCount (*1.25) is the most comparable one with our proposedShuffleCount, with a total of less than 40 GMacs. With almost no increase incomputational cost, our ShuffleCount improves the accuracy by 15%, reducing theMAE from 82.9 to 70.9. This vast difference is even more significant than the ac-curacy gain brought by tripling the computational cost from MobileCount (*1) toMobileCount (*2).Overall, the proposed ShuffleCount has the best accuracy compared with othermodels with a similar computational cost. At the same time, all methods with betteraccuracy require at least five times more computational cost. With a well-designedstudent network and crowd counting-specific knowledge distillation, ShuffleCountfulfills the motivation by successfully finding a good balance between accuracyand computational efficiency.55Chapter 4ConclusionDeep learning has achieved great success on many computation vision tasks vialearning through large-scale data. With the aid of deep learning models, complexproblems in daily life can be solved automatically and efficiently with minimalhuman involvement. Crowd counting is widely used in daily applications, includ-ing but not limited to surveillance cameras, traffic monitoring, public safety, andcity planning. Monitoring and controlling the number of people in a crowded areain real-time can help prevent potential accidents due to high people density. Inaddition to counting the number of people, further analysis such as the distancekept from people can be conducted with precise localization information in crowdscenes. In the era of Covid-19, detailed crowded analysis is critical in prevent-ing the spread of the virus. Real-time crowd estimation and human localizationare practical problems we need to address in crowd controlling. In this thesis,we first thoroughly reviewed existing state-of-the-art crowd counting approachesusing deep learning. We then introduced the Joint Crowd Counting and Localiza-tion model and the ShuffleCount to further improve existing interpretability andcomputational efficiency problems in crowd counting respectively. The methodsproposed in both the JCCL and ShuffleCount models can also be generalized toother ideas.First, we improve the model interpretability by providing precise localizationinformation along with crowd counting results. Many existing crowd countingworks only focus on density estimation, but a conventional density map is inca-56pable of providing localization results due to the impreciseness of density distribu-tion. Some other works use an additional branch to provide localization results andhelp improve the quality of crowd counting. We believe crowd counting and humanlocalization complement each other and can be solved in a single branched networkwith no additional cost. In Chapter 2, we model crowd counting as a classificationproblem using the proposed Pedestrian-Aware Density Map. The novel PADM ishighly interpretable and can handle inconsistent head center annotation due to hu-man labeling error. Three modules are introduced to address the scale variationchallenge in crowd counting. By precise localization results in crowded scenes, wecan acquire accurate crowd counting results at the same time. Joint crowd countingand localization (JCCL) improves the computational efficiency of crowd countingin the sense it can address both problems simultaneously without any extra cost. Inthe situation where detailed human location information is needed for crowed anal-ysis, our proposed JCCL model should be deployed since the simple yet effectivejoint approach yields higher accuracy and lower complexity.In the second part, we aim to build compact and highly accurate real-timecrowd counting models that can be used in edge devices where computational re-sources are very limited. Based on this motivation, we borrowed and redesignedan Optimized ShuffleNet specifically for the crowd counting task. Since we areaiming for extreme compactness, only the counting estimation is provided by ourmodel. In Chapter 3, we take advantage of ideas from knowledge distillation totransfer rich knowledge presentation learned by complex teacher network intocompact student network. Hierarchic feature regulation helps guide the studentnetwork to selectively learn feature representations from different feature layersof the teacher network. Diminishing teacher used in the proposed framework re-laxes the feature constraint in the later stage, allowing student networks to learnfrom their own experience effectively. The proposed ShuffleCount can achieve agood trade-off between accuracy and computational efficiency. The proposed Shuf-fleCount network can significantly improve the deployment feasibility of crowdcounting applications on various portable devices. Not only for real-time analysistasks, ShuffleCount can also boost productivity when analyzing a large amount ofcrowded data offline.574.1 ContributionThe main contribution of this thesis work can be summarized as the following:â¢ We propose a smartly simple framework JCCL to tackle the joint crowdcounting and localization problem efficiently.â¢ We propose a unique pedestrian-aware density map for crowd counting usedas the learning target. Based on this map, each person can be both countedand accurately localized. Compared with the traditional density map, theproposed PADM is more interpretable and can better deal with dense localareas. PADM reports better counting/localization accuracy.â¢ We are the first attempt that thoroughly investigates knowledge distillationfor crowd counting, and the accuracy of the compact network got boostedwith guidance from a complex teacher network.â¢ Knowledge is selectively transferred from the teacher network by using theproposed hierarchic feature regulation.â¢ In our proposed framework, the student network can have its own task-specific judgment for the crowd counting task through a teacher diminishingway.â¢ An optimized ShuffleNet is proposed to act as the highly compact yet accu-rate crowd counting network. We, therefore, refer to the proposed frameworkas ShuffleCount.4.2 Future worksIn this thesis work, we explored the crowd counting tasks from two different per-spectives. Both localization and crowd counting can be solved together with ourproposed JCCL. Compact ShuffleCount can give very accurate counting resultswith a minimal computational cost. It will be nice if we can combine the jointwork with knowledge distillation to build a compact yet accurate small networkthat can jointly predict precise localization and count in crowd scenes. There arestill many challenges that need to be resolve in the area of crowd counting. There58are many unlabeled crowded data that can be used to improve the accuracy of ex-isting models, and domain adaption techniques can be applied to crowd countingmodels to improve the prediction accuracy in other object counting tasks.The following future works are suggestions that can be further explored on topof the proposed method in this thesis:â¢ Build crowd counting and localization models in a semi-supervised way. La-beling crowd counting data is extremely labor-intensive work because thou-sands of heads need to be annotated in a single crowd image, limiting thenumber of labeled images in existing crowd counting datasets. But, thereare a tremendous amount of unlabeled crowd data that are not utilized. Withsemi-supervised learning, combining the small number of labeled data witha large number of unlabeled crowded data, we can further improve the accu-racy and generalizability of crowd counting models.â¢ Additional detailed information can be extracted simultaneously with crowdcounting and localization results. Just like localization can help improvecrowd counting models, adding other related estimation tasks can enrich thefeature extraction ability of the network. One example will be age estimationin dense scenes, which can support more detailed crowd analysis. At thesame time, accurate age estimation can help eliminate false positive objects,improving the accuracy of crowd counting results.â¢ Domain adaptation can be applied to the task of crowd counting. Use do-main knowledge learned from large datasets to improve the generalizabilityof the crowd counting model in more conditions. Generated synthetic datacan greatly increase the diversity of crowd counting datasets, and domainadaption from synthetic to real-world data can help to build more power-ful models. Also, the crowd counting model can be adapted to other objectcounting tasks, including but not limited to vehicle counting or animal count-ing, where data is even harder to collect.â¢ Knowledge distillation should be extended and utilized on differently struc-tured crowd counting models, taking advantage of complex but powerfulteacher networks designed explicitly for crowd counting. In this thesis work,59knowledge distillation has only been done between teacher and student net-works with similar structures. But many state-of-the-art crowd countingmodels have substantial knowledge gaps with simply-structure student net-works by involving more complicated backbones and post-processing sub-networks. It will be excellent if we can use some knowledge distillationtechniques to take advantage of rich feature representation in those networksto further improve the accuracy of the compact student network.â¢ Meta-learning can be applied to knowledge distillation to better answer thequestion that what is the most suitable way of connecting cumbersome teachernetwork layers with simple student networks. In this works, we use our priorknowledge to guide student networks to put more weight on learning deeperfeatures, but such type of prior knowledge can only be applied to networkswith similar structures, and the ratio of weight between multiple feature lay-ers can be sub-optimal. We can use the meta-learning framework to let mod-els decide which and where knowledge from teacher networks should bedistilled into student networks.60Bibliography[1] D. Babu Sam, S. V. Peri, M. Narayanan Sundararaman, A. Kamath, andV. B. Radhakrishnan. Locate, size and count: Accurately resolving people indense crowds via detection. IEEE Transactions on Pattern Analysis andMachine Intelligence, pages 1â1, 2020. doi:10.1109/TPAMI.2020.2974830.â pages 19, 32, 39[2] S. Bai, Z. He, Y. Qiao, H. Hu, W. Wu, and J. Yan. Adaptive dilated networkwith self-correction supervision for counting. In 2020 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pages4593â4602, 2020. doi:10.1109/CVPR42600.2020.00465. â pages 12, 22[3] S. Bai, Z. He, Y. Qiao, H. Hu, W. Wu, and J. Yan. Adaptive dilated networkwith self-correction supervision for counting. In 2020 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pages4593â4602, 2020. doi:10.1109/CVPR42600.2020.00465. â page 40[4] L. Boominathan, Kruthiventi Srinivas SS, and Babu, R Venkatesh.Crowdnet: A deep convolutional network for dense crowd counting. InProceedings of the 24th ACM international conference on Multimedia, pages640â644, 2016. â page 11[5] X. Cao, Z. Wang, Y. Zhao, and F. Su. Scale aggregation network for accurateand efficient crowd counting. In Proceedings of the European Conference onComputer Vision (ECCV), pages 734â750, 2018. â pages 11, 12, 40, 54[6] A. B. Chan and N. Vasconcelos. Bayesian poisson regression for crowdcounting. In 2009 IEEE 12th international conference on computer vision,pages 545â551. IEEE, 2009. â pages 11, 40[7] A. B. Chan, Zhang-Sheng John Liang, and N. Vasconcelos. Privacypreserving crowd monitoring: Counting people without people models ortracking. In 2008 IEEE Conference on Computer Vision and PatternRecognition, pages 1â7. IEEE, 2008. â pages 11, 1261[8] Z.-Q. Cheng, J.-X. Li, Q. Dai, X. Wu, J.-Y. He, and A. G. Hauptmann.Improving the learning of multi-column convolutional neural network forcrowd counting. In Proceedings of the 27th ACM International Conferenceon Multimedia, pages 1897â1906, 2019. â page 12[9] B. C. Csa´ji et al. Approximation with artificial neural networks. Faculty ofSciences, Etvs Lornd University, Hungary, 24(48):7, 2001. â page 6[10] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformableconvolutional networks. In Proceedings of the IEEE internationalconference on computer vision, pages 764â773, 2017. â pages 8, 22[11] D. Deb and J. Ventura. An aggregated multicolumn dilated convolutionnetwork for perspective-free counting. In 2018 IEEE/CVF Conference onComputer Vision and Pattern Recognition Workshops (CVPRW), pages308â30809, 2018. doi:10.1109/CVPRW.2018.00057. â page 11[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: ALarge-Scale Hierarchical Image Database. In CVPR09, 2009. â page 7[13] J. Gao, Q. Wang, and X. Li. Pcc net: Perspective crowd counting via spatialconvolutional network. IEEE Transactions on Circuits and Systems forVideo Technology, 2019. â pages 12, 13[14] W. Gerstner and W. M. Kistler. Spiking neuron models: Single neurons,populations, plasticity. Cambridge university press, 2002. â page 4[15] J. Guo, M. Chen, Y. Hu, C. Zhu, X. He, and D. Cai. Reducing theteacher-student gap via spherical knowledge disitllation. arXiv preprintarXiv:2010.07485, 2020. â page 42[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 770â778, 2016. â page 21[17] Z. He, T. Dai, J. Lu, Y. Jiang, and S.-T. Xia. Fakd: Feature-affinity basedknowledge distillation for efficient image super-resolution. In 2020 IEEEInternational Conference on Image Processing (ICIP), pages 518â522.IEEE, 2020. â page 40[18] A. L. HODGKIN and A. F. HUXLEY. A quantitative description ofmembrane current and its application to conduction and excitation in nerve.The Journal of physiology, 117(4):500â544, 08 1952.62doi:10.1113/jphysiol.1952.sp004764. URLhttps://pubmed.ncbi.nlm.nih.gov/12991237. â page 4[19] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand,M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neuralnetworks for mobile vision applications. arXiv preprint arXiv:1704.04861,2017. â page 43[20] P. Hu and D. Ramanan. Finding tiny faces. In 2017 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), pages 1522â1530, 2017.doi:10.1109/CVPR.2017.166. â page 18[21] S. Huang, X. Li, Z. Zhang, F. Wu, S. Gao, R. Ji, and J. Han. Body structureaware deep crowd counting. IEEE Transactions on Image Processing, 27(3):1049â1059, 2018. doi:10.1109/TIP.2017.2740160. â page 32[22] Z. Huang and N. Wang. Like what you like: Knowledge distill via neuronselectivity transfer. arXiv preprint arXiv:1707.01219, 2017. â page 40[23] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, andM. Shah. Composition loss for counting, density map estimation andlocalization in dense crowds. In Proceedings of the European Conference onComputer Vision (ECCV), September 2018. â page 29[24] X. Jiang, L. Zhang, M. Xu, T. Zhang, P. Lv, B. Zhou, X. Yang, and Y. Pang.Attention scaling for crowd counting. In 2020 IEEE/CVF Conference onComputer Vision and Pattern Recognition (CVPR), pages 4705â4714, 2020.doi:10.1109/CVPR42600.2020.00476. â page 13[25] X. Jin, B. Peng, Y. Wu, Y. Liu, J. Liu, D. Liang, J. Yan, and X. Hu.Knowledge distillation via route constrained optimization. In Proceedings ofthe IEEE International Conference on Computer Vision, pages 1345â1354,2019. â page 42[26] S. D. Khan, H. Ullah, M. Uzair, M. Ullah, R. Ullah, and F. A. Cheikh.Disam: Density independent and scale aware model for crowd counting andlocalization. In 2019 IEEE International Conference on Image Processing(ICIP), pages 4474â4478, 2019. doi:10.1109/ICIP.2019.8803409. â page19[27] P. Kidger and T. Lyons. Universal approximation with deep narrownetworks. In Conference on Learning Theory, pages 2306â2327. PMLR,2020. â page 763[28] K.-H. Kim, S. Hong, B. Roh, Y. Cheon, and M. Park. Pvanet: Deep butlightweight neural networks for real-time object detection. arXiv preprintarXiv:1608.08021, 2016. â page 40[29] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learningapplied to document recognition. Proceedings of the IEEE, 86(11):2278â2324, 1998. â page 7[30] X. Li, S. Li, B. Omar, and X. Li. Reskd: Residual-guided knowledgedistillation. arXiv preprint arXiv:2006.04719, 2020. â page 42[31] Y. Li, X. Zhang, and D. Chen. Csrnet: Dilated convolutional neuralnetworks for understanding the highly congested scenes. 2018.doi:10.1109/cvpr.2018.00120. â pages 12, 32, 39, 54[32] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dolla´r. Focal loss for denseobject detection. IEEE Transactions on Pattern Analysis and MachineIntelligence, 42(2):318â327, 2020. doi:10.1109/TPAMI.2018.2858826. âpage 20[33] C. Liu, X. Weng, and Y. Mu. Recurrent attentive zooming for joint crowdcounting and precise localization. In 2019 IEEE/CVF Conference onComputer Vision and Pattern Recognition (CVPR), pages 1217â1226, 2019.doi:10.1109/CVPR.2019.00131. â pages 13, 15, 19, 30, 32, 33, 34, 35[34] J. Liu, C. Gao, D. Meng, and A. G. Hauptmann. Decidenet: Countingvarying density crowds through attention guided detection and densityestimation. In 2018 IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 5197â5206, 2018. doi:10.1109/CVPR.2018.00545. âpage 11[35] Y. Liu, M. Shi, Q. Zhao, and X. Wang. Point in, box out: Beyond countingpersons in crowds. 2019. doi:10.1109/cvpr.2019.00663. â page 18[36] Y. Liu, Q. Wen, H. Chen, W. Liu, J. Qin, G. Han, and S. He. Crowd countingvia cross-stage refinement networks. IEEE Transactions on ImageProcessing, 29:6800â6812, 2020. doi:10.1109/TIP.2020.2994410. â page40[37] J. Ma, Y. Dai, and Y.-P. Tan. Atrous convolutions spatial pyramid networkfor crowd counting and density estimation. Neurocomputing, 350:91â101,2019. â page 1164[38] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practicalguidelines for efficient cnn architecture design. In Proceedings of theEuropean conference on computer vision (ECCV), pages 116â131, 2018. âpages 42, 43[39] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, andH. Ghasemzadeh. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34,pages 5191â5198, 2020. â page 42[40] J. Patterson and A. Gibson. Deep Learning: A Practitionerâs Approach.OâReilly Media, Inc., 1st edition, 2017. ISBN 1491914254. â pages xi, 6[41] M. Peng, C. Wang, T. Chen, G. Liu, and X. Fu. Dual temporal scaleconvolutional neural network for micro-expression recognition. Frontiers inpsychology, 8:1745, 2017. â pages xi, 8[42] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-timeobject detection with region proposal networks. In Advances in neuralinformation processing systems, pages 91â99, 2015. â page 18[43] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio.Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. âpage 48[44] D. Ryan, S. Denman, C. Fookes, and S. Sridharan. Crowd counting usingmultiple local features. In 2009 Digital Image Computing: Techniques andApplications, pages 81â88, 2009. doi:10.1109/DICTA.2009.22. â page 11[45] D. B. Sam, S. Surya, and R. V. Babu. Switching convolutional neuralnetwork for crowd counting. In 2017 IEEE Conference on Computer Visionand Pattern Recognition (CVPR), pages 4031â4039, 2017.doi:10.1109/CVPR.2017.429. â pages 11, 32[46] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings ofthe IEEE conference on computer vision and pattern recognition, pages4510â4520, 2018. â page 43[47] E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks forsemantic segmentation. IEEE Transactions on Pattern Analysis andMachine Intelligence, 39(4):640â651, 2017.doi:10.1109/TPAMI.2016.2572683. â page 865[48] M. Shi, Z. Yang, C. Xu, and Q. Chen. Revisiting perspective information forefficient crowd counting. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 7279â7288, 2019. âpages 12, 13[49] K. Simonyan and A. Zisserman. Very deep convolutional networks forlarge-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. âpage 21[50] K. Simonyan and A. Zisserman. Very deep convolutional networks forlarge-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. âpage 46[51] V. A. Sindagi and V. M. Patel. Generating high-quality crowd density mapsusing contextual pyramid cnns. In 2017 IEEE International Conference onComputer Vision (ICCV), pages 1879â1888, 2017.doi:10.1109/ICCV.2017.206. â pages 11, 12, 21, 32[52] V. A. Sindagi and V. M. Patel. Cnn-based cascaded multi-task learning ofhigh-level prior and density estimation for crowd counting. In 2017 14thIEEE International Conference on Advanced Video and Signal BasedSurveillance (AVSS), pages 1â6. IEEE, 2017. â pages 40, 54[53] V. A. Sindagi, R. Yasarla, and V. M. Patel. Pushing the frontiers ofunconstrained crowd counting: New dataset and benchmark method. InProceedings of the IEEE International Conference on Computer Vision,pages 1221â1231, 2019. â pages 29, 30[54] Y. Tian, Y. Lei, J. Zhang, and J. Z. Wang. Padnet: Pan-density crowdcounting. IEEE Transactions on Image Processing, 29:2714â2727, 2019. âpage 40[55] B. Wang, H. Liu, D. Samaras, and M. Hoai. Distribution matching for crowdcounting. arXiv preprint arXiv:2009.13077, 2020. â pages 46, 47[56] L. Wang, B. Yin, X. Tang, and Y. Li. Removing background interference forcrowd counting via de-background detail convolutional network.Neurocomputing, 332:360 â 371, 2019. ISSN 0925-2312.doi:https://doi.org/10.1016/j.neucom.2018.12.047. URLhttp://www.sciencedirect.com/science/article/pii/S0925231218315042. âpage 1166[57] P. Wang, C. Gao, Y. Wang, H. Li, and Y. Gao. Mobilecount: An efficientencoder-decoder framework for real-time crowd counting. Neurocomputing,407:292â299, 2020. â pages 40, 54[58] Q. Wang, J. Gao, W. Lin, and Y. Yuan. Learning from synthetic data forcrowd counting in the wild. In 2019 IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), pages 8190â8199, 2019.doi:10.1109/CVPR.2019.00839. â pages 32, 39[59] Q. Wang, J. Gao, W. Lin, and X. Li. Nwpu-crowd: A large-scale benchmarkfor crowd counting and localization. IEEE Transactions on Pattern Analysisand Machine Intelligence, pages 1â1, 2020.doi:10.1109/TPAMI.2020.3013269. â page 35[60] X. Wu, Y. Zheng, H. Ye, W. Hu, T. Ma, J. Yang, and L. He. Counting crowdswith varying densities via adaptive scenario discovery framework.Neurocomputing, 397:127â138, 2020. â page 11[61] Z. Yan, Y. Yuan, W. Zuo, X. Tan, Y. Wang, S. Wen, and E. Ding.Perspective-guided convolution networks for crowd counting. InProceedings of the IEEE International Conference on Computer Vision,pages 952â961, 2019. â pages xi, 12, 13[62] B. Yang, W. Zhan, N. Wang, X. Liu, and J. Lv. Counting crowds using ascale-distribution-aware network and adaptive human-shaped kernel.Neurocomputing, 390:207â216, 2020. â page 11[63] Y. Yang, G. Li, Z. Wu, L. Su, Q. Huang, and N. Sebe. Reverse perspectivenetwork for perspective-aware object counting. In 2020 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pages4373â4382, 2020. doi:10.1109/CVPR42600.2020.00443. â pages 12, 13[64] F. Yu and V. Koltun. Multi-scale context aggregation by dilatedconvolutions. arXiv preprint arXiv:1511.07122, 2015. â page 8[65] F. Yu, D. Wang, E. Shelhamer, and T. Darrell. Deep layer aggregation. InProceedings of the IEEE Conference on Computer Vision and PatternRecognition (CVPR), June 2018. â page 21[66] L. Zeng, X. Xu, B. Cai, S. Qiu, and T. Zhang. Multi-scale convolutionalneural networks for crowd counting. In 2017 IEEE International Conferenceon Image Processing (ICIP), pages 465â469. IEEE, 2017. â pages 11, 2167[67] C. Zhang, H. Li, X. Wang, and X. Yang. Cross-scene crowd counting viadeep convolutional neural networks. In 2015 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 833â841, 2015.doi:10.1109/CVPR.2015.7298684. â page 32[68] L. Zhang, M. Shi, and Q. Chen. Crowd counting via scale-adaptiveconvolutional neural network. In 2018 IEEE Winter Conference onApplications of Computer Vision (WACV), pages 1113â1121, 2018.doi:10.1109/WACV.2018.00127. â page 11[69] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficientconvolutional neural network for mobile devices. In Proceedings of theIEEE conference on computer vision and pattern recognition, pages6848â6856, 2018. â page 43[70] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single-image crowdcounting via multi-column convolutional neural network. In 2016 IEEEConference on Computer Vision and Pattern Recognition (CVPR), pages589â597, 2016. doi:10.1109/CVPR.2016.70. â pages xi, 11, 21, 32, 40, 54[71] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single-image crowdcounting via multi-column convolutional neural network. In 2016 IEEEConference on Computer Vision and Pattern Recognition (CVPR), pages589â597, 2016. doi:10.1109/CVPR.2016.70. â pages 29, 51[72] X. Zhou, D. Wang, and P. Kra¨henbu¨hl. Objects as points. In arXiv preprintarXiv:1904.07850, 2019. â page 21[73] Z. Zhou, C. Zhuge, X. Guan, and W. Liu. Channel distillation:Channel-wise attention for knowledge distillation. arXiv preprintarXiv:2006.01683, 2020. â pages 40, 5068
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Deep learning approaches for crowd counting in images
Text