UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Superstitious perception in humans and convolutional neural networks Laflamme, Patrick 2017

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2017_september_laflamme_patrick.pdf [ 1.19MB ]
Metadata
JSON: 24-1.0354254.json
JSON-LD: 24-1.0354254-ld.json
RDF/XML (Pretty): 24-1.0354254-rdf.xml
RDF/JSON: 24-1.0354254-rdf.json
Turtle: 24-1.0354254-turtle.txt
N-Triples: 24-1.0354254-rdf-ntriples.txt
Original Record: 24-1.0354254-source.json
Full Text
24-1.0354254-fulltext.txt
Citation
24-1.0354254.ris

Full Text

Superstitious perception in humans and convolutionalneural networksbyPatrick LaflammeBSc. Psychology, University of Waterloo, 2015A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ArtsinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Psychology)The University of British Columbia(Vancouver)August 2017c© Patrick Laflamme, 2017AbstractThe advent of complex Hierarchical Convolutional Neural Networks (HCNNs) hasled to great progress in the field of computer vision, with modern implementationsof HCNNs rivalling human performance in object recognition tasks. The design ofHCNNs was inspired by current understanding of how the neurons of the humanvisual system are organized to support object recognition. There are researcherswho claim that the computations undertaken by HCNNs are approximating thoseof the human visual system, because of their high accuracy in predicting the neu-ral activity of regions of the brain involved in object classification (Cichy, Khosla,Pantazis, Torralba, & Oliva, 2016; Khaligh-Razavi & Kriegeskorte, 2014; Yaminset al., 2014). However, there has been little investigation of how HCNNs and hu-mans compare on other tasks that HCNNs have not been trained on. Our studycompared the similarity of one HCNN, AlexNet, and humans on a superstitiousperception task that involves falsely recognizing a learned object in the absenceof strong evidence for its presence. We began by validating a new technique thatquantifies human performance on the superstitious perception task. The first phaseof the research revealed that human behaviour in the task is dependent on whetherparticipants employed an active or passive task strategy. Next, the responses ofour HCNN to the same images were analyzed in a similar manner. The resultsshowed that HCNNs behaved similarly to humans in some ways and differently inothers. Specifically, the classification images generated for the HCNN were similarto those derived from human participants, but the HCNN was also more consistentin its responses than humans. A second finding was that the differences in humanparticipants classification images (created by adopting active versus passive strate-gies) could not be accounted for by simply altering the proportion of false alarmiiresponses in the HCNN. This suggests that HCNNs may be using criteria similarto humans’ perception when evaluating the likelihood of an object being present.The higher similarity between humans and HCNN in the passive condition sug-gests that the criteria similarities are largest when humans recruit minimal centralexecutive resources in the decision making process.iiiLay SummarySome claim that new computational models of vision mimic the human visualsystem. However, to date, the comparisons have been quite superficial. A time-honored way to study the brain is to identify unexpected ways in which it behaves.For example, visual illusions have often been used to understand the approaches thebrain employs in order to identify objects in a visual scene. Superstitious percep-tion, the phenomenon of seeing objects in meaningless noise, is the illusion usedhere to compare the computational models to the visual system. After verifyingthat superstitious perception works as expected in humans, we tested the perfor-mance of the models on the same task. The models showed similar behaviour tothat of humans, although the similarity was reduced when humans actively usedexecutive functions to perform the task.ivPrefaceAll work presented in this thesis is original intellectual property of the author, P.Laflamme, under the supervision of Dr. James Enns. All data reported in chap-ters 2, 3 and 4 were collected under the UBC Ethics certificate H16-00071 entitled”Seeing through the noise”.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1vi1.1 Superstitious perception . . . . . . . . . . . . . . . . . . . . . . . 11.2 Task strategy and its influence on task outcome . . . . . . . . . . 51.3 Hierarchical convolutional neural networks and the human visualsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 182.1.2 Stimuli and procedures . . . . . . . . . . . . . . . . . . . 182.1.3 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . 192.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.2.1 Practice phase . . . . . . . . . . . . . . . . . . . . . . . . 212.2.2 Test phase . . . . . . . . . . . . . . . . . . . . . . . . . . 222.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 343.1.2 Stimuli and procedures . . . . . . . . . . . . . . . . . . . 343.1.3 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . 353.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35vii3.2.1 Practice phase . . . . . . . . . . . . . . . . . . . . . . . . 353.2.2 Test phase . . . . . . . . . . . . . . . . . . . . . . . . . . 363.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 Experiment 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 454.1.2 Stimuli and procedures . . . . . . . . . . . . . . . . . . . 454.1.3 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . 464.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.1 Practice phase . . . . . . . . . . . . . . . . . . . . . . . . 474.2.2 Test phase . . . . . . . . . . . . . . . . . . . . . . . . . . 474.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.3.1 Internal prediction . . . . . . . . . . . . . . . . . . . . . 554.3.2 Classification images . . . . . . . . . . . . . . . . . . . . 564.3.3 External prediction . . . . . . . . . . . . . . . . . . . . . 565 Experiment 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.1.1 Model description . . . . . . . . . . . . . . . . . . . . . 595.1.2 Stimuli and procedures . . . . . . . . . . . . . . . . . . . 60viii5.1.3 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . 615.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.2.1 Practice phase - training and validation stimuli . . . . . . 625.2.2 Test phase . . . . . . . . . . . . . . . . . . . . . . . . . . 625.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.3.1 The effect of false alarm rate on CI quality . . . . . . . . . 726 General Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.1 Superstitious perceptions in an HCNN and humans . . . . . . . . . 766.2 Implications for the study of superstitious perception . . . . . . . 78References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80ixList of TablesTable 1 The effect of bin size on the correlation between the proportionof images selected as containing a target and the IWC betweenthe trial image and the CI. . . . . . . . . . . . . . . . . . . . . 25Table 2 The magnitude and reliability of relation between the propor-tion of images selected as target and the IWCs between the trialimage and the target image, for the data binned at various levelsof resolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Table 3 How the number of bins effects the strength of the relationshipbetween the IWCs for participants’ CI and their trial images. . 38Table 4 How the number of bins effects the strength of the relationshipbetween the IWCs for participants’ assigned target and their trialimages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Table 5 How the number of bins effects the strength of the relationshipbetween the IWCs for participants’ assigned target and their trialimages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Table 6 How the number of bins effects the strength of the relationshipbetween the IWCs for participants’ assigned target and their trialimages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54xTable 7 How the number of bins effects the strength of the relationshipbetween the IWCs for the neural network’s target and the testtrial images. . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Table 8 How the number of bins effects the strength of the relationshipbetween the IWCs for the neural network’s target and the testtrial images. . . . . . . . . . . . . . . . . . . . . . . . . . . . 69xiList of FiguresFigure 1 A visual summary depicting the difference between reversecorrelation, internal prediction, and external prediction. (A)Participants were assigned to one of four possible targets. (B)Reverse correlation is the process of generating the CI by look-ing at the sum of stimuli that led to a false alarm, and subtract-ing the sum of stimuli that did not lead to a false alarm. (C)Internal prediction is the process of using a generated CI topredict participants responses to the stimuli used to generatethe CI. This process is referred to as backward prediction inthe neural spike train literature. Such a process will lead to anover-estimation of the CI’s goodness of fit. (D) External pre-diction is mathematically identical to internal prediction, butdiffers in using a generated CI to predict responses to novelwhite noise stimuli. This process is referred to as forward pre-diction in the neural spike train literature, and serves to give abetter estimation of goodness of fit. In this experiment, how-ever, we will use participants’ assigned target images in lieu ofa generated CI. . . . . . . . . . . . . . . . . . . . . . . . . . 15Figure 2 Summary of sensitivity and decision bias in the practice phaseof Experiment 1. A) Mean sensitivity by target visibility. B)Mean decision bias by target visibility. Error bars represent thestandard error of the mean. . . . . . . . . . . . . . . . . . . . 22xiiFigure 3 Classification Images generated across participants with eachtarget. Note that the target that participants were trying to de-tect is difficult to discern, but present. . . . . . . . . . . . . . 24Figure 4 A scatter plot showing the relationship between the IWC of theCI and each trial image, and the proportion of images in eachbin selected as containing a target. The relationship is sigmoidin nature, and the line drawn in is the sigmoid of best fit, withslope of 93.76 (SE = 1.34) and a center constant of 0.008 (SE= 0.0002). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Figure 5 The relationship between the likelihood of an image being se-lected as a target, and the IWC between that image and thetarget image. . . . . . . . . . . . . . . . . . . . . . . . . . . 28Figure 6 Summary statistics for the training phase of the experiment.A) Mean sensitivity score across the easy and hard practiceconditions. B) Decision bias, lnβ , across the easy and hardpractice conditions. Error bars represent the standard error ofthe mean. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Figure 7 Classification images for each target assigned to participants.These CIs are generated by aggregating across the participantsthat were assigned to each target. The label under each imagedenotes which target participants were assigned. . . . . . . . 37Figure 8 Scatter plot showing the relationship between the IWC for eachtrial image and the participants’ CI with 400 bins. Solid lineshows the sigmoid function that was fit to the data. The slopefor the sigmoid was 17.6 (SE=0.2), with a center constant of0.008 (SE = 0.0005). . . . . . . . . . . . . . . . . . . . . . . 39xiiiFigure 9 Scatter plot showing the relationship between the IWC for eachtrial image and the target participants were assigned. Data isaggregated into 400 bins. . . . . . . . . . . . . . . . . . . . . 40Figure 10 A summary of the practice phase of the experiment. A) Meansensitivity (d’) for both the easy and hard target difficulties,across groups. B) Mean decision bias (lnβ ) for both the easyand hard target difficulties across groups. Error bars representthe standard error of the mean. . . . . . . . . . . . . . . . . . 48Figure 11 The proportion of false alarms in the test phase, across groups.Error bars represent the standard error of the mean. . . . . . . 49Figure 12 The classification images generated across participants for eachtarget, in each condition. Blue shows the CIs generated for thepassive condition, red shows the CIs generated for the activecondition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Figure 13 The relationship between IWC for participants’ CI and each in-dividual trial image, and the proportion of images in each binidentified as containing a target. Data from the active conditionare plotted in red, and data from the passive condition are plot-ted in blue. All data were aggregated into 400 equally sizedbins for each condition. Lines show the sigmoid functions thatwere fit to the data for each condition. . . . . . . . . . . . . . 52Figure 14 The relationship between the IWC for participants’ assignedtarget and each individual trial image, and the proportion ofimages in each bin identified as containing a target. Data forthe active condition are shown in red, and data for the passivecondition are shown in blue. All data were aggregated into 400bins of equal size per condition. . . . . . . . . . . . . . . . . 54xivFigure 15 A summary of the practice phase of the experiment. A) Sensi-tivity (d′) of the neural network over 50,000 trials of easy andhard target difficulties. B) Decision bias (lnβ ) of the neuralnetwork over 50,000 trials of easy and hard target difficulties.These point estimates have no variability. . . . . . . . . . . . 63Figure 16 A histogram of the confidence ratings given by the neural net-work for the presence of a target image in a white noise trialimages. The red vertical line denotes the decision cutoff forthe liberal condition. The blue vertical line denotes the deci-sion cutoff for the conservative condition. . . . . . . . . . . . 64Figure 17 The classification images generated from the responses of theneural network that was searching for an X in each trial. Blueshows the CI generated for the conservative condition, whereresponses were generated from the conservative decision bound-ary. Red shows the CI generated for the liberal condition,where responses were generated from the active decision bound-ary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Figure 18 The relationship between IWC for the neural network’s CI andeach individual trial image, and the proportion of images ineach bin identified as containing a target. Aggregated responsesfrom the liberal condition are plotted in red, and aggregatedresponses from the conservative condition are plotted in blue.All data were aggregated into 400 equally sized bins for eachcondition. Lines show the sigmoid functions that were fit tothe data for each condition. . . . . . . . . . . . . . . . . . . 67xvFigure 19 The relationship between the IWC for neural network’s targetX image and each individual trial image, and the proportionof images in each bin identified as containing a target. Aggre-gated responses from the liberal condition are shown in red,and aggregated responses from the conservative condition areshown in blue. All data were aggregated into 400 bins of equalsize per condition. . . . . . . . . . . . . . . . . . . . . . . . 68xviGlossaryFMRI functional magnetic resonance imagingMEG magnetoencephalographyCI Classification ImageSGD stochastic gradient descentRSA representational similarity analysisHCNN Hierarchical Convolutional Neural NetworkIWC the Image-Wise CorrelationxviiAcknowledgmentsA big thank you to Dr. Jim Enns, whose patience and careful guidance made thiswork possible. Through the two years that spanned this work, his direction keptmy efforts on course. He led by example, showing me how to develop the skillsthat make a great scientist.I would also like to thank Ana Pesquita, whose mentorship helped me gainconfidence in myself, and Stefan Bourrier, who supported me in too many ways tocount.Last but not least, I would like to thank my incredible parents, without whomI would not have become who I am today. Their support and guidance have beencrucial to my professional and personal growth.xviiiDedicationI dedicate this work to Jenny Wan, who tolerated my busy lifestyle during this workand helped to keep me sane through the process.xixChapter 1: Introduction1.1 Superstitious perceptionWhen studying a system as complex as the human brain, one method that can beused to understand its inner mechanisms is to investigate the instances in which itperforms unexpectedly. One example of unexpected behaviour in the human visualsystem is the phenomenon of apophenia. Apophenia is defined as the experienceof seeing patterns or connections in random or meaningless data, and was firstcoined by Klaus Conrad, a German neurologist in the early 20th century (Fyfe,Williams, Mason, & Pickup, 2008; Shermer, 2008). In the visual domain, thisphenomenon describes instances in which one experiences a percept of an objectthat is not there. Common examples of visual apophenia include seeing animalsin the clouds, or seeing Jesus in a piece of toast (Liu et al., 2014). Importantly,it is highly unlikely that the percept is accurate in these cases. That is to say thatJesus is likely not in the toast, and animals are unlikely to have grown in size andbegun floating amongst the clouds. Nonetheless, the experience of apophenia isnear ubiquitous. Indeed, it is possible that the visual system is detecting evidencethat supports these percepts that were given rise by aphophenia, despite the groundtruth being an absence of such perceived objects. The difficulty comes in trying toquantify the extent of such evidence, if any is present.One human behavioural technique that has been developed to investigate thisphenomenon, and allows for quantification of available evidence is the supersti-tious perception task (Gosselin & Schyns, 2003). This technique involves asking1people to identify targets in noisy images (like static on an old TV). Participantswere asked to identify a target (e.g., the letter ′s′) within the noisy images that werepresented to them one at a time in the center of a computer screen. Participantswere asked to report whether they saw the target of interest within each noise im-age as it was presented. Unbeknownst to them, the images contained no target atall and consisted of pure random noise. Surprisingly, when Gosselin and Schyns(2003) added together all of the images identified as having a target in them, andthen subtracted all of the images that had not been identified, a representation ofthe target became visible (Gosselin & Schyns, 2003). The image that resulted fromsuch a process was named a classification image (CI). This CI was claimed to bean image of the internal representation of the target that participants were trying todetect within the image (Gosselin & Schyns, 2003).The stated theoretical goal of the superstitious perception task (Gosselin &Schyns, 2003) is to measure the internal representation participants are using whenthey falsely detect target signals in noise. The rationale for this method is built uponthe same logic that underpins the use of reverse correlation to estimate the receptivefield properties of individual neurons (see Ringach & Shapley, 2004, for a review).In the initial introduction of reverse correlation to that field, researchers stimulatedsingle neurons from the primary visual cortex of cats with random white noisevisual input to the retina. Then, they recorded the timing of each of the neuron’sspikes in response to the stimulation (Jones & Palmer, 1987). The strength ofthe white noise input in the moments leading up to a spike allowed researchers tomodel a given neuron’s receptive field, by computing the average stimulus patternwithin a specified time window before the neuron spiked. This spike-triggeredaverage was interpreted as an ideal template for the neurons receptive field becauseit represented the stimulus that was most likely to cause the neuron to fire.Critics of this interpretation cautioned that before accepting spike-triggered av-erages computed in this manner, it was important to describe how well the receptivefield predicted the neurons activity (DeAngelis, Ohzawa, & Freeman, 1993). Thiswas done by first estimating a neuron’s receptive field using reverse correlation.Once the receptive field was estimated, one could predict the neurons firing rate2in response to the stimulus used to estimate the receptive field. This is a test ofinternal validity, and it is illustrated in Figure 1C. Internal prediction was thus ameasure of how well the estimated receptive field predicted the neuron’s responseto the individual stimuli that had contributed to the estimation. If the neural re-sponse of individual stimuli was not sufficiently predicted by the receptive field,one could conclude that the calculated receptive field of the neuron was not ade-quately descriptive of the neurons function.DeAngelis et al. (1993) performed this internal prediction technique on the re-ceptive fields of cat early visual neurons and was able to show a high degree ofpredictive accuracy, thus validating the procedure for further use. Reverse correla-tion, used to estimate the receptive fields of single neurons in this manner, becameinstrumental for interpreting neuron spike trains and reconstructing representationsof the receptive fields of visual and auditory neurons alike (Jones & Palmer, 1987;Ringach, G., & Shapley, 1997; Theunissen et al., 2001, for example).Ringach and Shapley (2004) suggested another important step to the validationprocedure when they argued that one could also use CIs to make predictions of aneuron’s response to novel stimuli if its receptive field was already known. This isillustrated in Figure 1D. This could be done by performing an external predictionof a neuron’s firing rate. That is, one could evaluate a hypothesized receptive fieldin response to novel stimuli. The critical difference between the two approaches iswhether the stimuli used to generate the receptive field are the same as the stimulifor which you are trying to predict responses (internal prediction) or whether thereceptive field was generated with different stimuli (external prediction). Whenthe same stimuli are used for creation of the CI and for its evaluation of predictivepower, the measurement procedure is vulnerable to capitalization on chance (ormeasurement error). When novel stimuli are tested instead, a more stringent test ofa receptive field’s predictive power is given.In the domain of superstitious perception, participants do not respond withspikes, but rather with false alarms or correct rejections in a signal detection task.In addition, it is not possible to assess a temporal component of these responsesanalogous to neural firing times. Instead, discrete responses are made that depend3only on the current image. If the response is a false alarm, we gain informationimplying that the image is target-like in some respects. This is comparable to aneuron’s spike. Moreover, if the response is a correct rejection, we gain informa-tion implying that the image is not target-like. Note that this is even more informa-tion than we get from a neuron, which typically only responds in a positive way byincreasing its activity above some baseline level. Gosselin and Schyns (2003) putthis extra information to good use in applying reverse correlation to psychophysi-cal data, by creating an average-target image, and an average-non-target image, asshown in Figure 1B.Furthermore, participants in a behavioural experiment do not necessarily be-have in the same way as neuron when it comes to being presented with ambiguousstimuli (Rieth, Lee, Lui, Tian, & Huber, 2011), and instead they require a moreelaborate ruse to respond to superstitious perceptions. Rieth et al. (2011) achievedthis by having two practice phases: (1) participants were shown targets that weresimple to identify in noise — they called this the easy practice session; and (2)participants were asked to identify targets that were very heavily overlaid withnoise, making them hard to identify — they called this the hard practice session.The goal here was to encourage participants to see superstitiously. Notably, forour purposes, this also aids in stressing to participants that the targets are homoge-neous. The target shown to them at the beginning of the experiment is identical toall targets hidden in the noise, a consideration that helped to increase the strengthof our experimental design.There is one final difference worth noting between the application of reversecorrelations in neural and perceptual contexts. In the neural realm, a predictionis calculated based on an aggregate of spikes over time, in the units of firing rate(Hz). This is because while firing rate is predictable, neural firing time seems to bestochastic in nature (Dyan & Abbott, 2001). In the superstitious perception realm,we can instead interpret the aggregate prediction as a likelihood of false alarm for agiven trial. For this reason, it is best to aggregate trials into bins of comparable tar-get similarity when estimating the extent of explained variance in responses. Sincethere are no guidelines for bin size, a range of bin sizes will be used throughout4the following experiments to ensure that bin size has no effect on our conclusions.The statistical logic behind the reverse correlation technique, however, remains thesame as that in the neural spiking domain (Gosselin & Schyns, 2001).Given the conclusions reached in the reverse correlation literature with respectto the need to test a generated CI to evaluate its predictive capacity (DeAngelis etal., 1993; Ringach & Shapley, 2004), there is a critical step missing in the proce-dure described by Gosselin and Schyns (2003). Once a CI is generated, one shouldattempt to predict participants’ false alarm rates in response to new white-noiseimages, called external predictions, in order to estimate the degree to which the CIexplains participants’ perceptions. Without this crucial step, one does not knowthe degree to which the CI over-fit the data from which it was generated. With theimportance of this step in mind, we intended to implement the external predictionstep for the superstitious perception task. In doing so, we aimed to verify that anyconclusions made as a result of the CIs generated from this task apply to stimulibeyond the specific noise images from which the CI was generated. Furthermore,we intended to compare the conclusions drawn from internal predictions with thosedrawn from the novel external prediction procedure.1.2 Task strategy and its influence on task outcomeThe influence of task strategy has been well documented with respect to its effectson task outcomes for a variety of cognitive tasks (Jacoby & Brooks, 1984; Marcel,1983; Smilek, Enns, Eastwood, & Merikle, 2006; Van Selst & Merikle, 1993; Whit-tlesea & Brooks, 1994). In the realm of categorization, participants were observedto be more accurate during an item classification task when adopting a feature-focussed, analytical strategy when compared to a more wholistic, non-analyticalstrategy (Jacoby & Brooks, 1984; Whittlesea & Brooks, 1994). Even when per-cepts never reached conscious awareness, the effects of unconscious stimuli ontask outcome were stronger when participants let the stimulus ‘pop’ into mind, asopposed to trying to determine whether a stimulus was present in a preceding trial(Van Selst & Merikle, 1993). In the field of visual search, participants’ search5efficiency can be modulated by the instructions they are given before the task. Par-ticipants were faster and more accurate in their search when told to let the target oftheir search ‘pop’ into their minds as opposed to when they were told to activelysearch for the target of interest (Smilek et al., 2006). While the literature makesit clear that task strategy influences task outcome, the optimal strategy appears tovary by task, with analytical, active strategies being the better strategy in sometasks, but not others.Some researchers suggest that the differences in task outcome are due to dif-ferences in the recruitment of the various cognitive systems across task strategies(Smilek et al., 2006). For instance, a passive search strategy in a visual search taskmay recruit executive functions to a smaller degree than an active strategy (Smileket al., 2006). Such an interpretation would be in line with the previous findings(Jacoby & Brooks, 1984; Marcel, 1983; Van Selst & Merikle, 1993; Whittlesea& Brooks, 1994), where the optimal strategy is consistent for a given task, butvaries between different tasks. This is could be because tasks differ with respectto the requirement of various cognitive functions to effectively complete the taskin question. For example, a memory task might benefit more from an increase inrecruitment of memory functions, while an executive functions task may see lessof an improvement from the same change in recruitment. In this way, identifyingthe optimal strategy for a given task can give important insights into the cognitivefunctions upon which the task depends.To our knowledge, there has yet to be an investigation into the effects of taskstrategy on the outcomes of the superstitious perception task. Given that the super-stitious perception task is intended to measure internal representations (Gosselin &Schyns, 2003), identifying the optimal task strategy for the task may give insightinto the functions from which those internal representations are derived. To thisend, we investigated the effect of two task strategies on the outcomes of the task.In a passive task strategy, participants were asked to let the target ‘pop’ into theirmind when viewing the trial images. In an active task strategy, participants wereasked to actively search for evidence of the targets’ presence in each trial image.If a passive task strategy is more effective than an active task strategy, the inter-6nal representations are likely intrinsic to the visual system. However, if the activetask strategy is most effective, then executive functions are likely involved in themanifestation of these internal representations.1.3 Hierarchical convolutional neural networks and thehuman visual systemRecent advances in computing technology have expanded the capabilities of com-putational models used for simulating neural connections in the human brain. Theadvent of highly parallelized processing means that we can increase the number ofsimultaneously simulated neurons. From these new technologies, a particular set ofmodels have emerged that model the way that the human visual system translatespoints of light detected in the retina into a representation of the world around it.These models, known as Hierarchical Convolutional Neural Networks (HCNNS),have even been used to predict neural activity relating to object recognition withremarkable success (Cichy et al., 2016; Khaligh-Razavi & Kriegeskorte, 2014;Yamins et al., 2014). However, we are not aware of any attempts to test thesemodels by using their outputs to predict human behaviour beyond testing catego-rization accuracy. Comparing the behaviours of humans and those of these modelsat a deeper, more fundamental level will allow us to probe the accuracy of the mod-els. Do HCNNs behave similarly to humans on a task that probes the boundaries ofthe abilities of the human visual system to detect patterns? In this paper, we hopeto provide an answer to such a question by creating a neural network that is ableto perceive superstitiously. We then hope to compare the outcomes of the supersti-tious perception task between humans and HCNNs to identify their similarities anddifferences.The hierarchical nature of HCNNs mimic our current understanding of the ar-chitecture of the human visual system, with neurons at higher levels of the pro-cessing hierarchy having receptive fields that are larger relative to the visual input,and that select for more complex visual features (Dyan & Abbott, 2001; Yamins &DiCarlo, 2016). These HCNNs are a class of neural network models that search for7patterns in incoming data using filters, labelling sections of data that match a spe-cific pattern through high levels of activation in the individual nodes representingthat area of the input. In order to distinguish the biological neurons of the humanbrain from the simulated neurons of the HCNNs, the simulated neurons will bereferred to as nodes. In the first layer of the network, a set of filters are applied tothe input image, to detect patterns of interest (e.g., Gabor filters of various orienta-tions). For each section of the image, there is a node whose activity represents thedegree to which the input image matches one of the filters. When taken together,this means that the first layer of an HCNN is a set of spatial maps, one for each filter,indicating the locations in the image where patterns in the input image match thepattern in a filter. The activations of the nodes in the first layer are then used as theinput data for the second layer, where the same procedure is followed. In essence,the first layer of the network searches for specific patterns in the image, the secondlayer searches for specific second-order patterns in the collection of patterns iden-tified by the first layer, the third layer searches for specific patterns in the collectionof patterns identified by the second layer, etc.One example of an implemented HCNN is that of Krizhevsky, Sutskever, andHinton (2012), commonly known as AlexNet, which consists of 8 layers. In thefirst 5 layers (layers 1-5), the filters are smaller than the width of the previouslayer, meaning that the spatial locations are preserved from one layer to the next.The next 2 layers (layers 6 & 7) have filters that cover the entirety of the previouslayer, meaning that each node has a unique filter - each node is sensitive to patternsin the entirety of the image. The final layer is another layer where the filters arethe size of the previous layer except that it is used as the classification layer. Inthe classification layer, each node corresponds to a class that the neural networkis trying to identify in the images. The activation of nodes in this layer can beinterpreted as the networks’ confidence in the presence of that object in the inputimage. This procedure is similar to the human visual system in that the receptivefields of neurons in the visual system increase in size relative to the visual input asone increases from low-level areas to higher level areas (Dyan & Abbott, 2001).The similarities between HCNNs and the human visual system go further.8AlexNet, trained to identify a wide array of object classes, shows early recep-tive fields that show a surprising similarity to receptive fields of neurons in V1(Krizhevsky et al., 2012). These computational receptive fields of the HCNN werenot determined intentionally, but were settled upon as the most effective meansof classifying the images it was presented through a learning process known asstochastic gradient descent (SGD; Bottou, 1991, 2010). In short, SGD involvesshowing an example image to the network and estimating how wrong the neu-ral network was in classifying that image. This margin of error is then reduced bymodifying the strength of the connections between nodes within the network, eitherstrengthening or weakening each connection, so that the network becomes closerto being correct in its classification of the objects in the images it was shown. Thisprocess is repeated with new images until a certain error threshold is met and theresearchers conclude that the model has reached the approximate global minimumin classification error (i.e. it is as accurate as its going to get without over-fitting tothe specific images that it has been shown).Early attempts to model the primate visual system have now been recognizedas falling under the umbrella of HCNN models (e.g., HMAX; Yamins & DiCarlo,2016). These early models share similar functional architecture as the human vi-sual system and AlexNet, but were constructed by attempting to mimic the filters,or receptive fields, of visual processing neurons measured by neural recordings inthe early visual areas of various mammals (Poggio & Riesenhuber, 1999). This dif-fers from AlexNet in that it does not adapt to best distinguish between the stimuliit is presented. Instead, all features for which the system is sensitive are hand-designed to match those observed in the human visual system.It is possible that the higher order representations are similar in the human vi-sual system and in HCNNs, but as the neural signal is abstracted from the rawimage input, it becomes harder to interpret its receptive field, thus making it harderto compare with human neurons. This is mostly because of the difficulty in visu-alizing and interpreting the receptive fields of these higher-order artificial neurons.In the first layer of the HCNN, the filters are directly referencing the image itself.As a result, the receptive fields of those neurons can be directly interpreted, as the9strength of connection between a given pixel and a given neuron is directly relatedto what the neuron is detecting in the image. However, once you abstract beyondthe first layer of the network, the interpretability of the strength of connectionsbetween pixel and image weakens. This is because the upper layers receive theactivation of the lower level layers as input.The similarity in higher order representations of visual stimuli, in both HC-NNs and the human visual system, can be compared using an analysis known asrepresentational similarity analysis (RSA; Kriegeskorte, Mur, & Bandettini, 2008).This technique involves comparing activity within a system in response to a widevariety of stimuli. From there, the similarity in activation between each combina-tion of pairs of stimuli gives an indication of the degree to which the system seesthose two stimuli as being similar. For example, consider a system that displays thesame pattern of activation in response to a house and a car. Due to the similarity inactivation, one can conclude that the system represents houses and cars as similarobjects. On the other hand, if the system shows very different patterns of activity inresponse to those stimuli, one could say that the system represents cars and housesvery differently. If two systems show similar patterns of representational similaritybetween a wide variety of stimuli, one can consider them to be similar with respectto how they represent objects.In a HCNN, the activity of each layer of the network can be saved directlyto memory during the simulation, since its activity is being calculated during thesimulation. As such, HCNN activity is easy to acquire, and is easily reproducible.Since there is no noise introduced in most HCNNs, the resulting activation for agiven image will be exactly the same each time that it is presented. In addition,it is important to note that the unit of such activity is arbitrary, typically rangingfrom 0 to 1, or -1 to 1. However, it can be interpreted as a relative firing rate forthe purposes of these types of comparisons, with the minimum value indicating theleast activation, and the maximum value indicating the most activation.In contrast to the HCNNs, neural activity in the primate visual system is muchharder to measure as there is no direct access to neural activity, but it can be done bya variety of approaches, including functional magnetic resonance imaging (FMRI)10(e.g., Kriegeskorte et al., 2008), magnetoencephalography (MEG) (e.g., Cichy etal., 2016), or acfEEG (e.g., Cichy et al., 2016), each of which has advantages anddisadvantages. Critically for RSA analysis, though, is that the analysis techniquewould be more or less the same regardless of the method of neural measurement- the only thing that would change is the interpretation of the results. In fMRI,which has good spatial resolution but poor temporal resolution when measuringneural activity, the representational similarity matrix generated from the data willbe predominantly interpreted as a measurement of similarity in spatial encoding(i.e. which neurons fired). For MEG or EEG, which have better temporal resolutionbut worse spatial resolution than fMRI, the data will be predominantly interpretedas a measurement of similarity in temporal encoding (i.e. when neurons fired).Recently, researchers have attempted to compare activity in the macaque visualcortex with that of the high-order layers of a HCNN using RSA (Khaligh-Razavi& Kriegeskorte, 2014; Yamins & DiCarlo, 2016). Yamins and DiCarlo (2016)showed that a performance optimized HCNN was able to predict spatial patternsin neural activity in the macaque inferotemporal cortex more accurately than otherleading computational models of human vision (Yamins & DiCarlo, 2016; Yaminset al., 2014). In addition, recent comparisons of the primate visual system andHCNNs showed that low-level activity in a similar HCNN model predicted low levelvisual activity in the macaque visual system, both temporally and with respect tospatial activity (Cichy et al., 2016). Taken together, these observations have ledsome to believe that they are the best models of human vision currently available(Yamins & DiCarlo, 2016). At a behavioural level, we see similar trends in thecomparison between human vision and our computational models. Recent HCNNmodels have been able to achieve accuracy in view-variant classification tasks thatare indistinguishable from human performance on the same task (Kheradpisheh,Ghodrati, Ganjtabesh, & Masquelier, 2016). All of this evidence suggests that theHCNN model may be well on its way to accurately modelling human vision bylooking at the internal activations of the two systems while performing the task forwhich the HCNN was trained.On the other hand, there is also evidence suggesting that HCNNs do not fully11model the functional processes that underlie human vision. For instance, recentattempts to compare human image content similarity judgments to those made byan HCNN suggest that parallels in HCNN performance may not extend beyond im-age classification (Peterson, Abbott, & Griffiths, 2016). Moreover, in experimentstrying to compare the view-variant classification accuracy between the model andthe human visual system, visual masks were used to reduce re-entrant processingeffects (Kheradpisheh et al., 2016). This procedure, suggests that the HCNN visualmodel is an incomplete model of the human visual system, through the implicationthat the re-entrant processes in human vision are not included in the HCNN model.When examining the architecture of HCNNs, processing proceeds linearly and uni-directionally, with information being passed in only one direction up the hierarchy,from early layers to late layers. Re-entrance involves information also being passeddown from high-order processing areas to lower-order areas (Fahrenfort, Scholte,& Lamme, 2007). However, it is important to note that these findings do not pre-clude the possibility that the HCNN is simply an incomplete, but accurate, modelon human vision. Further comparisons such as those made by Peterson et al. (2016)and Kheradpisheh et al. (2016) should be considered in order to identify the lim-its of the HCNN-human visual system comparison. In the following experiments,such comparisons were used to identify whether the human visual system and theHCNNs use similar criteria to perform object recognition.There are some notable similarities between the human visual system andHCNN models. Often, they are compared at the level of representational simi-larity in neural activity (Cichy et al., 2016; Khaligh-Razavi & Kriegeskorte, 2014;Yamins & DiCarlo, 2016). To our knowledge, there has been relatively little com-parison between the behavior of human and computational models of vision. Thepresent work aimed to further the process of making such behavioural comparisonsbetween systems. Specifically, we aimed to compare the behavior of HCNN andhuman participants in a way that did not mean directly testing the neural networkon a task for which it was trained, using a superstitious perception paradigm (Gos-selin & Schyns, 2003).In summary, the primary goal of this manuscript was to evaluate the validity12of the superstitious perception technique (Gosselin & Schyns, 2003) with respectto two assumptions. The first assumption is that the Classification Images (CIS)generated using this technique reflect a measure of the internal representations ofa participants’ visual system. In order to test this assumption, we attempted topredict participants’ false alarm rates when they attempt to detect a target withinnovel white noise stimuli. If the predictions were accurate, we can conclude thatthe technique met this assumption. However, if prediction of participants’ falsealarm rates was unsuccessful, then it suggests that the CI does not represent theinternal representation of a target. The second assumption is with respect to thetask procedure. Specifically, in the original experiments by Gosselin and Schyns(2003), participants were not given a particular strategy to use during the task. Webelieve that by identifying the optimal task strategy, we can locate the cognitivefunctions from which the internal representations are derived. Here, we investigatethe effect of task strategy on both internal and external prediction ability.In addition to the primary goal, we also compared the behaviour of an HCNN tothat of the human visual system during the superstitious perception task. The goalwas to probe the ability of a HCNN to model human vision by testing its ability toperceive superstitiously as humans do. This test has a number of advantages overprevious comparisons between HCNNs and humans. Most notably, the HCNN willbe trained on data that are not generated by human responses, as is the case forobject recognition in natural scenes. If the HCNN bears similarities in response tohumans despite its training on objective, non-human data, then we can concludethat the model itself, and not the training on human data, is what is leading to theHCNN - human visual system similarities.13Chapter 2: Experiment 1The overall rationale and design of Experiment 1 are illustrated in Figure 1. Themain goal of this experiment was to replicate the methodology of Gosselin andSchyns (2003) in a context in which the participant’s search images were known.As such, participants were assigned to search for one of the four specific targets(shown in Figure 1A) in noisy images. Each participant was shown an image oftheir specific target prior to beginning the task, as is illustrated for the X-target(shown in Figure 1A). This differed from Gosselin and Schyns (2003), who did notcontrol the target shapes participants searched for beyond telling them to “look forthe letter ′S′, present in half of the following images.” The four search targets inthe experiment were selected so that participants could be assigned to targets thatdiffered from one another only a little (e.g., the X and the italic X share a highdegree of similarity) or by a large margin (e.g., the X and the two plus signs have alow degree of similarity).Given that there was a defined target for participants to use as an internal repre-sentation, we reasoned that fewer trials were necessary to achieve the same strengthof test. The goal here was to bring the relative power of the two experiments intoa comparable range. Since the targets are more clearly defined in our paradigm,fewer trials would be required to reduce the statistical noise to levels comparableto that of Gosselin and Schyns (2003). As such, just under half as many trials werecollected per target in this experiment, 8,000, relative to the Gosselin & Schynsexperiment, 20,000.Figure 1B illustrates the process of creating a classification image from the14Figure 1: A visual summary depicting the difference between reverse corre-lation, internal prediction, and external prediction. (A) Participants wereassigned to one of four possible targets. (B) Reverse correlation is theprocess of generating the CI by looking at the sum of stimuli that led to afalse alarm, and subtracting the sum of stimuli that did not lead to a falsealarm. (C) Internal prediction is the process of using a generated CI topredict participants responses to the stimuli used to generate the CI. Thisprocess is referred to as backward prediction in the neural spike train liter-ature. Such a process will lead to an over-estimation of the CI’s goodnessof fit. (D) External prediction is mathematically identical to internal pre-diction, but differs in using a generated CI to predict responses to novelwhite noise stimuli. This process is referred to as forward prediction inthe neural spike train literature, and serves to give a better estimation ofgoodness of fit. In this experiment, however, we will use participants’assigned target images in lieu of a generated CI.15responses of participants who were trying to detect a given target. The four classi-fication images, corresponding to each of the four targets, could then be evaluatedto see how much they differed from one another. The average difference in cor-relation magnitude between target CI and low versus high similarity non-targetswas thus a way to quantify the extent to which differences in target images werereflected in differences in the corresponding classification images.In Experiment 1, we first repeated the Gosselin and Schyns (2003) methodof preparing CIs using 50 pixel by 50 pixel images and participants’ responses toa signal detection task using the reverse correlation logic. But we did not endour analysis there, at steps A and B in Figure 1, as Gosselin and Schyns (2003)did. Stopping here implies that the success of a superstitious perception study isevaluated primarily by an intuitive visual comparison made by the experimenter(and/or the reader). Rather, we went on to use the CIs to predict participants’responses in two more ways, in an effort to put the evaluation of superstitiousperception results onto a more objective footing.First, we used the CIs to make internal predictions, following the logic ofDeAngelis et al. (1993) research on single cell receptive fields (Figure 1C). Thisallowed us to estimate how much variance in the responses made to the individualstimuli giving rise to the CI were explained by this model. Second, we followedthe suggestion of Ringach and Shapley (2004) to compute external predictions forcomparison with the internal predictions (Figure 1D). Any discrepancies betweenthese two measures would provide an estimate of the extent to which reverse cor-relations on their own are capitalizing on chance variation in the data, as suggestedby DeAngelis et al. (1993).In summary, Experiment 1 had three goals. The first goal was to replicatethe Gosselin & Schyns procedure in a context where the shape of the participant’starget images are known in advance and can be compared for their similarity to oneanother. The second goal was to examine the internal prediction accuracy of theclassification images to estimate the degree of predictive accuracy achievable bythe generated image, as argued by DeAngelis et al.. The third goal was to comparethese internal predictions with external predictions, in order to estimate the degree16to which reverse correlations are fortuitously capitalizing on measurement error.The addition of the internal and external prediction analyses removes the visual-interpretive role of the experimenter in estimating the strength of a superstitiousperception effect.The design of Experiment 1 leads to two sets of hypotheses. The first set con-cern the CIs and their correlation with the images that gave rise to them (reversecorrelations). If the CIs generated in the superstitious perception task differentiatebetween the low-similarity target images assigned to participants (e.g., X versusplusses), then the superstitious perception technique of Gosselin and Schyns canbe considered to have low-resolution discriminability between differing internalrepresentations. If the resulting CIs are further able to differentiate between be-tween high-similarity targets (e.g., X from italic X), then the technique must havehigh-resolution discriminability. The alternative outcome of no discriminabilityamong the four target shapes would imply that the procedure cannot differentiatebetween internal representations that differ in the way a + differs from an x, thusseriously undermining the logic of the superstitious perception task.A second set of critical predictions concern the relative strength of internal andexternal predictions. If the external predictions are similar in accuracy to the inter-nal predictions, then it suggests that the CI is a reliable estimate of a participant’sinternal representation. Alternatively, if the external prediction accuracy is muchsmaller, it suggests that the CIs generated from the superstitious perception task areover-fitting the data that comes from presenting noise images to participants and,as such, are less accurate estimates of participant’s internal representations thanhas previously been assumed (Gosselin & Schyns, 2003).172.1 Methods2.1.1 ParticipantsNine participants (five females) between the ages of 18 and 25 were recruited froma paid subjects pool at the University of British Columbia. The only restriction onparticipation was that participants must have had normal- or corrected-to-normalvision, verified by self-report upon participants’ arrival to the lab. All participantswere tested one at a time on the same computer.2.1.2 Stimuli and proceduresParticipants came into the lab twice and repeated the same set of procedures eachtime. This was done in order to collect 4000 experimental trials from each partic-ipant, without participants becoming overly fatigued during the experiment. Eachtime the participants came into the lab, they completed a two-phase task. Thistask included a practice phase to acquaint participants with the procedures, andan experimental phase in which there were no targets, and from which a CI wasconstructed for each participant. The practice phase was further broken into twoconditions: (1) an easy condition, intended to allow for practice identifying the tar-get for which they were searching; and (2) a hard condition, intended to encourageparticipants to identify images in the experimental phase by giving them the im-pression that targets were present but very difficult to identify. This practice phaseprocedure is similar to that used by Reith et. al 2011, who were able to obtainreliable results in a similar task.While two phases were completed per session, the core task throughout theexperiment was the same. All participants were asked to view images, one at atime, in the centre of the screen. They were then asked to identify whether a target,shown to them earlier, was present in the noisy image in front of them. They werethen asked to indicate its presence or absence with different key presses.18Participants were randomly assigned to one of four groups: X, italic X, +,and italic + such that there were at least two participants per group. The groupnames denote the type of target the participants were given during the experiment.All participants completed the task on an iMac late 2009 model, and the task wasconstructed using Matlab 2010a and PsychToolBox v11 (Brainard & Vision, 1997;Kleiner et al., 2007; Pelli, 1997). Images presented to participants occupied about2◦ of visual angle, and were 50px X 50px in resolution.The target images in the practice phase were generated by overlaying whitepixel noise over top of the target image in question. Non-targets were generated bysimply overlaying pixel noise onto a solid grey square with the same average lumi-nance as the target image. The contrast in the target image was adjusted betweenthe easy and hard practice phases in order to make the target harder to identify. Theoverlay noise was generated by sampling from a normal distribution with a mean of0, and a standard deviation of 0.2 for each pixel in the image which was then addedto the target image. Notably, the average luminance of each trial image stayed con-stant throughout the experiment, with minor variations due to random sampling ofthe noise. During the experimental phase of the experiment, all stimuli were gen-erated by overlaying white pixel noise atop the solid grey square, whose luminancematched that of the average luminance in the target.2.1.3 Data analysisSensitivity (d′) and decision bias (lnβ ) was used to assess the degree to which par-ticipants understood the task and participated as per the instructions. One partici-pant who showed near zero sensitivity to the target in the easy and hard conditionswas removed from subsequent analyses.Classification Images (CIS) were generated for each participant based on theresponses they gave for each trial image during the test phase. This was performedby adding together all of the trial images in which a target was identified and thensubtracting all of the images in which a target was not identified. This procedure isexactly as performed by Gosselyn & Schyns 2003. When visualising the images,19a gaussian filter was used to smooth the image in order to eliminate random noise,yielding a fully processed CI for each target. Since the spatial frequency of thetargets presented in this experiment were at most 3 cycles per image, the gaussianfilter was configured to eliminate frequencies higher than 3 cycles per image. Thisspatial smoothing procedure is as performed in previous literature (Gosselin &Schyns, 2003).These CIs were then assessed with regards to their similarity to the target forwhich each participant was searching. This was done by taking the IWC betweeneach participant’s CI and each of the 4 targets. The predicted target for each partic-ipant was the target with which their CI was most correlated. CIs that discriminatedthe participant’s assigned target from the other, irrelevant targets in the experimentwere taken to be accurate.Once the CIs were created, and their accuracy was quantified by using them topredict the participants’ targets, another approach was used to assess consistency inresponses across participants. We attempted to internally predict the likelihood thateach participant would select a given trial image as containing a target. This wasdone first by computing Image-Wise Correlations (IWCS) between the CI and thegiven trial image. These IWCs were interpreted as an index of the degree to whicha given trial image matched the participant’s generated CI. They were generatedfor every single image that was presented to participants. Then the trials weresorted from least to greatest with respect to their IWCs, and then split into 400 binsin order to estimate the likelihood of an image with a given IWC being selectedas containing a target. The number of bins was selected due to its allowance forenough trials in each bin for the resulting proportion to be considered continuous.To ensure that bin size had no effect on the reliability of any resulting relationships,the same procedures were also conducted using 50 and 1000 bins. The proportionof images identified as having a target was then calculated for each bin, alongwith the mean IWC value. The relationship between these two variables was thenobserved and interpreted. This procedure, predicting participants’ responses usingthe CI will be referred to as internal prediction.A similar procedure was used to estimate predictive power of the target image20that was shown to participants during the training phase of the experiment. To dothis, the IWC between the participants’ respective target image and each trial imagewas calculated. This yielded a single number for each trial image, interpretedas the degree of similarity between the trial image and the target the participantswere assigned to identify. As performed with the CIs, the same binning procedurewas used to generate the proportion of images selected as containing a target, asthe degree of similarity between the image and the target varies. This procedure,predicting participants’ responses using the known target they were trying to detect,will be referred to as external prediction.2.2 Results2.2.1 Practice phaseAnalysis of participants’ individual performance on the practice phase was as-sessed by examining sensitivity (d′) and decision bias (lnβ ) for both the easy andhard conditions. The purpose of this preliminary assessment is to identify partici-pants who did not understand the task. One participant was removed due to havingnear zero sensitivity in both the easy and hard conditions.Figure 2A shows that there were generally very high sensitivity scores (d′ >4),and that there was no difference between easy targets (mean d’ = 4.15, SE = 0.24)and hard targets (mean d′ = 4.34, SE = 0.25), t(15.95) = -0.56, p = 0.59, 95% CI =[-0.92 , 0.54]. Figure 2B shows participants’ average decision bias, as assessed bylog transformed β (lnβ ) in both the easy and the hard conditions. The results showthat participants were generally more liberal (more willing to say ‘target present’when uncertain) in the easy condition (mean lnβ = -0.77, SD = 0.3889410) than inthe hard condition (M = 0.59, SD = 0.44), t(15.73) = -2.31, p = 0.035.21012345easy  hardExperiment_Phased'A101easy  hardExperiment_PhaseLog()BFigure 2: Summary of sensitivity and decision bias in the practice phase ofExperiment 1. A) Mean sensitivity by target visibility. B) Mean decisionbias by target visibility. Error bars represent the standard error of themean.2.2.2 Test phaseParticipants detected targets in the test phase of the experiment, where no truetargets were presented, at a rate of 32.8% on average (SD = 18.0%). This comparesto a false alarm rate of 4.9% (SD = 5.0%) in the easy and 1.3% (SD = 1.5%) inthe hard conditions of the practice phase, suggesting that participants were indeedseeing superstitiously throughout the task. They also took considerably longer torespond in the test phase (M = 1661ms, SD = 1022ms), than in the practice phase(M = 793ms, SD = 145ms). They responded to an average of 2729 trials (SD =854) within their 2 sessions of 1 hour. No sensitivity or criterion measures couldbe computed in the test phase because no targets were presented, meaning that allresponses were correct rejections or false alarms.Figure 3 shows the CIs generated for each target. CIs were calculated for each22target by adding together all of the images in which the target was identified andthen subtracting the images in which no target was identified, as performed by Gos-selin & Schyns (2003). In order to display them as an image, linear transformationswere performed on the data, subtracting the minimum value, and then dividing bythe range. This was performed so that the minimum pixel luminance was 0, andthat the maximum pixel luminance was 1. Finally, a spatial filter was applied toallow spatial frequencies around 3 cycles per image, as performed by Gosselin andSchyns.Target CIs were correlated with the 4 possible targets that were assigned toparticipants to estimate the degree to which they could be used to discriminatebetween the target it was generated to represent and those that it was not. Tothis end, an image-wise correlation was calculated between the CI and each targetimage for each CI. These IWCs were grouped into two groups: those generatedwith the target associated with the given CI, those generated with the target thathad high similarity to the target associated with the given CI, and those that had lowsimilarity to the target associated with the given CI. For example, for the ‘x’ CI,the IWC associated with the ‘x’ target image was grouped as a “correct target”, theIWC associated with the italic ‘x’ was grouped as a “High Similarity Target”, andthe two associated with the ”+” targets were grouped as “Low Similarity Targets”.Overall, the average IWC in the “Correct Target” group, r = .33 (SD = .04), wasthe same as the average IWC in the “High Similarity Target” group, r = .33 (SD =0.06). The average IWC in the “Low Similarity Target” group , r = 0.25 (SD = .05),was significantly smaller than the “Correct Target” group, t = 2.30, p = 0.045.These data, therefore, replicates important aspects of previous studies that usedsuperstitious perception as a technique (Brown-Iannuzzi, Dotsch, Cooley, & Payne,2017; Gosselin & Schyns, 2003, to name a few). The question we turn to now iswhether the data can be used to make predictions about which specific imagesparticipants will be most likely to report a false alarm (report a “target” when itis not present). The slope of relationship when trying to make predictions withparticipants’ CIs will serve as a measure of the degree of sensitivity for the targetparticipants’ mental representations. The steeper the slope, the more consistent23Figure 3: Classification Images generated across participants with each tar-get. Note that the target that participants were trying to detect is difficultto discern, but present.participants were in their selections.Evaluation of internal prediction accuracyNext, each image that was shown to a participant was correlated with the partici-pant’s CI in order to obtain an Image-Wise Correlation. The IWCs were calculatedby computing the correlation between the two images at the pixel-by-pixel level.This meant that each image shown to a participant in the experiment was assigneda correlation value indicating the degree to which it was similar to (i.e., resem-bled) the mental representation that the participant was using to make decisions. Apositive correlation indicated that the pattern of dark and light pixels in a specificimage are similar to the pattern of dark and light pixels in the target image; a neg-ative correlation indicated that dark regions in the target image tend to correspondwith light regions in the trial image, and vice versa.Figure 4 shows the proportion of ‘target’ responses (i.e., the proportion of im-ages in each bin in the data in which a false alarm occurred) plotted against theaverage IWC between each image and the CIs shown in Figure 2. The data in theirentirety consisted of nearly 32,000 points (8 participants x 4000 trials, minus afew dozen missing trials for some participants). The 32,000 points were groupedinto various numbers of equally sized bins. This was both to smooth the data, andto convert the binary response variable into a pseudo continuous variable, inter-24pretable as the proportion of images selected as containing a target. To understandthe effect of the binning procedure on the observed relationship, we examined therelationship between ‘target’ responses and the IWCs at bin numbers of 50, 400,and 1000. The results are shown in Table 1.These correlations were obtained after fitting a sigmoid function to the data,yielding a slope of 93.76 (SE = 1.34), and a centre constant of 0.008 (SE = 0.0002).The maximum of the sigmoid function was fixed at 1, as participants cannot iden-tify more than 100% of the images shown as containing a target. Using these con-stants, a sigmoid transformation was performed on the IWCs between the target im-ages and CIs. There was a strong linear relationship between sigmoid-transformedIWCs and proportion of ‘yes’ responses by participants, r(398) = 0.99, p <.001.It is important to note that this relationship is a strong over-estimate of the truepredictive power of the CI, as we are making predictions of responses using infor-mation generated from the decisions we are trying to predict. The crucial pointhere is that the slope of the sigmoid function is very steep, suggesting that partici-pants could easily discriminate between which images matched their criterion, andwhich images did not.Table 1: The effect of bin size on the correlation between the proportion ofimages selected as containing a target and the IWC between the trial imageand the CI.Number of Bins Observed Correlation Statisical Significance50 r(48)= .99 p <.001400 r(398) = .99 p <.0011000 r(998) = .98 p <.001These results suggest that participants are quite consistent in their internal rep-resentations, as they’ve been measured by the superstitious perception technique.The next question we wanted to answer is whether our technique of estimating theirmental representations is actually externally valid, given that they were assigned atarget to try to identify in the images. In other words, given that we’ve told themwhat to look for in the images, we have some degree of certainty of what theirmental representation should be. To estimate the degree of match between these25−0.10 −0.05 0.00 0.050.00.20.40.60.81.0Image−wise correlation between trial image and classification imageProportion of "yes" responses by participantsFigure 4: A scatter plot showing the relationship between the IWC of the CIand each trial image, and the proportion of images in each bin selectedas containing a target. The relationship is sigmoid in nature, and the linedrawn in is the sigmoid of best fit, with slope of 93.76 (SE = 1.34) and acenter constant of 0.008 (SE = 0.0002).two concepts, we will attempt to predict participants’ responses to trial images us-ing the image of the target they were assigned. The degree to which we can predictparticipants’ responses using the targets they were assigned will help quantify howaccurate our mental representations are.Evaluating image selection using external predictionWith the target image that was shown to participants to depict what they weretrying to detect, a second set of IWCs were calculated. The IWCs were calculatedby computing the correlation between the two images at the pixel-by-pixel level.The IWCs between the target image and each trial were binned into 400 bins.26The proportion of images identified as containing a target was calculated for eachbin. This allowed for a continuous measure of the participants’ collective likeli-hood of perceiving a target within an image that has a given IWC score with thetarget that they were trying to detect. A positive IWC indicated that the respectivepixels in the given trial image and the target image are positively correlated, anddark regions in the target image tend to correspond to dark regions in the trial im-age. In contrast, negative IWC indicated that the respective pixels are negativelycorrelated, and dark regions in the target image tend to correspond with light re-gions in the trial image. In other words, the more positive the IWC becomes, themore “target-like” the image at a pixel-by-pixel level. The more negative the IWCbecomes, the less ‘target-like’ the image, again at the pixel-by-pixel level.Figure Figure 5 shows the proportion of ‘yes’ responses (i.e. the proportionof images in each bin in which a target was identified) plotted against the averageIWC in each bin.The data shown in Figure 5 are from binning the data into 400 discrete cate-gories. At this level of resolution, there was a moderate linear relationship betweenthe target-generated IWCs and the proportion of ‘yes’ responses among partici-pants, r(397) = .34, p <.001. One outlier was identified in this dataset, shown ingrey. This outlier was removed for the subsequent analyses. However, its removalhad no influence on the conclusions drawn. In its most detailed form, the dataconsisted of 32,000 points (8 participants x 4000 trials). In the non-aggregatedform, responses are binary in nature with participants having to choose either toidentify a target or not, even if their confidence is approximately 50/50. In order tosmooth the noise in the data, and to better estimate participants’ confidence in theirresponse, we examined the relationship between ‘target’ responses and the IWCsat bins ranging from 50 to 1000. The results are shown in Table 2. The apparentmagnitude of the relation increases as the number of bins is decreased, though thereliability of the relationship is approximately equal across bin size. This is pre-sumably because the number of data points (degrees of freedom) trades off withthe increase in measurement error that accompanies increasing sample sizes. Theimportant point is that there is a reliable relation between false alarms rates and the27−0.08 −0.06 −0.04 −0.02 0.00 0.02 0.04 0.060.00.10.20.30.40.5Image−wise correlation between trial image and target imageProportion of "yes" responses by participantsFigure 5: The relationship between the likelihood of an image being selectedas a target, and the IWC between that image and the target image.resemblance of trial images to the target template defined by the experimenter.Table 2: The magnitude and reliability of relation between the proportion ofimages selected as target and the IWCs between the trial image and thetarget image, for the data binned at various levels of resolution.Number of Bins Observed Correlation Statisical Significance50 r(47)= .65 p <.001400 r(397) = .34 p <.0011000 r(997) = .23 p <.0012.3 DiscussionIn Experiment 1, we had three primary goals. First, we aimed to reproduce theresults of Gosselin and Schyns (2003) and their superstitious perception technique.We also hoped to estimate the resolution of the technique through the use of tar-28gets with varying degrees of similarity. Second, we introduced the concept ofinternal prediction in order to estimate the degree of participants’ confidence intheir responses, and thus the strength of their internal representation of the target.This was quantified using the slope of the sigmoid function of their responses. Fi-nally, we validated the concept of external prediction for predicting participants’responses to stimuli using a known representation of their internal template thatwas not derived from their responses.Classification imagesThe CIs generated from the responses in the testing phase were less clear thanprevious work (Gosselin & Schyns, 2003; Rieth et al., 2011, , for example). Thiswas not surprising, however, since we used less than half the number of imagesper target. Attempts to evaluate which target was associated with which CI yieldedmixed results. On one hand, the CIs were able to distinguish between high- andlow-similarity targets. However, they were unable to distinguish between high-similarity targets and the correct target. This could be explained by participants notmentally distinguishing between these targets - participants remembered both the“x” and italic “x” as an “x”. On the other hand, the failure to distinguish betweenhigh similarities could indicate that the technique itself is not sensitive enough tomake such distinctions. Regardless, these results are in line with those of previousstudies.Internal predictionUsing internal prediction, or predicting responses associated with the images usedto create the CI, was useful in this case to estimate participants’ consistency in re-sponse. Aggregating data into bins allowed for the conversion of the binary yes/noresponses into a continuous proportion of false alarms. The false alarm could beinterpreted as an indirect measure of participants’ confidence in a target presence.Since it is unlikely that participants’ confidence in their perceptions is binary, thebinning procedure allowed us to estimate their aggregate confidence that a target is29present. This aggregate confidence was measured as a false alarm rate for a givenIWC range. Analyzing the relationship between this proportion of responses and theIWC between targets and trial images allowed us to get a measure of participants’consistency in responses. For example, a steeper slope of the resulting sigmoidcurve between the two variables suggests that the participants had a clearer, moredistinct decision boundary between their decisions. On the other hand, a shallowerslope would have indicated more uncertainty. In this case, there was a slope of93.76. It remains to be seen if this is a comparatively steep or shallow slope. Fur-thermore, the internal prediction process verified that the IWC technique is valid inestimating the likelihood of image selection in the superstitious perception task. Itsuggests that the pixel-wise similarity between the trial image and the CI containsinformation about the confidence of participants’ decisions.External predictionMeanwhile, external prediction, or predicting participants’ responses based on theimage of the target they were assigned, is a more rigorous validation of the super-stitious perception technique. Since participants’ targets were explicitly assignedto them, we knew the ground truth of the target that participants were using in theirdecisions. As such, we can estimate the degree to which the participant’s responsescould be predicted by the image of the target they were trying to detect in the im-ages. Not surprisingly, the external prediction method proved much less effectivein predicting participants’ responses than the internal prediction method. This islikely because the images we were using to calculate the IWC were not directlygenerated from the participants’ responses in the external prediction method, andthus there can be no over-fitting of the data. The moderate relationship betweenthe proportion of images identified as containing a target and the IWC for the tar-get image suggests that the technique does, in fact, measure the internal templateto some extent. It is important to note that the experimenter will not always haveaccess to the exact image that participants were trying to detect. In Gosselin &Schyns’ experiment, 2003, for example, participants were told to look for an ′S′ inthe image, but the exact qualities of the “S” that participants were trying to detect30was unknown. In such CI experiments, where participants’ internal templates fora concept are unknown, a CI for each participant should be used to predict theirresponses to novel white noise stimuli.The near-ceiling performance in the practice phase of the experiment suggeststhat it was too easy - even in the hard condition - and suggests that participantsnever struggled to detect the target. This means that participants were never en-couraged to make false alarms due to increasing task difficulty, as intended in thehard phase. If participants were not making a considerable number of false alarmsduring the experiment, this would suggest that they were not seeing superstitiously.If this were the case, participants false alarms could have been the result of a fewtrials in which the noise looked extremely target-like. However, participants werestill reporting false alarms in the testing phase at an average rate of 32%. Partici-pants were reporting these superstitious perceptions at a rate comparable to thosereported in previous work (Gosselin & Schyns, 2003; Rieth et al., 2011). For thesereasons, the ease of the second practice phase is unlikely to have had a significantimpact on the results of the testing phase.Despite the promising results further validating the superstitious perceptiontechnique, there is still a considerable amount of unexplained variance in the par-ticipants’ responses that is not explained by their assigned template. Some of thisvariance could be explained by slight differences in the participants’ memory of thetarget (Holden, Toner, Pirogovsky, Kirwan, & Gilbert, 2013, for example). How-ever, given the amount of unexplained variance in responses, it is likely that thereis another mechanism at play that is being indirectly measured by this technique.One possible mechanism that could be impacting the responses of participantsis their decision-making process. Over the past decade, it has come to light thatdecision-making processes and response strategies can dramatically alter observedresponse patterns (e.g., Lifshitz, Bonn, Fischer, Kashem, & Raz, 2013; Smilek etal., 2006). This is particularly likely, given the reported strategy of a participantin Gosselin & Schyns’ original paper: “[She] simply waited to see if the [target]jumped out at [her].” In chapter 3, we examine the effects of encouraging a par-ticular task strategy for participants on the outcome of a superstitious perception31task.32Chapter 3: Experiment 2In Experiment 2, we aimed to employ the techniques we developed, in order toevaluate the effect of task strategy on the measurement error associated with thesuperstitious perception task. This allowed us to ask pointed questions about theaccuracy of associated experimental outcomes when performed under different cir-cumstances.The purpose of the superstitious perception task is to study the latent repre-sentations of objects or concepts within the visual system. Therefore, one keyassumption is that the task is dependent predominantly upon the human visualsystem. However, research areas investigating behavioural measures in both thevisual and attention domains have shown the influence of task strategy on experi-mental outcomes (e.g., Seli, Jonker, Solman, Cheyne, & Smilek, 2013; Smilek etal., 2006). The mechanism behind these differences is thought to be differentialrecruitment of secondary brain networks to the task. Unfortunately, there is limitedinformation available with respect to the effect of task strategy on the outcomes ofthe superstitious perception task.Testimony reported by Gosselin and Schyns (2003) suggests that participantsseem to naturally adopt a passive strategy to the task, claiming to check if the targetof interest “jumps out at them” in each trial image during the task. Such a findingsuggests that participants adopted a strategy of trusting their initial impression ofeach trial image throughout the task. Unfortunately, subsequent studies using thesuperstitious perception technique neglect to specify the type of strategy adoptedby participants while performing the task, and thus this hypothesis remains uncon-33firmed. However, a task strategy that is dependent on initial impressions could beinduced by reducing the exposure time to each trial image, forcing a reliance onfirst impressions to guide responses (Kheradpisheh et al., 2016).The goal of this experiment was therefore to probe the effect of task strategy, asinfluenced by image presentation time, on the extent to which the CIs generated bythe superstitious perception task estimate participants’ internal representations. Bytesting the influence of task strategy on participants’ response patterns and gener-ated CI accuracy, we can gain a more thorough understanding of how strategy influ-ences the ability of the superstitious perception task to achieve its stated purpose:to measure and make participants’ latent representations accessible. The previousliterature, including the testimony of participants from Gosselin & Schyns work2003, suggests that adopting a passive strategy will yield the most accurate CIs. Ifthis is the case, it suggests that a passive task strategy should be used in order tomake the measurements of latent representations as accurate as possible.3.1 Methods3.1.1 Participants8 participants (seven females) between the ages of 20 and 26 were recruited fromthe paid subjects pool at the University of British Columbia. Only participants withnormal or corrected-to-normal vision were recruited, verified by self report uponparticipants’ arrival in the lab. All participants were tested one at a time on thesame computer.3.1.2 Stimuli and proceduresAll stimuli were generated in a manner identical to that described in chapter 2.Similarly, all procedures were identical except for the following deviations. First,the trial images were limited in presentation time, meaning that the trial image34would appear on screen for only 500ms upon trial onset, after which the imagewould be replaced with a blank screen. No visual mask was used after the stimuluspresentation during this experiment due to the fact that the stimuli were noise, andpresenting a noise mask may lead to serious confounds (Eriksen, 1980). The trialwould not end until the participant responded, but participants were encouraged torespond as quickly and as accurately as possible. This change in procedure wasmade to encourage participants to respond based on intuition so as to minimizenon-visual decision cues (e.g., deciding to identify a target because they haven’treported seeing one in a while). In other words, the change in procedure wasan attempt to force participants to follow a similar decision making strategy asreported by the participants in work by Gosselin and Schyns (2003).3.1.3 Data analysisData analysis procedures were identical to those described in chapter 2.3.2 Results3.2.1 Practice phaseFigure 6A shows that there was a large difference in average sensitivity (d′) be-tween two conditions of the practice phase, with the easy condition (mean d′ =3.36, SD = 0.89) being significantly larger than the hard condition (mean d′ = 0.38,SD = 0.44), t(7) = 10.14, p <.001. Figure 6B shows that there was also a differencein average decision bias, where participants tended to be more liberal (more likelyto identify a target as being present when uncertain) in the easy condition (M =-0.71, SD = 0.88) than in the hard condition (M = 0.22, SD = 0.16), t(7) = -3.18, p= .016.3501234easy  hardTarget Difficultyd'A101easy  hardTarget DifficultyLog()BFigure 6: Summary statistics for the training phase of the experiment. A)Mean sensitivity score across the easy and hard practice conditions. B)Decision bias, lnβ , across the easy and hard practice conditions. Errorbars represent the standard error of the mean.3.2.2 Test phaseSimilar to chapter 2, participants mistakenly identified an average of 36.4% (SD= 6.4%) of trials as containing a target, despite no trial containing a true target, incontrast to an average false alarm rate of 10.7% (SD = 11.5%) in the easy condition,and 26.9% (SD = 14%) in the hard condition. However, contrary to chapter 2,participants took less time to respond in the test phase (M = 503ms, SD = 197ms)than in the practice phase (M = 855ms, SD = 254ms). Overall, participants tookmuch less time to respond in experiment 2 compared to chapter 2. This suggeststhat the manipulation worked, forcing participants to think less about the reasoningbehind their response. No sensitivity or bias scores could be calculated as no targetswere presented in the test phase, making all responses either a correct rejection, ora false alarm.36Figure 7: Classification images for each target assigned to participants.These CIs are generated by aggregating across the participants that wereassigned to each target. The label under each image denotes which targetparticipants were assigned.Figure 7 shows the CIs generated for each target in Experiment 2. These im-ages were generated by adding together all of the trial images that were selectedas containing a target, and then subtracting all of the trial images that were notselected as containing a target. Visually, these CIs have much less visual similarityto the targets that they were trying to detect.Target CIs were correlated with each of the 4 possible targets that were assignedto participants. The goal of this procedure was to assess the ability of the CIs toidentify the target that participants were assigned. These correlations were groupedinto 3 groups. First, the correlation between the target CI and the image of thecorrect target were put into a “Correct Target” group. Next, the correlation betweenthe CI and the target with high similarity (X and italic X, or + and italic +) wereput into a “High Similarity” group. Finally, the remaining correlations betweenthe CI and the two low similarity targets were put into the “Low Similarity” group.Overall, the “Correct Target” group had an average correlation of r = .15 (SD =.22), the “High Similarity” group had an average correlation of r = .12 (SD = .19),and the “Low Similarity” group had an average correlation of r = .15, (SD = .15).37Evaluation of reverse correlation sensitivityUsing the participant’s classification images, the same technique as described inchapter 2 was applied to try to predict participants’ response to each trial basedon their CI. An image-wise correlation was calculated between each trial imageshown to participants and that participants’ classification image. This image-wisecorrelation serves as an index of how similar each trial image is to the participants’CI. This approach yielded nearly 32,000 data points (8 participants x 4000 tri-als, with some participants missing a few dozen responses due to time constraintsduring data collection). In order to convert the binary yes/no responses into a con-tinuous proportion of targets identified, trials were grouped into equally sized binsbased upon similar image-wise correlation values. For each bin, a proportion of‘target-present’ responses was calculated. A few bin numbers, 50, 400, and 1000were calculated with the intention of understanding the effect of bin number on theobserved relationship.Figure 8 shows a very strong relationship between the proportion of ‘yes’ re-sponses by participants and the image-wise correlation between each trial imageand the participants’ respective CIs, grouped into 400 bins. Due to the sigmoidalnature of the curve in chapter 2, a sigmoid was fit to the data. The resulting sig-moid had a slope of 17.58 (SE=0.2), and a center constant of 0.008 (SE = 0.0005).The relationship was equally reliable across bin size, though the magnitude of theobserved effect decreased slightly as the number of bins increased. The relation-ship’s reliability likely stays consistent due to the trade-off between sample-sizeand effect size. A summary of the relationship observed at the various bin sizes isshown in Table 3.Table 3: How the number of bins effects the strength of the relationship be-tween the IWCs for participants’ CI and their trial images.Number of Bins Observed Correlation Statistical Significance50 r(48)= .997 p <.001400 r(398) = .98 p <.0011000 r(998) = .96 p <.00138−0.15 −0.10 −0.05 0.00 0.05 0.100.00.20.40.60.8Image−wise correlation between trial image and classification imageProportion of "yes" responses by participantsFigure 8: Scatter plot showing the relationship between the IWC for each trialimage and the participants’ CI with 400 bins. Solid line shows the sigmoidfunction that was fit to the data. The slope for the sigmoid was 17.6(SE=0.2), with a center constant of 0.008 (SE = 0.0005).Evaluating image selection using forward correlationUsing the same technique as described in chapter 2, an image-wise correlation wascalculated between each trial image shown to participants and the target that theywere assigned to identify (x, +, italic x, or italic +). This image-wise correlationserves as an index of how similar each trial image is to the participants’ assignedtarget. In order to convert the binary responses into a continuous proportion, trialswere binned into 400 bins based upon similar image-wise correlation values. Foreach bin, a proportion of ‘target-present’ responses was calculated.Figure 9 shows a weak relationship between the proportion of ‘target-present’responses in each bin, with the mean image-wise correlation for each bin. The pos-itive linear relationship between proportion of ‘target-present’ responses and meanimage-wise correlation in each bin for 400 bins was significant, r(398) = .10 , p =39−0.06 −0.04 −0.02 0.00 0.02 0.04 0.060.20.30.40.5Image−wise correlation between trial image and target imageProportion of "yes" responses by participantsFigure 9: Scatter plot showing the relationship between the IWC for each trialimage and the target participants were assigned. Data is aggregated into400 bins..042, 95% CI = [.003, .198]. Again, as in chapter 2, the chosen number of bins wassomewhat arbitrary. A range of bin sizes were chosen to map the effect of bin sizeon the magnitude of the relationship, with 50 bins and 1000 bins being calculatedin addition to the 400 bins mentioned above. The effect appeared to be similarto that observed in chapter 2, with the magnitude of the relationship increasing aswe decreased the number of bins. However, the reliability remains approximatelyequal. The relationship, in this case, seems to be only somewhat reliable as it hov-ers around the decision boundary for statistical significance. In addition, it is muchsmaller than the relationship observed in chapter 2. A summary of the observedrelationship for the different numbers of bins can be found in Table 4.40Table 4: How the number of bins effects the strength of the relationship be-tween the IWCs for participants’ assigned target and their trial images.Number of Bins Observed Correlation Statistical Significance50 r(47) = .25 p = .080400 r(397) = .10 p = .0421000 r(997) = .06 p = .0453.3 DiscussionThe changes to the experimental procedure in Experiment 2 were intended toforce participants to make decisions based upon their first impressions. This wasachieved by limiting the time for which each trial image was presented on thescreen. Specifically, participants saw each image for 500ms before the image dis-appeared, and participants were asked to make a decision as to the presence of a tar-get. This manipulation seems to have decreased deliberation as the mean responsetime per trial decreased considerably between chapter 2 (at 1661ms) and this study(at 503ms). Additionally, the standard deviation of response times decreased aswell, from 1022ms in chapter 2 to 197ms in this experiment. This supports the hy-pothesis that a variety of strategies were employed in experiment 1, whereas herewe have successfully encouraged participants to use a specific strategy — namelyone of intuition.The observed differences in sensitivity between the two target difficulty ses-sions of the practice phase suggest that the difficulty between the two sessions waslikely to have been more effective at drawing participants to see superstitiously, asobserved by Rieth et al. (2011). However, there seems to have been only a negli-gible increase in rates of superstitious perception, with the false alarm rates in thetest phase (36.4%) being comparable to that observed in chapter 2 (32.8%). Thissuggests that such a method is not necessary for inducing superstitious perceptionsin participants.There was very little signal detected in the target CIs, shown in Figure 7. Thecorrelations between these images and the target images confirmed this, with neg-41ligible difference between the average IWC for “correct targets” and for “low simi-larity”. This suggests that the task lost its utility in detecting participants’ internaltemplates when they were asked to make fast decisions about the presence of thetarget in the image. One CI, however, seems to bear some similarity with its asso-ciated target image: the ’+’ target CI. This could be due to the possibility that themanipulation did not universally have the effect of increasing passive engagementin searching for evidence of a target. Nonetheless, as a group, this study showeda marked loss in the ability to measure participants’ mental representations of thetarget compared to chapter 2.This decrease in performance is corroborated by the noticeable decrease inslope observed in the backward prediction of participants’ responses based on theIWCs between their CIs and each trial image. The slope, 17.58, was much shallowerthan that observed in chapter 2, at 93.76. This suggests that participants were lessconsistent with their responses, and had a less rigid decision boundary betweendeciding whether an image contains a target or not. We would expect that theseparticipants would require far more trials in order for a template of their internalrepresentation would become visible, as their internal consistency in responses hasdecreased.When evaluating participants’ performance using the forward prediction, asimilar picture becomes evident. Participants’ responses were predicted less ac-curately using their assigned target image in this study than in chapter 2. In theprevious study, participants’ responses could be predicted with moderate accuracy,r(397) = .34, when the 400 bins were used. However, in this study, we see thatthis accuracy has dropped to a very low level of accuracy, r(397) = .10, suggestingthat their responses were guided less by the target that was assigned to participants.On the other hand, participants’ decision criteria in the easy section of the practicephase remained largely unchanged, ln(β ) = -0.77, when compared to chapter 2,ln(β ) = -0.71. These results could be explained by an increase in the internal noiseof participants’ decision making (Neri & Levi, 2006). In other words, by reducingparticipants’ exposure time to the trial images, their information about each imageis decreased and the amount of uncertainty that participants experience is increased.42This would lead to a noisier decision process despite the criterion for the decisionremaining unchanged. In chapter 4, we will address this question directly.43Chapter 4: Experiment 3The goal of experiment 3 was to build upon the assessment of task strategy onsuperstitious perception performed in experiment 2. One weakness in experiment2 was the lack of direct manipulation of task strategy, opening the possibility forconfounds. To strengthen our design, and build a better understanding of the ef-fect of task strategy on the accuracy of generated classification images estimatinginternal representations, a direct manipulation of strategy was necessary. In thisexperiment, a direct manipulation was employed by comparing and contrastingsuperstitious perception performance between two groups who were assigned dif-ferent task instructions, wherein differing task strategies were suggested.To perform a direct manipulation of task strategy, we borrowed from a previousmanipulation performed for the same purpose (Smilek et al., 2006). This experi-mental manipulation had two groups: (1) In the passive group, participants wereasked to follow their intuition — intended to minimize the engagement of execu-tive functions when performing the task; (2) In the active group, participants wereasked to critically examine each trial in the task — intended to maximize the en-gagement of their executive functions during the task. Thus, at a theoretical level,this experiment was examining the effect of executive function engagement on theefficacy of the superstitious perception task in estimating internal representations.444.1 Methods4.1.1 Participants14 participants (nine females) between the ages of 18 and 66 (mean age = 26.57,SD = 12.02) were recruited from the paid subjects pool at the University of BritishColumbia. Only participants with normal or corrected-to-normal vision were re-cruited. This was verified by self report upon participants’ arrival in the lab. Allparticipants were tested on at a time on the same computer, in the same room.4.1.2 Stimuli and proceduresStimuli were generated in the same manner as described in chapter 2. Procedureswere also the same, except an additional manipulation was added, and the numberof targets was reduced from 4 to 2, using only the standard “x” and “+” from chap-ters 2 and 3. Specifically, in order to more precisely manipulate the task strategyemployed by participants, the participants were assigned into either an Active taskgroup, or a Passive task group. The task was identical for both groups except for adifference in instructions at the beginning of the experiment.The instructions for the two groups were adapted from Smilek et al. (2006).They used a similar set of instructions to influence search strategy in a visual searchtask. The only changes that were made to the instructions were changing the con-text to be relevant to the superstitious perception task, and not a visual search task.In the Active group, participants were told:The best strategy for this task, and the one that we want you to use inthis study, is to be as active as possible and to “search” for the targetin the image as you look at the screen. The idea is to deliberatelydirect your attention to determine your response. Sometimes peoplefind it difficult or strange to “direct their attention” but we would likeyou to try your best. Try to respond as quickly and accurately as you45can while using this strategy. Remember, it is very critical for thisexperiment that you actively search for the target in the image. If youcannot find any reason to suspect that the target is present, respondthat the target is not present.In the Passive group, participants were told:The best strategy for this task, and the one that we want you to use inthis study, is to be as receptive as possible and see if the target popsinto your mind as you look at the image. The idea is to let the displayand your intuition determine your response. Sometimes people find itdifficult or strange to tune into their gut feelings but we would likeyou to try your best. Try to respond as quickly and accurately as youcan while using this strategy. Remember, it is very critical for thisexperiment that you allow the target to just pop into your mind. If thisdoes not happen, respond that the target is not present.Participants from both groups completed identical tasks, and differed only intheir task instructions.4.1.3 Data analysisData analysis procedures were identical those performed in chapter 2 and chapter 3,but were adapted slightly to accommodate the two instruction groups. Namely,all analyses were performed on the two groups separately, and then the resultsstatistically compared.Analysis of the practice phase was performed using a mixed effects ANOVA,with instruction group as a between-subject factor with two levels (Active and Pas-sive), and target difficulty as a within-subject factor with two levels (easy and hard).One mixed effects ANOVA was used to compare sensitivity (d’) across factors, anda second was used to compare decision bias, ln(β ).46The analysis of the test phase was identical to those of chapter 2 and chap-ter 3, but was split across instruction group, so the passive condition was analyzedseparately from the active condition. When comparisons between groups was nec-essary, an appropriate statistical test was employed. When comparing the meanproportion of false alarms across groups, a t-test was performed. When comparingthe slopes of the two sigmoid curves in the backward prediction analysis, a z-testwas performed to test the statistical significance of the difference between the twoslopes. Finally, a Fischer’s Z-test was used to compare the strength of the rela-tionships between target image IWCs and proportion of false alarms for the twogroups.4.2 Results4.2.1 Practice phaseFigure 10A shows that there was no interaction between condition and target diffi-culty for sensivity (d’), F(1,22) = 0.58, p = .46. Nor was there an effect of conditionon sensitivity, F(1,22) = 1.52, p = .23. There was, however, an effect of target dif-ficulty, with participants being more sensitive to targets in the easy condition thanin the hard condition, F(1,22) = 16.90, p <.001. Similarly, Figure 10B shows thatthere was no interaction between condition and target difficulty in decision bias(lnβ ), F(1,22) = 0.05, p = .82. There was no main effect of condition on decisionbias, F(1,22) = 0.68, p = .42, nor was there a main effect of target difficulty, F(1,22)= 2.23, p = .15.4.2.2 Test phaseFigure 11 shows that the difference in proportion of false alarms between con-ditions was trending towards significance, with the active condition (mean Pyes =44.7%, SD = 28.8%) reporting more false alarms than the passive condition (meanPyes = 20.3%, SD = 11.7%), t(9.74) = 2.17, p = .056. In addition, participants in4701234easy hardTarget Difficultyd'A1.51.00.50.00.5easy hardTarget DifficultyLog() GroupActivePassiveBFigure 10: A summary of the practice phase of the experiment. A) Mean sen-sitivity (d’) for both the easy and hard target difficulties, across groups.B) Mean decision bias (lnβ ) for both the easy and hard target difficultiesacross groups. Error bars represent the standard error of the mean.the two conditions responded about equally quickly, with participants in the activecondition (mean RT = 1438ms, SD = 942ms) being non-significantly slower thanparticipants in the passive condition (mean RT = 855ms, SD = 364ms) in response,t(9.54) = 1.60, p = 0.14.Figure 12 shows that there were much clearer patterns in the CIs from the pas-sive condition than in those from the active condition. These CIs were generatedthe same way as they were in experiments 1 and 2, by adding together all of theimages that were false alarms, and then subtracting all of the images that werecorrection rejections. Some linear transformations were applied to each image toensure that the minimum pixel luminance in the image was 0, and the maximumwas 1.In order to estimate how well the generated CIs were able to discriminate be-480.00.20.4Active PassiveConditionProportion of false alarm responsesFigure 11: The proportion of false alarms in the test phase, across groups.Error bars represent the standard error of the mean.tween their respective target and other potential targets, an IWC was calculatedbetween the CI and each of the 4 possible targets from experiments 1 & 2 (x, italicx, plus, italic plus). Discriminability ratings were averaged across assigned targetsfor each condition to estimate how well the overall condition was able to generatediscriminating CIs. In both conditions, discriminability was very low. In the activecondition, CIs had an average similarity score of r = 0.003 with their respectivetargets, a similarity score of r = - 0.001 for the high similarity target, and r = -.00249for the low similarity score. In the passive condition, CIs had an average similarityscore of r = 0.045 for their respective targets, a score of r = 0.040 for high sim-ilarity targets, and r = 0.027 for low similarity targets. The CIs from the passivecondition, while having a low degree of similarity with any targets, were able todiscriminate more effectively between the possible targets than were CIs from theactive condition.Evaluation of reverse correlation sensitivityAn IWC was calculated between each participant’s CI and each of their trial im-ages. The IWC was interpreted as being a measure of the similarity between eachtrial image and the CI. Larger, positive IWCs indicate higher similarity betweenthe participant’s CI and a trial image. Negative IWCs indicate lower similarity be-tween the participant’s CI and the trial image. These IWCs were used to predict thelikelihood of a participant identifying a target in a trial image.Figure 13 shows that the sigmoidal relationship between the IWC for a partici-pants’ CI and each trial image, and the proportion of false alarms for images withthat IWC value, grouped into 400 bins, is steeper in the active condition than in thepassive condition. This was verified by fitting a sigmoid curve to each condition,showing that the active condition data, slope = 48.7 (SE = 0.7), did indeed have asteeper slope than the passive condition, slope = 36.7 (SE = 0.6). A test of signifi-cance of the difference between two independently observed slopes was conductedas suggested by Cohen, Cohen, West, and Aiken (2003), z = 13.22, p <.001. How-ever, the relationship between bin size and the strength of the relationship observedin chapter 2 and chapter 3 was still present in both conditions, with a very stronglinear relationship between the proportion of false alarms in each bin, and sigmoidtransformed IWC values in both the passive and active conditions. These values areshown in Table 5.The aforementioned results and relationships were also tested with 50 and 1000bins to examine how they are affected by bin size. The observed slope for thetwo conditions remained the same for all 3 bins, suggesting that bin size does not50Figure 12: The classification images generated across participants for eachtarget, in each condition. Blue shows the CIs generated for the passivecondition, red shows the CIs generated for the active condition.51−0.10 −0.05 0.00 0.05 0.10 0.150.00.20.40.60.81.0Image−wise correlation between trial image and target imageProportion of "yes" responses by participantsActivePassiveFigure 13: The relationship between IWC for participants’ CI and each indi-vidual trial image, and the proportion of images in each bin identifiedas containing a target. Data from the active condition are plotted in red,and data from the passive condition are plotted in blue. All data wereaggregated into 400 equally sized bins for each condition. Lines showthe sigmoid functions that were fit to the data for each condition.affect these values. In addition, the relationship between the proportion of falsealarms and the sigmoid-transformed mean IWC for each bin decreased slightly asthe number of bins increased. The decrease in the strength of the relationship islikely due to less aggregation of the data as the number of bins increases, andthus a weaker signal-to-noise ratio is obtained. Nonetheless, the reliability of theserelationships remained relatively constant, likely due to an increase in degrees offreedom as the bin number increases, that strongly suggests that the relationship ispresent. These results are shown in Table 552Table 5: How the number of bins effects the strength of the relationship be-tween the IWCs for participants’ assigned target and their trial images.Condition Number of Bins Observed Correlation Statistical SignificanceActive50 r(48) = .998 p <.001400 r(398) = .0.99 p <.0011000 r(998) = .97 p <.001Passive50 r(48) = .997 p <.001400 r(398) = .98 p <.0011000 r(998) = .94 p <.001Evaluating image selection using forward correlationA second set of IWCs were calculated between the target assigned to each partici-pant, and their respective trial images. As was the case for the IWCs involving theCIs, these can be interpreted as the similarity between a given trial image and thetarget assigned to the participant. As in chapter 2 and chapter 3, the IWC scorescan be used to predict the likelihood with which participants will identify a targetin a given trial image.Figure 14 shows the relationship between the IWC with participants’ target im-ages, and the proportion of false alarms in each of 400 equally sized bins of IWCobservations. In the Passive condition, there was a moderate relationship betweenthe observed IWC and the proportion of false alarms, r(397) = .25, p <.001. How-ever, the active condition shows no relationship between the IWC with participants’target images and the proportion of false alarms, r(397) = .01, p = .87. These rela-tionships were significantly different from one another, z = 3.45, p <.001, but thedifference is in the opposite direction than expected. To verify that this differenceis not a result of the number of bins, the two correlations were re-calculated for 50and 1000 bins. The results were similar, showing the same trends as observed inthe previous experiments, but with reliability remaining consistent. A summary ofthese results can be found in Table 6.53−0.05 0.00 0.050.00.20.40.60.8Image−wise correlation between trial image and target imageProportion of "yes" responses by participantsActivePassiveFigure 14: The relationship between the IWC for participants’ assigned targetand each individual trial image, and the proportion of images in each binidentified as containing a target. Data for the active condition are shownin red, and data for the passive condition are shown in blue. All datawere aggregated into 400 bins of equal size per condition.Table 6: How the number of bins effects the strength of the relationship be-tween the IWCs for participants’ assigned target and their trial images.Condition Number of Bins Observed Correlation Statistical SignificanceActive50 r(47) = .04 p = .76400 r(397) = .01 p = .871000 r(997) = .00 p = .95Passive50 r(47) = .57 p <.001400 r(397) = .25 p <.0011000 r(997) = .15 p <.001544.3 DiscussionThe goal of experiment 3 was to directly compare the accuracy of the classificationimages with respect to estimating participants’ internal templates across task strate-gies. To do so, we used task instructions similar to those of Smilek et al. (2006) todirectly compare two task strategies using the techniques developed in chapter 2.The two task strategies included a passive strategy, which is presumed to reducethe recruitment of executive functions to the task, and an active strategy, presumedto increase the recruitment of executive functions to the task. Investigations of thisnature in other domains has suggested that task strategy can have a notable impacton task outcome (Smilek et al., 2006). Furthermore, indirect comparisons betweenExperiments 1 and 2 suggested that if participants adopt a passive task strategy,classification images would be less accurate. However, the work of Smilek et al.(2006) suggested the opposite outcome; Adopting a passive task strategy yieldsbetter outcomes in visual search.4.3.1 Internal predictionAs observed in chapters 2 and 3, the relationship between the proportion of falsealarms and trial image similarity to the classification image was sigmoid in naturein both conditions. Moreover, the slope of the sigmoid curve in the active condi-tion was significantly steeper than that in the passive condition, a result that is inagreement with the results of chapter 3, where we induced a passive task strategyby reducing stimulus presentation time. chapter 3 also yielded a shallower slope inthe sigmoid function in comparison to the unfettered responses of participants inchapter 2. This suggests that participants were less consistent in their responses inthe passive condition, meaning that there was a more distributed decision boundarybetween trial images that were target-like, and those that were not. Alternatively,this could be interpreted as participants in the passive condition being less confidentin their responses, yielding a noisier decision process than in the active condition.554.3.2 Classification imagesThe classification images, shown in Figure 12, showed very little resemblance totheir respective targets in either condition, with the highest IWC of the 4 CIs being r= 0.049. Qualitatively, there is very little signal in either set of CIs, but those in thepassive condition seem to have more detail than do those from the active condition.However, the analysis of the discriminative ability of the generated CIs, comparedacross the two conditions suggests that the passive condition yielded CIs that werebetter able to discriminate between the correct target and other targets of high orlow similarity. This suggests that when using the superstitious perception task toestimate internal representations of a specific nature, the passive task strategy is amore reliable method to employ.4.3.3 External predictionFigure 14 shows a very large difference in predictive ability of human responsesbetween the active and passive conditions in experiment 3. In the active condition,the non-significant correlation between target-image similarity and false alarm ratesuggests that the participants’ target images did not describe the signal participantswere detecting. This could be caused by participants in the active condition be-coming focused on small subsets of the trial image that appear target-like, whiledisregarding the rest of the image. This may indicate that while participants mayfocus on one subset of a trial image (e.g., participants focus on the top half of theimage which may be target-like while the bottom half of the image, disregarded byparticipants, may be particularly un-target-like). Since our image-wise correlationprocedure considers the image as a whole, and not subsets thereof, the image de-scribed would appear to have no similarity to the target since the two subsets of theimages would be collapsed together for analysis. In contrast, the passive conditionshows that as image similarity increases, false alarm rate increases, a result thatwas comparable in direction and magnitude to chapter 3. This suggests that par-ticipants considered the image as a whole when making predictions in the passivecondition.56With respect to the practice phase of the experiment, no significant differencewas found in sensitivity or decision bias between conditions. This is as expectedgiven the above explanation of results, since target trials in the practice conditiongenuinely contained a target hidden in noise, and thus all subsets of the image werelikely to contain some portion of signal. As a result, whether one looked at the trialimage as a whole, or focused on a particular subset, one was likely to come tothe same conclusion regarding the target’s presence. However, future work couldreplicate this finding with increased power for between-subject analyses to increaseconfidence in the null results observed in this experiment.Notably, the false alarm rate differed significantly between the passive and ac-tive conditions, with participants reporting more false alarms in the active conditionthan in the passive condition. An alternative explanation of the results observed inexperiment 3 is that the differences in false alarm reports could have driven the restof the experimental results. Specifically, that by increasing the false alarm rate,participants’ responses became more difficult to predict, and the generated CIs losttheir discriminability. The superstitious perception task is based upon the isola-tion of minute traces of signal that occur spontaneously within randomly generatednoise images. It is conceivable that by changing participants decision biases to be-come more liberal, the underlying statistical processes upon which the task reliesbecame less powerful and thus were unable to isolate the signal participants weredetecting within the noise. Experiment 4 addresses this issue through the use ofcomputer simulations in an attempt to show that the active-passive differences arenot simply a result of shifting decision biases.57Chapter 5: Experiment 4In this experiment, we used a neural network that was trained to identify objects inimages, and modified it such that it could identify targets within white noise. Giventhe body of literature that suggests a similarity in representation between HCNNsand the human visual system, we expected that the low-level representations of thetargets in the network would approximate those in the human visual system (Cichyet al., 2016; Khaligh-Razavi & Kriegeskorte, 2014; Yamins & DiCarlo, 2016). Ifthese low-level representations in the HCNN truly did approximate those of thehuman visual system, then they would be sufficient to perform tasks tangential tothat for which the network was trained.To this end, we attempted to induce superstitious perception in the AlexNetHCNN (Krizhevsky et al., 2012). However, since the AlexNet network was trainedto classify objects in images contained within the ImageNet database, some sec-ondary training was required in order to teach the network to identify targets innoise. Specifically, the network was trained to identify an “X” in white noise. Inorder to test the degree to which the low-level representations are generalizable,the low-level layers (layers 1-5) of the AlexNet HCNN were left unchanged, mean-ing that the weights connecting nodes were not affected during learning. Only twofully connected layers (layers 6 & 7) were trained to perform the new task.The goals of experiment 4 were two-fold. First, we intended to test the neuralnetwork’s ability to perceive superstitiously in the manner that humans do withoutdirect training on the task based upon human responses. If the neural network dis-plays a similar pattern of results to the human participants, it would suggest that58there is a similarity in their performance. Specifically, we would expect its re-sponses to be comparable to humans who are not actively engaging their executivefunctions on the task (i.e. the ‘passive’ condition in chapter 4), given the beliefsthat the neural network is similar to humans only in visual processing.Second, if the neural network could see superstitiously, we intended to deter-mine whether the loss of CI quality in the liberal condition in experiment 3 wassimply due to increased proportion of false alarms. If this were the case, thenselecting a liberal criterion such that 44% of images that were most target-like ac-cording to the neural network - like in the liberal condition - would result in a CIof poor similarity to the target. On the other hand, if CI quality is equal between aliberal criterion, the analogue of the liberal condition in experiment 3, and a moreconservative criterion, the analogue of the passive condition in experiment 3, thenwe could conclude that the decrease in CI quality in Experiment 3 was due to in-terference of executive decision making on superstitious perception.5.1 Methods5.1.1 Model descriptionThe HCNN model used in this experiment was a modified, pre-trained form ofAlexNet (Krizhevsky et al., 2012), an HCNN which attained state of the art imageclassification accuracy for the ImageNet benchmark in 2012. This model was cho-sen due to its relative popularity in comparing HCNNs to the human visual system(Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014). With the additionalinformation given by the aforementioned comparisons, AlexNet was the modelwith the highest documented similarity with the human visual system, making itthe ideal model to use when comparing the ability of neural networks and humansto perceive superstitiously.Since the model was not originally trained to identify an ′x′ in noise, the net-work needed to be modified in order to perform such a task. To do so, the weights of59the nodes in the final 2 fully connected layers of AlexNet were randomly initiatedand a new binary classifier replaced the original classification layer of the network.These new layers were trained to perform the novel detection task. The decisionto retrain the final 2 fully connected layers was made due to these layers being thepoint in the network hierarchy in which spatial information is lost. All previouslayers of the network were convolutional layers, meaning that spatial informationin those previous layers is preserved from one layer to the next (Krizhevsky et al.,2012). Notably, however, the weights in the 5 convolutional layers were not modi-fied from their trained state, as provided by its creators (Krizhevsky et al., 2012).Training of the 3 randomly initiated layers of the network was performed usingstimuli generated in the same fashion as the easy training images used in chapters 2,3 and 4. These stimuli were generated either by reducing the contrast slightlybetween the target and the image background (target present trials) or by creatinga consistent grey image, luminance matched to the target (target absent trials). Alltraining images were then overlayed with gaussian random noise. Unlike previouschapters, the network was trained exclusively on images containing an ‘x’ as thetarget image, learning to distinguish between images that contained an ‘x’ hidden innoise from images that contained pure noise. The network was trained on 100,000training images (50% of them containing a target) for 19 epochs, meaning that thenetwork was shown each training image 19 times during training.The number of epochs for training was chosen in a very specific manner. Aftereach training epoch, the network was testing on novel “easy training” images andon “hard training” images from chapters 2, 3 and 4. From the networks’ responses,a sensitivity and bias score was calculated. Training continued until the sensitivityand bias at the end of the epoch were approximately matched in sensitivity and biasto the participants observed in chapter 4.5.1.2 Stimuli and proceduresWhite noise trial images were generated in the same way as in chapters 2, 3 and 4.That is to say that each image was generated pixel by pixel, randomly sampling a60luminance from a gaussian distribution centered at 0.5, which a standard deviationof 0.1 for each of the 2500 pixels. As an analogue to the superstitious perceptiontask completed by humans, the network was fed 16,000 50px x 50px noise imagescentered in a 227px x 227px grey image exactly as done for the human participants.The size of the overall image, grey border included, was set to match the 227px x227px input the AlexNet architecture was built to accept (Krizhevsky et al., 2012).The network’s reported confidence in the presence of a target was recorded for eachof the 16,000 images resulting in a 16,000 response vector.Once the network confidence was recorded for each trial image, decision crite-ria were set at two different points. In a liberal condition, the decision criterion wasset such that the false alarm rate during the test phase was matched with the meanfalse alarm rate observed in the active condition of chapter 4. In a conservativecondition, the decision criterion was set such that the false alarm during the testphase was matched with the mean false alarm rate observed in the passive condi-tion of chapter 4. Once again, due to the complete lack of any targets among theimages presented to the network in the test phase, any positive identification of thepresence of a target was a false alarm.5.1.3 Data analysisData analysis procedures were identical to those performed in chapter 2 and chap-ter 3, but were adapted slightly to accommodate the two criterion settings. Namely,all analyses were performed on the two groups separately, and then the results sta-tistically compared.Analysis of the practice phase was performed using a mixed effects ANOVA,with criterion setting as a between-subject factor with two levels (liberal and con-servative), and target difficulty as a within-subject factor with two levels (easy andhard). One mixed effects ANOVA was used to compare sensitivity (d′) across fac-tors, and a second was used to compare decision bias, ln(β ).The analysis of the test phase was identical to that of chapter 4. However, the61liberal and conservative criterion groups were analyzed separately. When compar-isons between groups was necessary, and appropriate statistical test was employed.When comparing the mean proportion of false alarms across groups, a t-test wasperformed. When comparing the slopes of the two sigmoid curves in the backwardprediction analysis, a z-test was performed to test the statistical significance of thedifference between the two slopes. Finally, a Fischer’s Z-test was used to comparethe strength of the relationships between target image IWCs and proportion of falsealarms for the two groups.5.2 Results5.2.1 Practice phase - training and validation stimuliFigure 15 shows a summary of the neural network’s performance on trials gen-erated in the same way as the easy and hard trials in Experiments 1-3. For thisphase of the experiment, the network was making its own decisions. Specifically,if the network was more than 50% confident that a target was present, it wouldrespond with “Target Present”. As a result, the network’s response to each trial(either “Target Present” or “no target present”) was used to estimate the network’ssensitivity to the ‘x’ target. Figure 15A shows a large difference in the network’ssensitivity (d′) to targets in the easy (d′= 6.34) and hard target (d′= 0.57) difficultysessions. Figure 15B shows a large difference between the neural network andhuman participants. Namely, the neural networks shows little to no bias in its de-cisions, unlike in the human data observed in previous chapters, in both the easy(lnβ = −1.95×10−14), and hard (lnβ = 9.00×10−5) target difficulty sessions.5.2.2 Test phaseFigure 16 shows the distribution of confidence ratings output by the neural networkfor the presence of a target image in the white noise test trials. The confidence rat-ings could range from zero to one, with zero denoting a high degree of confidence620246easy hardTarget Difficultyd'A0.0e+002.5e055.0e057.5e05easy hardTarget DifficultyLog()BFigure 15: A summary of the practice phase of the experiment. A) Sensitiv-ity (d′) of the neural network over 50,000 trials of easy and hard targetdifficulties. B) Decision bias (lnβ ) of the neural network over 50,000trials of easy and hard target difficulties. These point estimates have novariability.that a target is not present, and a one denoting a high degree of confidence thata target is present in the given trial. The standard approach to making decisionsbased on the neural network output is to consider a target to be present if the confi-dence is above 0.5, and then consider a target to be absent if the confidence rating isbelow 0.5. Based on these criteria, the network did not perceive superstitiously atall, with the network confidence ratings ranging from 0.0006, to 0.0117. However,by pushing the network to be more liberal in its decisions, namely by lowering thedecision boundary to be a specific percentile point of observed confidence ratings,we can induce superstitious perceptions.In order to mimic the observed proportions of false alarms in the liberal andconservative conditions of chapter 4, two separate decisions boundaries were cal-culated. One decision boundary was at the 55.3 percentile point, to mimic the63Network ConfidenceCount0.002 0.004 0.006 0.008 0.010 0.01205001000Figure 16: A histogram of the confidence ratings given by the neural networkfor the presence of a target image in a white noise trial images. The redvertical line denotes the decision cutoff for the liberal condition. Theblue vertical line denotes the decision cutoff for the conservative condi-tion.44.7% false alarm rate observed in the liberal condition. The other boundary wasat the 79.7 percentile point to mimic the 20.3% false alarm rate observed in theconservative condition. These decision boundaries are depicted in Figure 16 asvertical bars on the histogram. If the network’s confidence was above the deci-sion boundary, the trial was labeled as a false alarm. On the other hand, if thenetwork’s confidence was below the decision boundary, the trial was labeled as acorrect rejection.Figure 17 shows the CIs calculated from the responses generated by the twodecision boundaries. As in the previous chapters, these CIs were generated bytaking the sum of all the images for which there was a false alarm, and subtractingall of the images for which there was a correct rejection. The two CIs are verysimilar in form. The image-wise correlation between the two CIs was r = 0.64.64Figure 17: The classification images generated from the responses of the neu-ral network that was searching for an X in each trial. Blue shows the CIgenerated for the conservative condition, where responses were gener-ated from the conservative decision boundary. Red shows the CI gener-ated for the liberal condition, where responses were generated from theactive decision boundary.An IWC was calculated between each CI and the target image in order to evaluatetheir similarity to the target. The CI generated from the conservative responsescorrelated with the target with a strength of r = .55, while the CI generated fromthe liberal responses correlated with the target with a strength of r = .54. To test theability of the generated CIs to discriminate what the target of the network from alow similarity target, the IWCs were calculated between the conservative and liberalCIs and the “+” target. The IWCs were both markedly smaller than for the correcttarget, r = .38 and r = .33 respectively.Evaluation of reverse correlation sensitivity using internal predictionFigure 18 shows a sigmoidal relationship between the IWC generated for each trialimage and the neural network’s CIs, and the proportion of false alarms. The fig-ure shows the relationship for both the conservative and liberal conditions. The65proportion of false alarms was generated by binning responses into equal groupsbased on similar IWC scores between the CI and each trial image. A sigmoid func-tion was fit to each set of data, resulting in a slope and center constant for boththe conservative and liberal conditions. In the liberal condition, a relatively steepslope of 71.7, (SE = 1.9), and a center constant of 0.004, (SE = 0.0003). In theconservative condition, there was also a relatively steep slope of 74.2, (SE = 2.2),and a center constant of 0.022, (SE = 0.004). The two slope coefficients were notsignificantly different from one another, z = 0.84, p = .40. Furthermore, in orderto test the goodness of fit of the two models, we calculated the correlation betweenthe sigmoid transformed IWC scores and proportion of false alarms per bin. Thegoodness of fit was very high in both the liberal condition, r(398) = .96, p <.001,and in the conservative condition, r(398) = .993, p <.001. A range of bin sizeswere calculated in order to estimate the effect of the number of bins on the mag-nitude of the of the observed goodness of fit. We performed the same analysis on50, 400, and 1000 bins generated from 16000 trial images fed to the network; theresults are summarized in Table 7.Table 7: How the number of bins effects the strength of the relationship be-tween the IWCs for the neural network’s target and the test trial images.Condition Number of Bins Observed Correlation Statistical Significanceliberal50 r(48) = .995 p <.001400 r(398) = .96 p <.0011000 r(998) = .90 p <.001conservative50 r(48) = .993 p <.001400 r(398) = .94 p <.0011000 r(998) = .87 p <.001Evaluating image selection using external predictionFigure 19 shows a linear relationship between IWC scores for each trial image withthe target “x” image, and the proportion of false alarm responses by the neuralnetwork at both the conservative and liberal decision points. The proportion of falsealarm scores was generated by taking the proportion of false alarms from equal bins66Ȃ0.10 Ȃ0.05 0.00 0.05 0.10 0.150.00.20.40.60.81.0ImageȂwise correlation between trial image and target imageProportion of "yes" responses byConservativeLiberalHCNNFigure 18: The relationship between IWC for the neural network’s CI andeach individual trial image, and the proportion of images in each binidentified as containing a target. Aggregated responses from the liberalcondition are plotted in red, and aggregated responses from the conser-vative condition are plotted in blue. All data were aggregated into 400equally sized bins for each condition. Lines show the sigmoid functionsthat were fit to the data for each condition.of responses whose trial images had similar IWC scores with the target. Figure 19shows the relationship as observed when the data are grouped into 400 equallysized bins. In the liberal condition, there was a relatively strong linear relationshipbetween IWC scores and proportion of false alarms, r(398) = 0.58, p <.001. In theconservative condition, there was also a relatively strong relationship between IWCscores and the proportion of false alarms, r(398) = 0.60, p <.001. Most notably,there was no significant difference in the strength of the relationship between theconservative and liberal conditions, z = 0.43, p = 0.67. In order to verify that thenumber of bins had no effect on our conclusions, we tested the relationship at binsizes of 50, 400, and 1000. A summary of the results for the different bin sizes are67−0.05 0.00 0.050.00.20.40.60.81.0Image−wise correlation between trial image and target imageProportion of "yes" responses by sPassiveActiveHCNNFigure 19: The relationship between the IWC for neural network’s target Ximage and each individual trial image, and the proportion of images ineach bin identified as containing a target. Aggregated responses fromthe liberal condition are shown in red, and aggregated responses fromthe conservative condition are shown in blue. All data were aggregatedinto 400 bins of equal size per condition.found in Table 8 1.1In order to ensure the stability of the HCNN model, all model weights and biases were truncatedto four decimal places as opposed to the unlimited number of significant digits allowed under normalCNN procedures. All analyses covered in this chapter were performed anew using the model withtruncated weights. The changes had no influence on the various analyses, but did noticeably reducethe quality of the generated CIs. The shape of the target was still present in both the liberal andconservative conditions. Nonetheless, the CIs lacked the clarity observed in the CIs generated fromthe model without truncated weights. These observations demonstrate the sensitivity of the HCNNmodel to restricting the significant digits of the model weights. However, the observations do notchange the overall conclusions of this work.68Table 8: How the number of bins effects the strength of the relationship be-tween the IWCs for the neural network’s target and the test trial images.Condition Number of Bins Observed Correlation Statistical Significanceliberal50 r(48) = .87 p <.001400 r(398) = .58 p <.0011000 r(998) = .42 p <.001conservative50 r(48) = .90 p <.001400 r(398) = .60 p <.0011000 r(998) = .41 p <.0015.3 DiscussionExperiment 4 had two goals, with success in the first goal being a requirement forpursuing the second one. The first goal was to compare the behaviour of the neuralnetwork’s performance on the superstitious perception task with the performanceof human observers in the conservative condition of Experiment 3. We chose tocompare them on two specific measures: (1) the ability of the generated CI to dis-criminate between the assigned target, and other possible targets; (2) the strengthof the internal and external predictions. If the neural network and human visualsystem represent visual information in a similar way, we would expect very ac-curate internal predictions, with a very large correlation between IWC for the CIand proportion of false alarms, and moderate accuracy in the external predictions,with a moderate correlation between IWC for the target image and proportion offalse alarms. Furthermore, we would expect that the generated CI would be able todistinguish between the target, and low-similarity targets.The second goal was to investigate the effect of task strategy on the reliabilityof the CI. In Experiment 3 we found that adopting a liberal cognitive strategy ledto increased response consistency, but decreased external prediction ability of thegenerated CI. We compared the performance of the neural net under both conserva-tive and liberal decision biasing conditions, to see if the results would mimic thosefound in human subjects.To train an HCNN that can see superstitiously, we first needed to ensure that the69system could identify the relevant target image at a level comparable to our humanobservers (i.e., essentially perfect identification and discrimination of noisy ‘X’sfrom pure noise). We began by using the 5 pre-trained convolutional layers of theAlexNet HCNN (Krizhevsky et al., 2012).These 5 layers were then left unchanged for the duration of the training process.In order to allow the neural network to perform the superstitious perception task,its higher-level layers needed to be retrained to identify an ‘x’ within noise basedon the output of the 5th layers of AlexNet. In other words, the final 2 layers ofthe network that were trained to perform the superstitious perception task had onlythe representations derived from the 5 pre-trained layers of the AlexNet to use asinput. This meant that any success the network had in identifying the targets wasdue to the information present in the AlexNet convolutional layers.During training, the modified AlexNet was taught to distinguish noisy imagescontaining an X’ from those containing pure noise. 100,000 images were used fortraining, of which 50% were targets containing an X’, and 50% were non-targetscontaining only noise. Upon training completion, the HCNN was able to achievea very high standard of performance, correctly labelling near all of the imagesthat were presented to it, with a sensitivity that was near ceiling (d′ = 6.34) inthe easy target condition. Since the only part of the network that was learningto perform the task was the final 2 fully connected layers, one can conclude thatsufficient information is preserved about the images in the first 5 layers of AlexNetto correctly identify targets in noise. This finding suggests that the HCNN had asufficiently complex representation of the image presented to it within the first 5layers that it can perform simple tasks without any retraining. It also suggests that,much like the human visual system, the low-level representations can be recruitedto perform tasks that are tangential to that for which it was trained.Classification imagesThe Classification Images generated from the responses of the HCNN both qualita-tively and quantitatively approximate the target for which the network was search-70ing. Figure 17 shows that both CIs appear to contain an “x”-like form in the centerof the image. Furthermore, the two CIs have high IWC scores with the target image,and lower IWC scores with a low similarity target, the + image, suggesting that thetwo CIs generated from the neural network’s responses were of adequate quality todistinguish the network’s internal template of an “x” from a target of low similarity,the “+”. These data suggest that the neural network did indeed see superstitiouslyduring the task. However, it is important to note that these classification imagescould only be generated after the reduction of the decision criterion to match thefalse alarm rates of participants.Internal predictionWith regards to the internal prediction of neural network responses, the observedpatterns were similar to those generated by human observers in Experiment 1. Theshape of the relationship between IWC for trial images and CI, and the proportionof false alarms was sigmoid in nature. The sigmoidal nature of the relationship islikely an artifact of the restricted range of a proportion, and so the similar shapeshere should not be used as definitive evidence that the neural network and humansresponded similarly.Nonetheless, it is worth noting that there was similarity in the strength of thenet and human’s internal predictions. Furthermore, the response consistency, asmeasured by the slope of the sigmoid, was within the (admittedly large) range ofobserved consistencies in humans.External predictionThere was a moderate-to-strong relationship between the trial-target IWC and theproportion of false alarms, as shown in Figure 19. Such a relationship suggests thatthe neural network’s responses were guided by the similarity of the trial imagesto the target. However, the strength of this relationship based on the responsesof the neural network was considerably larger than that observed based on the71responses of human participants, suggesting that the neural network’s responseswere governed more by their similarity to the target than were those of the humanparticipants.5.3.1 The effect of false alarm rate on CI qualityOverall, the neural network’s performance on the superstitious perception task, asindicated by the CIs and measures of internal and external prediction, were similarto that observed in humans. Qualitatively, the CIs showed a resemblance to thetarget for which the neural network was looking. Quantitatively, the same CIs wereable to distinguish the network’s target from other targets with similar features. TheHCNN responses exhibited similar patterns to those observed in human responses.Given these findings, we conclude that the network did indeed see superstitiously.Following this conclusion, we used the HCNN responses to meet our second goalof the experiment: to evaluate the degree to which the rate of false alarm reportsthroughout the task affects the end CI quality. In short, to manipulate the decisioncriterion of the neural net, through the increase or decrease of its false alarm rate,has a negligible effect on CI quality, and quantitative target similarity.Looking at the strength of the CI-target IWCs for the liberal and conservativeresponse conditions shows a negligible difference in image similarity between thetwo conditions. In addition, the two CIs showed a similar pattern in the degree towhich they were able to distinguish the network’s target from a low similarity targetimage. This suggests that the proportion of false alarms has little or no impact onCI quality.The same can be said of the effect of manipulating the false alarm rate onthe quantitative measures used to assess response patterns during the superstitiousperception task. With respect to the internal prediction, the two conditions had sim-ilar slopes, suggesting that response consistency between the two conditions wascomparable. Furthermore, the slopes found for the two conditions in the externalprediction procedure were also very similar. Both of these findings are contrary towhat would be expected if the difference in false alarm rate between the liberal and72conservative conditions in chapter 4 was the key factor leading to the qualitativeand quantitative differences in task performance for human observers.Given the array of results in this experiment, we can conclude that the differ-ences observed in chapter 4 between the liberal and conservative task strategieswere not due to differences in false alarm rate. More specifically, that modifyingneural network false alarm rate yielded no significant changes in the quantitativemeasures developed in chapter 2 to evaluate how well the generated CI estimatesthe system’s internal representation. If the differences in CI estimate accuracy ob-served in chapter 4 across task strategy were not due to false alarm rate differences,one can conclude that task strategy influences the underlying mechanism that is be-ing measured.73Chapter 6: General DiscussionIn chapter 5, we compared the performance of Hierarchical Convolutional Neu-ral Network (HCNN) with the performance of human observers in a superstitiousperception task (chapters 2, 3 and 4). To our knowledge, past comparisons of thebehaviour of the human visual system and that of HCNNs have been limited toconditions in which the neural networks were trained to respond based on imagelabels generated by humans, and then tested on the same task (Cichy et al., 2016;Khaligh-Razavi & Kriegeskorte, 2014; Yamins & DiCarlo, 2016).In the present work, we took the comparison an important step further, becausewe first trained HCNNs to identify objects in noisy images, before testing them onimages in which there was no objective signal. Any relation between the falsealarms of the net and the images that yielded those false alarms were thus an indi-cation of the internal template (i.e., the mental representation) that was learned bythe net and used to make best guesses in noisy images.With respect to the process that generates the behaviours, comparisons of theneural processing through which the human visual system and select HCNNs iden-tify objects has revealed that they represent features of an object in similar ways(Cichy et al., 2016; Khaligh-Razavi & Kriegeskorte, 2014; Yamins & DiCarlo,2016). These comparisons were performed using representational dissimilarityanalysis (Kriegeskorte et al., 2008), an analysis that compares the spatial relation-ships in neural activity in response to different stimuli. The logic underlying thetechnique states that the degree of difference in a neural system’s representationsbetween two stimuli can be indexed using the dissimilarity in neural activation in74response to the two stimuli. Put differently, imagine an experiment in which a re-searcher was comparing the neural activation in a brain area in response to lettersand houses. One set of neurons in the brain area that were highly active in responseto houses became very inactive in response to letters, whereas another set followedthe opposite pattern. Since the spatial correlation in activation between these twosets of neurons is very low, we would say that the activation is very dissimilar.From this, we can conclude that the brain area from which the recordings weremeasured represents the letters and houses very differently. If such an analysis isdone comparing activity between a large set of different object classes, one cancreate a matrix of dissimilarity scores comparing each object class with each of theother object classes.If one does such an analysis for two separate systems, such as the human vi-sual system (using fMRI) and an HCNN, one can measure the degree to which thetwo systems share representations by correlating the two associated representa-tional dissimilarity matrices. In short, Yamins and DiCarlo (2016), among others,showed a high degree of similarity between the representation of objects in thehuman visual system, and in an HCNN trained to identify these objects in images.This means that neurons in the human brain and the nodes of the neural networkshow similar activation patterns when exposed to the same set of varied stimuli ofdifferent object classes, and suggests that the two systems share representations ofeveryday objects.Representational similarity between HCNNs and the human visual system hasalso been observed at different levels of visual complexity (Cichy et al., 2016).Early processing areas of the HCNN seem to share the highest degree of represen-tational similarity with early processing areas of the human visual system, such asV1. Likewise, late processing areas of the HCNN seem to share the highest degreeof representational similarity with late processing areas of the human visual systemsuch as IT. In sum, not only do HCNNs show representation similarity with the hu-man visual system at the global level, but representations seem to be similar fromthe lowest to the highest levels of processing.To date, the HCNN has only been compared to human performance on tasks75that the neural network was specifically trained to perform. Moreover, the neuralnetwork was trained based upon the labels created by humans. Unsurprisingly, theneural network showed human-like patterns in response to the task for which it wasspecifically trained. The novel aspect of the present study was that we comparedneural network behaviour on a superstitious perception task — a task that is bydefinition subjective and therefore has no right answer — to the behavior of humanson the same task.6.1 Superstitious perceptions in an HCNN and humansIn chapter 5, we used the newly developed techniques to compare a HCNN’s be-haviour to that of our human participants. The comparison between the HCNN andhumans yielded a number of similarities, and some differences. First and foremost,the network, like the human participants, was able to perceive superstitiously. Thelow level representations in the network trained to identify objects in natural imagescontained sufficient information about novel images to detect very minute traces ofsignal in white noise. The flexibility in the task is similar to that of humans, whocan use the visual system to perform a plethora of tasks, from negotiating objectsin a room in order to sit in front of a computer, to completing an exercise such asthat performed in these studies. We can conclude that a neural network trained toidentify objects in naturalistic images can be co-opted to perceive superstitiously.Such a conclusions suggests that, to some degree, the procedures through which itidentifies objects mimics the human visual system in more than just object recog-nition.Beyond the HCNN’s ability to perceive superstitiously, its response patternsduring the superstitious perception task showed marked similarities to humans aswell. Like humans, the relationship between the IWC for classification imagesand false alarm rate was sigmoid in nature. However, as mentioned earlier, thesigmoidal nature of the relationship may simply be due to the restricted range ofthe false alarm rate measure, which is a proportion and thus is limited in rangefrom 0 to 1.76When comparing the external prediction capability between the human re-sponses and HCNN responses, the HCNN responses are significantly better predictedby IWCs between the target and trial image than were the human responses. Thissuggests that the neural network’s responses were based on trial images’ likeness tothe target, but to a greater extent than were the human responses. This differenceis highlighted by the fact that the neural network will give identical confidenceratings when shown the same trial image due to the construction of the neural net-work, and the algorithm upon which it relies. This consistency is likely not presentin humans, who are much noisier in their responses.It was in getting the network to perceive superstitiously, however, that the mostnotable difference between the network and humans was observed in this exper-iment. The decision boundary of the network — the point of how target-like animage needs to be before being considered a target — is not flexible in a neuralnetwork as it is in humans. Instead, the decision boundary needed to be manuallyset post-hoc, otherwise the network would not have reported any targets amongthe noise with the default decision boundary used in most cases (Krizhevsky et al.,2012; LeCun et al., 1989). While the HCNN and human visual system may seemto be similar under certain circumstances, the aforementioned lack of flexibility inresponse criterion suggests that HCNNs as a model of the human visual system isincomplete.Overall, the similarities displayed between humans and HCNNs in the super-stitious perception task adds to the growing body of evidence pointing towardsHCNNs as viable models for human vision. Most notable in the present researchis the observed similarities in response patterns, despite the network never beingexposed to human response data. All data on which the network was trained weregenerated algorithmically, so the network could not have learned patterns of hu-man response from its training data. The two systems, HCNN and human visualsystem, appear to behave similarly on tasks in which the network was trained onhuman labels of images (Kheradpisheh et al., 2016; Peterson et al., 2016), but thesesimilarities could originate from the networks simply learning to match inputs withhuman responses through trainning, and not by simulating the computations occur-77ring in the human visual system. However, the present work, where the trainednetwork was co-opted to perform a task in which it was never exposed to humandata, suggests that the similarities between HCNN models and the human visualsystem may be more fundamental in nature.One area worthy of further exploration is the necessity of manually setting thedecision criterion in the HCNN in order for the network to perceive superstitiously.In humans, this seems to occur automatically, with participants reporting very fewfalse alarms during the training phases, but reporting as many as 50% false alarmsduring the test phase, despite all images being pure noise. On the other hand,the neural network required a manual shift in criterion to match the false alarmrate observed in humans. Without the human data available, such a shift wouldbe completely arbitrary. For this reason, the development of a dynamic decisioncriterion algorithm may be beneficial to understanding how this process occurs inhumans. Specifically, humans seem to take into account high-level information,such as the expected frequency of targets during a task, in order to modify theirdecision criterion.6.2 Implications for the study of superstitious perceptionIn chapters 2, 3 and 4, we developed a method to quantitatively evaluate a classi-fication image generated from responses in the superstitious perception task (Gos-selin & Schyns, 2003). Using a combination of internal and external prediction,we evaluated a number of key features of participants’ performance on the task.First, we evaluated participants’ response consistency by finding the slope of thesigmoidal relationship between the proportion of false alarms and the IWC scorebetween participants’ CI and each trial image using internal prediction. Second, weestimated how much the participants’ internal representations, estimated by the CI,influences their responses using external prediction. These three measures togethergive an estimate of how well the CI captured participants’ internal representations.We also demonstrated that the accuracy of such classification images variesbased upon the cognitive strategy employed during the task. Namely, we showed78that when one engages executive functions to identify targets in the superstitiousperception task, we see a marked decrease in CI similarity to one’s target. On theother hand, one becomes more consistent in their responses relative to those whodo not engage their executive functions. By employing a neural network to seesuperstitiously, we were able to show that simply changing the decision criteriondoes not account for the differences in sigmoid slope in internal prediction andexternal prediction accuracy observed between the passive and active conditions inchapter 4.Comparing the results of chapter 2 to those of chapters 3 and 4, there is a no-table difference across all analyses. Comparing the results of chapter 2 to thoseof the subsequent experiments, the CIs were much better able to discriminate thetargets, the slope of the sigmoid was notably steeper, implying greater responseconsistency, and the false alarm rate was better explained by the target image. Allof these results imply that participants performed better on the task when theirtask-strategy was not imposed by the experimenter. Such an implication is surpris-ing given that, in chapter 3, and the passive condition of chapter 4, we directedparticipants to adopt the strategy that Gosselin and Schyns (2003) reported theirparticipants to be using. However, results from our experiments suggest that taskstrategy has a profound impact on the conclusions that can be drawn from the re-sults of this task. For this reason, we suggest that future experiments attempt toisolate the optimal task strategy for superstitious perception results. Before resultsgenerated from the superstitious perception task can be taken at face value, wemust generate techniques and measures to ensure that participants adhere to theoptimal strategy when performing the task.79ReferencesBottou, L. (1991). Stochastic gradient learning in neural networks. Proceedingsof Neuro-Nmes.Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent.In Proceedings of compstat’2010 (pp. 177–186). Springer.Brainard, D. H., & Vision, S. (1997). The psychophysics toolbox. Spatial vision,10, 433–436.Brown-Iannuzzi, J. L., Dotsch, R., Cooley, E., & Payne, B. K. (2017). TheRelationship Between Mental Representations of Welfare Recipients andAttitudes Toward Welfare. Psychological Science, 28(1), 92–103. doi:doi:10.1177/0956797616674999Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016). DeepNeural Networks predict Hierarchical Spatio-temporal Cortical Dynamicsof Human Visual Object Recognition. arXiv, 15. doi:doi:10.1038/srep27755Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multipleregression/correlation analysis for the behavioral sciences (3rd. ed.).Mahwah, N.J.: L. Erlbaum Associates.80DeAngelis, G. C., Ohzawa, I., & Freeman, R. D. (1993). Spatiotemporalorganization of simple-cell receptive fields in the cat’s striate cortex. II.Linearity of temporal and spatial summation. Journal of Neurophysiology,69(4), 1118–1135.Dyan, P., & Abbott, L. (2001). Theoretical neuroscience. computational modelingof neural systems. Cambridge, Mass.: MIT Press.Eriksen, C. W. (1980). The use of a visual mask may seriously confound yourexperiment. Attention, Perception, & Psychophysics, 28(1), 89–92.Fahrenfort, J. J., Scholte, H. S., & Lamme, V. A. (2007). Masking disruptsreentrant processing in human visual cortex. Journal of cognitiveneuroscience, 19(9), 1488–1497.Fyfe, S., Williams, C., Mason, O. J., & Pickup, G. J. (2008). Apophenia, theoryof mind and schizotypy: perceiving meaning and intentionality inrandomness. Cortex, 44(10), 1316–1325.Gosselin, F., & Schyns, P. G. (2001). Bubbles: a technique to reveal the use ofinformation in recognition tasks. Vision Research, 41(17), 2261–2271. doi:doi:10.1016/S0042-6989(01)00097-9Gosselin, F., & Schyns, P. G. (2003). Superstitious Perceptions Reveal Propertiesof Internal Representations. Psychological Science, 14(5), 505–509. doi:doi:10.1111/1467-9280.03452Holden, H. M., Toner, C., Pirogovsky, E., Kirwan, C. B., & Gilbert, P. E. (2013).Visual object pattern separation varies in older adults. Learning & memory,20(7), 358–362.Jacoby, L., & Brooks, L. (1984). Nonanalytic cognition: Memory, perception,and concept learning. Psychology of learning and motivation.81Jones, J. P., & Palmer, L. A. (1987). The two-dimensional spatial structure ofsimple receptive fields in cat striate cortex. Journal of neurophysiology,58(6), 1187–211.Khaligh-Razavi, S. M., & Kriegeskorte, N. (2014). Deep Supervised, but NotUnsupervised, Models May Explain IT Cortical Representation. PLoSComputational Biology, 10(11), e1003915. doi:doi:10.1371/journal.pcbi.1003915Kheradpisheh, S. R., Ghodrati, M., Ganjtabesh, M., & Masquelier, T. (2016).Deep Networks Can Resemble Human Feed-forward Vision in InvariantObject Recognition. Scientific reports, 6, 32672. doi:doi:10.1038/srep32672Kleiner, M., Brainard, D., Pelli, D., Ingling, A., Murray, R., & Broussard, C.(2007). Whats new in psychtoolbox-3. Perception, 36(14), 1.Kriegeskorte, N., Mur, M., & Bandettini, P. (2008). Representational similarityanalysis - connecting the branches of systems neuroscience. Frontiers insystems neuroscience, 2, 4. doi: doi:10.3389/neuro.06.004.2008Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). Imagenet classification withdeep convolutional neural networks. Advances in Neural InformationProcessing Systems 25.LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W.,& Jackel, L. D. (1989). Backpropagation applied to handwritten zip coderecognition. Neural computation, 1(4), 541–551.Lifshitz, M., Bonn, N. A., Fischer, A., Kashem, I. F., & Raz, A. (2013). Usingsuggestion to modulate automatic processes: From stroop to mcgurk and82beyond. Cortex, 49(2), 463–473.Liu, J., Li, J., Feng, L., Li, L., Tian, J., & Lee, K. (2014). Seeing jesus in toast:neural and behavioral correlates of face pareidolia. Cortex, 53, 60–77.Marcel, A. J. (1983). Conscious and unconscious perception: Experiments onvisual masking and word recognition. Cognitive Psychology, 15(2),197–237. doi: doi:10.1016/0010-0285(83)90009-9Neri, P., & Levi, D. M. (2006). Receptive versus perceptive fields from thereverse-correlation viewpoint. Vision research, 46(16), 2465–74. doi:doi:10.1016/j.visres.2006.02.002Pelli, D. G. (1997). The videotoolbox software for visual psychophysics:Transforming numbers into movies. Spatial vision, 10(4), 437–442.Peterson, J. C., Abbott, J. T., & Griffiths, T. L. (2016). Adapting deep networkfeatures to capture psychological representations. CoRR.Poggio, T., & Riesenhuber, M. (1999). Hierarchical models of object recognitionin cortex. Nature Neuroscience, 2(11), 1019–1025. doi:doi:10.1038/14819Rieth, C. A., Lee, K., Lui, J., Tian, J., & Huber, D. E. (2011). Faces in the mist:illusory face and letter detection. i-Perception, 2(5), 458–76. doi:doi:10.1068/i0421Ringach, D., G., S., & Shapley, R. (1997). A subspace reverse correlationtechnique for the study of visual neurons. Vision Research, 37(17),2455–2464.Ringach, D., & Shapley, R. (2004). Reverse correlation in neurophysiology.Cognitive Science, 28(2), 147–166.Seli, P., Jonker, T. R., Solman, G. J. F., Cheyne, J. A., & Smilek, D. (2013). A83methodological note on evaluating performance in asustained-attention-to-response task. Behavior Research Methods, 45(2),355–363. doi: doi:10.3758/s13428-012-0266-1Shermer, M. (2008). Patternicity: Finding meaningful patterns in meaninglessnoise. Scientific American, 299(5).Smilek, D., Enns, J. T., Eastwood, J. D., & Merikle, P. M. (2006). Relax!cognitive strategy influences visual search. Visual Cognition, 14(4-8),543–564.Theunissen, F. E., David, S. V., Singh, N. C., Hsu, A., Vinje, W. E., & Gallant,J. L. (2001). Estimating spatio-temporal receptive fields of auditory andvisual neurons from their responses to natural stimuli. Network (Bristol,England), 12(3), 289–316. doi:doi:10.1088/0954-898X/12/3/304Van Selst, M., & Merikle, P. M. (1993). Perception below the ObjectiveThreshold? Consciousness and Cognition, 2(3), 194–203. doi:doi:10.1006/ccog.1993.1018Whittlesea, B., & Brooks, L. (1994). Journal of Experimental Psychology:Learning, Memory, and Cognition. Journal of Experimental.Yamins, D., & DiCarlo, J. J. (2016). Using goal-driven deep learning models tounderstand sensory cortex. Nature Neuroscience, 19(3), 356–365. doi:doi:10.1038/nn.4244Yamins, D., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J.(2014). Performance-optimized hierarchical models predict neuralresponses in higher visual cortex. Proceedings of the National Academy ofSciences of the United States of America, 111(23), 8619–24. doi:84doi:10.1073/pnas.140311211185

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.24.1-0354254/manifest

Comment

Related Items