Open Collections

UBC Undergraduate Research

D.R.O.P.S. : drop-off recognition optical processing system Saxena, Pranav; Yee, Rosanna 2011

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
Saxena_Pranav_et_al_APSC_459_2011.pdf [ 1.35MB ]
Metadata
JSON: 1.0074462.json
JSON-LD: 1.0074462+ld.json
RDF/XML (Pretty): 1.0074462.xml
RDF/JSON: 1.0074462+rdf.json
Turtle: 1.0074462+rdf-turtle.txt
N-Triples: 1.0074462+rdf-ntriples.txt
Original Record: 1.0074462 +original-record.json
Full Text
1.0074462.txt
Citation
1.0074462.ris

Full Text

  D.R.O.P.S  Drop-off Recognition Optical Processing System       Pranav Saxena Rosanna Yee   Project Sponsor: Dr. Cyril Leung   Project Number 1122 Applied Science 459 Engineering Physics Project Laboratory University of British Columbia April 23, 2011   ii  Executive Summary   The purpose of this project is to design and develop a drop-off detection system to assist people with vision loss in navigating their surroundings. The system uses two cameras to successfully detect sudden drop-offs, such as stairs, curbs, edges of sky train platforms, etc. The system then provides aural feedback to appropriately notify the user of drop-offs.  The research objectives involved learning how to analyze and process images from two cameras to detect drop-offs, using edge detection in conjunction with stereo vision to accurately and consistently locate drop-offs.  From testing, it was found that the depth calculations were reasonably accurate with an error of 4.22 cm and meet our error margin of 7.62 cm (3 inches). However, the coefficient of variation was much higher than expected at 48.8%. The average processing speed was 6.21 frames per second which is one-fifth of the Minoru’s sampling speed. In our testing we found that the Minoru camera was not designed for outdoor use in very bright conditions, as the lighting can overwhelm the input and make it difficult to perform analysis. It was also required for street curbs to have an edge on the road, so that there is an edge to compare with the edge of the curb as seen in the image.  There are various improvements that could be made to the current system. Currently, speech feedback is not implemented; instead, a series of beeps are used to warn the user. Outdoor performance could be improved by adding an optical filter to the Minoru camera. Calculations for the distance between the user and the drop-off are inaccurate, due to correlation errors from edge-noisy drop-offs. The fixture holding the Minoru could be redesigned to be more user-friendly. Optimization of the algorithms could also improve processing speeds.   A working prototype has been demonstrated to the project sponsor Dr. Leung. This project has the potential to greatly benefit people that are visually impaired by providing a cost-effective, user-friendly, and useful tool to help them navigate their everyday environment.      iii   Table of Contents  Executive Summary       ii. Table of Contents       iii. List of Figures and Tables      iv.   1. Background   1.1 Rationale and Motivation    pg 1  1.2 State of the Art Technologies  pg 1  1.3 Sponsorship      pg 2  1.4 Previous Projects     pg 2  1.5 Project Objectives     pg 3  1.6 Alternative Strategies     pg 3  1.7 Overview of Report      pg 4   2. Discussion  2.1 Technical Background     pg 5  2.2 Theory       pg 5  2.3 Final Design      pg 6 2.4 Alternative Methods     pg 15 2.5 Testing and Reviewing the System   pg 16 2.6 Results       pg 17 2.7 Discussion of Results     pg 19  2.8 Key Issues      pg 21    3. Conclusions        pg 23  4. Project Deliverables      pg 24  5. Recommendations      pg 25  6. Appendices  Appendix A – Key Software Code    pg 36  Appendix B – Technical Specifications   pg 43        Appendix D – Bill of Materials    pg 49   7. References       pg 50   iv  List of Figures and Tables  Figures: Figure 1 - Shift in angles seen by left and right “eyes”    pg 6 Figure 2 - Comparing disparity for objects at different distances  pg 6 Figure 3 –Sobel kernel weighting for x-gradient and y-gradient  pg 7 Figure 4 - An example of edge detection      pg 8 Figure 5 – Flow Graph depicting the Edge Location Algorithm   pg 9 Figure 6 – Detailed description of pixel checking process   pg 9 Figure 7 – Pixel priority configurations      pg 10 Figure 8 – An example of Edge Location      pg 10 Figure 9 – Disparity and depth        pg 11 Figure 10 - Minoru 3D webcam       pg 12 Figure 11 - Stereo vision using two parallel cameras    pg 12 Figure 12- Finding corresponding pixels       pg 13 Figure 13 – Depiction of a large blind spot for horizontal facing cameras pg 13 Figure 14 - New orientation of cameras to reduce blind spots   pg 14 Figure 15- Mini2440 SBC with labelled components    pg 15 Figure 16- Velcro fixture to hold the Minoru camera    pg 17 Figure 17- Overview of system       pg 18 Figure 18- Measured depth vs. calculated depth    pg 19  Tables:  Table 1 – Average Processing Times      pg 18 Table 2 – Key Issues          pg 21 Table 3 – Bill of Materials         pg 24 1   1. Background   1.1 Rationale and Motivation For most, much of urban navigation and interaction with our environment is based on visual signals (i.e. road signs, traffic lights, etc). Being able to see road conditions and avoid obstacles in our path or see oncoming dangers from a distance is something most may take for granted. However, navigation in a dynamic, unstructured environment becomes more challenging when one has impaired vision. As most electronic interfaces are visually based, simple information can be difficult to access. Because of this, the user interface for those visually impaired becomes very different, mainly based on aural feedback, tactile feedback, and voice recognition.   After speaking to a representative from CNIB (Canadian National Institute for the Blind) it was found that “drop-offs”, sudden descending steps in one’s path (i.e. sidewalk curbs, stairs, train platforms, etc) were a major concern for the safety of a person with vision loss. Especially for those who use wheelchairs and walkers, it is often difficult to detect the edge of a drop-off from a safe distance to the edge.   1.2 State of the Art Technologies Technologies that have been developed to assist visually impaired users in navigation are becoming more sophisticated in the past few decades than the simple, traditional white cane. The newest version of the electronic white cane is the Smart Cane, developed by Dr. Ye of UALR, which features 3D Flash LADAR (LAser Detection And Ranging) sensing. The information collected from LADAR is processed in real time using VRO (Visual and Range Odometry) to compare intensity and range of images from one frame to the next. Using this and matching invariant features, the transition can be used to derive roll, pitch, yaw, and Cartesian positional change. However, LADAR is still very expensive and would not be an economically feasible option for the user. A similar effect can be achieved by ultrasonic detection, but is less accurate and has a lower resolution.   An alternative method for range detection and sensing one’s surroundings was stereo vision. The use of two cameras, slightly offset from each other (i.e. one on the left and one on the right),  allows the distance from an object to be calculated by comparing invariant and variant features. By comparing the images from both cameras, closer objects would appear to be more different as the angles at which it is seen by the cameras is larger, while farther objects would appear the same. Based on how much deviation an object has from one camera’s point of view to the other, the relative distance from the user can be found. Similar research has been conducted by James Coughlan and Huiying Shen from The Smith-Kettlewell Eye Research Institute.   2    Another branch of computer vision in development is image processing for reading road signs and characters. In general, in the image obtained from a camera eye, the road sign is recognized using colour discretion, and then the ROI (region of interest) is selected. From there, the shape of the sign is found using edge detection, and the centre point found. Matching the features and colours of the imaged region to the database of road signs and symbols, this information can be provided to the user. However, this algorithm has not been optimized or the capability made available to pedestrian users at an affordable price.   The development of state of the art electronic white canes is not at the stage where it can be produced on a commercial scale, where one can purchase such a cane on a meagre budget.  Rather, commonly sold electronic white canes are more cost effective; Sound Foresight Ltd uses ultrasonic detection and tactile vibration feedback, while J & E Semiconductor uses RFID (Radio Frequency IDentification), and is meant to be used indoors through installation of RFID-tagged tiles.    1.3 Sponsorship  This project was sponsored by Dr. Cyril Leung, and focused on an aspect of technology that is innovative and not fully explored. The Goal of the project is not to optimize existing tools, such as GPS devices like the Trekker Breeze, but instead provide a unique solution or approach to this issue of navigation for the vision impaired.   1.4 Previous Projects There are two projects done in the past two years by Engineering Physics students, both attempted to solve the issues of finding specific objects in a general location such as doors and bus stops. While GPS can offer general guidance to the user’s location, it is only accurate to about 35 ft and gives no information about the immediate surroundings. The first project was based on using RFID tagged tiles to guide the user, while the second project was based on using a wireless RF transmitter. Dr. Leung had also sponsored other students in electrical engineering on similar projects, which span from using GPS to guide the user around the UBC campus, to character recognition to read labels for the user.      3   1.5 Project Objectives  To deliver a working prototype of a drop-off detection system that will serve to aid people who are visually impaired in their daily routines by detecting drop-offs and notifying them  This includes:  Detecting drop-offs include sidewalk curbs, stairs, and other sudden drops greater than 4 inches, within 3 feet away from the user.   Stereo vision that can accurately differentiate between object distances within 3 inches, within 3 feet away from the user.     Edge detection that can locate main edges (i.e. sidewalk curb, platform edges, possible obstacles, etc)    User feedback consists of an aural message, with warning given in an appropriate amount of time to allow user enough time to react.    1.6 Alternative Strategies Many alternatives were considered in the approach to this project, but were eliminated due to time and budget restraints. For this project, stereo vision, edge detection, and other image processing techniques was employed to help guide the user. This requires two cameras and a controller for receiving and outputting signals and processing the information. One strategy was to use two high-end cameras with its own processing units, and a simple microcontroller to deal with user feedback. One popular camera system is the Blackfin Camera Stereo Vision system. However, due to the high cost, this method was not chosen. Other negative aspects of this choice are the lack of proper documentation and tutorials for developers new to the camera system, and the lack of flexibility in handling the image data, as the simple microcontroller may only query and receive output from the camera system which carries out all processing internally.  Ultrasonic detection was not chosen for range detection as it did not have a high enough resolution to function fully. The wide beam range of sonar would also introduce noise into the system and the user may be warned of drop-offs to their far right. Infrared sensing was also not selected as IR signals may be confused in an outdoor environment, where there can be multiple sources of heat signatures.   From researching, it was found that the Arduino microcontroller was not able to handle image processing at the rate required. While the Arduino can process images and perform the image manipulation in edge detection and stereo vision, the time required for the Arduino to do so would be too long to be effective. By the time the Arduino interprets the feed from the camera and can produce a warning to the user, it could be too late.  4   1.7 Overview of Report In the following sections, the project would be described in further detail. In Section 2. Discussion, the implementation and theory behind the methods used is given. In addition, the tests carried out to characterize the system and the results are discussed. Section 3. Conclusion summarizes the findings of the project. Section 4. Project Deliverables gives a list of the deliverables and the financial breakdown of the project. Section 5. Recommendations gives a series of suggestions surrounding the issues encountered.     5   2. Discussion   2.1 Technical Background Basics of digital images An image is essentially a matrix of pixels, with each pixel representing an element. A pixel holds a very specific colour value, and is represented in the matrix. An image matrix is actually comprised of its 3 layers, the red, green, and blue image matrices. These three primary colours, together, can produce any known colour. Each pixel in a specified layer holds an 8-bit value representing that pixel’s red, green, or blue component. The higher the value, the higher the intensity of that colour is represented. For example, the colour purple would be represented by using only red and blue values, with green values set to 0, while the colour yellow would be represented by only red and green values, with blue values set to 0. White is represented with all the colours set to 255, and black is represented with all the colours set to 0.   2.2 Theory Theory – Edge detection To implement drop-off detection in a more reliable manner, edge detection would be executed to supplement stereo vision. Wherever there is a sharp change in the height of a surface, there will be a visible edge. Thus, where a change in depth is detected and there is an edge, then there exists a drop-off. Having both types of detection would help reduce noise and the likelihood of a misreading.   Edge detection is done by comparing each pixel to its neighbouring pixels. If there is a large difference between the surrounding pixels, then there would be an edge. However, before this step is done, the image needs to be simplified. The method chosen would sort the pixels in terms of brightness, and pixels that are considered darker would be converted to black, and pixels that are lighter would be converted to white. From a coloured image, the image is converted to a black and white image, where the edges are identified in black. How this is done is explained in greater detail in Section 2.4 Final Design.   Theory – Stereo vision The two cameras employed allow us to mimic a natural human process. Stereo vision allows us to find the 3D location of objects in a 2D image. Stereo vision allows those with vision in both eyes to interpret depth and objects visually in 3D. When observers with full vision look at an object, each eye sees a similar but slightly different image of the object. The image seen by the left eye shows the object shifted to the right, while the image seen by the right eye shows the object shifted to the left. These two images are combined and processed in the brain. How one interprets depth visually is from the disparities between these two images. For an object that is father from the observer, 6   there is less difference between what is seen from the left eye and the right eye, as the angled difference seen by both eyes is smaller. Similarly, for a closer object, the difference in angle seen by both eyes is greater, so the object is seen to be more shifted, as shown in Figure 1.   Figure 1. Shift in angles seen by left and right “eyes” (Source: http://upload.wikimedia.org/wikipedia/commons/7/7c/Aufnahme_mit_zwei_Kameras.svg)   Because of this geometry, the greater the disparity in the images, the closer the object is, while less disparity represents a farther object (Figure 2). Implementing stereo vision will be discussed further in Section 2.4 Final Design.   Figure 2. Comparing disparity for objects at different distances (Source: http://www.vision.deis.unibo.it/smatt/Seminars/StereoVision.pdf)   2.3 Final Design In order to implement the image processing capabilities of DROPS, development has been carried out using the Microsoft Visual Studio 2010 C++ IDE. C++ is a powerful language which allows object-oriented programming principles to be employed. Also, in order to manipulate image data and run low level image processing functions, the Open CV Image Processing library was used. This is a widely used, open source library of programming functions for real time Computer Vision. Lastly, the software application has been developed and run on a portable laptop machine. This laptop has a 2.67GHz processor, and is able to rapidly process image data.  7   Edge detection algorithm  There are many steps necessary to take input images and identify valid drop-offs. The first step will involve using the process of edge detection to locate any edges seen in the image. This is useful because any drop-off seen by the camera is essentially an edge in the image. The edge detection algorithm used was a Canny edge detection algorithm, based on the Sobel kernel. The kernel is applied to every pixel, and gives weights to surrounding pixels to compute the intensity gradient in the horizontal and vertical direction (x and y direction respectively), as shown in Figure 3. The sum of the pixel intensities multiplied by the weights gives the gradient. From finding the gradients in these two directions, the magnitude of intensity change can be found as                         (1)  where letting the centre pixel be at (i,j), in the ith row, and jth column                                                   (2a)                                                   (2b)       Figure 3. Sobel kernel weighting for x-gradient (left) and y-gradient (right).    Where there is a large degree of change in intensity, there is likely an edge. By comparing to a threshold value to eliminate noise and assigning a value to the pixel where an edge is detected, this will create an image that only contains edges in the image, against a white background (see Figure 4). The edges seen afterwards depend greatly upon the threshold used. A very low threshold will detect too many edges, and a very high threshold will do the opposite.   Other edge detection algorithms were tried and it was found that the Canny algorithm with the Sobel kernel was the best, though marginally better than the Prewitt kernel for producing stronger edges. It seems from research that the Canny method is the standard and preferred algorithm used for detecting edges.      1 2 1 0 pixel 0 -1 -2 -1  1 0 -1 2 pixel -2 1 0 -1  8     Figure 4. An example of edge detection    The edges were detected using the intensity values of the blue channel of each pixel, as red, green, and yellow are usually related to ambient light and noise. (In RGB colour coding, yellow is a result of combining red and green). Also, in conditions with little lighting, the blue edges gave a better result. Thus, the edges used were based on blue values, and are represented in the output edge detected image as having blue lines.   To determine which edges are important, the top third of the image is ignored (i.e. attention should be focused on edges seen in the bottom of the images). This is useful in eliminating any other edges in the image, but does ultimately limit the range of drop- off detection. This is also helpful, because it is not desirable for the end-user to be warned of drop-offs many meters away. It will also be important to disregard completely or nearly vertical edges (45-135º from the horizontal) in the analysis. This is because it is not very likely that any user would happen to walk off a drop-off at such an angle. On the reverse, it will be more important to analyse any edges at an angle between 45º and -45º. The angle an edge makes can be determined by utilizing the matrix positions of several localized black pixels of an edge and the elementary slope equation to determine the average slope they generate.     9   Edge location algorithm  In order to use our edge data to analyze depth, it is necessary to define and locate all pertinent edges. To satisfy this requirement, an edge location algorithm was developed and implemented. This process searches an edge-detected image for series of 100 connected [blue] pixels, stores their locations, and outputs a red outline on each edge. The edges are scanned for in a horizontal fashion (left-to-right) row by row; this strategy prefers edges closer to the horizontal, which is desirable. Edges of various angles are allowed, as are curved edges, as long as they are not disjointed. The flow graph in Figure 5 illustrates this method. Note that pixels are checked with 3 different configurations in order to vary the priority of adjacent pixels. This strategy allows flat and sloped edges to be found easily. If only configuration 1, 2, or 3 is used, flat, positively sloped, or negatively sloped edges are predominantly found, respectively.    Figure 5: Flow Graph depicting the Edge Location Algorithm   Figure 6: Detailed description of pixel checking process  10              Figure 7: Pixel priority configurations, higher values reflect lower priority during checks  In the event of a partial edge being rejected (white pixel), or an edge being detected (pixel count == 100), the algorithm finds the location of the last pixel of the edge that is on the row the edge started on. The edge scan continues at the pixel to the right of this one.  Lastly, a search is abandoned if the partial edge moves off the image. Overall, this algorithm is successful in finding and storing edge locations efficiently for later use (Figure 8).    Figure 8: An example of Edge Location   Configuration 1   Configuration 1 Configuration 2   Configuration 1 Configuration 3   Configuration 1 4 2 Last Pixel 1 5 3 Priority Configurations for Pixel Checking 4 1 Last Pixel 3 5 2 4 2 Last Pixel 3 5 1 11   Stereo vision algorithm To detect a drop-off, it is necessary to perceive a change in depth. The solution chosen is to use stereo vision, which uses input from two cameras, parallel to and slightly offset from each other. The images from the cameras are compared, such that objects with greater disparities (i.e. larger changes in its features) are closer, and objects that are invariant are farther (see Figure 5).   Figure 9. Disparity and depth (Source: http://www.vision.deis.unibo.it/smatt/Seminars/StereoVision.pdf)  Here, both cameras are positioned in the standard orientation, facing forward. The camera on the left is the reference camera, and the camera on the right is the target camera. The object observed is at location P, distance Z away from the cameras, where the image seen at point p and p’ in the image planes (pink).    and    are known values. From knowing the width/length of the cameras’ image planes and one can find the resolution of the images (pixel width/length); this creates a handy pixel to distance ratio on the image plane. By measuring the distance in pixels to the desired point p or p’ in the image, one can determine the actual distance to p or p’ in the image plane (   and   ).  Let   be the distance between the two cameras and   be the focal length. By similar triangles OR-P-OT and p-P-p’, the following is derived:                      (3a)                    (3b)  The disparity, , is calculated as          .                (4) 12              (4b)                   (4c)   Using  , the horizontal distance between the object and the reference camera,  is noted by similar triangles:                                      (5)   The Minoru 3D camera (Figure 10) was used, which has two identical cameras with the same resolution and frequency. These cameras are already aligned such that the centre of the lens sits on the same epipolar line (Figure 11). However, instead of having the cameras sit to the left and right of each other, the camera is intended to be rotated 90o such that one camera sits on top of the other. The epipolar line is now along the columns of the matrix.         Figure 10. Minoru 3D webcam            Figure 11. Stereo vision using two parallel cameras  (Source: http://walyou.com/minoru-3d-webcam/)                (Source:http://123nonstop.com/biography/Stereo_Vision)   The logic behind rotating the camera is to simplify calculations. When using horizontal cameras on plain (same colour pixels) surfaces, such as curbs, it becomes extremely difficult to find the corresponding pixel from the reference camera’s image, in the target camera’s image (Figure 12). Since all the pixels have very similar values in that region, there is no distinct feature to search for in both images. When using vertically placed cameras, the pixels along the central vertical axis can be chosen to find the corresponding pixel. For example, one can search along this axis, from the bottom of the images to the first identified edges. This easily gives two pixels that directly correspond with each other. Having found point p and p’ from Figure 9, one can now proceed with finding XR and XT, and the rest of the calculation. 13    Figure 12. Finding corresponding pixels  One will note that in the derivation above, the cameras are pointing directly ahead in the horizontal (parallel to the ground), for simplicity. For the purposes of this project, stereo vision is applied on points just above and below drop-offs, which are located on the ground in front of the user. The lenses of the Minoru 3D Webcam is not wide-angle, and will need to be a minimum distance away from a drop-off just to see it, essentially creating a blind-spot. This means the user will not be able to see any drop-offs that are very close to them (Figure 9).  Figure 13: Depiction of a large blind spot for horizontal facing cameras In order to effectively reduce this blind spot and allow the system to detect drop-offs that are closer to them, the cameras are angled down from the horizontal (42º) at angle ϕ, shown in Figure 10.  14    Figure 14. New orientation of cameras to reduce blind spots This strategy is effective in reducing this blind spot, but it complicates the derivation from above. Using the stereo vision calculations from above, Z’ (replaced Z) will now be solved. Z’ is the straight distance along the bottom [reference] camera’s axis, from the camera to the orthogonal plane upon which P sits. Using equation 2, where Z’ replaces Z, Y’ replaces X, and Ybottom (distance from centre of image plane to point p on image plane) replaces XR, one can easily calculate Y’. The difference to this calculation comes in finding h, the actual height from the point of interest to the reference camera. The simple realization from these calculations is that                     (6) One can now repeat this process on the next desired point of interest, P2. This gives h and h2, the real world estimates of heights from the reference camera to the top and bottom of the drop-off, respectively. Of course, the height of the drop-off, h2-h, can then be determined, and a warning can be issued to the user if this exceeds 4 inches. Audio and vibration output The audio output of a warning to the user when a drop-off is detected would be implemented. The algorithm is as follows:   Drop- off detected after edge detection and depth detection  Set drop-off detection flag to 1  If drop-off detected o Send signal to vibration motor to vibrate  o Output warning message  o After warning message is played, stop vibration o Set detection flag to 0  Else, do nothing 15   2. 4 Alternative methods  There is an alternative method which was considered in implementing DROPS system. This option would allow the system to be fully portable, but was not pursued in order to focus on the feasibility of a drop-off detection system.   Mini 2440  The Mini 2440 (Figure 15) is a Single-Board Computer capable of reading and processing images from the Minoru 3D webcam. This controller board has a 400 MHz Samsung ARM9 Processor. The Minoru will interface via the USB host provided by the Mini2440 to provide image data for processing. However, in order to carry out image processing via the Mini 2440, it is suggested to use the Eclipse IDE along with the embedded Linux 2.6 OS. Although Windows CE 6 is available for this board, it is not able to run the Visual Studio IDE. Please note that the Open CV library has full support for the Eclipse IDE, which runs on Linux.     Figure 15. Mini2440 SBC with labelled components (Source: http://www.friendlyarm.net/products/mini2440)  16    Another option over the Mini2440 is a Pico-ITX motherboard form factor. This is a very small motherboard complete with CPU, built-in memory, SATA ports for a hard drive, and various ports (e.g. USB) to essentially create a small computer. These tend to be more expensive, but much more powerful than a Mini2440 (processor ranging from 1-2 GHz). These would definitely be able to process images at a faster rate. The negative aspect of this option, disregarding the price, is the physical size of the system (including a hard drive and full enclosure). The Pico-ITX option may be smaller than the laptop, but the ease of use of a laptop in developing the code outweighs the potential difficulties found in configuring and troubleshooting a Pico-ITX for the first time.     2.5 Testing and Reviewing the System  In order to test and rate the system’s performance, certain criteria were considered. The following system capabilities were tested:  1. Detect all types of drop-offs with a height of 4 inches or more 2. Detect a given drop-off at least 4 times out of 5 tries (minimum 80% detection rate) 3. Detect drop-offs that make an angle between +75º and -75º relative to the horizontal of the image 4. Detect a drop-off if it is within a 2 metre range of the user, but outside of the above mentioned blind spot 5. Only detect drop-offs directly in front of the user (not side of the camera’s view) 6. Process and output a warning to the user within 1 second   To meet condition 1, the system was tested on various types of drop-offs that are greater than or equal to 4 inches. These drop-offs included curbs, stairs (indoor & outdoor), and sky train platforms. The system’s response to small drop-offs (less than 4 inches) and lines made in flat pavement was also tested. The system should not output any warning for these scenarios. The user does not need to be aware of small drop-offs or lines drawn into the floor pattern.  To meet condition 2, four different drop-offs of various types were chosen and tested on at various angles and ranges. The tester stood still while testing the specific scenario. In total, 14 different tests were carried out for 250 frames each. If 80% of scenarios result in a consistent warning to the user (constantly warning user throughout extended test), this condition will be properly met.  17   To meet condition 3, the cameras were placed within this range of angles, relative to various drop-offs, while testing condition 2. The system should be able to detect any drop-offs within this range, but not necessarily any outside. If condition 2 is met, so is condition 3.  To meet condition 4, the cameras were randomly placed within the specified range, while testing for condition 2 (mentioned above). If the system meets condition 2, then it meets condition 4.   To meet condition 5, the cameras will be oriented so that their central axes intersect with all drop-offs while testing for condition 2. Once again, if condition 2 is met, then condition 5 is also met. It will not be necessary to detect drop-offs which do not intersect with the central axes of the cameras.  To meet condition 6, the response time will be recorded in all tests for condition 2. If the average response time is less than or equal to 1 second, condition 6 will have been met.  In order to test and use the system, a Velcro fixture was made to hold the camera at the specified angle, without any user interaction (as seen in Figure 16).   Figure 16: Velcro fixture to hold the Minoru camera   2.6 Results  Currently, a prototype has been developed according to the project proposal, with the overview of the system shown in a flowchart in Figure 17. After capturing images with the Minoru 3D camera, the system reduces the images to edges, and then locates the edges that are significant and most likely to be associated with a drop-off. Next, only edges which intersect with the center axis of the camera are filtered through. 18   Corresponding edges are matched based on their sorted order from the bottom of the image. Depth calculation is done using the disparity found between each set of matched edges (one from each camera), as described in Section 2.4 Final Design. Five pairs of corresponding edges are checked per frame. The breakdown of processing speeds can be found in Table 1. The average number of frames per second, after all processing, was 6.21 fps. (Minoru camera has maximum of 30fps).   Overview of System   Figure 17. Overview of system    Table 1. Average Processing Times  Algorithm Average processing time  Minoru start-up reaction time (turning cameras on) 1300-1500 milliseconds Edge detection  30 ms (15ms each frame) Edge location 80 ms (40ms each frame) Depth calculation 51 milliseconds Average time for processing (without start-up Minoru start-up reaction time) 161 milliseconds  As seen above, there is a start-up cost of turning the cameras on, but over time, the average fps approaches 6.21fps. With more edges, the fps is closer to 4fps, and in the absence of edges, the fps is closer to 9fps. An audible beep is emitted for every detected drop-off in the desirable range, for every frame. The system is also, in most cases, able to detect and warn of obstacles, such as chairs, couches, or stairwells (seen from the bottom). A graph of the actual depth vs. the calculated depth values can be seen below in Figure 18.   19     Figure 18. Measured depth vs. calculated depth  As seen above, the calculated depths are close to the real depth values, with an average error of 4.22 cm. This is within the desired error margin of 3 inches (7.62cm). However, there is a high average coefficient of variation of the results – 48.8%. When only the first drop-off was considered for each set of frames, the average error improved to 3.33 cm, but the coefficient of variation was still high – 49.2%.   2.7 Discussion of Results Testing and reviewing of the system was completed, as per Section 2.6. The physical limitations of the system have been tested and determined. The system can effectively find drop-offs within a range of 0.4m to 3m in front of user, and the camera has a field angle span of 42o. The edge location algorithm was able to locate edges between about -30o to 30o, because of the nature of how the pixels are searched - with a preference along the horizontal. This angle is relative to the edge detected; i.e. when the edge is horizontal with the user facing the edge, it is considered to be at 0o.   In terms of the condition listed in Section 2.6, all were met except for condition 3: 1. All types of drop-offs with a height of 4 inches or more were detected. 2. All tested drop-offs were detected and resulted in a warning to the user (100% detection rate). 3. Only drop-offs that make an angle between +30º and -30º relative to the horizontal of the image were detected. 4. Drop-offs within a range of 0.4m to 3m in front of the user can be detected. 5. Drop-offs in front of the user were focused on and easily found. 6. A warning to the user was issued well within 1 second. As noted above, the system is able to process 6.21 frames per second, on average. 20    Although condition 3 was not fully met, the system is still able to identify and warn of most drop-offs in front of the user. The ±30° range is still desirable, as it covers the most common angles at which users may encounter a drop-off.  As mentioned above in the Results Section, the average depth calculation error (4.22 cm) was found to be within the desired error range of 7.62 cm (3 inches). This error improved, when only considering the very first drop-off in each image taken, to 3.33 cm. This improvement is due to most drop-offs being located at the very bottom of the camera image (edges are not found on the floor pattern prior to seeing a real drop-off). The real discrepancy in the results is due to the high standard of deviation of the calculated depths. This is not desirable, as many individual measurements lie far away from the actual depth value. This issue occurs due to the edge matching strategy employed. As corresponding edges are matched from the bottom up, the matched pairs do not always correspond to the same real edge. The mismatching of edges results in incoherent output, and thus the results have a high coefficient of variation. However, day-to-day use of the system is not affected, as all real drop-offs are detected, and the user is quickly warned.  There is a concern for false positives in the results. Although false positives do not occur often, warning the user when there is no drop-off is not ideal. However, this only happens for heavily patterned surfaces (i.e. bricked flooring). This occurs again due to the edge matching strategy. With an edge-heavy surface, the edges are not successfully matched, and so many incoherent readings result.   The lighting condition also affects the Minoru webcam’s performance. In low lighting conditions, there was a slower response time, and it did not detect edges when there is no light. This is not as significant a concern since it is not as likely that visually impaired users would be navigating at night or in low light conditions. The performance outdoors was also affected by ambient light on sunny and bright days. Because the Minoru 3D camera is a web camera intended for indoor use, the intensity of the light outdoors causes the images to “wash out” and become white where there is intense light reflected. When this occurs, the images have low contrast as information is lost, so it is difficult to detect any meaningful edges and perform calculations. This issue could be resolved by applying an optical filter to block out the excess light.   It was found that the Minoru 3D camera needs to be able to find edges in order to perform depth analysis. If there is no edge found on the bottom of the drop-off, it cannot register a drop-off. To detect street curbs, it requires a painted line on the road (i.e. the bottom of the drop-off) for detection. For most roads, there is already a painted line. We also attempted to calculate and compare the distance from the drop-off, but had inaccurate results. This is possibly due to errors in edge correlation and possible instability of the camera fixture.   21   2.8 Key Issues From developing the drop-off detection system, several key issues were found that were difficult to address, and affected the accuracy and robustness of the system.   Table 2. Key Issues Issue Description Noise  When an image is processed for edge detection, there is inevitably some amount of noise associated with the processed result. These distortions could arise from a number of causes, such as the imperfections in the surfaces of the objects captured in the image. However, it was found that using a smoothing process before edge processing, the resulting image had less noise.  Poor lighting conditions With poor lighting, it was difficult to find edges as the input to the camera was not ideal. As such, it is a limitation of the system to have adequate lighting in order to find drop-offs. Heavily patterned surfaces/ Ambiguous and repetitive patterns Surfaces with intricate patterns are more time-consuming and complicated to process, since there are more edges. Often being able to find the correct corresponding pixels is harder when there are more potential matches in a given region. Reflective surfaces and shadows Light being reflected off a surface would distort and be seen as completely different for each camera. Also, the reflection “whites out” the surface, so possible identifying characteristics are eliminated. Similarly, shadows cause distortions that are seen differently in each camera. Blurring due to fast moving camera The movement of the camera impacts the image captured, as quick rotations would result in blurring, or streaking of the images. Because of this, the camera requires the user to not turn too quickly, or allow a second for the cameras to readjust.  Processing speed The processing speed of the images is on average 5 fps, which is reasonable. Since the Minoru webcam takes input at a maximum speed of 30 fps, it was expected that the processing speed would be slower. However, the system could be optimized so that the processing speed is increased.  Robustness and accuracy The system needs improvement in its robustness. While the system catches all drop-offs, there are false positives and its performance may be affected by lighting conditions. Also, although the system calculates good average for the depth, it has a large standard deviation.         22   Implications of Results The system is not yet ready for commercial use, and needs some improvement before it is a reliable system. However, the improvements suggested are relatively simple to carry out and should have a significant effect.    Sources of Error Error in measurement arises mainly from two general factors: poor lighting conditions and errors in correlation. Specular highlights resulting from reflecting light interfere with the depth analysis by creating edges, shapes of different colour intensities, and noise that is not anticipated. Implementing correct correlation is another key issue. False positive readings can arise due to noise. This can lead to mismatching edges, which can give results that are much greater or much smaller than expected.     Limits of Performance/ Measurement For the drop-off detection system to function effectively, it requires adequate lighting for the cameras to capture meaningful images. The system may also detect depth incorrectly when there are ambiguous patterns or complex patterns on surfaces. Another limitation of the system’s performance is if the user is moving or rotating quickly, such that the images obtained by the cameras become blurred. Moving quickly impacts the processing results and the computation time.    Suggestions for further testing procedures It is recommended that further testing is done with users trying to navigate their surroundings using the drop-off detection system. While the testing we have detailed in Section 2.6 is crucial in determining the performance of the system, satisfying technical requirements is not sufficient for qualifying a consumer product. Also, it would beneficial to carry out testing with a user who is visually impaired.   23    3. Conclusions  In the feasibility study of a robust drop-off detection system, it was found that, with minimal financial investment, such a system is able to efficiently detect drop-offs and obstacles, and audibly warn the user without delay. It has been found that such a system can be implemented with a relatively cheap stereo webcam, Velcro fixture, and [powerful] CPU running image processing software. In terms of image processing techniques, edge detection, edge location, and stereo vision calculations were sufficient in identifying drop-offs.  In testing and rating the system, a low error margin of 4.22 cm was found, well within the desired bounds of 7.62 cm. However, a high coefficient of variation of 48.8% was found, signifying a high discrepancy between individual results. This issue occurs due to the edge matching algorithm employed, which will not always match edges correctly. Another by-product of this algorithm is incorrect estimates of a user’s distance to the drop-off (for edge-noisy drop-offs). It has also been observed that in bright [sunny] conditions, camera intake is “washed” out, and no meaningful processing can take place. On average, the system is able to intake and process images from both cameras at 6.21 frames per second. Lastly, the system is able to identify drop-offs within a range of 0.4m to 3m, and an azimuthal range of ±30°.   Overall, DROPS shows that it is quite feasible to use such a drop-off detection system on a day-to-day basis. It is successfully able to identify and warn the user of all drop-offs over 4 inches in depth. DROPS has several limitations, but also a high range of flexibility in other areas. In a future project, with the specified improvements made (mentioned below in Section 5), DROPS could eventually become a relatively cheap, practical, commercial product.        24   4. Project Deliverables   4.1 List of deliverables The deliverable includes a working prototype which satisfies all of the capabilities mentioned above in Section 1.5 Project Objectives. However, due to time constraints and issues implementing, we have not completed the functions for outputting aural message and vibration; instead, an audible series of beeps are issued.   4.2 Financial Summary  Table 3. Bill of Materials Item #  Description  Quantity  Vendor  Cost  Order by  Funded by  1  Minoru 3D Camera  1  RobotShop  $65.21  Pranav  Dr. Leung  2  Vibrating disk motor VPM2  1  RobotShop  $3.50  Pranav  Dr. Leung          Shipping and taxes   $20.67           Total cost   $89.38        25   5. Recommendations  There are many recommendations for how to improve the project. These include:  1) Implementing speech feedback  a) Currently speech feedback is not implemented.  b) It is unknown why attempts at text to speech using Visual Studios SAPI (Speech Application Programming Interface), or playing pre-recorded warning messages did not output audio despite the code compiling.  c) Investigating why such methods are not working should bring more clarity and allow this function to be executed  2) Installing an optical filter to improve performance outdoors  a) An optical filter would likely improve the performance of the Minoru 3d camera outdoors and reduce errors in depth calculation b) It has yet to be determined which spectrums of light needs to be filtered (i.e. visible light, UV, IR, etc)  3) Improving calculations for range a) Currently range calculations are inaccurate for edge-noisy drop-offs, due to correlation errors. Improving the matching algorithm may give better results. See below.  b) Another contributing factor may be that the fixture holding up the Minoru camera is not stable, and the shift in tilt would yield inaccurate results as the calculations are based on a fixed angle the camera is tilted at. See below.   4) Redesigning strap and fixture for Minoru 3d camera to be more stable a) Current method could be more user friendly, as it requires adjusting the camera and ensuring it is aligned correctly. b) A more stable camera could produce better and more reliable results.   5) Improving processing speeds a) The current average processing speed is ~6fps. This may not be fast enough for the user to react within the closer range of finding a drop-off at around 0.5m.  b) A faster processing speed would also allow the user to turn quickly without having the images captured by the cameras blur.   6) Improving correlation algorithm a) Edge matching doesn’t work well for “noisy” drop offs/obstacles, being that there are many “edges” in ambiguous patterns.  b) A smarter and more effect correlation algorithm would help reduce miscalculations in range and depth, and enhance the overall performance. 7) Portability a) Use the suggested strategies in Section 2.4 to develop and run DROPS (or a similar system) portably, on a smaller processing unit.  8) Further testing a) With the recommended improvements made to the system, further testing is necessary not only to characterize the system, but also note how the potential user interacts with the system.  26   6. Appendices Appendix A – Key Software code  Application Dialog:  cvisionDlg.h:  // cvisionDlg.h : header file //  #pragma once #include "highgui.h" #include "cxcore.h" #include "cv.h" #include "Edger.h" #include <time.h> #include "Stereoer.h" #include "stdafx.h" #include "sapi.h"  // CcvisionDlg dialog class CcvisionDlg : public CDialogEx { // Construction public:  CcvisionDlg(CWnd* pParent = NULL);  // Dialog Data  enum { IDD = IDD_CVISION_DIALOG };   protected:  virtual void DoDataExchange(CDataExchange* pDX); // DDX/DDV support   // Implementation protected:  HICON m_hIcon;  IplImage *image;  // This is the image pointer (input)  Edger edger;  Stereoer stereoer;  ofstream timefile;   // Generated message map functions 27    virtual BOOL OnInitDialog();  afx_msg void OnSysCommand(UINT nID, LPARAM lParam);  afx_msg void OnPaint();  afx_msg HCURSOR OnQueryDragIcon();  DECLARE_MESSAGE_MAP() public:  afx_msg void OnProcess();  afx_msg void OnBnClickedButton2();  afx_msg void OnEditInstances();  afx_msg void OnLocateEdges();  afx_msg void OnBnClickedOk();  afx_msg void OnBnClickedCancel();  afx_msg void OnOpenImages();  afx_msg void OnDepth();  afx_msg void OnQuickSnap();  afx_msg void OnRunApp(); };  cvisionDlg.cpp: // cvisionDlg.cpp : implementation file //  #include "stdafx.h" #include "cvision.h" #include "cvisionDlg.h" #include "afxdialogex.h"  #ifdef _DEBUG #define new DEBUG_NEW #endif   // CAboutDlg dialog used for App About  class CAboutDlg : public CDialogEx { public:  CAboutDlg();  // Dialog Data  enum { IDD = IDD_ABOUTBOX };   protected:  virtual void DoDataExchange(CDataExchange* pDX);    // DDX/DDV support  // Implementation protected:  DECLARE_MESSAGE_MAP() };  CAboutDlg::CAboutDlg() : CDialogEx(CAboutDlg::IDD) { }  void CAboutDlg::DoDataExchange(CDataExchange* pDX) {  CDialogEx::DoDataExchange(pDX); }  BEGIN_MESSAGE_MAP(CAboutDlg, CDialogEx) END_MESSAGE_MAP()   // CcvisionDlg dialog     CcvisionDlg::CcvisionDlg(CWnd* pParent /*=NULL*/)  : CDialogEx(CcvisionDlg::IDD, pParent) {  m_hIcon = AfxGetApp()->LoadIcon(IDR_MAINFRAME);  timefile.open("time.txt"); }   28   void CcvisionDlg::DoDataExchange(CDataExchange* pDX) {  CDialogEx::DoDataExchange(pDX); }  BEGIN_MESSAGE_MAP(CcvisionDlg, CDialogEx)  ON_WM_SYSCOMMAND()  ON_WM_PAINT()  ON_WM_QUERYDRAGICON()  ON_BN_CLICKED(IDC_BUTTON1, &CcvisionDlg::OnProcess)  ON_BN_CLICKED(IDC_BUTTON2, &CcvisionDlg::OnBnClickedButton2)  ON_EN_CHANGE(IDC_EDIT2, &CcvisionDlg::OnEditInstances)  ON_BN_CLICKED(IDC_BUTTON4, &CcvisionDlg::OnLocateEdges)  ON_BN_CLICKED(IDOK, &CcvisionDlg::OnBnClickedOk)  ON_BN_CLICKED(IDCANCEL, &CcvisionDlg::OnBnClickedCancel)  ON_BN_CLICKED(IDC_BUTTON5, &CcvisionDlg::OnOpenImages)  ON_BN_CLICKED(IDC_BUTTON6, &CcvisionDlg::OnDepth)  ON_BN_CLICKED(IDC_BUTTON7, &CcvisionDlg::OnQuickSnap)  ON_BN_CLICKED(IDC_BUTTON3, &CcvisionDlg::OnRunApp) END_MESSAGE_MAP()   // CcvisionDlg message handlers  BOOL CcvisionDlg::OnInitDialog() {  CDialogEx::OnInitDialog();   // Add "About..." menu item to system menu.   // IDM_ABOUTBOX must be in the system command range.  ASSERT((IDM_ABOUTBOX & 0xFFF0) == IDM_ABOUTBOX);  ASSERT(IDM_ABOUTBOX < 0xF000);   CMenu* pSysMenu = GetSystemMenu(FALSE);  if (pSysMenu != NULL)  {   BOOL bNameValid;   CString strAboutMenu;   bNameValid = strAboutMenu.LoadString(IDS_ABOUTBOX);   ASSERT(bNameValid);   if (!strAboutMenu.IsEmpty())   {    pSysMenu->AppendMenu(MF_SEPARATOR);    pSysMenu->AppendMenu(MF_STRING, IDM_ABOUTBOX, strAboutMenu);   }  }   // Set the icon for this dialog.  The framework does this automatically  //  when the application's main window is not a dialog  SetIcon(m_hIcon, TRUE);   // Set big icon  SetIcon(m_hIcon, FALSE);  // Set small icon   // TODO: Add extra initialization here  image = 0;  return TRUE;  // return TRUE  unless you set the focus to a control }  void CcvisionDlg::OnSysCommand(UINT nID, LPARAM lParam) {  if ((nID & 0xFFF0) == IDM_ABOUTBOX)  {   CAboutDlg dlgAbout;   dlgAbout.DoModal();  }  else  {   CDialogEx::OnSysCommand(nID, lParam);  } }  // If you add a minimize button to your dialog, you will need the code below //  to draw the icon.  For MFC applications using the document/view model, //  this is automatically done for you by the framework.  void CcvisionDlg::OnPaint() {  if (IsIconic())  { 29     CPaintDC dc(this); // device context for painting    SendMessage(WM_ICONERASEBKGND, reinterpret_cast<WPARAM>(dc.GetSafeHdc()), 0);    // Center icon in client rectangle   int cxIcon = GetSystemMetrics(SM_CXICON);   int cyIcon = GetSystemMetrics(SM_CYICON);   CRect rect;   GetClientRect(&rect);   int x = (rect.Width() - cxIcon + 1) / 2;   int y = (rect.Height() - cyIcon + 1) / 2;    // Draw the icon   dc.DrawIcon(x, y, m_hIcon);  }  else  {   CDialogEx::OnPaint();  } }  // The system calls this function to obtain the cursor to display while the user drags //  the minimized window. HCURSOR CcvisionDlg::OnQueryDragIcon() {  return static_cast<HCURSOR>(m_hIcon); }      void CcvisionDlg::OnProcess() {  clock_t start, finish;  double  duration;  CString output;   start = clock();  edger.processImage(image);  finish = clock();  duration = (double)(finish - start);  output.Format("%f", duration);  SetDlgItemText(IDC_EDIT3, output);   cvShowImage("Processed Image", edger.getOutputImage());  GetDlgItem(IDC_BUTTON4)->EnableWindow(TRUE); }   void CcvisionDlg::OnBnClickedButton2() {  CFileDialog dlg(TRUE, _T("*.bmp"), NULL,   OFN_FILEMUSTEXIST|OFN_PATHMUSTEXIST|OFN_HIDEREADONLY,   _T("image files (*.bmp; *.jpg) |*.bmp;*.jpg|All Files (*.*)|*.*||"),NULL);    dlg.m_ofn.lpstrTitle= _T("Open Image");    if (dlg.DoModal() == IDOK) {     CString path= dlg.GetPathName();  // contain the selected filename   image= cvLoadImage(path);              // load the image  cvShowImage("Original Image", image);  // display it  GetDlgItem(IDC_BUTTON1)->EnableWindow(TRUE);  } }  void CcvisionDlg::OnEditInstances() {  // TODO:  If this is a RICHEDIT control, the control will not  // send this notification unless you override the CDialogEx::OnInitDialog()  // function and call CRichEditCtrl().SetEventMask()  // with the ENM_CHANGE flag ORed into the mask.   CString input;  int thresh;   GetDlgItemText(IDC_EDIT2, input);  30    thresh = atoi(input);  edger.setthreshold(thresh);  stereoer.setthreshold(thresh); }   void CcvisionDlg::OnLocateEdges() {  clock_t start, finish;  double  duration;  CString output;   start = clock();  edger.LocateEdges();  finish = clock();  duration = (double)(finish - start);  output.Format("%f", duration);  SetDlgItemText(IDC_EDIT3, output);   cvShowImage("Edges Located", edger.getOutputImage()); }   void CcvisionDlg::OnBnClickedOk() {  // TODO: Add your control notification handler code here  cvDestroyAllWindows();  cvReleaseImage(&image);  //edger.Edger::~Edger();  //stereoer.Stereoer::~Stereoer();  timefile.close();  CDialogEx::OnOK(); }   void CcvisionDlg::OnBnClickedCancel() {  // TODO: Add your control notification handler code here  cvDestroyAllWindows();  cvReleaseImage(&image);  //edger.Edger::~Edger();  //stereoer.Stereoer::~Stereoer();  timefile.close();  CDialogEx::OnCancel(); }   void CcvisionDlg::OnOpenImages() {  CFileDialog dlg(TRUE, _T("*.avi"), "",                     OFN_FILEMUSTEXIST|OFN_PATHMUSTEXIST|OFN_HIDEREADONLY|OFN_ALLOWMULTISELECT ,  "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.pgm;*.ppm;*.tif;*.tiff) |*.bmp;*.jpg;*.jpeg;*.png;*.pgm;*.ppm;*.tif;*.tiff|All Files (*.*)|*.*||",NULL);    char title[]= {"Open Image Sequence"};    const int maxFiles = 10; // you can select a maximum of 200 files  // MAX_PATH is constant specifying  // the maximal number of characters in a path  const int bufferSize = (maxFiles * (MAX_PATH + 1)) + 1;   dlg.m_ofn.lpstrTitle= title;  dlg.m_ofn.lpstrFile= new TCHAR[bufferSize];  dlg.m_ofn.lpstrFile[0]= '\0';                dlg.m_ofn.nMaxFile = bufferSize;  // should be maxFiles but there is a bug in this class...                                      if (dlg.DoModal() == IDOK) {     std::vector<CString> files;     // This is an iterator that extracts all filename   POSITION pos= dlg.GetStartPosition();   while (pos) {      files.push_back(dlg.GetNextPathName(pos));   }     // do something with the vector of filename   stereoer.setFiles(files); 31     if (files.size() > 1){    GetDlgItem(IDC_BUTTON6)->EnableWindow(TRUE);   }   else    GetDlgItem(IDC_BUTTON6)->EnableWindow(FALSE);  }    delete[] dlg.m_ofn.lpstrFile; }   void CcvisionDlg::OnDepth() {  clock_t start, finish;  double  duration;  CString output;  CWnd* Edit1 = GetDlgItem(IDC_EDIT1);   start = clock();  stereoer.run(0, Edit1);  finish = clock();  duration = (double)(finish - start);  output.Format("%f", duration);  SetDlgItemText(IDC_EDIT3, output);   cvShowImage("Edges Located", stereoer.getOutImage());  cvShowImage("Edges Located2", stereoer.getOutImage(1));    //output.Format("%f", depth);  //SetDlgItemText(IDC_EDIT1, output); }   void CcvisionDlg::OnQuickSnap() {  IplImage* leftimg = 0;  IplImage* rightimg = 0;  CvCapture* leftCam = 0;  CvCapture* rightCam = 0;   leftCam = cvCaptureFromCAM(0);  rightCam = cvCaptureFromCAM(1);  leftimg = cvQueryFrame(leftCam);  rightimg = cvQueryFrame(rightCam);   cvWaitKey(20);  cvShowImage("Bottom Img", leftimg);  cvShowImage("Top Img", rightimg);  cvSaveImage("Left.jpg", leftimg);  cvSaveImage("Right.jpg", rightimg);    cvReleaseCapture(&leftCam);  cvReleaseCapture(&rightCam); }  void CcvisionDlg::OnRunApp() {  clock_t start, finish;  double duration;  CString output;  CWnd* Edit1 = GetDlgItem(IDC_EDIT1);   start = clock();  stereoer.run(1, Edit1);  finish = clock();   duration = (double) (finish - start);  timefile << duration << endl;  //output.Format("%f", depth);  //SetDlgItemText(IDC_EDIT1, output);  output.Format("%f", duration);  SetDlgItemText(IDC_EDIT3, output); }   32   Edger.h: #if !defined EDGER #define EDGER  #include <cv.h> #include <highgui.h> #include <cxcore.h> #include <math.h>  #define DEFAULT_THRESHOLD 25  class Edger {    private:   IplImage *output;  int threshold;  int counter;  // private attributes    public:   // empty constructor  Edger() : output(0), threshold(DEFAULT_THRESHOLD), counter(0) {     // default parameter initialization here  }   // Add here all getters and setters for the parameters  void setthreshold(int n) {   threshold= n;  }  int getthreshold() {   return threshold;  }  IplImage* getOutputImage() {   return output;  }  // to check if an initialization is required  virtual bool isInitialized(IplImage *image) {   return output && (output->width == image->width) && (output->height == image->height);  }  // for all memory allocation  virtual void initialize(IplImage *image) {   cvReleaseImage(&output);   output= cvCreateImage(cvSize(image->width,image->height), image->depth, image->nChannels);  }  virtual void process(IplImage *image);  void LocateEdges();  // the method that checks for initilization  // and then process the image  inline void processImage(IplImage *image) {   //ensure a new image is used each time we run LocateEdges   if ((!isInitialized(image)) || (counter > 0)) {     initialize(image);   }   cvSmooth(image, image, 2,3,0,0,0);   cvSmooth(image, image, 2,3,0,0,0);   process(image);  }   // memory de-allocation  virtual void release(){   cvReleaseImage(&output);  }   ~Edger() {    release();  } };  #endif  33   Edger.cpp: #include "stdafx.h" #include "Edger.h"   void Edger::process(IplImage *image) {    int pixelwidth = image->nChannels;  int nl= image->height;  int nc= image->width * pixelwidth;  int step= image->widthStep;  int surr_pixels[8];  int gradx = 0;  int grady = 0;       unsigned char *data= reinterpret_cast<unsigned char *>(image->imageData);  unsigned char *dataout= reinterpret_cast<unsigned char *>(output->imageData);    data+= step;  // skip to 2nd line  dataout+= step;    for (int i=2; i<=(nl-1); i++) {   for (int j=5; j<=(nc-3); j+=pixelwidth) {    // process each pixel ---------------------    //read previous line    surr_pixels[0] = data[j-3-step];    surr_pixels[1] = data[j-step];    surr_pixels[2] = data[j+3-step];    //read current line    surr_pixels[3] = data[j-3];    surr_pixels[4] = data[j+3];    //read next line    surr_pixels[5] = data[j-3+step];    surr_pixels[6] = data[j+step];    surr_pixels[7] = data[j+3+step];     //canny/sobel method             gradx = surr_pixels[0]-surr_pixels[2]+2*surr_pixels[3]-2*surr_pixels[4]+surr_pixels[5]- surr_pixels[7];             grady = surr_pixels[0]+2*surr_pixels[1]+surr_pixels[2]-surr_pixels[5]-2*surr_pixels[6]- surr_pixels[7];     if ((sqrt(float(gradx*gradx+grady*grady)) > threshold) )    {                 dataout[j] = 0;     dataout[j-1] = 0;     dataout[j-2] = 255;    }             else    {                 dataout[j] = 255;     dataout[j-1] = 255;     dataout[j-2] = 255;    }      } // end of line         data+= step;  // next line   dataout+= step;  }  counter++;  //cvShowImage("Processed Image", output);  //cvSmooth(output, output, CV_MEDIAN, 3, 0, 0, 0);  //cvShowImage("Smooth Image", output);  //cvDilate(output, output, NULL, 1);  //cvShowImage("Dilated Image", output);  //cvErode(output, output, NULL, 1);  //cvShowImage("Eroded Image", output); }    void Edger::LocateEdges() {    int pixelwidth = output->nChannels; 34    int nl= output->height;  int nc= output->width * pixelwidth;  int step= output->widthStep;  int blkpcnt = 0;  int edgepoints[100] [2] = {0};  int height_tracker = 0;  bool going_up = false;  bool going_down = false;  bool skipRpixel = false;  bool skipTRpixel = false;  int checkpixel = 0;   int cnt = 0;    unsigned char *dataout= reinterpret_cast<unsigned char *>(output->imageData);   dataout+= (nl-2)*step; // skip to 2nd last line    for (int i=(nl-1); i>(nl/3); i--) {   for (int j=3; j<=(nc-3); j+=pixelwidth) {        //allow pixel detection priority to change every 10 dots    if ((blkpcnt >= 5) && ((blkpcnt % 5) == 0)){     if ((blkpcnt >= 10) && ((blkpcnt % 10) == 0)){      skipTRpixel = true;     }     else{      skipRpixel = true;     }    }   // begin locating edges ---------------------    //check current pixel / check pixel to right    if ((!skipRpixel) && (!skipTRpixel) && (dataout[j+2] == 0)){     blkpcnt++;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;    }    //checking top-right pixel    else if ((!skipTRpixel) && (blkpcnt>0) && (dataout[j+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;    }    //checking bottom-right pixel    else if ((blkpcnt>0) && (dataout[j+2+step] == 0)){     blkpcnt++;     dataout+=step;     height_tracker++;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;    }    //checking top-right pixel during priority switch    else if ((skipTRpixel) && (blkpcnt>0) && (dataout[j+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;    }    //checking top pixel    else if ((blkpcnt>0) && (!going_down) && (dataout[j-3+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     j-=3;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;     going_up = true;    }    //checking bottom pixel    else if ((blkpcnt>0) && (!going_up) && (dataout[j-3+2+step] == 0)){     blkpcnt++;     dataout+=step;     height_tracker++;     j-=3;     edgepoints[blkpcnt-1] [0] = j;     edgepoints[blkpcnt-1] [1] = height_tracker;     going_down = true; 35      }    else{//stop pixel count, return to latest pixel on original row     blkpcnt = 0;     dataout+=-height_tracker*step;     height_tracker=0;     for (int m = 0; m<=99; m++){      if (edgepoints[m] [1] != 0){       j = edgepoints[m-1] [0];       break;      }     }     for (int m = 0; m<=99; m++){      edgepoints[m] [0] = 0;      edgepoints[m] [1] = 0;     }     going_up = false;     going_down = false;    }    if (i+height_tracker >= nl){     break;    }    if (blkpcnt == 100){     //check angle of edge      //colour in edges for viewing     for (int n = 99; n >= 0; n--){       dataout[j-1*step] = 0;      dataout[j+1-1*step] = 0;      dataout[j+2-1*step] = 255;      dataout[j+1*step] = 0;      dataout[j+1+1*step] = 0;      dataout[j+2+1*step] = 255;            if (n>0){       j = edgepoints[n-1] [0];       dataout+=((edgepoints[n-1] [1]) - (edgepoints[n] [1]))*step;      }     }     blkpcnt = 0;     height_tracker=0;     for (int m = 0; m<=99; m++){      if (edgepoints[m] [1] != 0){       j = edgepoints[m-1] [0];       break;      }     }     for (int m = 0; m<=99; m++){      edgepoints[m] [0] = 0;      edgepoints[m] [1] = 0;     }     going_up = false;     going_down = false;     cnt++;     if (cnt > 799)      return;    }    skipRpixel = false;    skipTRpixel = false;      } // end of line   //ensure variables are reset and the proper row is reset before moving to next row   //(this is in case we are in the middle of detecting an edge and we hit the right end of the matrix)   blkpcnt = 0;   dataout+=-height_tracker*step;   height_tracker=0;   going_up = false;   going_down = false;   dataout-=step;    }  counter++; }   36   Stereoer.h:  #if !defined STEREOER #define STEREOER  #include <cv.h> #include <highgui.h> #include <cxcore.h> #include <math.h> #include "stdafx.h" #include "Edger.h" #include <time.h> #include <windows.h> // WinApi header #include "resource.h" #include <iostream> #include <fstream> using namespace std;   #define MAXEDGES 500 #define CIRCLECOUNT 30  #define FOCALLENGTH 0.003 #define IMGLENGTH 0.00236 #define IMGWIDTH 0.00176 #define BASELINE 0.0597 #define PHIDEG 35*(3.1415926535/180)  class Stereoer {  private:   std::vector<CString> files;   DWORD pause;   Edger edgeobject1, edgeobject2;   IplImage** outimages;   int edges[2][MAXEDGES][100][2];   int numedges[2];   int centreedges[4] [CIRCLECOUNT];   double depth, range;   ofstream myfile;   ofstream beepfile;   ofstream rangefile;  public:     Stereoer() : pause(0), outimages(0), edges(), numedges(), centreedges(), depth(0), range(0) {    outimages = new IplImage*[2];    myfile.open("depth.txt");    beepfile.open("beep.txt");    rangefile.open("range.txt");   }     void setFiles(std::vector<CString> &f) {      files= f;   }     void setPause(DWORD p) {            pause= p;   }     DWORD getPause() {            return pause;   }    void setthreshold(int n) {       edgeobject1.setthreshold(n);    edgeobject2.setthreshold(n);   }     IplImage* getOutImage(int chout = 0) {    return outimages[chout];   }    int LocateEdges(IplImage* output, bool wimage);  37     IplImage* transposeImage(IplImage* image) {     IplImage *rotated = cvCreateImage(cvSize(image->height,image->width), IPL_DEPTH_8U,image->nChannels);     CvPoint2D32f center;     float center_val = (float)((image->width)-1) / 2;    center.x = center_val;    center.y = center_val;    CvMat *mapMatrix = cvCreateMat( 2, 3, CV_32FC1 );     cv2DRotationMatrix(center, 90, 1.0, mapMatrix);    cvWarpAffine(image, rotated, mapMatrix, CV_INTER_LINEAR + CV_WARP_FILL_OUTLIERS, cvScalarAll(0));     cvReleaseMat(&mapMatrix);     return rotated;   }    virtual void initialize(IplImage *image, bool wimage = 0) {   if (!outimages[wimage])    cvReleaseImage(&outimages[wimage]);   outimages[wimage]= image;   }    void AnalyzeDepth(CWnd* Edit1);    void Mesh(IplImage* image1, IplImage* image2);    void run(bool appmode, CWnd* Edit1) {    IplImage *image, *image2;    CvCapture* leftCam = 0;    CvCapture* rightCam = 0;    int repetitions;     //CvVideoWriter *writer = 0;    //int fps = 5;    //writer=cvCreateVideoWriter("out.avi",CV_FOURCC('D', 'I', 'V', 'X'), fps,cvSize(480,640),1);     if (appmode) {    leftCam = cvCaptureFromCAM(0);    rightCam = cvCaptureFromCAM(1);    }     if (appmode) repetitions = 200;    else repetitions = 2;             for (int i=0; i<repetitions; i+=2) {       if (appmode) {      image = cvQueryFrame(leftCam);      image2 = cvQueryFrame(rightCam);      cvWaitKey(20);     }     else {      image = cvLoadImage(files[i],-1);      image2 = cvLoadImage(files[i+1],-1);     }           image = transposeImage(image);     image2 = transposeImage(image2);      edgeobject1.processImage(image);     edgeobject2.processImage(image2);      initialize(edgeobject1.getOutputImage());     initialize(edgeobject2.getOutputImage(), 1);     numedges[0] = LocateEdges(outimages[0], 0);     numedges[1] = LocateEdges(outimages[1], 1);     //analyze outimages     AnalyzeDepth(Edit1);          if (appmode) {     cvShowImage("Bottom Img", outimages[0]); 38       cvShowImage("Top Img", outimages[1]);     //cvWriteFrame(writer,outimages[0]);     }     //else cvSaveImage("Bottom.jpg", outimages[0]);     Sleep(pause);     cvReleaseImage(&image);     cvReleaseImage(&image2);    }    cvReleaseCapture(&leftCam);    cvReleaseCapture(&rightCam);    //cvReleaseVideoWriter(&writer);       }    virtual void release(){    myfile.close();    beepfile.close();    rangefile.close();    }    ~Stereoer() {     release();   } }; #endif  Stereoer.cpp:  #include "stdafx.h" #include "Stereoer.h"  void Stereoer::AnalyzeDepth(CWnd* Edit1) {   //Code is set up for this function, use:  //edges 4d array for all known detected edges  //and calculate corresponding points and then  //carry out depth calculations  //for debugging: output images for testing   int i = 0;  //int halfpoint = outimages[0]->width * (outimages[0]->nChannels)/2;  int j = 0;  int edgecount = 0;  bool wimage = 0;  bool edgezero = 0;  bool beep = 0;  bool rangecheck = 0;  double height1 = 0, height2 = 0, tempdepth;  double Ydown = 0, Yup = 0;  CString output;   for (int m = 0; m<CIRCLECOUNT; m++){   centreedges [0] [m] = 0;   centreedges [1] [m] = 0;   centreedges [2] [m] = 0;   centreedges [3] [m] = 0;  }    //Find and store relevant edges into centreedges array  while (i < MAXEDGES) {   for (j = 0; j < 100; j++) {    if (edges[wimage][i][j][0] == 720) {     //j = 240 - edges[wimage][i][j][0];     //cvCircle(outimages[wimage],cvPoint(240, (edges[wimage][i][j][1])), 20, cvScalar(0, 255, 0, 0),1, 8, 0);     if (edgecount < CIRCLECOUNT) {      if (edgecount > 0) {       if (abs(edges[wimage][i][j][1] - edges[wimage][centreedges[wimage][edgecount-1]][j][1]) <= 10) {        break;       }      }      centreedges[wimage] [edgecount] = i;      centreedges[wimage+2] [edgecount] = j;      edgecount++; 39       }     break;    }   }   i++;   if ((i>numedges[wimage]-1) && (!wimage)) {    i = 0;    edgecount = 0;    wimage = 1;   }   else if (i>numedges[wimage]-1)    break;  }   //Draw Circles on centre edges  wimage = 0;  for (int k = 0; k < CIRCLECOUNT; k++) {   if (centreedges[wimage][k] != 0) {    cvCircle(outimages[wimage],cvPoint(240, (edges[wimage][centreedges[wimage][k]][centreedges[wimage+2][k]][1])), 20, cvScalar(0, 255, 0, 0),1, 8, 0);   }   else if ((centreedges[wimage][k] == 0) && (!edgezero)) {    cvCircle(outimages[wimage],cvPoint(240, (edges[wimage][centreedges[wimage][k]][centreedges[wimage+2][k]][1])), 20, cvScalar(0, 255, 0, 0),1, 8, 0);    edgezero = true;   }   if ((k == (CIRCLECOUNT - 1)) && (!wimage)) {    wimage = 1;    k = -1;   }  }    //calculate depth  for (int m = 0; m < 5; m++) {   int n = m;   int o = m+1;    if ((abs(edges[0][centreedges[0][n]][centreedges[2][n]][1] - edges[1][centreedges[1][n]][centreedges[3][n]][1]) >= 35) || (abs(edges[0][centreedges[0][o]][centreedges[2][o]][1] - edges[1][centreedges[1][o]][centreedges[3][o]][1]) >= 35)) {    continue;   }    if ((m > 0) && ((centreedges[0][n] == 0) || (centreedges[0][o] == 0) || (centreedges[1][n] == 0) || (centreedges[1][o] == 0))) {    continue;   }    Ydown = ((640 - edges[0][centreedges[0][n]][centreedges[2][n]][1])*(IMGLENGTH/640));   Yup = ((640 - edges[1][centreedges[1][n]][centreedges[3][n]][1])*(IMGLENGTH/640));   height1 = ((BASELINE/(Ydown-Yup))*((abs(Ydown- 0.5*IMGLENGTH))*cos(PHIDEG)+FOCALLENGTH*sin(PHIDEG)));      range = ((BASELINE*FOCALLENGTH)/(Ydown-Yup))*cos(PHIDEG);   if ((!rangecheck) && (abs(range)<= 3.5)){    rangefile << range << endl;    rangecheck = 1;   }    Ydown = ((640 - edges[0][centreedges[0][o]][centreedges[2][o]][1])*(IMGLENGTH/640));   Yup = ((640 - edges[1][centreedges[1][o]][centreedges[3][o]][1])*(IMGLENGTH/640));   height2 = ((BASELINE/(Ydown-Yup))*((abs(Ydown- 0.5*IMGLENGTH))*cos(PHIDEG)+FOCALLENGTH*sin(PHIDEG)));      tempdepth = abs(height2 - height1);       if ((tempdepth > 0.10) && (tempdepth <=0.50)){    Beep(723,50);    if (!beep) {     beepfile << tempdepth << endl;     beep = 1;    }    myfile << tempdepth << endl;    output.Format("%f", tempdepth);    Edit1->SetWindowTextA(output);    depth = tempdepth;     40      //make a sound   }   //REMOVE THIS SLEEP   //Sleep(300);  }   }  void Stereoer::Mesh(IplImage *image1, IplImage* image2) {    int pixelwidth = image1->nChannels;  int nl= image1->height;  int nc= image1->width * pixelwidth;  int step= image1->widthStep;    IplImage* output = cvCreateImage(cvSize(image1->width,image1->height), image1->depth, image1- >nChannels);  unsigned char *left= reinterpret_cast<unsigned char *>(image1->imageData);  unsigned char *right= reinterpret_cast<unsigned char *>(image2->imageData);  unsigned char *dataout= reinterpret_cast<unsigned char *>(output->imageData);    left+= step;  // skip to 2nd line  right+= step;  // skip to 2nd line  dataout+= step;    for (int i=2; i<=(nl-1); i++) {   for (int j=3; j<=(nc-3); j+=pixelwidth) {    // process each pixel ---------------------    //read previous line    dataout[j] = 0;    dataout[j+1] = 255;    dataout[j+2] = 156;      } // end of line         left+= step;  // next line   right+= step;  // next line   dataout+= step;  }  cvShowImage("testing", output);   }  int Stereoer::LocateEdges(IplImage* output, bool wimage) {    int pixelwidth = output->nChannels;  int nl= output->height;  int nc= output->width * pixelwidth;  int step= output->widthStep;  int blkpcnt = 0;  //int edges[100] [2] = {0}; //MAKE A 4XD ARRAY EDGEPOINTS, AND USE WIMAGE HERE TO PICK WHICH ONE YOU ARE USING  int height_tracker = 0;  bool going_up = false;  bool going_down = false;  bool skipRpixel = false;  bool skipTRpixel = false;  int checkpixel = 0;  int edgenum = 0;    unsigned char *dataout= reinterpret_cast<unsigned char *>(output->imageData);   dataout+= (nl-2)*step; // skip to 2nd last line    for (int i=(nl-1); i>5; i--) {   for (int j=3; j<=(nc-3); j+=pixelwidth) {        //allow pixel detection priority to change every 10 dots    if ((blkpcnt >= 5) && ((blkpcnt % 5) == 0)){     if ((blkpcnt >= 10) && ((blkpcnt % 10) == 0)){      skipTRpixel = true;     }     else{      skipRpixel = true;     }    }   // begin locating edges ---------------------    //check current pixel / check pixel to right 41      if ((!skipRpixel) && (!skipTRpixel) && (dataout[j+2] == 0)){     blkpcnt++;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;    }    //checking top-right pixel    else if ((!skipTRpixel) && (blkpcnt>0) && (dataout[j+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;    }    //checking bottom-right pixel    else if ((blkpcnt>0) && (dataout[j+2+step] == 0)){     blkpcnt++;     dataout+=step;     height_tracker++;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;    }    //checking top-right pixel during priority switch    else if ((skipTRpixel) && (blkpcnt>0) && (dataout[j+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;    }    //checking top pixel    else if ((blkpcnt>0) && (!going_down) && (dataout[j-3+2-step] == 0)){     blkpcnt++;     dataout-=step;     height_tracker--;     j-=3;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;     going_up = true;    }    //checking bottom pixel    else if ((blkpcnt>0) && (!going_up) && (dataout[j-3+2+step] == 0)){     blkpcnt++;     dataout+=step;     height_tracker++;     j-=3;     edges[wimage][edgenum][blkpcnt-1] [0] = j;     edges[wimage][edgenum][blkpcnt-1] [1] = height_tracker;     going_down = true;    }    else{//stop pixel count, return to latest pixel on original row     blkpcnt = 0;     dataout+=-height_tracker*step;     height_tracker=0;     for (int m = 0; m<=99; m++){      if (edges[wimage][edgenum][m] [1] != 0){       j = edges[wimage][edgenum][m-1] [0];       break;      }     }     for (int m = 0; m<=99; m++){      edges[wimage][edgenum][m] [0] = 0;      edges[wimage][edgenum][m] [1] = 0;     }     going_up = false;     going_down = false;    }    if ((i+height_tracker >= nl) || (i+height_tracker <=5)){     break;    }    if (blkpcnt == 100){     //check angle of edge      //colour in edges for viewing     for (int n = 99; n >= 0; n--){       dataout[j-1*step] = 0;      dataout[j+1-1*step] = 0;      dataout[j+2-1*step] = 255;      dataout[j+1*step] = 0; 42        dataout[j+1+1*step] = 0;      dataout[j+2+1*step] = 255;            if (n>0){       j = edges[wimage][edgenum][n-1] [0];       dataout+=((edges[wimage][edgenum][n-1] [1]) - (edges[wimage][edgenum][n] [1]))*step;      }     }     //reset all values     blkpcnt = 0;     height_tracker=0;     for (int m = 0; m<=99; m++){      if (edges[wimage][edgenum][m] [1] != 0){       j = edges[wimage][edgenum][m-1] [0];       break;      }     }     for (int m = 0; m<=99; m++){      //edges[wimage][edgenum][m] [0] /= pixelwidth;      edges[wimage][edgenum][m] [1] += i;     }     going_up = false;     going_down = false;     edgenum++;     if (edgenum > (MAXEDGES - 1))      return edgenum;    }    skipRpixel = false;    skipTRpixel = false;      } // end of line   //ensure variables are reset and the proper row is reset before moving to next row   //(this is in case we are in the middle of detecting an edge and we hit the right end of the matrix)   blkpcnt = 0;   dataout+=-height_tracker*step;   height_tracker=0;   going_up = false;   going_down = false;   dataout-=step;    }  return edgenum; }     43   Appendix B - Technical Specifications  Camera chip for Minoru 3D camera   44   Appendix B - Technical Specifications Vibrating disc motor  49   Appendix D - Bill of Materials  Item #  Description  Quantity  Vendor  Cost  Order by  Funded by  1  Minoru 3D Camera  1  RobotShop  $65.21  Pranav  Dr. Leung  2  Vibrating disk motor VPM2  1  RobotShop  $3.50  Pranav  Dr. Leung          Shipping and taxes   $20.67           Total cost   $89.38      50   References  Related research conducted:  H. Shen, K.Y. Chan, J. Coughlan, and J. Brabyn. A Mobile Phone System to Find Crosswalks for Visually Impaired Pedestrians. Technology and Disability. 2008; 20(3):217–224.  V. Ivanchenko, J. Coughlan, and H. Shen. Staying in the Crosswalk: A System for Guiding Visually Impaired Pedestrians at Traffic Intersections. Assist technol Res Ser. 2009; 25(2009): 69–73.   A. Ruta, Y. Li, and X. Liu. Traffic Sign Recognition Using Discriminative Local Features. School of Information System, Computing and Mathematics, Brunel University. <http://videolectures.net/ida07_ruta_tsr/> (Oct. 8, 2009)  J. Hatzidimos. Automatic traffic sign recognition in digital images. Proceedings of the International Conference on Theory and Applications of Mathematics and Informatics (ICTAMI). Acta Universitatis Apulensis, University of Alba Iulia, Romania. (2004)  M. Shneier. Road Sign Detection and Recognition. National Institute of Standards and Technology, Gaithersburg. (2005)  D.  Shaposhnikov, L. Podladchikova, A. Golovan, N. Shevtsova, K. Hong, and X. Gao. Road Sign Recognition by Single Positioning of Space-Variant Sensor Window. Proc. 15th International Conference on Vision Interface, 213-217, 2002. Calgary, Canada.  A. Lorsakul and J. Suthakorn. Traffic Sign Recognition for Intelligent Vehicle/Driver Assistance System Using Neural Network on OpenCV. 4th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2007)  A. Ruta,  Y. Li, and X. Liu. Real-time traffic sign recognition from video by class-specific discriminative features. Pattern Recognition 43 (2010) 416 - 430  J.S. Ju, E. Ko, and E. Y. Kim. EYECane: navigating with camera embedded white cane for visually impaired person. ASSETS ’09, Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility (Oct 2009) 233- 234  C. Ye and M. Bruch. A Visual Odometry Method Based on the SwissRanger SR4000. Proc. SPIE, Vol. 7692, 76921I (2010)  C. Ye and G.M. Hegde. Robust Edge Extraction for Swissranger SR-3000 Range Images. ICRA'09 Proceedings of the 2009 IEEE international conference on Robotics and Automation (2009) 2396-2401  J. Coughlan and H. Shen. Terrain Analysis for Blind Wheelchair Users: Computer Vision Algorithms for Finding Curbs And Other Negative Obstacles. Conference and Workshop on Assistive Technology for People with Vision and Hearing Impairments (CVHI '07). August 2007  51    Components research: Surveyor SRV-1 BlackFin Camera - http://www.surveyor.com/blackfin/ Microcontroller Frame Grabber (uCFG) - http://www.digitalcreationlabs.com/uCFG.htm Microcontroller UART tutorial - http://www.societyofrobots.com/microcontroller_uart.shtml  Software research: Computer Vision Tutorial - http://www.societyofrobots.com/programming_computer_vision_tutorial.shtml C library for Atmel AVR controllers - http://webbot.org.uk/iPoint/30.page Computer Vision Software - http://www.cs.cmu.edu/~cil/v-source.html Octave Image Package - http://octave.sourceforge.net/image/index.html Octave/ Open CV tutorial - http://www.embedautomation.com/content/setting-octave- and-open-cv-image-processing Octave image processing guide - http://www.cheric.org/education/eduaids/octave/octman/octave_88.html  Existing electronic visual aids JES Semiconductor (white cane product using RFID) - http://rfidepc.com.tw/EWC.htm I-CANE – Intelligence in a Cane - http://www.i-cane.nl/en/system  Smart cane -  http://www.innovations- report.com/html/reports/medicine_health/scientists_creates_smart_cane_blind_162791.html cane - http://www.gizmag.com/go/3827/ Trekker Breeze - http://www.afb.org/afbpress/pub.asp?DocID=aw100202 Tormez GPS - http://www.esa.int/esaNA/SEMVQOS1VED_index_0.html  Researching user perspective Video guide to white cane techniques - http://www.youtube.com/watch?v=4hlqAPH8Ulg&NR=1 Forum for the visually impaired - http://www.afb.org/message_board_replies2.asp?TopicID=6132&FolderID=3 D. Tzovaras,  K. Moustakas, G. Nikolakis, and M. Strintzis. Interactive mixed reality white cane simulation for the training of the blind and the visually impaired.  Personal and Ubiquitous Computing, Volume 13, Number 1, January 2009 , pp. 51-58(8)  B.S. Hoyle, J.M Fowler, D.A.Waters, D.J.Withington. Development Of The Electronic Guide Cane For Enhanced Primary Mobility For The Vision Impaired. Conference and Workshop on Assistive Technologies for Vision and Hearing Impairment (CVHI) (2004)  

Cite

Citation Scheme:

    

Usage Statistics

Country Views Downloads
China 4 0
France 4 0
United States 3 0
United Kingdom 2 0
Canada 2 0
Turkey 2 0
Japan 2 0
Iran 1 0
City Views Downloads
Unknown 5 0
Ashburn 3 0
Beijing 3 0
Corsham 2 0
Izmir 2 0
Tokyo 2 0
Vancouver 1 0
Angers 1 0
Shenzhen 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}
Download Stats

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.7410.1-0074462/manifest

Comment

Related Items