UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Hemispheric vision with resolution enhancement 1998

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata


ubc_1999-0002.pdf [ 9.64MB ]
JSON: 1.0065287.json
JSON-LD: 1.0065287+ld.json
RDF/XML (Pretty): 1.0065287.xml
RDF/JSON: 1.0065287+rdf.json
Turtle: 1.0065287+rdf-turtle.txt
N-Triples: 1.0065287+rdf-ntriples.txt

Full Text

Hemispheric Vision with Resolution Enhancement by Dean M . H . Anderson B . A . S c , The University of Br i t i sh Columbia , 1996 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE D E G R E E OF Master of Applied Science in THE FACULTY OF GRADUATE STUDIES (Department of Electrical and Computer Engineering) We accept this thesis as conforming to the^qtfcjred standard T H E U N I V E R S I T Y O F B R I T I S H C O L U M B I A December 1998 © D e a n M . H . Anderson, 1998 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. The University of British Columbia Vancouver, Canada DE-6 (2/88) 9 ABSTRACT Representing the whole world around a given point is an important goal in computer vision and many other applications. Most camera lens systems have been designed and optimized to provide excellent perspective images; however, this leaves out al l the information behind and around the camera. Previous methods of obtaining a hemispheric (half-world) and omnidirectional (full-world) view of the world have involved fisheye lenses as well as pan and tilt cameras. In this thesis, an omnidirectional vision system using two parabolic reflectors is described along wi th a technique and apparatus to obtain a higher resolution from an imaging system. The system has two parabolic mirrors for imaging. It allows a substantially hemispheric or half omnidirectional view of the world. T w o of these sensors placed back to back would provide an omnidirectional view of the world from a single viewpoint. The results from the hemispheric, double parabolic mirror system that we buil t are shown in the thesis. The main disadvantage of mapping such a large field of view onto a single sensor is the loss of resolution. We therefore describe a method to obtain better resolution using several image frames wi th control over the displacement of these frames. This involves meshing together several image frames displaced from each other by known amounts. The images are obtained from a variable angle pr ism normally used for image stabilization in camcorders. The results are promising. 11 Table of Contents Abstract ii List of Tables vi List of Figures vii Acknowledgements x 1 Introduction 1 1.1 The Opt ica l Problem 3 1.2 The Resolution Problem 7 1.3 Overview of the Thesis 9 2 Hemispheric Imaging 11 2.1 Proposed System 11 2.2 System Parameters 16 2.3 Advantage of Using Parabolic Reflectors 20 2.4 Limitat ions 21 2.5 Mapping Hemispheric Image into a Perspective Image 22 2.6 Results 23 2.6.1 Computing Requirements 25 m t. 2.7 Conclusion 25 3 Super-Resolution 27 3.1 Theory 27 3.1.1 Approach 27 3.1.2 One Dimensional Analysis 28 3.1.3 Image Formation 32 3.1.4 Methods for Reconstruction 33 3.1.5 Registration Problem 34 3.1.6 Restoration, Inverse Fi l te r ing 36 3.1.7 High Resolution Reconstruction 37 3.2 Simulation 38 3.2.1 Reconstruction wi th Known Shifts 38 3.3 F ina l Remarks on Super-resolution Problem 46 4 A Sub-Pixel Resolution System 47 4.1 Description of V A P 47 4.2 Appl ied Super-Resolution 57 4.2.1 Centroid Cal ibrat ion 60 4.3 Control of the V A P 62 4.4 Experimental Results using ES-750 66 4.4.1 Init ial Tests 66 4.4.2 Open-Loop Results -. 67 4.4.3 Sampling at Subpixel Intervals 76 4.5 Issues and Problems 80 5 Conclusions 83 5.1 Future Work for Combined System 84 iv 5.2 Summary of Results 86 Bibliography 88 Appendix A Aberrations (see Chapter 2) 92 v List of Tables 2.1 Mi r ro r Specifications 20 4.1 U S A F 1951 Resolution Test Chart (lppm) 76 v i List of Figures 1.1 Image from a Fisheye Lens 4 1.2 Orthographic Projection 5 1.3 Orthographic View of A r i e l ( N A S A web site) 7 1.4 O M N I C A M E R A ( T m ) 8 2.1 C O n i c Projection Image Sensor(COPIS) 12 2.2 View of C O P I S Sensor 12 2.3 Two Mir ror Hemispheric Imaging System 13 2.4 Convex Parabolic Reflector 14 2.5 Two Mi r ro r Hemispheric Vis ion system 15 2.6 Two Mir ro r Hemispheric Vis ion System 17 2.7 Hemispheric Scene taken with Double Parabolic System 24 2.8 Perspective Image from Hemispheric View(F ig . 2.7) 24 2.9 Perspective Image found by Deconvolving F i g . 2.7 24 3.1 Ideal pixel P S F array wi th non-zero separation 28 3.2 Spatial Signals of the Sensor and Image 30 3.3 Averaging of a higher resolution image sequence 31 3.4 Original 256x256x8bit Lena Test Image 38 3.5 Simulation Model 40 v i i 3.6 Low Resolution 64x64x8bit Lena Image Frame (obtained by applying H2 to F i g . 1) 41 3.7 High Resolution Reconstructed Image (no deblurring applied) . . . 42 3.8 Wiener Fil tered 256x256x8bit Lena Test Image (from Noisy Frames) 44 3.9 Image Correlation between Frames 45 4.1 Complete V A P Assembly (US 5481394) . 48 4.2 Front and Rear Plates of V A P (US pat. 5481394) 50 4.3 Large P r i sm 51 4.4 One Dimensional Structural V iew of V A P 51 4.5 Geometry of V A P movement (US pat. 5481394) 53 4.6 Restriction of V A P movement (US pat. 5481394) 54 4.7 V A P actuator unit (US 5481394) 55 4.8 Side View of V A P Actuator (US 5623305) 56 4.9 Our Front E n d Differential Position Amplif ier 58 4.10 V A P Deflection Angle versus Drive Voltage (US 5623305) 59 4.11 Open Loop representation of V A P 62 4.12 V A P Bode Plots (US 5623305) 64 4.13 Closed Loop representation of V A P 65 4.14 Dua l C o i l Wrapping of V A P Actuator (US 5623305) 66 4.15 Our Control Circui t 67 4.16 Vert ical Displacement with Sinewave Input 68 4.17 Horizontal Displacement wi th Sinewave Input 69 4.18 Horizontal Direction V A P Frequency Response (0.5 V Sinewave) . 69 4.19 Vert ical Direction V A P Frequency Response (0.5 V Sinewave) . . . 70 4.20 Square Wave Response 70 4.21 Displacement wi th Compensating Op A m p 71 v in 4.22 V A P run open loop in one direction 72 4.23 Hysteresis of V A P 73 4.24 V A P Drift over T ime 74 4.25 V A P Displacement versus Appl ied Voltage 75 4.26 Zoom of Raw U S A F 1951 Test Image 77 4.27 Image Reconstruction 78 IX ACKNOWLEDGEMENTS I would like to acknowledge Ray Burge, Dr . Greg Grudic , Henry Wong, Dr . Peter Lawrence and Dr . T i m Salcudean for their help, equipment use, and technical assistance. In addition, I would like to thank the technical staff who provided me with enormous assistance and patience (Don Dawson, Bruce Dow, Tony Leugner, A l Prince, Rob Ross and L loyd Welder). Final ly , I would like to thank Ray Burge again for his enthusiasm and exceptional help throughout the entire project. x Chapter 1 Introduction Hemispheric sensors are being developed[Y + 95, Nay96] to allow imaging wi th a field of view of 360 degrees. This means that every lateral direction of a scene can be viewed. The idea is to capture the whole scene around a fixed object and create a panorama or an omnidirectional view of the world at that point. These sensors are extremely useful in such applications as video conferencing and surveillance. W i t h the hemispheric view, several perspective images can be viewed of the world about the camera at the same instant. A perspective image is one which is linear and undistorted, as in normal eyesight. In a video conferencing application, a single sensor could sit on the table and the receiver at the other end could pick which participant to view at a given time or choose to view some or al l of them at once. Similarly, i n surveillance, the ideal situation is to have a camera that cannot be compromised. No one can sneak up on or dodge an omnidirectional camera. W i t h normal cameras the security is l imited if the person being watched knows where it is pointing. The most popular solution to this security problem for normal non-omnidirectional cameras is to mount them inside a silvered dome so 1 2 that no one knows if they are being watched. More applications for omnidirectional cameras wi l l materialize from the benefit of seeing the whole scene around a fixed point. They have a larger field of view than the unaided eye and as such can image more of the surroundings than a casual observer or a conventional camera. W i t h zooming capability, the possibilities for use increase. In this thesis we wi l l discuss a new design for an omnidirection camera. We then discuss a system that can be used to overcome the main l imitat ion of the camera, which is a lack of resolution. The first attempts at omnidirectional imaging involved a fish eye lens[H + 86]. These lenses were built as an extreme wide angle lens and are st i l l popular today for many applications. More recent methods have involved pan and ti l t cameras[KA96]. These are cameras mounted on a platform containing two joints that can be panned and t i l ted to capture the scene at each angle. The multiple scenes so derived are pieced together to create either a panorama(when there is only panning and no t i l t ing of the camera) or an omnidirectional view. The pan and ti l t is popular in speed independent applications such as real estate imaging. In this application, the client can then look at each scene around different rooms or landscapes without visit ing the site. Clearly, it is not important for the views to be in real-time. Recent omnidirectional cameras are using mirrors to perform the imaging. They are formed as standard conic sections and offer the best promise for less distortion of the reconstructed perspective images. Our system is part of this subset of om- nidirectional sensors, the advantages of which w i l l become apparent to the reader later in this thesis. 3 1.1 The Optical Problem Conventional imaging systems are very l imited in their field of view. Cameras are designed to image plane perspective scenes with no distortion in the image plane. Attempts to recover a hemispheric scene originated with fish eye lenses. We propose a system wi th two parabolic mirrors to accurately image a scene about a complete hemisphere. In addition, a resolution recovery scheme is proposed to compensate for the loss of information inherent in imaging such a large field of view. Put t ing two of our cameras back to back allows almost a full omnidirectional view. Fish Eye Lens The common fish eye lens, such as that seen on door peepholes, is a lens designed for a wide field of view. The problem with these lenses is that it is difficult to make two lenses identical in diameter and focal length that have the same distortion pattern[Nal96]. It is difficult to determine how the distortion affects the image. Perspective images cannot be recovered from a fish eye lens without knowing the exact geometry of the lens or the exact distortion caused by the lens, which varies from lens to lens and takes considerable time to measure. F i sh eye lenses are also notoriously expensive to manufacture. A scene imaged from a fisheye lens([SA94]) is shown in F i g . 1.1. Pan and Tilt Another approach for the imaging of a hemispheric scene has been the use of a pan and t i l t camera. The camera is essentially mounted on a platform that can be pan and t i l ted to acquire images from a full hemisphere. The images are then  5 Projection plane (top view) (front view) Figure 1.2: Orthographic Projection stitched together to recreate the hemispheric scene. This method suffers from serious . problems. Stitching the scenes together is difficult because the overlapping image boundaries must be identified for each scene. The mechanical nature of the mount used for panning and t i l t ing can also create accuracy problems. There can be calibration problems i n keeping the center of the imaging surface at the center of movement. The two main disadvantages of this system are the required high mechanical accuracy and the slow speed. A stepping motor must be used and it cannot be driven fast or backlash wi l l plague the response. 6 Omnicamera The Omnicamera(Tm) (Fig. 1.4) was developed by Shree Nayar at Columbia Universi ty[NPB98]. He investigated omnidirectional catadioptric (which means a combined mirror and lens system) sensors using single mirrors wi th a conical cross section across the optical axis. The goal was to find systems wi th a single view- point. The proposed solution was to use a single parabolic primary mirror. Above the primary mirror, a camera system wi th an orthographic lens is used to image the parallel lines from the lower mirror. A n orthographic lens is a telecentric(centred on the optical axis) lens that images the full side of the object wi th the line of sight perpendicular to that object. F ig . 1.2 shows several orthographic projections of an object[Spe84]. This type of view is commonly used in satellite imaging of planets (Fig . 1.3). In the Omnicamera, the lower mirror in his system performs the same purpose as the one we describe later, which is to allow for a single viewpoint. The orthographic lens must be the same size as the lower mirror in order to get the full image. Because of the imaging geometry, the camera must be mounted outside the dome (see F i g 1.4). This is a disadvantage because the camera must be large to accomodate the orthographic lens size and is therefore easily detectable, damaged or knocked out of alignment due to its position. In addition, the orthographic (telecentric) lens is expensive and requires a fair amount of calibration to set up wi th the camera and align wi th the optical axis. 7 Figure 1.3: Orthographic View of Ar i e l ( N A S A web site) 1.2 The Resolution Problem In many imaging situations it is desirable to get a higher resolution than is obtained through the standard optics. This is the especially the case in wide field of view systems. Every optical system, however, wi l l have a l imi t ing resolution based on the optics or the detector grid spacing and size. Even the most expensive components degrade the resulting imaged view of the real world to some extent. For further resolution improvement, it becomes necessary to choose a different approach than just purchasing the best optics and detectors. Resolution is an often misunderstood term. The common misinterpretation involves thinking it is the pixel size of an image. A fixed 320x320 picture does not necessarily 8 Figure 1.4: O M N I C A M E R A ( T m ) have its resolution doubled along each dimension when it is resized to 640x640. A n image manipulation program simply doubles the pixel size of the original image. There is no new information to be obtained. Resolution of a picture is measured by the bandwidth of the signal from the sensor (lines per m m or otherwise) [Gro86]. If one defines 'resolution' as the number of lines visible in the image divided by the field of view, there are three factors that affect the resolution. First is the sharpness of the optics. Second is the number of pixels in the sensor used to cover the field of view. The third factor affecting resolution is the size of each pixel . The smaller the pixel size, the more ideal the sampling but the lower the light sensitivity of the camera. Resolution as defined in this thesis is measured in lines per millimetre. 9 Recovering the actual real world from a single low resolution image is not possible because the optics and sensor wi l l always have a low pass filtering effect on the image. What is needed is to use multiple images in a constructive manner to recover that lost information. One approach for recovering high resolution is to look at several low-resolution images of the same specimen. If each image is shifted from each other by a known amount, a higher resolution reconstruction can be done. When the amount of shift is not known, it is possible to model the registration mismatch to determine the relationship between images[KBV90, MS88 , SPS87]. In summary, al l components of the optical system affect the final image and image resolution. The main degradation however, with today's quality optics, almost always occurs at the detector or sensor[Luk66, Luk67]. This is particularly the case for C C D or C M O S cameras and imaging systems [Nor78]. In this thesis, we wi l l model the detector and use a linear procedure to recover the high resolution by using multiple images. 1 1.3 Overview of the Thesis This thesis directly addresses the problems identified in the state of the art by introducing a new optical approach to solving the problems in a hemispheric imaging system. Firs t , i n Chapter 2, a hemispheric imaging system is proposed. The issues involved in imaging a complete scene, including angle of view, blindspot and resolution, w i l l be examined. Final ly , our proposed system is described as we constructed i t . The theory for obtaining higher resolution from low resolution frames, also known 10 as the super-resolution problem, is described in Chapter 3. This is followed by a test of the theory using standard image-processing test images. Subsequently, in Chapter 4, a method of controlling an optical system to carry out the increased resolution is described. It involves using a prism system wi th electronic feedback control through P S D sensors to shift individual image frames. The subsequent shifted images have a known mismatch and can be pieced together in a manner to get improved resolution. The final chapter summarizes the results and briefly describes how the two systems should be integrated for an optimal imaging system. Chapter 2 Hemispheric Imaging 2.1 Proposed System The newest systems for hemispheric imaging involve the use of a single mirror wi th a conic section of revolution. Yamazawa et al . have proposed an omnidirec- tional image sensor for robot navigation called C O P I S (COnic Projection Image Sensor)[Y +95] using a conical sensor sitting on top of the robot. The robot wi th the sensor attached is shown in F i g . 2.1 and a close-up of another version of the same mirror system, not attached to the robot (and mounted in a different dome), is shown in F i g . 2.2. Another researcher, Shree Nayar, has expanded the C O P I S research by studying the family of omnidirectional sensors that use one mirror wi th a conical cross-section[Nay96]. He found that the best results occur wi th a parabolic reflector. His final system uses one parabolic reflector with an orthographic lens and camera above the mirror (F ig . 1.4). We propose a system with two parabolic mirrors as shown in F i g . 2.3 and F i g . 2.5. It is similar to Nayar's system but does not require a large and expensive orthographic 11 Figure 2.2: View of C O P I S Sensor Figure 2.3: Two Mir ror Hemispheric Imaging System 14 focus I I Z Figure 2.4: Convex Parabolic Reflector lens. The first mirror or primary mirror in the system is a convex paraboloid. F i g . 2.4 of the primary mirror shows that rays passing through the focus are reflected parallel to the mirror's axis of symmetry. The diameter of the primary mirror is equal to four times the focal length, allowing points on the horizon to be imaged. The equation for the primary is that of a standard parabola, z\ = ^jyj - , where hi is the radius ^J(xi)2 + ( j / i ) 2 from the vertex (origin) of the primary. The upper or secondary mirror is a concave parabloid with the same clear aper- ture(diameter) as the bottom mirror, but wi th a longer focal length. It has the equation, z2 — where hi is the radius \Jix-i)2 + (y2)2 from the vertex(origin) of the secondary mirror. The parallel rays reflected from the primary mirror are reflected off the secondary and converge at its focus. A pinhole or a small lens located at the vertex of the primary is located at the secondary mirror's focus. This allows light to pass through the primary mirror. A sensor located under the primary mirror detects the image formed by the mirror system. Figure 2.5: Two Mirror Hemispheric Vis ion system 16 The pinhole has the purpose of blocking out any stray light which would produce noise in the image. Dr i l l ing a pinhole directly on top of the primary parabola is very difficult. Therefore, it is proposed to have a larger aperture in the primary. This aperture is then covered by a sheet with a pinhole in i t . This makes calibration easier without the possibility of ruining the primary mirror by dri l l ing an off-center hole. The focal length of the secondary mirror is equal to the distance from the secondary vertex to the pinhole. Al though the new pinhole location is not right at the primary vertex, it is st i l l located on the optical axis. The image formed by the system has a single viewpoint (located at the primary mir- ror focus) and provides a hemispheric field of view. As outlined by Nayar [Nay96], a single viewpoint allows for demapping of the omnidirectional image into perspective images with the geometry of the scene preserved. A raw C C D or C M O S sensor can be placed below the pinhole in the primary mirror. '•; Perspective images are formed by first choosing the viewing direction and then using appropriate software to decode the simple mapping done by the mirrors. In addition, by putt ing two of the systems back to back, a complete omnidirectional system is achieved with min imal bl ind spots. 2.2 System Parameters Figure 2.6 shows the entire two mirror system and two rays. In the system, both mirrors have the same clear aperture (defined as the optical diameter of the com- ponent). The clear aperture of the primary was chosen so that a ray (ray a in the figure), from the horizon (at 90 degrees wi th respect to the optical axis) would pass through the focus such that it is reflected parallel to the optical axis. This prerequisite requires that the clear aperture of the primary (and subsequently of Figure 2.6: Two Mir ror Hemispheric Vis ion System 18 the secondary because they share the same clear aperture) be 4 / i . The parameters involved with Figure 2.6 are defined as follows: hbi : radius of ray b on surface i (2-1) fi : focal length of mirror i (2-2) 6Zxi : distance from mirror i in z-direction to where ray X hits the mir(r2ir3) pa : radius of sensor surface (2-4) PaS, Pbs '• radius of ray a, b on sensor surface (2-5) fs : distance of sensor from P (defined as the sensor focal length) (2.6) 6 : angle subtended from optical axis by ray b (2-7) The symbol ' P ' in the diagram indicates the pinhole at the vertex of the primary. The symbol 'S ' i n the diagram shows the location of the sensor ( C C D ) below the primary mirror. The figure shows that there is a bl ind spot where the environment cannot be imaged. This bl ind spot caused by the upper mirror is centred about the optical axis of the system. The bl ind spot diameter, 2 * pbs, can be found by looking at ray b, which was chosen to show how the secondary mirror blocks the imaging. From ray b, the following equation can be written to determine the bl ind spot diameter on the C C D surface: t a n f l = 2 / 1 (2.8) J1 + J2- 0Z2 where 6z2 is the local (with the local mirror apex as the origin) z coordinate of where ray b grazes mirror 2. This coordinate, 8z2, is found from the equation for the parabolic surface of mirror 2. Substituting h2 — 2 / i , simplifies the equation for the point in question to 8z2 = The equation for the bl ind spot angle (where 19 this angle specifies the ray which creates the bl ind spot on the C C D ) then becomes: tan0 = 2 ^ , , , 2 (2.9) Now consider mirror 1 wi th the focus as the origin. A n equation for the radius of a point on the mirror wi th respect to the axis of symmetry (optical axis) is as follows (standard parabolic equation as defined from the focus): fti = , 2 ^ . s ing (2.10) 1 + cos 6 v ' A n approximation of how effectively the C C D surface is used can be found by looking at the radius of the extreme rays a and b on the imaging surface. For ray a the following can be written by using similar triangles: Pas 2 / a Surfaces 1 and 2 are parabolas. Therefore, (2.11) «*> = ^ (2-12) where fs is the distance of the imaging surface to the center of the pinhole or lens at the apex of surface 1. Using similar triangles, a similar equation can be written for ray b: Js h - ozb2 which simplifies using the parabola equation for surface 2 (z2 = to: r 2 / 1 / 2 , s ,6S = / , _ _ (2.14) 20 Pr imary Focal Length ( / i ) 1.02 cm Secondary Focal Length ( / 2 ) 7.24 cm Pr imary Mir ror Diameter (4 / i ) 4.06 cm Secondary Mir ror Diameter (4 / i ) 4.06 cm Mir ro r Separation ( / 2 ) 7.24 cm Blindspot Angle (0) 14.0 deg P ixe l Use Fraction (Eq. 2.15) 0.35 Table 2.1: Mir ror Specifications Recall that pis is the bl ind spot radius on the C C D . In one dimension, the number of pixels that are covered can be found from the ratio T ^ v J ^ i r ^ r ^ r m - ^ ° f ° r t w o dimensions, the number of pixels that are covered in a square C C D of dimension / by / is: P where N is the number of pixels. Table 2.1 summarizes the mirror parameters for the system. (2.15) 2.3 Advantage of Using Parabolic Reflectors Having a single viewpoint is advantageous because al l the information from the environment is seen from this reference point. If there were more than one viewpoint, as is the case for C O P I S and fish eye lenses (where each scene.point is imaged as viewed from slightly different positions), then the world representation would require 21 mapping al l images into a common reference frame. It would also require the depth of points in the scene to be estimated or determined. This makes omnidirectional mapping difficult. A sensor that can view the world from a single viewpoint allows the construction of perspective images easily. A l l that needs to be done is to look and find the mapping from the sensor to the point that is to be examined a certain distance (also called the focal depth) away from the sensor [Nay 96]. As shown by Nayar[Nay96], the easiest mirror curvature to obtain a single viewpoint in a catadioptric system is the parabola. In our system, the primary mirror is also a parabola. In addition, the secondary mirror is a parabola that focuses those rays that are reflected parallel to the optic axis through the aperture in the top of the primary. The mapping is simple because the parallel rays reflected from the primary are those rays that are aimed at the focus of the primary. The single viewpoint is evident and perspective images can be found by finding the ray intersections of a given perspective plane on the primary mirror. The final hemispheric image obtained from the C C D is in fact just this but scaled down proportionally by the upper or secondary mirror. 2.4 Limitations The advantages of our system have previously been examined. In summary, they are as follows: a single viewpoint, a reduced bl ind spot, a hemispheric field of view, no complex lenses, an omnidirectional field of view with two cameras back to back and compact design. The limitations come from several considerations. In reality, the C C D has a fast exposure time. The pinhole imaging technique requires long exposures. For some applications, however, real-time imaging is a necessity. In this case, the pinhole must 22 be replaced by a small lens. In fact, pinhole C C D cameras on the market have a small pinhole lens in them. The lens is inexpensive and does not degrade the image significantly. For our system, using a pinhole lens wi l l not ruin the fundamental imaging properties of the system as a whole. A l l that needs be done is choose a focal length to image the scene onto the C C D sensor a fixed distance away. Another problem is wi th the C C D sensor itself. C C D s and C M O S sensors typically have 512x512 or 480x640 pixels. When the mapping to perspective images is done, the number of pixels is reduced below this number. So to achieve similar image quality on the perspective images as conventional cameras, it is necessary to have a C C D with more pixels. This l imitat ion in C C D resolution also hampers the quality of images obtained from the Omnicamera and C O P I S mentioned earlier. 2.5 Mapping Hemispheric Image into a Perspec- tive Image One of the benefits of having a hemispheric image is that perspective images can be formed in any direction. The image is a compact storage of an entire scene. The perspective image can be formed by specifying the viewing direction, the focal length and the desired image size. Once the above parameters are set, it becomes easy to produce the desired per- spective image. First , the center of the omnidirectional image must be known or determined. After this is known, the radius of the omnidirectional image is found in pixels. Using the desired viewing direction, focal depth (zoom distance away from the viewpoint) and image size, the hemispheric image is stepped across in incre- ments and mapped to a new perspective grid. The resulting data is interpolated to 23 form a final digital image. A brief description follows: • F i n d the radius and the center of the hemispheric view • Choose the number of steps to cross the image • Choose the image size and location (angle of view) as well as the focal depth • Define a matr ix of X and Y co-ordinates on the perspective image • M a p the perspective image points from the hemispheric image using the parabola equations and the inputs given above • Interpolate the points (from the previous step) to the corresponding ones on the hemispheric image to form a perspective image 2.6 Results Using a standard stock lens, a hemispheric scene was imaged. The image is shown in F ig . 2.7. The results would be better wi th a better C C D camera wi th more pixels (such as the recently released M E G A - p i x e l cameras wi th 16 times as many pixels). In addition, the fraction of pixels used on the C C D (see E q . 2.15 derived earlier) was only 0.35. The camera could not be brought close enough inside the primary mirror because there were obstructions on the camera board thereby leaving much of the C C D surface unused. A custom C C D would have been required to fit the mirror system properly. Using the interpolation algorithm, perspective images were obtained wi th a software unmapping. These are shown in F i g . 2.8 and F i g . 2.9. A n y direction can be chosen Figure 2.9: Perspective Image found by Deconvolving F i g . 2.7 25 with similar results. Near the lower extremities, a larger scene area is mapped to the fixed C C D space. The pipes on the ceiling in F i g . 2.9 are straight as expected, but they are not very clear. This is because of the lack of resolution and the unfinished calibration of the system. The lens d id not map the hemispheric image onto the whole C C D surface. In fact, much of the C C D surface was wasted. The results do show the apparatus works and that better results would be obtainable by using a better C C D and having a pinhole lens wi th a focal length matched to the system so less of the C C D surface was left unused. 2.6.1 Computing Requirements The unmapping was done using the Mat l ab (Tm) matr ix manipulation program. Performing an unmapping to a 70 by 128 pixel image took 544047 floating point operations (flops) on a S P A R C 5 wi th a 8 5 M H z microsparc II processor. This corresponded to a C P U time of 18 seconds. For a m by n image, the computing power order of magnitude in flops is O(60mn). Using the data above, a 640 by 480 picture would take about 1.9E7 flops. W i t h a fast enough processor and a good compiler, real-time unmapping may be possible at 640 by 480 resolution. 2.7 Conclusion Although double parabolic systems have been used extensively in astronomical tele- scopes, they have yet to be used for wide angle imaging. We have proposed a lenless or pinhole lens imaging system for hemispheric imaging. It can be easily manufac- tured using two inexpensive plastic mirrors and a single image sensor. The design 26 is much less cumbersome than Nayar's, yet it s t i l l provides similar image quality for a C C D with the same number of pixels, and a single viewpoint for distortion free perspective images. A n omnidirectional (360 degrees in every direction) sys- tem is possible by putt ing two of our systems back to back, but this would require two sensors. The main application of our system would be for security and video- conferencing. The system would be fully integrated and not require any external mounting of cameras. A preliminary sketch of the system without opto-mechanical components is shown in F i g . 2.3 Chapter 3 Super-Resolution 3.1 Theory This chapter deals wi th the problem of resolution enhancement. In the previous chapter, a hemispheric imaging system was described for use with a C C D sensor. The main drawback is wi th the l imited number of pixels on the sensor which means less information to create the perspective images a person would see normally wi th their own eyesight. The chapter starts wi th the simplest case, a one dimensional analysis. The more general problem in 2D is then discussed and a simulation per- formed using standard test images. 3.1.1 Approach For our resolution simulations, we wi l l assume a knowledge of the high-resolution image. The degrading of the image by the sensor point spread function (PSF) wi l l also be known or approximated. Restoration of images without knowledge of 27 28 A Figure 3.1: Ideal pixel P S F array with non-zero separation the imaging procedure has been well studied in the literature[MS88, K B V 9 0 ] . High resolution reconstruction is different from restoration in that a higher final sampling rate and higher detail can be achieved. For the purpose of our analysis, we wi l l assume that the image incident on the sensor array is continuous. The sensors used for imaging have a point spread function (PSF) with a much larger magnitude than a single object point source. A typical C C D sensor has pixels separated by 13 to 20 \irn. Each pixel has a P S F that is very small but non-zero width. We wi l l assume that the resolution of the picture is l imited only by the P S F of the detectors. This is a reasonable assumption for C C D cameras as outlined by Gross[Gro86]. 3.1.2 One Dimensional Analysis A linear sensor wi th ideal pixels is shown in F i g . 3.1. A n actual sensor wi l l have pix- els wi th a non-zero point spread function. The non-zero width P S F of an individual pixel is important to consider because it imposes the final l imi t on the resolution improvement. The resolution cannot be improved by more than this PSF[JR84] . Consider a linear image, y, formed on a pixel array. It wi l l be assumed that each pixel has identical an P S F , h, and that they are separated by A . The actual object, 29 x, is a continuous function. The image can then be written as: M - l y(z) = h(z) * x(z)6(z - nA) + v(nA) (3.1) 71=0 where M is the number of pixels, * is the convolution operator, and v is the observa- tion noise. F i g . 3.2 shows what the spatial signals of the sensor and the image might look like. Notice the large space between the pixel PSFs . Without this space, there could be no resolution improvement wi th the method we describe later. Lucki ly , this is a valid assumption as most C C D s have large separations between relatively sharp pixel PSFs[Gro86]. W i t h just a single image, it is not possible to recover the high resolution content of the original scene. If several images are taken, and the displacement of these images relative to each other is known, then it is possible to recover a higher resolution. Because the individual detectors are non-ideal low pass filters, the high frequency information is contained in the original image in the form of aliasing. If the individ- ual detectors were perfect low-pass filters, then it would not be possible to recover the lost details. Consider now M-i Vi(z) = h(z~ &i) * x(z)8(z - nA) + v(nA) (3.2) 71=0 where yi is the image frame at a displacement of at- relative to the base frame(ao = 0). Suppose that each frame is displaced from each other by a fraction of the pixel size of the low resolution frame, If there are L distinct frames taken at different displacements but wi th a; less than the pixel size of y (i.e. between the low-res pixels), then it is possible to achieve a I D resolution improvement by a factor of L. A n example of the above follows. Figure 3.3 shows (in a simplified manner) how yi averages the high resolution pixel sequence x. Let yx = [?/i(0),?/i(l),yi(2),?/i(3)] (a) Single Detector Response (b) Sensor Array Impulse Response (c) Image Signal t y (d) Local Response of Sensor to Image Figure 3.2: Spatial Signals of the Sensor and Image 31 2Yl(0) 2 Y l ( D 2Yl(2) 2Yl(3) I Sensor Frame 1 x(0) 2 x ( l ) 1 x(2) 2 x(3) 0 x(4) 3 x(5) 4 x(6) 1 x(7) 2 1 I I I I I I 1 2YiO) 2 Y2d) 2Yi2) 2Y2(3) Sensor Frame Figure 3.3: Averaging of a higher resolution image sequence where y a (0) = ' W W , = 5&HsGl , y i (2 ) = £ l 2 l ± £ i l i ; y i ( 3 ) = El5l±£M. N o w consider that the sensor frame is shifted by a2 — A where A is the spacing be- tween the pixels of x. We then have y2 = [2/2(0), 2/2(1), 2/2(2), 2/2(3)] where 2/2(0) = E i 9 l ± £ i l i ) y 2 ( l ) = £i2l±£(3l j 2 / 2 (2) = £ill±£(M Y 2(3) = £(6l±^n_ I f A N D O N L Y I F ^ . J J I S known to be zero, then the above gives a solvable set of equations for the high res- olution image. Otherwise, shifting the sensor wi l l result in extraneous information being mapped onto the edge pixels, thus corrupting the procedure. If the images, ?/i, had larger pixels, then more images would be required to recover the resolution. A n example of the process for 1-D reconstruction wi th known shifts follows, using an image at the higher level, x = [2,1,2,0,3,4,1,2], finite in extent such that x(n) = 0 : n < 0 ,n > 7. The low resolution images y\ and y2 are obtained as above but wi th the second image y2 shifted by one half p ixel to the right wi th respect to 2/i- The result is y± = | [2,3,3,5], and y2 = | [3,2,7,3]. Meshing the series to a larger grid yields ymesh = |[j/i(0), j/ 2(0), 2/i(l), y 2 ( l ) , 2/i(2), y2(2), 3/1 (3), y2(3)] = | [2 ,3 ,3 ,2 ,3 ,7 ,5,3] . The same result, ymeshi could be obtained with the simple z 32 transform filter, Ymesh(z) — K The inverse filter is then simply 1 + ^ _ t or In the space domain this gives a recursive relationship for x(n) as x(n) = 2ymesh(n) — x(n — 1). Using x(—1) = 0, the original series x(n) can be obtained perfectly. In practice, however, the actual image cannot have perfectly sharp boundaries. In other words, x{n) would be infinite in extent; and this would cause some error when the above method is used. Interpolation is therefore used for real images as described later. 3.1.3 Image Formation Imaging always involves the application of non-ideal sensors that degrade the op- t imal obtainable image. The narrower the P S F of the optics and sensor system, the better the resolution that can be achieved. The degradation of the ideal image x(j, k) by a digital imaging system can be modeled as follows: M - l N-l yi(jjk)= £ x(m,n)h(j - m , k - n ) + n(j,k) (3.3) m=0 n—0 where h(j, k) is the degradation from the imaging process, n(j, k) the additive noise, M is the number of pixels in the j (horizontal) direction, iV the number of pixels in the k or vertical direction, and y;(j, k) the ith observed low-resolution image. Equation 3.3 can be rewritten in vector-matrix form as: y = Hx + n (3.4) Ideally, the degradation should be minimized by using the best possible optics and sensors. Unfortunately, real detectors can only achieve a given P S F and become very expensive when they are of high quality. As mentioned earlier, we assume 33 that the sensor P S F is the primary l imi t on the resolution. Our goal is to recover the high-resolution image (by upsampling the image and increasing the maximum represented spatial frequency), x(j,k), from several low-resolution images. The following summarizes the issues in the imaging system: • Sensor PSF: blurs the image • Undersampling: results in aliasing errors • Noise 3.1.4 Methods for Reconstruction Reconstruction is possible in both the spatial and frequency domains. Currently, the preferred technique is in the spatial domain[ST90] because lower noise and higher resolution have been achieved spatially. Spatial domain techniques include projec- tion onto convex sets(POCS)[ST90], tomographic backprojection[IP91], interpola- t ion, and minimizat ion of error[TG94]. The frequency domain approach involves spectrum cancellation and eliminating aliasing artifacts[KS93]. The frequency do- main method is currently only used when there is a registration problem. The reconstruction problem has three parts to it. The first is part is image registra- tion. The amount of shift between the low-resolution frames must be determined to some accuracy. The second part is the restoration and filtering of the frames to compensate for degradation and noise. Final ly, the high resolution image is reconstructed or pieced together using interpolation or other techniques. Algor i thm for high-resolution reconstruction of 2-D images: 1. Registration: determine how much each frame is shifted from each other 34 2. Restoration or Fi l ter ing: remove the degradation and noise from each low- resolution frame 3. Reconstruction or Interpolation: use the results of above to piece together a high-resolution image It turns out that the above three steps are interdependent. There have been at- tempts to combine al l three steps i n a global procedure, but so far only steps 1,2 and 2,3 have been combined (in separate instances) successfully[KBV90, TG94] (in the spatial domain). 3.1.5 Registration Problem Known Shifts In the ideal case, we would know the exact amount that each low resolution image is shifted wi th respect to the reference frame. This is only possible when the imaging process is directly controllable. When the sub-pixel shifts are random, the problem is more difficult and becomes one of registration. Only interframe translations of the low-resolution images wi l l be considered. Rap id movement imaging systems, such as aircraft radar, w i l l have to treat the case of translations wi th frames not spaced at sub-pixel amounts. Based on the assumption that resolution only depends on the number of pixels, an argument can be made to determine the improvement in resolution. W i t h control over the subpixel shifts, it is possible to achieve a resolution improvement of \[L where L is the number of low resolution frames[TOS92]. This is derived as follows. Let the low resolution images have a resolution of K x K, and the high resolution scene a resolution oi N x N ( M = N ) . There are then K x K equations from the 35 low resolution frames and N x N unknowns from the individual pixels (in the high resolution scene). This means that the total number of equations that can be formed is K x K x L. In order to satisfy the requirement that there is the same number of equations as unknowns, it is necessary that N = \f(L) X K. In an ideal world if the pixels were ideal samplers and there is no low-pass filtering by the optics then the max imum theoretical resolution improvement is the square root of the number of distinct sub-pixel frames[TOS92]. The above argument may lead some to believe that an infinite resolution improve- ment is possible. Unfortunately, this is not the case because of the zero-crossings in the sensor frequency response. In other words, real pixels have a non-zero point spread function (see F i g . 3.2). The shifted images must be separated by at least the width of the P S F in the spatial domain or there wi l l be overlap. Some high- frequency information wi l l inevitably be lost as a result of the smoothing effect of the sensor response. Noise, as usual, agravates the problem. Unknown Image Displacements When the shifts are unknown, there is a registration problem that must be solved before a high-resolution image can be reconstructed. The formation of the low- resolution images through the system is considered as before. The degradation is usually assumed to be a low-pass blurring or averaging function. Alternatively, the images are modelled by a Gauss-Markov covariance model. If the original image were known, it would be possible to use matched filtering to determine the best correlation for a given shift. In fact, even though the original scene is not known, matching and correlation can s t i l l be used. The high-resolution image is estimated and the shifts varied to produce the best possible reconstruction. 36 The original image, x(j,k), is not known with certainty, however, so several itera- tions must be performed with different estimates based on the samples, y{. Mor t and Srinath suggested a maximum likelihood image registration algorithm[MS88] that requires no a priori knowledge of k). 3.1.6 Restoration, Inverse Filtering Once the low-resolution frames have been sampled and their relative shift deter- mined, it is necessary to try to remove some of the noise through inverse filtering and restoration techniques. The problem wi th most inverse filters is that they are unstable and not physically realizable. Pseudoinverse filters attempt to solve this problem by setting the frequency response to zero at ill-posed points. The pseu- doinverse filter, however, remains highly sensitive to noise. The preferred technique is Wiener filtering because it removes the noise and some of the blur caused by the low-pass filtering. When there is no registration problem, the restoration procedure is straight forward and lends itself well to conventional inverse filtering techniques. If there is a reg- istration problem, it is preferable to combine the registration and inverse filtering techniques using an Expecta t ion-Maximiza t ion(EM) algorithm[TG94, T K 9 4 ] . In the E M algorithm, the degradation process is modelled as in E q . 3.3 and an image covariance model[Jai88]. A n ini t ia l guess is taken for the shift of each low-resolution frame. Each shift is then incorporated into the covariance model. The resulting modified probability density function of the observed image is then maximized as the shifts are iteratively estimated. Another similar iterative reconstruction technique for nonuniformly spaced samples is proposed by Sauer and Allebach[SA87]. 37 3.1.7 High Resolution Reconstruction After the opt imal shift estimation (if unknown) and restoration have been per- formed, the final step is to piece the images together. Ideally, a global procedure wi l l be developed to perform all the above taking the interdependences into account. If the shifts of the low-resolution images and a priori knowledge of the final image are known, a recursive approach combining restoration and reconstruction may be the best solution [ K B V90]. Otherwise one procedure is to interpolate based on the shifts and amount of noise in the images. Interpolation can be done using any of the available techniques. Linear interpolation may be the best option, but the choice ult imately depends on the relative shift of each frame. Other interpolation methods include such novel techniques as fractals and more conventional ones involving polynomials. The reconstruction can be seen as a problem of solving a set of simultaneous equa- tions formulated by E q . 3.3 for each frame. The equations are linearly independent if the displacement between frames is at subpixel amounts. Since the equations are coupled together wi th each other, the solution can be simplified by either to- mographic backprojection[IP90] or projection onto convex sets[S089]. Addi t iona l constraints (such as finite enery and l imited support) have been added to the above methods to account for observation noise[TOS92]. 38 Figure 3.4: Original 256x256x8bit Lena Test Image 3.2 Simulation 3.2.1 Reconstruction with Known Shifts In order to demonstrate how the resolution recovery procedure is achieved, a sim- ulation was performed. The Lena test image (Fig. 3.4) with 256x256 pixels and 8 bits per pixel was used. The Lena image is a standard reference in image processing. Low resolution samples were taken from the picture. These samples were then be manipulated in an attempt to recreate the high-resolution image. The Lena test image was originally taken with a sti l l camera onto conventional film. The image has since been digitized and standardized for use in imaging processing tests. The above test image (Fig . 3.4) is the reference image, i. It is impossible to recover 39 this image perfectly because of the quantization effects and added noise. We consider the model of the sensor shown in F i g . 3.2 where there is space between the pixels where no imaging occurs. A model of the simulation process is shown in F i g . 3.5. Image A is the original image (i), B is the test image after low-pass filtering (yi), and C is the ensemble of low-resolution images shifted with respect to each other. Fi l ter H2 performs the low pass filtering to obtain a lower resolution image and (G\), (G2), (G3), and (G4) perform the operations to get the low resolution images. To obtain the low-resolution images, a simple moving average ( M A ) filter (see H2 below) is applied to the known high resolution image. The filter is essentially a low-pass filter that blurs the image onto larger pixels. The M A filter averages the adjacent pixels in a square neighbourhood around each of the high-resolution pixels in question. One such filter is as follows: ' l 1 1 ^ H i = — 16 1 1 1 1 1 1 1 1 1 1 1 1 (3.5) For our simulation, however, a weighted moving average, W M A , filter (#2) gives the closest neighbour pixels more significance. This more accurately reflects the situation on a sensor array. The individual sensors or pixels don't have a flat point spread function i n the space domain but rather one that is weighted about its center[Gro86]. The W M A filter is as follows: Ho = 1 36 / 1 2 2 1 ^ 2 4 4 2 2 4 4 2 V I 2 2 I) (3.6) O r i g i n a l Test Image i ( l , l ) ! id ,2 ) j i d , 3 ) ! i d , 4 ) - — - 4 — - 4 - L . i ( l , l ) • i(l ,2) | id ,3 ) ! i d , 4 ) - — — • j - _ _ L _ _ i ( l , l ) ] i d , 2 ) j id,3)] i d , 4 ) J ! J i d , l ) | id ,2 ) j i ! i d , 3 ) | i d , 4 ) i H2 LPF O r i g i n a l y i Low Resolution Images y i d . l ) y l d . l ) y i (2 , l y3d,i)j y i (3 , i yi(2,iS y id ,2 ) y2d , l ) yi(2,2)] y 4 d , i yi(3,2J y2(2,i yi(4, l | | yi(4,2)| y3(2,l)l y4(2,l y ld ,2 ) I yi(2,3 y3 d,2)| yi(3,3) yil2,2i; yid ,4 ) y2d,2) yi(2,4) y4d,2) L . yi(3,4) y2(2,2) yi(4,3)| yi(4,4l y3(2,2)l y4(2,2) ( G i l (G3 y l 1 (G4) (G2) y3 Figure 3.5: Simulation Mode l -II The L low resolution images are formed by subsampling the multiplication of the above filter wi th the high-resolution image. If the high resolution image is N x N, the low-resolution image is K x K, where K = In our simulation, TV = 256 and L = 16 such that K = 64 and the low-resolution images have pixels \J~(L) = 4 times as large. One of the low-resolution frames is shown in Figure 3.6. 10 20 30 40 50 Figure 3.6: Low Resolution 64x64x8bit Lena Image Frame (obtained by applying H2 to F ig . 1) In order to complete the simulated process of how the sensor array misses certain parts of the image (see F ig . 3.2) additional subsampling operations (Gi) ,(G3) ,and (G4) are applied to get the final low resolution images. These low-res images are considered the highest resolution images that our imaging system can achieve on its own. From here we work backwards to recover the resolution. 42 50 100 150 200 250 Correlation with original:0.958 Mean Square Error: 3505.979 Figure 3.7: High Resolution Reconstructed Image (no deblurring applied) W i t h the low-resolution images created, the high resolution image, i , (A in F ig . 3.5) can be artificially "reconstructed" by merging the low resolution images (yl,y2,j/3,y4 in F i g . 3.5). Interpolation is used to improve the results as described earlier[Gro86]. Figure 3.7 shows the reconstructed image using a bi-linear interpolation method after 2 dimensional filtering of the image with H2 and meshing the images to a higher grid. As expected the image appears blurred because in essence it is just the low-pass version of the reference image i. The example we have given is t r ivial but demonstrates how some high frequency content is lost and just how dramatic the improvement in resolution can be from F i g . 3.6 to F i g . 3.7. A conventional deblurring restoration filter (such as a Wiener filter) could be applied to the image. A n alternative method for creating low-resolution shifted frames is suggested in 43 [KBV90] . It involves interpolating the original image by expanding it wi th a zero- padded D F T . The higher resolution image can then be sub-sampled and noise added to get the shifted low-resolution frames. The advantage of this technique is that an arbitrarily large number of interframe samples can be obtained by increasing the number of points in the D F T . Since the low-resolution blurring function, i/2, is a low-pass filter, some of the high frequency information is lost. This means that the reconstruction problem cannot be uniquely determined and minimizat ion of error techniques converge to several solutions. Our M A filters, H\ and i f 2 , have a non-uniqueness problem because they are singular matrices. No inverse filter exists (because of the singularity), a fact that mandates the use multiple images shifted from each other to recover the resolution. The above simulation was then repeated but wi th white Gaussian noise added to each low-resolution image. The noise was then removed using an appropriate Wiener adaptive filter and the high-resolution image reconstructed by interpolation. The resulting image is shown in Figure 3.8. The high-resolution information is visible but the image suffers from greater Gaussian blur. The result is worse because of the added noise. Registration Problem In our simulation above, we directly controlled what the sub-pixel shifts were. Sup- pose, however, that the shifts between frames were not known. W i t h knowledge or an estimate of the ideal image, a correlation analysis can be performed to find the most likely shift. Two of the low resolution images are expanded by zero padding between samples to the size of the high-resolution original image. The two-dimensional correlation 44 Figure 3.8: Wiener Fil tered 256x256x8bit Lena Test Image (from Noisy Frames) 45 x 1 0 7 Figure 3.9: Image Correlation between Frames of each image with the original is then performed and plotted. The shift is found from the maximum of the correlation. The results obtained were consistent with the actual shifts given to the two frames. A plot of the correlation between the tenth low-resolution zero-padded image (shifted 2 pixels in j direction, 1 pixel in k direction) and the original image is shown in Figure 3.9. The maximum of this image was found at (254,255), shifted (-2,-1) from the reference frame at (256,256), which agrees with the shift ini t ial ly given to it. This simple example shows one way how image registration can be achieved. 46 3.3 Final Remarks on Super-resolution Problem When there are multiple low-resolution images at subpixel displacements it becomes possible to upsample and recover higher resolution information. It is apparent that the task becomes much easier with more information about the imaging process and the ideal image. It is also highly desirable to know the exact shift of each frame and make the shifts at sub-pixel levels. In aircraft radar and other difficult to control imaging systems, this is not always possible. If however, a device existed to control the imaging shifts exactly in a controlled manner, high-resolution reconstruction could be achieved without much effort. The whole process is optimized by combining the restoration and interpolation steps when there is no registration problem. It is clear from the procedure that the algorithm works best when there is much blank space in between pixel P S F s in the sensor array. This does not mean that the pixels on the C C D should be spaced further apart (because this would destroy the response of the C C D on its own), but that the individual detectors (pixels) should have a narrow P S F in the spatial domain. Conventional high-pass filtering or deblurring techniques should be applied to the final blurred high-resolution reconstruction. The resulting image resolution can thus surpass that possible wi th the best available sensors and optics. Further, cheaper optics and sensors can be used to produce higher resolution images by taking multiple frames. The main disadvantage is the longer processing time. Chapter 4 A Sub-Pixel Resolution System Achieving sub-pixel displacement can be done in one of two ways. The first method is to optically shift the image using a movable refracting element as part of the optics. This can be achieved simply using a prism, an etalon, or even a single refracting plate. The second method involves moving the sensor itself. The C C D could be mounted on a plate, which in turn could be displaced with actuators. The decision of which method to use is based on how easy it is to construct the actuators and how easy it is to set up the position sensing devices. 4.1 Description of VAP A method of obtaining higher resolution images from video frames is outlined be- low. The method involves a C A N O N ES-750 camera wi th optical image stabiliza- tion. The image stabilization is done by driving a variable angle prism based on a vibration input and the current pr ism angle. The prism itself is used to deflect the image on the C C D surface. Our goal was to use the vari-angle pr ism ( V A P ) to 47 Figure 4.1: Complete V A P Assembly (US 5481394) 49 control the separation between several images and obtain a higher resolution from this. The V A P was chosen (instead of an etalon or moving the C C D ) to achieve the resolution shifts because it was a readily available device wi th pre-made actuators and position sensing. The goal was to expand the functional use of inexpensive camcorders equipped wi th means for image stabilization without going through the laborious process of building a system from the ground up. The mechanical structure that we use to deflect the images at subpixel amounts is the Variable Angle Pr i sm shown in F ig . 4.1. The important components are summarized below: 31 Front V A P Surface 37 O-ring 33 Back V A P Surface 46a Permanent Magnets 47a,48a Magnet Casing 45a Voice Coils 43a,b P i t ch A x i s , Yaw Axi s 41a Infrared Diode ( I R E D ) 52a Position Sensing Device (PSD) 39a Window Slit 38ab Front and Back Plates (for moving the vap) 50 Figure 4.2: Front and Rear Plates of V A P (US pat. 5481394) Items 42ct6,49,50 are used to hold the V A P in a static position when the locking mechanism is on. Item 40a is just an arm attached to the front plate. Please see US patent 5481394 and US Patent 5623305 which describe the V A P as used as a 'Camera Shake Correcting Apparatus Having Damping Coils for Yaw and P i t ch ' . The Vari -Angle P r i sm is formed from two glass surfaces separated and filled by a rubber silicone l iquid with index of refraction, n. It is essentially a prism. The Front plate moves about a horizontal axis such that the image is shifted vertically on the C C D . The rear glass surface is confined to movements about a vertical axis, which allows for horizontal image displacement. The pitch angle (6P) of the front surface (31) can be changed, while the rear surface (33) controls the yaw (6y) as shown in F i g . 4.2. The prism has the function of bending light rays. The light is refracted through the Figure 4.3: Large Pr i sm Figure 4.4: One Dimensional Structural V iew of V A P 52 prism to the other side. The following equations can be derived wi th reference to F i g . 4.3 using Snell's law of refraction and simple geometry: sincci = n sin (4-1) s m a 2 = n s i n a 2 (4-2) a i + a' 2 = a (4.3) e = a a + a2 - a (4.4) where e is the angle between the incident and outgoing light rays, n is the index of refraction and a is the prism apex angle. When the variable apex angle is small as in F i g . 4.4, the equations above can be simplified to the equation: e = (n - l)cr (4.5) The deflection of an incoming ray is related to the index of refraction, n , of the prism and the prism apex angle. Equation 4.5 shows that the position of an image on the final C C D screen can be moved about with movement of the prism. The V A P itself is physically circular and the geometry becomes more complicated when both sides of the V A P (31 and 33 in F i g . 4.2) are moved at the same time as shown in Fig.4.5. Suppose that the rear plate is rotated at 9'p and the front plate at 0'y to produce a movement (f>. The inclination wi th respect to the angle <j> for the rear plate is given by 9[. From the geometry, the following can be written for 9[: tan 9[ = sin <f> tan 9P (4-6) Similarly, the following can be written for the inclination wi th respect to <f> for the front plate: tan 9'2 = cos <j) tan 9y (4-7) FIG. 3(A) PRIOR ART Figure 4.5: Geometry of V A P movement (US pat. 5481394) 54 Figure 4.6: Restriction of V A P movement (US pat. 5481394) The prism apex angle, a, is the sum of the two above angles: a = 6[ + 0'2 (4.8) Therefore a is larger than both 8y and 9P. If we now define 9 y m a x and 6 p m a x as the max imum angles that the V A P can be rotated in the respective yaw and pitch directions, then it is clear that for oblique movements, the total prism angle a can exceed the max imum allowable prism angle amax, such that the image overshoots the C C D . The solution is to restrict the movements in each direction as shown in F i g . 4.6 and the following equation: O2p + 02y< C (4-9) The V A P has 2 degrees of freedom allowing movement in two dimensions, which is all the movement wi th physical meaning for an image. A l l that needs to be achieved now is control of the V A P . This is described in the next section. 55 44 41b Figure 4.7: V A P actuator unit (US 5481394) Actuat ion of the prism surfaces is done through magnetic voice-coils. It is essentially a linear D C motor system. The voice-coils are driven by an external voltage source and move wi th in the permanent magnet stators. Each actuator has a characteristic winding resistance [ V / A ] , R. The actuator coils are wound wi th pancake wire. F i g . 4.7 and F i g . 4.8 show the actuator unit (with attached photosensors described later). Number 42 on F i g . 4.7 is the actuator winding coil . Numeral 43 on the same figure is an identical coil that w i l l be discussed later. The labels are summarized below: 6 or 15 C o i l Casing (see F i g . 4.14) 7ab Actuator Mounts 7ef Hole for Stem 53 or 54 26 or 32 C o i l Casing (see F i g . 4.14) 56 4 A Alfc> A l t o 5 5 5 3 o r 54J 2 6 o r 3 2 6 o r 1 5 A O Figure 4.8: Side View of V A P Actuator (US 5623305) 40 Stator 41 Actuator Casing 41b Slit Window 44 Infrared Emi t t i ng Diode ( I R E D ) 45 Posit ion Sensing Device (PSD) 53 or 54 Stem Connecting Actuator to V A P Frame 55 Vap Frame For position sensing, each side has an angle sensor, formed by an infrared photo- diode ( I R E D ) and photo-sensor (PSD) . F i g . 4.7 shows the position sensing in detail (44,41b,45). The I R E D (44) and P S D (45) combination is fixed in place while a small window slit (41b) on the chassis moves with the V A P outer surfaces. This 57 moves the dot of light on the photo-sensor allowing the angle of the pitch and yaw to be measured. We used the instrumentation amplifier circuit shown in Fig.4.9 to amplify the P S D output. I N A 118 is an instrumentation amplifier (amplifying the small differential signal from the V A P P S D ) set to a gain of 100 through the 560 ohm resistor. O P A 2 7 is just a buffer amplifier. The circuit attached to the non-inverting terminal of OPA27 is a t r imming circuit (made more stable by the current references (REF200)) used to tune the instrumentation amp INA118 through the buffer amp (OPA27) . It was important to use shielded twisted pair wire to prevent noise from being introduced. Va is a symbol notifying the voltage feed to the control circuit described later in F i g . 4.15. 4.2 Applied Super-Resolution Achieving super-resolution in practice is a non-trivial problem. In al l real world cases, the actual representation of an image is unknown. For moving camera sys- tems, such as aircraft surveillance, controlling the shift between image frames is next to impossible. It is therefore necessary to consider the full registration prob- lem. Our goal is to show a system that can take several images that are shifted by a known and repeatable amount and use these images to recover information that was lost in the optical process. In practice, determining the sub-pixel shift is a difficult problem. A test target must be carefully chosen to calibrate the system properly. In addition, an appropriate method for locating the target must be outlined. W i t h the above accomplished, a mapping of the system position to relative shift can be done. 58 ( i n s t r u m e n t a t i o n amp) (trimming network) -15V Figure 4.9: Our Front E n d Differential Position Amplif ier Figure 4.10: V A P Deflection Angle versus Drive Voltage (US 5623305) 60 4.2.1 Centroid Calibration In order to calibrate the system, a good test target is required that can be distin- guished from the environment. Issues such as external l ighting, camera automatic gain control and external vibrations make the consistent determination of the cen- troid a problem. In order to minimize the variations in the overhead light, it was found that a white target on black paper worked best. In addition, the camera automatic gain function adjusted the contrast less wi th this configuration than for a black target on white paper. The target itself was a 1cm by 1cm square centered i n the camera field of view. The camera zoom was set to the widest angle. Using different zooms causes changes in sensitivity to movement of the V A P . In fully zoomed mode, it is clear that small movements in the V A P creates larger displacements in the centroid across pixels because the target takes up more relative area on the sensor, which in turn translates to a larger area on the screen. Because of the significant changes i n zoom and V A P temperature during different test conditions, it was found that calibration was necessary before every test. Two approaches were used to find the centroid. The first was an iterative approach to examine the outer neighbourhood of a suspected spot. The second was to find the centroid of a specified small area of pixels. The difficulty with the first approach lies in the threshold value for a detected spot. O n the border, the contrast effect of the camera automatic gain control causes the pixels to be lighter than the background. Using a fixed threshold in a iterative algorithm throws out information on the border. The automatic gain control of the camera causes unwanted effects no matter which algorithm is used, but the local area is less sensitive to i t . The best solution 61 is to use a larger dot to make the boundary have less importance. Lighting problems also hamper the consistent determination of the centroid. As the brightness and direction of the overhead light shifts, the centroid can move about. For the sub-pixel work we are concerned with , this can cause problems. Because of the above problems of lighting and camera automatic gain control ( A G C ) as well as various vibration noise, a statistical approach was at first used to find the centroid. The centroid was found for several hundred image frames. A sinewave input was ini t ial ly used to drive the V A P . Vin = Asinut (4.10) Vout = kVin (4.11) Vrms = kA.rms (4.12) So experimentally, yTms = jy—, where N is the number of points taken. Obtaining data this way gets r id of the sensitivity to jiggle and averages out the noise effects. The centroid was actually located using a threshold technique. The algorithm is as follows: • Check if current pixel is within the threshold for target • If not, scan through image unti l one is found • Recursively check the neighbours of each pixel within the threshold • When a hit is found (as part of the intended target), mul t ip ly X coordinate index by the pixel value • A d d the above value to a running sum for all hits ( S U M X X B ) 62 l / (Js A 2+Ds) Ky Figure 4.11: Open Loop representation of V A P • Repeat above two steps for Y coordinates ( S U M Y X B ) • A d d current pixel darkness value to running sum ( S U M B ) • Repeat 5 previous steps unt i l no more hits are found • Calculate centroid location by division ( S U M X X B / S U M B , S U M Y X B / S U M B ) To eliminate false positives, the user was allowed to select from a number of possible hits. Once the desired centroid was selected, an image of the region in question was displayed on the screen. This made sure the right coordinates were chosen for the test target. W i t h the test target located, the above threshold technique or a localized area technique was used to calculate the centroid after each movement of the prism. 4.3 Control of the VAP The V A P can be run open-loop, but doing so results in^hysteresis in the displace- ment/voltage curve. The hysteresis is due to the magnetic nature of the voice coils and the permanent magnets. F i g . 4.10 of US patent 5623305 shows the hysteresis for the V A P . This same patent also shows how the V A P can be modeled in terms of physical parameters. The 2Kt/R O 63 assembly has a particular torque constant, Kt, and inertia J. The silicone l iquid inside the V A P has a viscosity resistance D [ ^ ^ ^ ] and the V A P itself has a spring constant, Ky. A block diagram for the V A P is shown in F i g . 4.11. From this diagram, the following equation can be written for the open loop gain: where R is the winding resistance in Ohms. This can be simplified to a generalized second-order equation: G M = i ( 4 - 1 4 ) where n is a normalization and £ is an attenuation coefficient. Only the form of the equation above is important; we do not need an expression for un or £ and the patent does not provide one either. A simulation yields Bode diagrams (Fig . 4.12) showing phase and gain versus frequency. The characteristic curves for the V A P reveal a pole at 100Hz. In addition, running the V A P open-loop prevents control over small angles due to the V A P non-linearities and the hysteresis curve. The l iquid inside the V A P changes mechanical (viscosity) and optical properties (index of refraction) significantly wi th small changes i n tem- perature. The strong temperature dependence of the device as well as the need to have control over small angles requires a more elaborate control system. In order to improve results, the position of the P S D is fed back. This alleviates the problem of the V A P non-linearity and allows smooth control of the variable angle prism. The phase margin is increased, which reduces the effect of the V A P mechanical properties. A block diagram showing the V A P control is shown in F i g . 4.13 wi th G(s) equal to the open-loop gain above. The actual circuit we designed and used to feedback the position is shown in Figure 4.15. Symbol Va is the voltage \ 64 Figure 4.12: V A P Bode Plots (US 5623305) 65 G ( s ) J K Figure 4.13: Closed Loop representation of V A P from the front end instrumentation amp (Fig. 4.9). The first operational amplifier TL071 is a high D C gain amplifier tuned through the lOOnF capacitor and 39Kohm resistor to roll off at a frequency of 20Hz. Without this rolloff, the parasitic pole at 100Hz causes the V A P to oscillate violently. The feedback loop is composed of the amplifier LF356 and the current mirror of 2N3904 and 2N3906. The mirror is there to get a higher current drive. In addition, there is a couple of 2.2Volt Zener diodes tied as shown in front of the current mirror to prevent the V A P actuator from being overdriven through Vout. The patent mentioned earlier is concerned wi th correction for image shake. It describes control based on a detected angular velocity from an on-camera gyroscope. A l l the signals are processed digitally through a microcontroller. In addition, the actuators have a second coil (Fig. 4.14-42,43) on them (physically separated from the first coil by a barrier) to pick up the velocity of the actuator (Fig . 4.14-56). The second coils are called damping or dump coils and are not driven. Their movement through the magnetic field inside the stator induces a voltage. The A C stability is increased by using the second damping coil , which introduces an derivative feedback term to increase the phase margin. In D C motor theory, the technique is known as velocity feedback. As mentioned earlier, the characteristic pole is at 100Hz wi th only the position being 66 Figure 4.14: Dua l C o i l Wrapping of V A P Actuator (US 5623305) fed back proportionally. Adding the derivative feedback helps push the pole to a larger frequency, thus reducing the sensitivity of the system to external vibrations. For our purposes, however, the proportional feedback is sufficent because we are not yet t rying to shift the images transparently in real t ime and are only working in one dimension. In order to stabilize both the vertical and horizontal directions of the V A P , velocity feedback must be used i n the control. 4.4 Experimental Results using ES-750 4.4.1 Initial Tests In essence, there are two resolution problems that we are concerned with . The first is wi th the C C D surface itself. A n image at the C C D surface is sampled by the individual pixels, which is then resampled by the frame grabber at the computer. 67 4.7M A / V V - 10K AA/V (from Front end) ^ ° r - 100K A A / V 2N3904 15V v Out -o 12: 4 -15V r -15V 2.2V 2N3906 10K 2.2V (user s u p p l i e d voltage) Figure 4.15: Our Control Circui t The digitized frame suffers loss of detail from quantization effects. The second loss of resolution occurs at the screen. The screen has its own pixels which sub-sample the digitized image when it is displayed. The issue we are concerned wi th is whether a subpixel movement on the C C D is equal to a subpixel movement in the viewed image on the screen. Luckily, the mapping from the camera image surface to the screen is one to one. Therefore, by studying the subpixel movement on the C R T surface, it is possible to recover the resolution caused by the smoothing effect of the C C D , as discussed earlier. 4.4.2 Open-Loop Results The first tests on the V A P involved driving it open-loop wi th a sine wave input and using Equation 4.12 to find the displacements on the C R T . F i g . 4.16 shows the horizontal mean rms displacement for sinewave inputs of several different frequen- cies. F i g . 4.17 shows the same information but wi th the control system attached to the vertical (pitch) actuators of the V A P . The results in both cases are linear up to a frequency of 25Hz. A t 3QHz, the plot is a flat line indicating no response to 68 Vertical Displacement with Sinewave Input 5Hz o 10Hz X 15Hz + 20Hz » 25Hz • 30Hz 150 200 Vrms(mV) Figure 4.16: Vert ical Displacement wi th Sinewave Input the signal. This shows that the frequency response deteriorates somewhere around 25Hz. Next, the frequency response of the V A P was measured to find where exactly the cut-off frequency is. This involved using a 0.5 Volt sinewave wi th an arbitrarily set frequency. The frequency was changed and the response measured. F i g . 4.18 is a plot of the results. The same experiment was repeated for the vertical V A P plate as shown in F i g . 4.19. The results for both sides coincided. The bandwidth of the device is situated around 20Hz. A step function frequency response was also performed. The results appear in F i g . 4.20. The square wave response is useful because it shows what would happen i f the voltage to the V A P were increased digitally at specific increments. The bandwidth is reduced to approximately \bHz when a squarewave is used to move the Variable Angle Pr i sm. 69 1.6 1.4 » 1.2 1 1 a | o . 8 0.6 Horizontal Displacement with Sinewave Input 5Hz 0 10Hz X 15Hz + 20Hz » 25Hz • 30Hz + I I , •$ s • • r» w • a • 50 100 150 200 250 300 350 400 Vrms(mV) Figure 4.17: Horizontal Displacement with Sinewave Input VAP Horizontal Frequency Response(V=0.5) <o 2 X ft 1.5 CO c II 1 4) « 0.5 ft w • 0 • • • • • • • • • • • • • • • • • • • • Seriesi 20 40 Frequency(Hz) 60 Figure 4.18: Horizontal Direction V A P Frequency Response (0.5 V Sinewave) 70 Normalized VAP Vertical Frequency Response w E 1.2 1 0.8 > 3) « 0.6 g a 0.4 0.2 a E L. 0 Z 0 -0.2 20 40 60 80 Frequency(Hz) Figure 4.19: Vert ical Direction V A P Frequency Response (0.5 V Sinewave) VAP horizontal Frequency Response(.2Vpeak square wave) 3.5 ^ 3 X a 2.5 c 2 i 1.5 s 1 a .2 0.5 • 0 • • • • • • • • Series 1 0 10 20 30 40 Frequency(Hz) Figure 4.20: Square Wave Response 71 Displacement with Compensating Op-Amp 0.6 0.55 0.5 0.45 $ 0 , 4 - of | 0.35 <v .- »' o •iS 0.3 o 0.25 0.2 0.15 [ I 0.1 19.3Hz • - 0 - - 20Hz • • • • » • • • 23.1Hz 24.75Hz 10 20 30 Frequency(Hz) 40 50 60 Figure 4.21: Displacement wi th Compensating Op A m p W i t h the frequency response determined, we set out to attempt pushing out the response using a compensator circuit wi th a variable frequency gain inverting op- amp to drive the circuit and attempt to increase the bandwidth. The feedback impedance was changed to adjust the location of the pole. The frequency response was then plotted with each different characteristic frequency, / c , in F i g . 4.21. The driving op-amp had a gain of wi th Z ; the input resistor and Zf the parallel combination of a resistor and capacitor to get the desired frequency fc. It is apparent that the response deteriorates above 20Hz. The pole at 20Hz was not significantly reduced with the op-amp. This indicates that there are higher order effects involved wi th the V A P . The op-amp was then run at D C . In one direction only, the response in F i g . 4.22 was found. The plot is linear in the middle section, but falls off at the edges as 72 Mean Position versus Applied Voltage oo 7 • — jj. 1 on 7 • oz. / o-i 7 • j 1. / Tt 7 OUV , j.?0 7 • * ?° 7 ^ ZO. / * * 97 7 — • — 1 — m - 1 1 -600 -400 -200 0 200 400 600 Voltage(mV) Figure 4.22: V A P run open loop in one direction expected considering the l imi t ing hysteresis of the actuators. Unfortunately, the results in F i g . 4.22 were only attainable under strict conditions and were not repeatable between tests. When the V A P was moved back and forth, a hysteresis pattern was immediately apparent. F i g . 4.23 shows the situation for some representative movements of the V A P . The hysteresis is similar to F i g . 4.10 except the scale is different. We are using a different circuit to drive the V A P than that described in the patent so a direct comparison is not possible. The same can be said about the frequency response. The 100Hz pole described in the patent does exist as is clearly apparent when the control falls out of loop and starts oscillating, but it is not the same as the bandwidth that we determined the V A P to have using an external driving circuit. In addition, the instability of the device was also a concern. A program was written to continuously sample the V A P over a period of 4 hours. The input was a constant magnitude and frequency square wave and the output was the displacement of the test target at each point as shown in F i g . 4.24. As mentioned earlier, the drift is 73 VAP vertical displacement at DC VAP Horizontal DC Displacement Voltage(mV) 270 Voltage (a) Tr ia l 1 (b) Tr ia l 2 Figure 4.23: Hysteresis of V A P due to several factors. The extreme points are most likely due to random vibration of the bench where the camera is mounted. In addition, a long term change in the displacement is evident and the separation between the high and low points becomes narrower. The narrowing of the displacement can be explained by a decrease in the index of refraction, n , caused by heating and subsequent expansion of the V A P fluid. The skew is explained by the V A P settling into an equil ibrium state. Feedback The above graphs and plots indicate a clear need for feedback. The feedback network in Chapter 3 (Figures 4.9 and 4.15) was constructed and attached to the V A P position sensors and actuators directly. Using feedback improves the situation considerably. The results show that subpixel 74 Time Analysis ~ 5000 Pixel Figure 4.24: V A P Drift over T ime movements can be achieved using the Variable Angle P r i sm. Initially, the following results in Figs. 4.25a and 4.25b. There is an obvious knee to the graphs. The knee is not due to hysteresis but rather to a spring loading effect in the actuator assembly. Over a smaller range wi th a single pixel , linear results are obtained as shown in F i g . 4.25c. W i t h the above linear V A P movement over a pixel range, it becomes easy to sample images displaced at subpixel amounts to each other. The procedure must be done quickly, however, because of the inevitable heating of the unit and subsequent drift. The above mentioned Figs. 4.25c and 4.25d do show some separation when the movement is reversed. A n explanation would be the slippage of the window as discussed in the previous Chapter. For this reason, it is recommended that the samples be taken while the movement is in the same direction. It should be noted that the feedback is stable for frequencies up to 100Hz. Also , a 75 VAP Disptacement with PSD Feedback V A P Displacement with PSD Feedback 52.21 1 1 1 1 1 1 M | 1 1 1 Applied Voltage(mV) Applied Voltage(mV) (c) (d) Figure 4.25: V A P Displacement versus Appl ied Voltage 76 Element 0 1 Grou 2 V 3 4 1 10.0 22.0 40.0 80.0 160.0 2 11.2 22.4 44.9 89.8 n / a 3 12.6 25.2 50.4 101.0 n / a : 4 14.1 28.3 56.6 113.0 n / a 5 15.9 31.7 63.5 127.0 n / a 6 17.8 35.6 71.3 143.0 n / a Table 4.1: U S A F 1951 Resolution Test Chart (lppm) large magnitude shake can bring the control out of the stable range. In addition, if the bench suffers large vibrations then the V A P wi l l oscillate at the 100Hz parasitic frequency. Such behavior was observed using an oscilloscope. 4.4.3 Sampling at Subpixel Intervals In this section, actual images were taken with the V A P calibrated using the above methods. In order to step the V A P at equal intervals, a D A C was used to drive the position. For each tr ia l , the V A P is calibrated and the graph of centroid dis- placement versus applied voltage used to find how much voltage was needed to get fractional increments. A l l the tests were done with a displacement of one quarter pixel i n one dimension (the vertical dimension). The max imum resolution improve- ment from the four images displaced at quarter pixel amounts is a factor of 2. The test image is the standard U S A F 1951 resolution chart (Table 4.1). Four images were taken displaced from each other by a quarter pixel each. 77 78 79 It is difficult to tell where the resolution cuts out on the above image. Zooming in on the middle section shows where it deteriorates (Fig. 4.26). Examinat ion of the zoomed image shows that the last clearly distinguishable group is Group 1 Element 5 giving an uncorrected resolution of 31.7 lppm. Group 1 is on the far right of the image. The high resolution reconstruction was done by meshing the image and fitting it to the higher resolution grid. This was done using Mat l ab (Tm) and the result is shown in Fig.4.27. Note that there are some jaggies present in the horizontal direction. These are due to,the fact that the image was not perfectly aligned. The 2nd element of Group 2 can be distinguished in this image putting the resolution at 44.9 lppm. Unfortunately the numbers are not clear on the image. Group 2 is the leftmost group in the center of the picture (to the right of the large Group 0). Element 2 is at the top of this group and element 1 of Group 2 is in the bot tom left corner of the two center groups (note that Element 1 of Group 0 is in the bottom left corner of the outer two groups (0 and 1)). The resulting resolution improvement is about 40 percent. The theory mentioned earlier predicts the maximum resolution recovery from the above method at double the resolution of the low-res pictures. The experimental results were lower than this predicted maximum value for two main reasons. Firs t , there was a delay of thirty minutes from when the V A P was calibrated to when the above reconstruction was done. This allowed for some drift in the V A P reference (from the P S D ) because of temperature changes in the V A P fluid. Second, there is noise introduced through the image acquisition which corrupts the images and can not be removed through our sampling process. In addition, the actual theoretical l imi t is measured by the P S F s of the actual C C D pixels. There was insufficient infor- mation to determine what the form and spacing of these P S F s was. The doubling of 80 resolution figure is just an upper bounds for four samples displaced at quarter-pixel amounts. Computing Requirements W i t h four 480 by 640 pictures, the above reconstruction took 4294833 flops in Ma t l ab (Tm) . The number of flops for reconstruction, using L images each of size m by n , is on the order of magnitude O(30ran + 2 * L * m * n). This maybe too slow for implementation i n real t ime. For real t ime operation, a faster processor or multiple processors may be required. In addition, the picture size might have to be reduced. 4.5 Issues and Problems The patent literature for Canon's proprietary V A P technology is geared towards image stabilization. They have constructed the device for rapid correction of high frequency camera shake. It incorporates a gyroscope and microprocessor for very fast correction. The microprocessor allows digital control and filters which makes the calibration of the system much easier. We effectively bypassed the microcontroller used by Canon. We do not need gy- roscope inputs because it is assumed that the camera is held stable. This cannot be the case, however, for a hand-held camcorder. In bypassing the microcontroller, we used analog circuitry for ease of in i t ia l measurements. Unfortunately, the V A P was not designed for long term D C stability. In addition, the instrumentation amps used to pick up the signals are subject to drift. The drift can be minimized, but since we are concerned with very small inter-pixel shifts, the measurements must 81 be done rapidly. In a final system, continuous calibration wi l l be required as well as digital control of the entire system. Examples of the drift were shown previously (Fig. 4.24). The control is ult imately l imited by the position sensing apparatus. Unfortunately, Canon does not publish the specifications for the infrared diodes and P S D sensor used in either the patent literature or the service manual. Modern PSDs have very good resolution and can track laser spots to an accuracy of lfim. In this device, however, the accuracy is clearly l imi ted by the mechanical window. Typ ica l infrared diodes have a broad spectrum with a viewing angle between 20 and 40 degrees. Without the mechanical window, the I R E D would spray itself onto the P S D making measurements difficult to obtain. The mechanical window, however is fairly large on the order of 1 — 2mm. Bo th the precision and accuracy of the angle sensor is therefore l imited primari ly by the mechanical window (Numeral 41b, F i g . 4.7). Another problem wi th the V A P assembly is the crosstalk from one side to another. When the angle is changed in the yaw direction, there is movement in one of the pitch directions. The converse is also true. According to the diagram of the V A P assembly in F i g . 4.1, the back plate is not completely independent from the front. The axis of the opposite plate can rotate on movement of the other plate because nothing holds the plates static and the axes aren't perfectly centered. Therefore, the control must hold the one side when the opposite side is moved. In a microcontroller system as implemented for the actual V A P controller in the camera, this could be directly implemented in the microcontroller itself. Unfortunately, when we connected our controller to both sides at the same time, the V A P would ini t ia l ly stay stable unt i l one side was moved. Afterwards, it would oscillate uncontrollably at the 100Hz parasitic frequency described in the patent literature. This is due to the noise and 82 crosstalk reducing the phase margin to an unacceptable level. A microcontroller based control is recommended to control both sides concurrently. In a future incarnation of a V A P image displacement device for the purposes of sub-sampling to increase resolution, it is recommended that a new angle sensing device be constructed. Instead of an I R E D , a laser diode could be mounted on the chassis where the existing window slit is. Also, more elaborate digital control using a microprocessor is required to improve stability and to get the images before drift occurs. Chapter 5 Conclusions This thesis has summarized the results of work in image processing. We have also given a simulation of how a higher resolution is achievable from several lower resolution frames. A device has been described that displaces images. This device is called a Variable- Angle P r i s m ( V A P ) . The displacement is linear wi th the pixel range. We have also shown how to control the V A P to get images displaced at subpixels to each other. We then used these images to get a higher resolution in the one dimension. The extension to 2 dimensions is nontrivial but possible. We have shown that the improvement in resolution from the above method is about 40 percent. This is less than the doubling of resolution that is theoretically pos- sible using 4 images. Further improvement would require digital control wi th a microcontroller to pick out the images rapidly and precisely. 83 84 5.1 Future Work for Combined System Ideally the system would incorporate both the Vari-Angle P r i sm and the hemi- spheric camera. The loss of resolution caused by the sampling of the C C D could be compensated quite nicely by dithering the images wi th the V A P . The main resolu- tion problem in the imaging system is at the C C D . If conventional film were used, there would be much less resolution lost. Unfortunately, current C C D s have pixel sizes of 10-20 micrometers, much larger than the resolution achieved with a photo- graphic f i lm. The advantage of using C C D s over film is the continuous sampling abil i ty and easy conversion into a standardized digital form. Unfortunately, the V A P is currently too large in its present form to fit under the primary mirror. It could be placed in front of the mirror but this would complicate matters enormously. The demapping would have to take into account the fact that the rays pass through the V A P twice. A simulation would have to be done and a calibrated table look-up formed to determine the amount of shift. This is not an efficient solution and the preferred method would be to have a miniature V A P or prism deflection system made to fit the hemispheric camera. After the V A P project was complete, we discovered that Sharp came up with a device for deflecting images wi th a single refracting element. This device d id not exist when the research for this thesis was performed. The reader who wants to do more work in this area is strongly urged to check out their patent. It is t i t led 'Imaging Apparatus Having Improved Resolution due to Control of the Inclination Angle of a Refracting Plate in More than One Direct ion ' (US patent 05637861). It maybe a better solution than the V A P because i t is far more simple and much less sensitive to temperature. In addition, the mirror has too much material inside obstructing the placement of 85 the C C D camera. The preferred mirrors need to be built using a mold technique so that they are very thin and strong. The cost is higher than for the a luminum ground mirrors that we used, but the bulk manufacturing prospects are great and subsequent mirrors are very cheap once the mold is buil t . Another problem is that the particular camera that we used has a potentiometer on the surface that prevents proper mounting inside the primary to coincide with the focal length of the lens. In addition, the lens does not fit inside the mirror hole. A custom lens needs to be built to fit exactly inside the hole on the top of the primary mirror or preferably the hole needs to be increased or decreased to fit a stock lens of choice. This involves choosing the right clear aperture of lens along wi th a focal length that w i l l project the image to the C C D surface correctly. It is my experience, however, that a standard C C D camera wi l l not work wi th the apparatus. A C C D camera needs to be built from scratch using raw C C D sensor components. Such a task is not t r iv ia l . Ideally, the camera would be digital. Our experiments wi th the camcorder used a frame grabber that resampled the N T S C format. The camera outputs the signal in analog form which is then chopped up by the A D C on the frame grabber. Using a digital camera would eliminate additional conversion loss and degradation by quantization noise. ^ Another big problem is the bl ind spot. Right now, the view can only be called substantially hemispheric. Perhaps in a later incarnation, the bl ind spot in the center of the scene could be reduced further by putting a lens system in a hole with a diameter the same size as the b l ind spot in the upper mirror (hb2 i n F i g . 2.6). This would be projected onto the unused centre spot that currently exists on the C C D because of the geometry. If the image from beyond the secondary mirror is l imited in scope to the same size as the blindspot, then it w i l l not interfere with the 86 hemispheric data. Such a modification would allow more of the actual hemispheric scene to be imaged. Future work wi th the V A P involves digitizing the control and connecting both front and back sides of the device together to shift the images in the horizontal and vertical directions and obtain 2 dimensional image resolution improvement. A microproces- sor is required to coordinate the movements properly and assure loop stability. Final ly , we have mentioned the possibility of incorporating the two technologies of omnidirectional vision and super-resolution into a single device. The prospects are very good as they complement each other nicely. Many more experiments need to be done, however, to develop a full working model that is commercially viable. 5.2 Summary of Results In this work we have shown a system for hemispheric or omnidirectional vision. It is compact and has a very large field of view of 360 degrees i n one hemisphere. The geometry and issues concerning the camera have al l been examined and a design proposed for minimizing some of the intrinsic optical problems. A resolution recovery method was also shown and built using a Variable Angle P r i sm to effect image shifts. The hemispheric vision system above was in fact built and demonstrated to work. There are certain things that need to be done to improve the resolution of the images, however. These include, but are not l imited to, making a custom digi tal C C D camera wi th a large number of pixels, rebuilding the mirrors wi th an injection moulding process so that they are very thin (one thousandth of a inch thick is possible), building custom optical mounts to hold the mirrors in place, and finally 87 choosing a new lens matched to the custom C C D and primary mirror aperture. M u c h work needs to be done before the system can be marketed commercially. The market for such a device has not been determined, but it may grow exponentially as knowledge of omnidirectional sensors and their applications is determined. Also in this thesis, a image displacement apparatus known as a Variable-Angle prism was outlined. It displaces rays of light and can be used for sub-pixel movements. Using several images displaced from each other at sub-pixel amounts, a higher res- olution can be obtained for the final image of the original scene. The V A P was used successfully to obtain resolution improvement in one dimension. Subpixel dis- placements were also confirmed using the device and our custom control hardware. Super-resolution was demonstrated successfully. Bibliography [Gro86] D . Gross. Super Resolution from Sub-pixel Shifted Pictures. Master's thesis, Te l -Av iv University, October 1986. [H + 86] E . Ha l l et al . Omnidirectional Viewing Using a F ish Eye Lens. In Optics, Illumination, and Image Sensing for Machine Vision, volume V o l . 728, pages pp. 250-256, Chicago, 1986. S P I E . [IP90] Micha l Irani and Shmuel Peleg. Super Resolution From Image Sequences. IEEE Transactions on Acoustics, Speech, and Signal Processing, V o l . 38( l ) :pp . l l5 -120 , January 1990. [IP91] M . Irani and S. Peleg. Improving Resolution by Image Registration. CVGIP: Graphical Models and Image Processing, V o l . 53:pp.231-239, M a y 1991. [Jai88] A . K . Jain. Fundamentals of Digital Image Processing. Prentice H a l l , New York, 1988. [JR84] L . S . Joice and W . L . Root. Precision Bounds in Superresolution Process- ing. Journal Optical Society of America, pages ppl49-168, Feb 1984. 88 89 [KA96] A . Krishnan and N . Ahuja. Panoramic Image Acquisi t ion. Proceedings of IEEE Conference on Computer Vision and Image Acquisition, pages pp.379-384, June 1996. [KBV90] S. P. K i m , N . K . Bose, and H . M . Valenzuela. Recursive Reconstruc- t ion of High Resolution Image From Noisy Undersampled Multiframes. IEEE Transactions on Acoustics, Speech, and Signal Processing, V o l . 38(6):ppl013-1027, June 1990. [Kor88] Dietr ich Korsch. Reflective Optics. Academic Press, I N C . , Boston, 1988. [KS93] S. P. K i m and Wen-Yu Su. Subpixel Accuracy Image Registration by Spectrum Cancellation. IEEE Procs., V o l . V(6):ppl53-156, 1993. [Luk66] W . Lukosz. Opt ical Systems with Resolving Power Exceeding the Classical L i m i t . Journal Optical Society of America, 56:pp. 1463-1472, November 1966. [Luk67] W . Lukosz. Opt ica l Systems with Resolving Power Exceeding the Classical L i m i t , II. Journal Optical Society of America, 57:pp. 932-941, Ju ly 1967. [MS88] M . S . Mor t and M . D . Srinath. M a x i m u m Likel ihood Image Registration wi th Subpixel Accuracy. Applications of Digital Image Processing XI, SPIE, V o l . 974:pp38-45, 1988. [Nal96] V . Nalwa. A True Omnidirectional Viewer. Technical report, Bell Labo- ratories, Holmdel, NJ 0773S, USA, February 1996. [Nay96] Shree K . Nayar. Catadioptric Omnidirectional Cameras. Technical Report, Dept. of Computer Science, Columbia University, October 1996. 90 [Nor78] A . Nordbryhn. The Dynamic Sampling Effect wi th C C D Imagers. In Applications of Elec. Imaging Systems, volume 143, pages pp42-51. S P I E , 1978. [NPB98] Shree Nayar, Shmuel Peleg, and Rafi Brada. Omnidirect ional Imaging Apparatus. US Patent 5760826, page 24, June 2 1998. [SA87] K e n D . Sauer and Jan P. Allebach. Iterative Reconstruction of Band- Limi ted Images from Nonuniformly Spaced Samples. IEEE Transactions on Circuits and Systems, V o l . 34(12):pp.1497-1506, December 1987. [SA94] S. Shaw and J . K . Aggarwal. A Simple Calibrat ion Procedure for Fish- Eye(High Distortion) Lens Camera. In Proc. IEEE Int. Conf. on Robotics and Automation, pages pp. 3422-27. I E E E , 1994. [S089] H . Stark and P. Oskoui. High-resolution Image Recovery from Image-plane Arrays, Using Convex Projections. J. Opt. Soc. Am. A, V o l . 6:ppl715- 1726, 1989. [Spe84] W . P . Spence. Engineering Graphics. Prentice H a l l , N J , 1984. [SPS87] Danny Keren S. Peleg and L . Schweitzer. Improving Image Resolution Using Subpixel Mot ion . Pattern Recognition Letters, V o l . 5:pp223-226, March 1987. [ST90] M . Ibrahim Sezan and A . Murat Tekalp. Adapt ive Image Restora- t ion wi th Artifact Suppression Using the Theory of Convex Projections. IEEE Transactions on Acoustics, Speech, and Signal Processing, V o l . 38(l):pp.l81-185, January 1990. 91 [TG94] Br ian Tom and Nikolas Galatsanos. Reconstruction of a High Resolu- tion Image from Registration and Restoration of Low Resolution Images. Image Processing, 1994 Internation Conference, V o l . 3:pp.553-557, 1994. [TK94] B . C . Tom and A . K . Katsaggelos. Mul t i -Channel Image Identification and Restoration Using the Expectat ion-Maximizat ion Algor i thm. In Appli- cations of Digital Image Processing XVII, volume V o l . 2298, San Diego, Ju ly 1994. S P I E . [TOS92] A . M . Tekalp, M . K . Ozkan, and M . I . Sezan. High-Resolution Image Re- construction From Lower-Resolution Image Sequences and Space-Varying Image Restoration. IEEE Proc. ICASSP-92, V o l . 3:pp.169-172, 1992. [Y + 95] K . Yamazawa et al. Obstacle Detection wi th Omnidirect ional Image Sen- sor HyperOmni Vis ion . IEEE Internal Conference on Robotics and Au- tomation, pages pp.1062-1067, M a y 1995. Appendix A Aberrations (see Chapter 2) Traditional optical design concerns maximizing the quality of mapping a planar object onto a planar sensor. There are several types of aberrations, called the Seidel aberrations. These aberrations are third order effects and include the phenomena of spherical aberration, coma, astigmatism, field curvature, and distortion. They are represented by the coefficients A , B , C , - (C+2D) , and E respectively[Kor88]. Our two mirror system (Fig.2.6) is similar in form to a double parabolic collimator described by Korch[Kor88] and has the same Seidel aberration coefficients[Kor88]: A = 0 ( A . l ) 5 = - ( r 1 ) - 2 ( c 1 2 + c 2 2 ) (A.2) C = 271(2*! + nd^B + 2 ( ^ - ( C l 2 + c 2 2 ) + C l - c 2 ) (A.3) D=l-C-Cl + c2 (A.4) E = - r 1 2 t 1 ( 3 * 1 + 2r1d1)B + nih + ^~)C + T&D - r 1 2 r f 1 ( ^ c 2 2 + C l + c 2 ) (A.5) where T\ = 1 — v\t\, V\ = c\ and c 2 the vertex curvatures of the mirrors(ci = ^r-), 92 93 t\ the entrance pupi l distance (ti = fi in our system), d\ the mirror separation, s\ the object distance and f i i the ratio of the clear aperture of mirror 2 to mirror 1. Because A is zero, there is no spherical aberration in our system. This comes by virtue of using parabolic reflectors. Please see Korch[Kor88] for a more detailed expression of the coefficients. For our system wi th an object at 5 metres the coeffi- cients are as follows: B = 0 . 0 6 7 - ^ c m 2 C = 2 . 4 8 — cm D - 1.06— cm E = 5.3


Citation Scheme:


Usage Statistics

Country Views Downloads
China 4 0
United States 3 1
City Views Downloads
Beijing 4 0
Redmond 2 0
Unknown 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}


Share to:


Related Items