UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

MPEG video coding with adaptive motion compensation and bit allocation based on perception criteria Barzykina, Ekaterina 1998

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_1998-0685.pdf [ 10.62MB ]
Metadata
JSON: 831-1.0065211.json
JSON-LD: 831-1.0065211-ld.json
RDF/XML (Pretty): 831-1.0065211-rdf.xml
RDF/JSON: 831-1.0065211-rdf.json
Turtle: 831-1.0065211-turtle.txt
N-Triples: 831-1.0065211-rdf-ntriples.txt
Original Record: 831-1.0065211-source.json
Full Text
831-1.0065211-fulltext.txt
Citation
831-1.0065211.ris

Full Text

MPEG VIDEO CODING WITH ADAPTIVE MOTION COMPENSATION AND BIT ALLOCATION BASED ON PERCEPTION CRITERIA by Ekaterina Barzykina B. A. Sc. (Electrical Engineering) St. Petersburg Film and Television School  A THESIS SUBMITTED IN PARTIAL FTJIJTLLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE STUDIES DEPARTMENT OF ELECTICAL AND COMPUTER ENGINEERING We accept this thesis as conforming to the required standard  THE UNIVERSITY OF BRITISH COLUMBIA . April 1998 © Ekaterina Barzykina, 1998  In presenting this thesis in partial fulfilment  of the requirements for an advanced  degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department  or  by his or  her  representatives.  It  is understood that  copying or  publication of this thesis for financial gain shall not be allowed without my written permission.  Department of  Mc&L  b t l  <f Q)ATZ^C/^  The University of British Columbia Vancouver, Canada Date  DE-6  (2/88)  tttf-M  4'93/  ^i^l/l££^U^y>  Abstract Picture quality and bandwidth are the two most important issues in digital video communications. A new method that guarantees consistent picture quality throughout each frame and each group of pictures (GOP) and also improves encoding efficiency of all versions of MPEG standard is proposed in this thesis. Addressing the bandwidth problem, we develop a unique Frame Partitioning technique that significantly improves the exploitation of temporal redundancies amongst video frames. This new technique works is used to modify the standard MPEG motion compensation strategy to allow for different temporal resolution within a single frame, thus improvinl compression. To further improve picture quality, we propose an original bit allocation and rate control algorithm complementing our motion compensation technique. We base our approach to bit allocation on macroblock distortion measurements, ensuring the consistency of quality on a frame and a GOP level. For the same picture quality, frame partitioning alone yields an additional increase of up to 20 percent or more in the encoding efficiency.  Within a single frame, the use of our bit  allocation and rate control algorithm can decreases the fluctuations in picture quality by as much as 70 percent. The combination of both techniques proposed in this thesis, allows an even greater improvement in either overall visual quality or bandwidth.  Table of Contents  Abstract  ii  T a b l e of Contents  iii  List of Figures  ix  List of Tables  '.  ix  x  Acknowledgment Chapter 1  INTRODUCTION  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION  ;  1  COMPENSATION BASED ON FRAME 6  PARTITIONING 2.1 2.2  INTRODUCTION  6  TRADITIONAL MPEG MOTION COMPENSATION A N D ITS D R A W B A C K S  8  2.2.A  M o t i o n Compensation in M P E G  8  2 . 2 . B D r a w b a c k s o f the traditional a p p r o a c h  13  C o n s e c u t i v e frames c o n t a i n i n g areas that are exactly the same  14  C o n s e c u t i v e frames c o n t a i n i n g areas w i t h "non-visible" changes  17  C o n s e c u t i v e frames c o n t a i n i n g areas w i t h "almost n o n - v i s i b l e " changes  iii  20  2.3 FRAME PARTITIONING  24  2.3.A Concept of Frame Partitioning  24  2.3.B Determination of the Frame Partitioning Mask  28  2.4 MOTION COMPENSATION WITH FRAME PARTITIONING  .34  2:5 MOTION COMPENSATION WITH FRAME PARTITIONING : RESULTS AND APPLICATIONS  39  Bit-rate improvement  40  Quality improvement  42  2.6 SUMMARY Chapter 3  54  BIT ALLOCATION AND R A T E CONTROL BASED ON M A C R O B L O C K QUALITY CRITERIA  56  3.1 INTRODUCTION  56  3.2 EXISTING BIT ALLOCATION AND RATE CONTROL TECHNIQUES  58  3.2.A Bit allocation  58  3.2.B Rate control  60  Rate-Distortion and Rate-Quantization in the TM5 algorithm  : 62  Rate-Quantization modeling on a macroblock level with pre-encoding  64  Drawbacks of the non-integrated approach to  iv  bit allocation and rate control  66  3.3 AN INTEGRATED APPROACH TO BIT ALLOCATION AND RATE CONTOL BASED ON QUALITY CRITERIA  69  3.3 A . Bit allocation based on macroblock quality  72  3.3.B. Rate control  76  Rate-Quantization modeling  76  Quality-Quantization modeling  80  3.4 A NOVEL BIT ALLOCATION ALGORITHM WITH THE EMPHASIS ON CONSITANCY OF IMAGE QUALITY  84  3.4.A. Bit allocation and rate control algorithm based on macroblock quality criteria in variable bit rate encoding  84  3.4.B. Bit allocation and rate control algorithm based on macroblock quality criteria in constant bit rate two-pass encoding  88  3.4.C. Bit allocation and rate control in constant bit rate real-time encoding 3.5 RESULTS  ... 92 .'•  95  Macroblock level quality statistics  95  Frame level quality statistics  96  3.6 SUMMARY  98  V  Chapter 4  CONCLUSIONS  100  Bibliography  102  vi  List of Figures  Figure 2.1  Relationship between I, P and B frames in MPEG video encoding  Figure 2.2  9  MPEG Motion Compensation: conceptual idea of an algorithm  Figure 2.3  10  Traditional MPEG Motion Compensation: block scheme of an algorithm  Figure 2.4  12  Traditional MPEG Motion Compensation in video sequences containing areas that are exactly the same  Figure 2.5  Traditional MPEG Motion Compensation in video sequences containing areas that visually identical  Figure 2.6  16  19  Traditional MPEG Motion Compensation in video sequences containing areas that are visually almost identical  22  Figure 2.7  Local temporal resolution in video encoding  26  Figure 2.8  Example of a Frame Partitioning Mask  28  Figure 2.9  Structure of a macroblock in MPEG-2 Main Video Profile  30  Figure 2.10  Motion Compensation with Frame Partitioning: block scheme of an algorithm  Figure 2.11  Comparison between encoding algorithms that use Motion Compensation with Frame Partitioning and  vii  37  algorithms that use traditional MPEG Motion Compensation Figure 2.12  38  Comparison of the quality performance between the traditional MPEG-2 encoding and MPEG-2 encoding with Frame Partitioning, example "Tennis"  Figure 2.13  43  Comparison of the quality performance between the traditional MPEG-2 encoding and MPEG-2 encoding with . Frame Partitioning, example "Ballet"  Figure 2.14  46  Comparison of the quality performance between the traditional MPEG-2 encoding and MPEG-2 encoding with Frame Partitioning, example "Garden"  49  Figure 3.1  Rate control strategy of the Test Model 5 algorithm  63  Figure 3.2  Buffer strategies in constant bit rate encoding  70  Figure 3.3  Bit allocation based on macroblock quality: block scheme of an algorithm  Figure 3.4  73  Rate-Quantization and Quality-Quantization modeling in a bit allocation algorithm based on macroblock quality criteria  Figure 3.5  76.  Determination of macroblock Variance Classes in an image  Figure 3.6  78  Example of the Rate-Quantization curves for Intra and Inter variance classes  80  viii  Figure 3.7  Example of the Quality-Quantization curves for Intra and Inter variance classes  Figure 3.8  83  Bit allocation and rate control algorithm based on macroblock quality criteria in variable bit rate two-pass encoding  Figure 3.9  86  Bit allocation and rate control algorithm based on macroblock quality criteria in constant bit rate two-pass encoding: first pass  87  List of Tables  Table 2.1  Size of the Frame Partitioning Mask in percent in for full size images  Table 2.2  40  Bit-rate improvement achieved by using adaptive Motion Compensation based on Frame Partitioning  Table 3.1  41  Comparison of macroblock level SNR statistics for I and P frames encoded with our bit allocation and rate control algorithm and encoded with the Test Model 5 algorithm  Table 3.2  96  Comparison of frame level SNR statistics for frames encoded with our bit allocation and rate control algorithm and with the Test Model 5 algorithm  X  97  Acknowledgment  I would like to thank my supervisors, professors Rabab K. Ward and Panos Nasiopoulos for their continuous encouragement and support that they have given in the last few years. I am very grateful for of all of advice and experience that they have shared with me. Also, I would like to express my gratitude to all other people who cared about me and wished me well.  Chapter 1 INTRODUCTION  Chapter 1 INTRODUCTION In the last few years, the world has witnessed a new wave of evolution in the communications, broadcasting and entertainment industries. This upsurge is directly connected with the activities of the Motion Pictures Expert Group (MPEG), a committee that overseas the development of standards for compression, decompression, processing, and coding of moving pictures and associated audio for digital storage media. M P E G is one of the working groups of the International Standardization Organization (ISO) and so far has produced two standards, MPEG-1 and MPEG-2, with work being continued on two more standards, MPEG-4 and MPEG-7. In 1991, the first version of M P E G , MPEG-1, standardized the coding of the combined audio-visual signal at the bit-rates around 1.5 Mbit/s. The MPEG-1 standard was primarily intended to process video at SIF (Source Input Format) resolution (images of 352x240 pixels at 30 frames per second) and to play back video off a C D - R O M or over telephone lines at a low bit-rate. Even though the MPEG-1 syntax had been shown to operate successfully on higher-resolution pictures and at higher bit-rates, some of its limitations, such as being able to accommodate only progressively scanned and not the interlaced pictures, have prompted the development of a second stage of the M P E G standard - MPEG-2. In 1993, the ISO committee introduced MPEG-2 - a standard designed for broadcast quality video and associated audio. The MPEG-2 allowed compression of the audio-visual signal at higher bit-rates ( around 6 Mbit/s and higher) for full-resolution images (720x480 pixels and higher). MPEG-2 supported interlaced video, multi-resolution scalability, as well  1  Chapter 1 INTRODUCTION  as a number of other new technical features. In addition, responding to a large number of potential applications of MPEG-2, the MPEG committee standardized five profiles to be used in different communications scenarios - Simple Profile, Main Profile, Spatially Scaleable Profile and others. Today, the MPEG-2 standard enjoys a worldwide acceptance and it is exclusively used by such applications as High Definition Television (HDTV), Digital Television and Digital Versatile Disc (DVD).  '  The newest phase of the MPEG standard, MPEG-4, is expected to reach its final stage in November 1998 [1] and is expected to provide a new level of interactivity for multimedia applications of the next generation. The MPEG compression works by exploiting two properties of video streams: the presence of considerable similarities between consecutive frames, which constitute temporal redundancies, and the fact that some details within a single image may be unnoticeable to the human eye, which constitute spatial redundancies. The main features supported by both MPEG-1 and MPEG-2 standards include block based Motion Compensation, Discrete Cosine Transform (DCT), quantization of the DCT coefficients and their variable length encoding. Motion Compensation takes advantage of the temporal redundancies in the video stream and reduces the initial amount of information to be encoded. Quantization exploits the spatial redundancies and matches the output to a given bit rate, allowing for most of the compression and determining the quality of the reconstructed images. The new MPEG-4 standard is expected to combine the mentioned above base features of MPEG with other new ideas. One of the unique features of MPEG-4 that is relevant to our work is the concept of a video object. A video object (VO) can be defined in a scene context, where a scene (video information of several consecutive frames) is composed of a  2  Chapter 1 INTRODUCTION  number of objects. For example, a person moving across a background would represent a scene with the person classified as V O l and the background as V02. Each VO can be of arbitrary shape and is encoded separately. The introduction of video objects opens great possibilities in many new applications such as interactive video. For simple video encoding, one of the advantages of video objects is that different objects can be encoded at different temporal/spatial scalability levels. However, although the concept of video object encoding is without doubt attractive, the amount of overhead information needed to define objects and reconstruct scenes may become a significant drawback. Depending on the application, this overhead can occupy 20 per cent or even more of the total stream bandwidth. For a fixed bit-rate, such overhead has a significant impact on quality. Video objects yet require additional processing and overhead information due to the contradiction between the arbitrary shapes of video objects and their block based encoding. The reason is that, while video objects are not necessarily rectangular, Motion Compensation and, more importantly, the DCT process remain block based. MPEG-4 handles this problem by first, defining an arbitrary boundary for a VO and then using zero padding to fill out a shape composed of 8x8 blocks.  Our aim in this thesis is to develop a different and a much simpler way to exploit the idea of video objects, while not introducing any additional overhead information and improving the encoding efficiency of the compression algorithm. We propose to modify the existing MPEG Motion Compensation procedure in order to achieve different temporal scalability encoding for different parts of a frame. We design a new Frame Partitioning technique - a technique that separates the temporal changes be-  3  Chapter 1 INTRODUCTION  tween consecutive frames into two categories: changes that are detected by the human eye and changes that are unnoticeable to the human vision. By utilizing the results of Frame Partitioning, our Motion Compensation with Frame Partitioning algorithm disregards the information corresponding to the unnoticeable changes, thus taking the exploitation of temporal redundancies to a new level - level of perceptual adaptability. Our method works within the standardized MPEG-2 (1) syntax and does not require any extra overhead information. The new technique of Motion Compensation with Frame Partitioning allows us to attain a significant reduction of the bit-rate for the same quality of reconstructed images or, reversibly, an increase in quality within the same bit budget. To maximize the advantages of the proposed Motion Compensation approach, we develop an original bit allocation and rate control algorithm. Our bit allocation strategy results in a greater consistency of quality in the reconstructed video sequence and can be beneficial to both constant bit-rate and variable bit-rate encoding.  The thesis is organized as follows. In Chapter 2 , we introduce our Motion Compensation algorithm. We review the existing approach to Motion Compensation and describe the situations where the traditional MPEG Motion Compensation can work to a disadvantage. We introduce a new concept of Frame Partitioning and incorporate this novel idea into our new Motion Compensation with Frame Partitioning algorithm. Our bit allocation and rate control strategy is discussed in Chapter 3. We examine different bit allocation and rate control techniques featured in current publications. We propose a  4  Chapter 1  INTRODUCTION  new approach to bit allocation which places the emphasis on consistency of quality, both throughout a frame and a sequence of frames. In both Chapter 2 and Chapter 3, we present the results of our research. We discuss the potential applications of Frame Partitioning and our bit allocation algorithm. We use tests results to illustrates the degree of improvement that can be achieved with these new techniques.  5  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Chapter 2 PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.1. INTRODUCTION Even though MPEG-2 encoding , at present, provides video at broadcast quality levels, it is still important for a compression algorithm to achieve yet higher encoding efficiency and, thus, quality. Since the output quality of a video compression algorithm is evaluated by people, an additional efficiency increase can be achieved by using the properties of human visual system to optimize the encoding process. So far, most of the work involving perceptual adaptability has been limited to the bit allocation (BA) and rate control (RC) algorithms. While different bit allocation techniques can sometimes without doubt significantly improve the visual quality of the decompressed sequence, at other times they are still not enough to provide acceptable encoding quality or are unable to improve the level of compression. In all versions of the MPEG standard, the compression at the stage of Motion Compensation is traditionally lossless and does not contain any perceptual adaptability, i.e. for the traditional Motion Compensation, all temporal changes that occur between the frames are encoded in full, whether or not these changes are perceived by the human eye. Our aim in this Chapter, is to propose an approach to Motion Compensation that is perceptually adaptive. We develop a novel Motion Compensation algorithm, which encodes only the changes that are "visible" to the human eye and disregards the "invisible" changes. By discarding the  6  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSA TION BASED ON FRAME PARTITIONING  non-visible temporal changes, our algorithm can achieve a significant improvement in the encoding efficiency without altering the perceived content of the video stream. The organization of this Chapter is as follows. In Section 2.2 we give an overview of the traditional approach to Motion Compensation in MPEG and illustrate the situations where such an approach can be improved. In Section 2.3 we introduce a new idea of Frame Partitioning and describe in detail how this new technique works. In Section 2.4, we present our Motion Compensation algorithm, which incorporates perceptual adaptability into this stage of compression. Finally, in Section 2.5 we discuss the results that can be achieved by using Motion Compensation with Frame Partitioning and the potential applications of this new technique.  7  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.2. TRADITIONAL MPEG MOTION COMPENSATION AND ITS DRAWBACKS  2.2.A. Motion Compensation in MPEG Motion Compensation (MC) is the fist stage in the MPEG algorithm. It takes advantage of the similarities between consecutive frames (temporal redundancies) and performs the initial compression through differential encoding of some frames in the input video sequence. The idea of Motion Compensation is that a current frame (and its constituent parts) can be modeled as a translation of another frame from some previous time, i.e. a Predicted Frame. With Motion Compensation, instead of encoding the contents of the actual image, the algorithm processes the information on the mismatch between parts of the current frame and their corresponding parts from the predicted frame. The output of Motion Compensation is the Prediction Error Frame, found as the pixel by pixel difference between the current frame and its predicted frame, and Motion Information, explaining how the predicted parts of the frame can be found. To support the random access to the frames in the video sequence and at the same time to exploit temporal redundancies to the maximum, the MPEG standard defines three types of frames: / (intra), P (predicted) and B (bi-directional interpolation). I frames are coded with no reference to any previous frame (no motion compensation). P and B frames are coded with respect to a preceding frame (P frames) or with respect to both preceding and following frames (B frames). The relationship between I, P and B frames in MPEG encoding is illustrated in Figure 2.1.  8  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  The MPEG standard divides the video sequence into groups ofpictures (GOP). A GOP always starts with an I frame, with the rest of the frames being of type P or of types P and B. To simplify our future discussions, we will assume that a GOP is composed of I and P frames only.  Frames in order of display  Motion Prediction  Figure 2.1. Relationship between I, P and B frames in MPEG video encoding.  In the MPEG standard, each frame in divided into blocks of 16 x 16 pixels, called a macroblock (MB). Each macroblock is composed of four 8x8 luminance blocks and their corresponding chrominance blocks. A macroblock serves as a unit for Motion Compensation and quantization, with a block being a unit for the DCT. For P and B frames the reduction of temporal redundancies is done through Motion Compensation, which in part is executed in two stages: the Motion Estimation (ME) and then the calculation of the prediction error. On the stage of Motion Estimation, each macroblock in the current frame is predicted from its Reference Frame, which can be a previous  9  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  frame, a future frames or both (see Figure 2.1). The results of Motion Estimation are the Predicted Frame, composed of all of the predicted macroblocks, and a set of parameters called motion information consisting mainly of motion vectors, which indicate the location of each predicted macroblock in the reference frame. After Motion Estimation is completed, the algorithm finds the pixel by pixel difference between the predicted and the current frame, the Prediction Error Frame. The steps of MPEG Motion Compensation strategy are illustrated in Figure 2.2. Reference Frame  Current Frame  Figure 2.2. MPEG Motion Compensation: conceptual idea of an algorithm. Since all of the MPEG standards are generic, they do not specify a particular implementation of any part of the compression algorithm. The standardization is limited to only the  10  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSA TION BASED ON FRAME PARTITIONING  syntax of the encoded bit stream. With respect to Motion Compensation, the MPEG specifies how to represent the motion information for each macroblock. It does not, however, specify how this information is to be found. In the traditional approach to Motion Compensation [4], the Motion Estimation algorithm uses a block-matching technique to find the motion vectors. For each macroblock, the vectors are obtained by minimizing a cost function measuring the mismatch between the reference and the current macroblock. Most often, ME algorithms use a macroblock pixel by pixel absolute difference to measure the mismatch. The absolute difference or absolute error (AE) for each macroblock is defined as follows: 15  15  AE( v , v )= ^ x  I cur(i,j) - ref(i-v ,j-v ) | .  y  x  y  (2.1)  ,where v and v are the motion vectors in the horizontal and vertical direction x  y  respectively; cur(i, j) is a pixel value in a macroblock from a current frame; ref(i, j) is a pixel value of the same macroblock in a reference frame; ref(i-v ,j-v ) is a pixel value the reference macroblock displaced x  y  by a vector (v , v ); x  y  The minimum absolute error within a search window will give the values of the motion vectors. Traditionally, motion vectors for each macroblock are first found in two stages. First the motion vectors are found by comparing the macroblocks in the Current Frame and the Original Reference Frame (the reference frame in its original state, i.e. before quantization). Then, optionally, these values can be further refined by comparing the macroblocks in the Current Frame and the Reconstructed Reference Frame (the reference frame that has been quantized and then inversely quantized, so as to simulate the decoding process). After  11  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  the motion vectors are identified, they are used to find the predicted frame. The pixel values of each macroblock in the Predicted Frame are found as the values of the corresponding macroblock in the Reconstructed Reference Frame displaced by the corresponding motion vectors. Figure 2.3 presents a blocks diagram, illustrating the steps of the traditional MPEG Motion Compensation algorithm.  Full pixel search Half pixel search Find motion vectors !|Mb|tiOT'vectorsj  |  Find predicted frame "|« Predicted lianic  |  Calculate predicted error frame **|  Figure 2.3. Traditional MPEG Motion Compensation: block scheme of an algorithm.  After Motion Compensation, each macroblock in a frame is identified as being of either Intra or Inter type. The pixel values of an Intra macroblock are just its original values in the  12  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  current frame, taken with respect to the average intensity value of 128. The pixel values of an Inter macroblock are equal to its prediction error, i. e., the pixel by pixel difference between the predicted and the current macroblock. I frames are composed of Intra macroblocks only. P frames can have both Intra and Inter type macroblocks, where an Inter macroblock can be either of two types: predicted with motion vectors or predicted with no motion vectors. An Inter macroblock of the first type has motion vectors whose values are not equal to zero, while a macroblock of the second type has zero motion vector. After Motion Compensation, each P and B frame is represented by the prediction error frame and its corresponding motion information. As was mentioned before, the traditional MPEG Motion Compensation is lossless, i.e. no error is introduced in this step of the encoding. The lossy part of the algorithm comes later, in the quantization of the DCT coefficients.  2.2.B. The drawbacks of the traditional approach to Motion Compensation Because the traditional MPEG Motion Compensation is a lossless procedure, it is very difficult to introduce perceptual adaptability into this stage of compression. The typical Motion Compensation algorithm treats all areas of the image equally, regardless of the visual importance of the temporal changes in different parts of the frames. In many cases, even changes that can not be detected by the human eye are still encoded. As a result, a significant percentage of the number of bits allocated for a frame can be wasted on these areas and, because of the limited bit budget, other parts of the image may suffer from visual artifacts.  13  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSA TION BASED ON FRAME PARTITIONING  Let us examine in detail how the traditional MPEG Motion Compensation algorithm works when it encounters different kinds of temporal changes that occur in a video sequences.  i) Consecutive frames containing areas that are exactly the same Suppose that an input video stream contains several consecutive frames, each composed of two regions: a "no-change" area that remains exactly the same for all of these frames and a "change" area that is continuously different from one frame to the next. For the "nochange" area, either the difference between the two frames for this area is zero or this area is an exact translation of another area in the reference frame and the difference between these two areas is zero When the Motion Compensation algorithm calculates the pixel values of the prediction error frame, it finds the mismatch between the current original frame and the reconstructed reference frame (Figure 2.3). Due to the quantization error, the reconstructed reference frame can significantly differ from its original version. Consequently, the values of the prediction error in a "no-change" area will always be non-zero. The algorithm will fail to recognize that in the original sequence this area is in fact exactly the same as it was in the previous frame. Instead, the encoder will just compute the prediction error frame and send this information for further processing: quantization and final coding. During quantization, most bit allocation (BA) and rate control (RC) algorithms will treat both "no-change" and "change" macroblocks equally, which is contradicted by the very different perceptual effects that will result from the loss of details in these two regions. The impact of the quantization in the "no-change" area will manifest itself in the lower reconstructed quality of this area  14  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  when it first appears, followed by its continuous improvement from so on from one frame to the next. The consequences for the "change" area are more severe, such as the loss of information vital to the understanding of the video content and the appearance of visual artifacts in areas that carry new information . With many BA and RC strategies, the result of encoding both areas in the same way will be an unevenness of quality levels between the two areas in the images, with higher quality in "no-change" areas and lower quality in "change" regions. If the algorithm could establish beforehand that the "no-change" areas in the consecutive frames are indeed exactly the same, then it could alter the encoding strategy, so that all or most of the available bit budget could be spend on the areas that correspond to the region with changes, thus avoiding artifacts and maintaining an even level of quality for all parts of the images. A good example of a video stream containing areas that are exactly the same is a computer, generated sequence with an object moving on top of a sill background. If the background of such sequence is reasonably complex, as represented in Figure 2.4.a, then, provided that the bit-rate is moderate, the encoding algorithm will need the time interval of several consecutive frames to exhaust the background data (see Figure 2.4.b). As a result, while the quality of the background continuously improves, areas with changes may not have enough bits to prevent artifacts (see Figure 2.4.c).  15  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSATION BASED ON FRAME  PARTITIONING  (a) Original consecutive frames FrameO, Frame 1 and Frame2.  (b) Prediction Error Frame for Framel and Frame2.  (c) Reconstructed FrameO, Framel and Frame2.  Figure 2.4. Traditional Motion Compensation in a video sequence containing areas that are exactly the same.  16  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  ii) Consecutive frames containing areas with "non-visible" changes In the previous example we examined a video stream containing picture areas that remain exactly the same during a period of several consecutive frames. Such pixel by pixel similarities can be encountered mostly in artificially generated images (computer graphics, cartoons, etc.). Due to the nature of the real-life based video, the areas will seem "the same" to the human eye are most of the times different on a pixel by pixel level. Many times, such conditions as subtle changes in lights and shadows or changes in details that are too intricate to recognize can account for the pixel level differences between images that are perceived as being the same. Since the encoding algorithm does not make any distinction between "visible" and "nonvisible" changes, it will give equal attention to macroblocks carrying information about nonvisible background fluctuations as well as macroblocks that correspond to visible motion. As a result, a significant part of the allowed bit budget can be spent on encoding the nonperceived differences between consecutive frames, while other areas of the image can suffer from visual artifacts. If the encoding algorithm could identify that some of the changes being encoded are not detected by the human eye, then a logical assumption would be that these changes can be completely omitted. By not encoding the "non-visible" changes that occur between the frames, we could increase the overall encoding efficiency without altering the perceived content of the video stream. Such an approach would either result in a decrease in the bitrate or in an improvement in quality, as it will enable the encoding algorithm to reduce the overall level of artifacts by refining the quantization.  17  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  An example of a sequence that contains the "non-visible" temporal changes is given on Figure 2.5. This sequence represents a panoramic view, where most of the changes between the frame macroblocks are caused by motion and not by the absolute changes in there visual content. Even thought the differences (apart from motion) between the original Frame 1 and Frame2 of Figure 2.5.a in most areas are not visible, the actual differences on the pixel by pixel level (after taking motion into account) in these same areas are non-zero. The non-zero differences are especially noticeable in the areas of high texture (flowers) in Figure 2.5.b, showing the difference between the two original frames after motion estimation. The quantization of Frame 1 forces the pixel values of the prediction error of Frame2 in the area of the "non-visible" changes to become yet greater (Figure 2.5.c), thus increasing the amount of "non-visible" information to be encoded. Figure 2.5.d illustrates the visual artifacts that could have been avoided if a perceptually adaptive motion compensation were used.  18  Chapter 2 PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  An example of a sequence that contains the "non-visible" temporal changes is given oh Figure 2.5. This sequence represents a panoramic view, where most of the changes between the frame macroblocks are caused by motion and not by the absolute changes in there visual content. Even thought the differences (apart from motion) between the original Frame 1 and Frame2 of Figure 2.5.a in most areas are not visible, the actual differences on the pixel by pixel level (after taking motion into account) in these same areas are non-zero. The non-zero differences are especially noticeable in the areas of high texture (flowers) in Figure 2,5.b, showing the difference between the two original frames after motion estimation. The quantization of Framel forces the pixel values of the prediction error of Frame2 in the area of the "non-visible" changes to become yet greater (Figure 2.5.c), thus increasing the amount of "non-visible" information to be encoded. Figure 2.5.d illustrates the visual artifacts that could have been avoided if a perceptually adaptive motion compensation were used.  18  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  (b) Difference between original Framel and Frame2 after estimating Motion.  (c) Prediction Error Frame for Frame2  Increased level of artifacts in the "change areas  (d)Reconstructed Framel and Frame2. Figure 2.5. Traditional Motion Compensation in a video sequence containing areas that are visually identical.  19  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  iii) Consecutive frames containing areas with "almost non-visible" changes Other kinds of temporal changes that can occur between consecutive frames are the "almost non-visible" changes. These changes correspond to such visual effects as minor fluctuations in the background and are generally not perceived by the human eye when frames are played as a sequence, but can be recognized if the images are examined one by one as still images. As was mentioned before, in the traditional MPEG encoding all kinds of temporal changes are treated in exactly the same way: the "non-visible", the "almost non-visible" and the "clearly visible" changes. In a situation of a limited bit-budget and a lot of motion between the frames, there can not be enough bandwidth to equally preserve all of these changes at the acceptable quality levels. The shortage of bandwidth may result in either an overall higher levels of artifacts in all areas of the image, and/or a high levels of distortion located exclusively in areas with a lot of motion changes. By decreasing the number of bits allocated to macroblocks whose distortion is less evident to the human visual system and making these bits available to macroblocks whose distortion is more perceptually apparent, a good bit allocation algorithm can sometimes solve the problem of artifacts, but only to a limited degree and only if the artifacts are minor and appearance is restricted to a relatively small area in a frame. Unfortunately, if the artifacts are severe or cover a large area, additional innovative measures are needed to ensure the acceptable quality of the reconstructed images. If the algorithm could distinguish the "almost non-visible" fluctuations from the visually important information, then it could find a better encoding solution as a compromise between the overall visual quality of the video stream and the accuracy of the representation of the "almost non-visible" changes. In other words, in certain circumstances we see it benefi-  20  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  cial to allow the loss of some of the information about the changes between consecutive frames in order to preserve the overall acceptable quality of the video stream. It is necessary to stress, that we are here talking only about those temporal changes which are not perceptually significant and only about the encoding situations which will otherwise result in a high levels of visible artifacts in the reconstructed images. Figure 2.6 illustrates a video stream that contain the "almost non-visible" temporal changes and the level of artifacts present in the frames which have been encoded with traditional MPEG, providing a good example where the "almost non-visible" changes can be sacrificed in order to preserve quality.  21  Chapter 2 . PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Almost non-visible changes:  (b) Prediction Error Frame for Framel  (c) Reconstructed FrameO and Framel. Figure 2.6. Traditional Motion Compensation in a video sequence containing areas that are visually almost identical.  22  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSA TION BASED ON FRAME PARTITIONING  A close examination of the traditional MPEG Motion Compensation strategy reveals its iack of flexibility when it comes to encoding temporal changes that are different in their perceptual manifestation. The examples that we have examined in this section, indicate the importance of the perceptually adaptive approach to Motion Compensation, which would provide more efficient ways to exploit temporal redundancies in the video stream. We would like to propose a new technique that will allow us to incorporate the characteristics of the human visual system into the process of Motion Compensation, so as to increase the overall quality or compression performance of the encoding algorithm.  23  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.3. FRAME PARTITONING 2.3.A. Concept of Frame Partitioning We introduce a concept of Frame Partitioning, a novel technique that is designed to separate temporal changes that occur between the frames into different categories based on the degree of the perceived importance of these changes. In order to simplify our discussion, we will talk only about two categories of temporal changes — changes that are detected by the human eye and changes that are invisible to human vision. If desirable, our technique can be extended to allow for the detection of the "almost non-visible" changes as well. The goal of our Frame Partitioning technique is to separate each frame in a video sequence into two areas - an area with "visible" changes and an area that contains only the "non-visible" temporal changes. In order to work within the MPEG syntax, Frame Partitioning has been designed as a block based technique. For each macroblock in a frame, Frame Partitioning makes a decision as to whether or not the changes contained within that MB are detected by the human eye. The macroblocks that carry the "non-visible" changes shall be referred to as Insignificant, while all other macroblocks are labeled Significant. A collection of all Insignificant macroblocks in a frame constitute a Frame Partitioning Mask, an area whose spatial information is not detected by human vision and, thus, can be disregarded during encoding.  Frame Partitioning carries an analogy to MPEG-4 video objects,  with only two instances of an object allowed within a single frame: one "object", composed  24  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  of all Significant macroblocks, and one "background", composed of all Insignificant macroblocks. Frame Partitioning identifies Significant and Insignificant areas in a frame based only on the degree of similarities between the original macroblocks and their predicted counterparts. Information about motion is not considered in the decision as to on whether or not a macroblock is Significant or Insignificant. In other words, a macroblock whose location in the consecutive frames changes but whose visual content remains the same is considered Insignificant by our Frame Partitioning technique. Another definition of Significant and Insignificant macroblocks can be given using the concept of local (instantaneous) temporal resolution or just temporal resolution. We define local temporal resolution as a virtual frame/macroblock rate at which we can encode the corresponding frame/macroblock video information without introducing visual discontinuity into the decoded stream. On a frame level, if two (or more) consecutive frames are visually the same, then the first of these frames will be considered as having its temporal resolution twice (or more) lower than the frame rate (see Figure 2.7, a). Similar reasoning is used to identify the temporal resolution on a macroblock level (see Figure 2.7, b). We are using the term "visually the same" to emphasize that the frames ( and in parallel the macroblocks, if we will be talking about temporal resolution for macroblocks) do not have to be exactly the same, but just have to be identified as visually unchanged. In Frame Partitioning, we propose to identify Significant and Insignificant macroblocks based on the minimum temporal resolution that is necessary for their encoding. Within a single frame, all macroblocks that must be encoded with a local temporal resolution equal to the frame rate, will be identified  25  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  as Significant macroblocks, while all macroblocks whose local temporal resolution can be half that of the frame rate or less, will be identified as Insignificant macroblocks.  temporal resolution = frame rate  temporal resolution = Vi frame rate  Framel, visually different from FrameO  Frame2, visually identical to Framel  temporal resolution = frame rate  Frame3, visually different from Frame2  Frame4, visually different from Frame3  (2.7.a) Local temporal resolution on a frame level.  temporal resolution = frame rate  Macroblock 0  MB 1, visually different from MB 0, labeled Significant  temporal resolution = 1/3 frame rate  MB 2, visually identical to MB 1, labeled Insignificant  temporal resolution = frame rate  MB 3, visually MB 4, visually MB 5, visually identical to different from different from MB 2, MB 3, MB 4, labeled labeled labeled Insignificant Significant Significant  (2.7.b). Local temporal resolution on a macroblock level.  Figure 2.7. Local temporal resolution on a frame and a macroblock level.  26  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  The exact rules by which each of the frame macroblocks is found to be Significant or Insignificant will be laid down in a later section. Now, in Figure 2.8, we will just give an example of a Frame Partitioning mask for several frames of an arbitrary video sequence. As one can see from Figure 2.8.a, in this simple example a Frame Partitioning Mask corresponds to the visually identified background, while the Significant macroblocks correspond to a moving object. In other sequences, the Significant macroblocks may not corre. spond to one visually distinctive area in an image, as illustrated on Figure 2.8.b.  FrameO  Framel  Frame2  (a) Original consecutive frames FrameO, Framel, Frame2 and Frame3 and their corresponding Frame Partitioning Masks.  27  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION, BASED ON FRAME PARTITIONING  li Framel  FrameO „  Insignificant macroblocks  Significant macroblocks  Frame Partitioning Mask 1 (b) Original consecutive frames FrameO and Framel and a Frame Partitioning Mask for Framel.  Figure 2.8. Example of a Frame Partitioning Mask.  2.3.B. Determination of the Frame Partitioning Mask. Frame Partitioning divides a frame into Significant and Insignificant macroblocks, with all Insignificant macroblocks constituting a Frame Partitioning Mask. As was mentioned before, a macroblock is considered to be Insignificant if it has not "visibly" changed from its predicted macroblock.  28  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  We use the following assumptions to emulate the way the human visual system distinguishes visible changes from those that are not visually recognizable: • the visual sensitivity to changes is greater for smoother parts of the images and less for those parts with texture; • changes in the luminance component are more noticeable than changes in the color component; • the human eye is less sensitive to changes that occur in very dark or very light parts of the image and is most sensitive to changes occurring in parts of the image with average luminance; •  motion has the ability to mask changes: it is more difficult to recognize changes in texture, luminance or color if there is motion associated with those changes; the changes are more obvious in parts of the picture with no motion;  Following our suppositions of the human visual perception, we chose to base the decision as to whether or not a macroblock belongs to the frame partitioning mask the following factors: ADC - the dc error of the luminance component, Aa - the variance error, AY — the 2  absolute error of the luminance component and AC — the absolute error of the chrominance component of the image. Figure 2.9 illustrates the structure of a macroblock to help understand the definitions of errors. For all macroblocks in a frame, we perform the comparison between a predicted and a current macroblock either on a block level or on a macroblock level. We choose the level of comparison based on whether or not a macroblock contains a luminance or a chrominance edge.  29  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSATION  BASED ON FRAME  Luminance Y, Luminance Y, Chrominance, 8x8 block #0 8x8 block #1 8x8 block #1 -C?(0) - Cr -DC(1) -DC(O) - Y(l) * - Y(0)  Luminance Y, 8x8 block #0 - ^(O) -DC (0)  Luminance Y, Luminance Y, Chrominance, 8x8 block #2 8x8 block #3 8x8 block #2 - ^(3) - <?(2) - Cb -DC(3) -DC(2) - Y(2) - Y(3)  Luminance Y, 8x8 block #2 -o (2) -DC (2) - Yp(2)  P  Luminance Y, Chrominance 8x8 block #1 ' 8x8 block #i  -C?P(1)  - Cr  -DC (1) - Y (l)  p  P  - P(°) '.. Y  PARTITIONING  p  Luminance Y, Chrominance 8x8 block #3 8x8 block #2  2  p  P  Original macroblock  -Cb  -DC (3) -Y (3)  p  p  p  Predicted macroblock  Figure 2.9. Structure of a macroblock in MPEG-2 Video Main Profile.  For a non-edge macroblock, the comparison is performed oh a macroblock level, where the ADC,. ACT and AY errors are found as the average of their corresponding four block values A DC, A^o and A Y, and the AC error is found as the maximum oftheZi Crand bl  2  bl  w  A Cb, the chrominance R and the chrominance B errors. The edge macroblocks are combt  pared on a level of blocks, with ADC,  ACT  2  and AY identified as the maximum of the four  corresponding block errors and^C being the same as for,a non-edge macroblock.  The errors are defined as follows. The DC error A DC(n) of block n is found as the difference between the DC component bl  of the corresponding original and the predicted blocks in a macroblock; A DC(n) bl  = abs\DC(n)-DC (n)\  (2.2a)  p  ,n = 0, 1,2,3  30  .  '  ;  ,  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  For the non-edge macroblock the final DC error ADC is found as the average of the four block errors: ADC = AVG{ A DC(0), A DC(l), A DC (2), A DC(3)} bl  bl  bl  bl  (2.2b)  For the non-edge macroblock the final DC error ADC is found as the maximum of the four block errors: ADC = MAX{ A DC (0), A DC(l), A DC(2), A DC(3)} bl  bl  bl  bl  (2.2c)  The block absolute luminance error A Y is defined as the average pixel by pixel absobl  lute difference between the original and the predicted block before the DCT.  J  7=7  i=7  AY =  '  :  i  % abs[ Y(i,j, n)-Y (i,j, n)]  bl  o4 ,  (2.3a)  p  j _  = 0  •  0  .  .  .  ,n = 0, 1,2,3 Similar to the dc error, the final value of the absolute error AY for a non-edge macroblock equals the average of four absolute block errors, AY = MAXf A Y(0),A Y(l), A Y(2), A Y(3)J bl  bl  bl  bl  (2.3b)  and for an edge macroblocks to their maximum. AY = MAX{ A Y(0),A Y(l), A Y(2), A Y(3)J bl  bl  bl  bl  (2.3c)  For the non-edge macroblock, the variance error ACT is the calculated as the absolute 2  difference between the average variances of the predicted macroblock and the current original macroblock and is found as  ACT  2  = abs \  4 ^ H  n=0  r j > ) - 7E ^  '  A ^  n=0  31  cr(n) \  (2.4a)  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  For an edge macroblock, the variance error Ao is calculated as the maximum of four block 2  variances error A G :. bl  .  l  "  A cr (n) = abs\cT (n)-o (n)\ bl  2  2  ..  2  p  (2.4b)  n = 0, 1,2,3; Ao = MAX { A cx (0), A o (l),A e2(2), 2  bl  2  bf  2  bl  A a (3) } bl  2  (2.4a)  And, finally, the absolute error of the chrominance component, chrominance error AC is found for both edge and non-edge macroblocks as the maximum of the absolute differences of the chrominance components and in accordance to Figure 2.9 : AC = MAX { A Cr, A (Cb) } bl  (2.5a),  bl  ,where  A Cr=^J f bl  abs[Cr(i,j)-Cr (i,j,)]  J J  J  and  p  (2.5b)  <=7 J=7  A Cb=— Z X abs[ Cb(i,j)-Cb (i,j,) J bl  y  "4 ,  p  jo  = 0  (2.5b)  =  To accommodate for the change in sensitivity of the human visual system to macroblocks of different illumination, we introduce a function of DC component, p o s i t i o n _ D C , which we define as: position_DC = (DC b - DC m  , where we chose DC  avg  avg  f  as 128 and  1^ DC  mb  =  T  4  £  D  C  (  N  )>-  „=o  32  (2.6)  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  A macroblock is classified as insignificant if all of its errors (ADC, AO*, AY and AC*) do not exceed their corresponding threshold values. In our algorithm, all of the error thresholds except for the absolute error threshold are found in a following form:  thresholdJorml = K*exp[ KK*log{(? +l.l) ]*(0.25*position_DC+1.0) mh  (2.7)  and the absolute error threshold is found in the form of thresholdJorrnl = K*log(0 +l.l)*(O.25*position_DC+l.O)  (2.8)  2  mb  I ,where o ^ is the average variance of a macroblock o ^ 2  2  =  3  ^("j  ' ;  n=0, 1,2,3; K and KK are coefficients whose values depend on a specific error for which a threshold is used and on the extent we wish to increase the use of temporal redundancies in addition to standard motion compensation. To exploit the masking effect of motion, for the macroblocks whose motion vectors are not zero we increase all of the above thresholds by 10 percent. For the macroblocks that are encoded with no Motion Compensation, the thresholds are kept unchanged.  33  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.4. M O T I O N C O M P E N S A T I O N W I T H F R A M E PARTIONING In the previous section we described a novel Frame Partitioning technique, which allows an algorithm to distinguish between the visible and non-visible temporal changes that occur between consecutive frames. By using the results of Frame Partitioning, a compression algorithm can have an option not to encode the changes that have been identified as nonvisible and to use different temporal resolutions for different parts of a frame, thus increasing the encoding efficiency and improving bandwidth. We propose a way of combining our Frame Partitioning technique with a traditional MPEG Motion Compensation approach in a new perceptually adaptive Motion Compensation algorithm - Motion Compensation with Frame Partitioning. This new method significantly improves the use of temporal redundancies in encoding and results in a higher efficiency and quality of encoding without impacting the perceived content of the video stream. By not encoding the non-visible changes that occur between frames, our method introduces a degree of lossiness into the Motion Compensation procedure and, thus, allows for its perceptual adaptability. Our Motion Compensation algorithm works in two stages. Frame Partitioning constitutes the first stage of the algorithm, followed by the second stage that consists of the modified MPEG Motion Compensation The objective of first stage of our algorithm, Frame Partitioning, is to separate each frame into Significant and Insignificant regions. To find the regions, the algorithm looks for the similarities between the two original consecutive frames through the use of MPEG Motion  34  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Estimation. Unlike in the traditional approach to Motion Estimation (remember Figure 2.3), our algorithm does not use the Reconstructed Reference Frame to find the Predicted Frame on the stage of Frame Partitioning in Motion Estimation. Instead, the Original Reference Frame is used in calculating the motion vectors (in both full-pixel and half-pixel search) and the Predicted Frame. By using an Original Reference Frame and not its reconstructed version, we eliminate the influence of the quantization error on the process of distinguishing between the visible and the non-visible changes in the input video stream. The results of the full-pixel motion vectors search performed in Frame Partitioning are saved and later utilized in the second stage of our Motion Compensation algorithm, so as to keep the increase of computation to a minimum. After finding the Predicted Frame and comparing it with the current original image following a technique described in Section 2.3.B, all macroblocks of the current frame are separated into two categories: significant and insignificant, where all insignificant macroblock are not "visibly" different from their corresponding predicted macroblocks and all significant macroblock have acquired new content which makes them "visibly" different from their predicted counterpart. After Frame Partitioning, the algorithm proceeds with its second stage, the Motion Compensation that results in a Prediction Error Frame. The Prediction Error for Significant macroblocks is calculated using the standard MPEG Motion Compensation procedure, which finds the pixel by pixel difference between the current and the predicted macroblocks. The Prediction Error of Insignificant macroblocks is made equal to zero. Motion information (constituting mainly of motion vectors) is transmitted for both Significant and Insignificant macroblocks.  35.  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  To further explain how the Motion Compensation with Frame Partitioning works, we present, on Figure 2 . 1 0 , a block diagram of our method and, on Figure 2 . 1 1 , a side by side comparison between a compression algorithm that uses our Motion Compensation technique and a compression algorithm with traditional MPEG Motion Compensation. Figure 2 . 1 2 gives additional illustrations for the comparison of two Motion Compensation techniques.  36  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSA TION BASED ON FRAME PARTITIONING  On»mal >i ^Reference Frame/  ( in ii m OIIJIII.I1;  <2> , FtppeV  Full pixel search Half pixel search Find Motion Vectors for Frame Partitioning  <T  motion, vectors  *\ . Find Predicted Frame for Frame Partitioning ~|^-^  | Identify significant and insignificant macroblocks | Frame Partitioning  Find Motion Vectors for final Motion Compensation  Set the values of the Prediction Error Frame to zero  Find Predicted Frame  T  <CIjPredictcdFramr> -w Find Prediction Error Frame r*Traditional M P E G Motion Compensation  Figure 2.10. Motion Compensation with Frame Partitioning: block scheme of an algorithm.  •37  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSATION BASED ON FRAME  PARTITIONING  MPEG encoding with Frame Partitioning  Traditional MPEG encoding  I Frame Partitioning p*"^  ^sp£i^ nii\nij^ ;  CjnsigiiilK.tiit  Mljs^  Find Predicted Frame Find Prediction Error  Zeroing Prediction Error  Prediction error Frame  Prediction lirror Frame  •..j-j*' ( ii  11 .HI-till 111  .Hid  I'.i I ran-Turin ,inJ Qiiatili/atinn  (JllLtllll/alliill  Figure 2.11. Comparison between encoding algorithms that use Motion Compensation with Frame Partitioning and traditional MPEG Motion Compensation. .  38  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.5. MOTION COMPENSATION WITH FRAME PARTITIONING: RESULTS AND APPLICATIONS Motion Compensation with Frame Partitioning significantly improves the use of temporal redundancies in video encoding and introducing perceptual adaptability into this, traditionally lossless, stage of video compression. By not encoding the visually undetectable temporal changes, our technique can achieve a significant improvement in either the quality of the reconstructed images or in the bit-rate reduction. For the same encoding quality, Frame Partitioning improves the encoding efficiency by up to 50 percent, with the average improvement at 20 to 30 percent. In this Chapter, we test the performance of our Motion Compensation algorithm and compare our technique with the traditional MPEG encoding. Using both algorithms, we encode a video stream in order to determine the following: 1) the bit-rate improvement that our algorithm can provide over the traditional encoding for the same level of quality and 2) the improvement in quality of the reconstructed images for the same bandwidth. For our inquiry, we use the standard MPEG-2 verification sequences: "Flower Garden", "Tennis", "Bicycle" and "Bus". All sequences are composed of full size images (720x480 pixels, 1350 macroblocks per frame). The test results are obtained by encoding five groups of pictures (10 frames each) for all four video streams. During the first set of evaluations, we encode 50 frames (five GOPs) of different sequences with the two algorithms at the same (flat) quantization level, so that the reconstructed images in both cases will have similar quality, and compare the resulting bit rates. For the second assessment, we encode the same 50 frames with the two algorithms at the same bit-rate and compare the visual quality of the reconstructed images.  .39  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  i) Bit-rate improvement Our Motion Compensation with Frame Partitioning algorithm achieved an improvement in bit rate by ndt encoding the spatial information about the temporal changes that have been identified as "non-visible". In most cases, the size of the Frame Partitioning Mask, which corresponds to a number of macroblocks that are found insignificant (i.e., carrying information about the non-visible changes), will be indicative to the degree of the bit-rate improvement provided by our algorithm. In the Table 2.1 we show the number of macroblocks that were found to be insignificant in encoding in one GOP if all four sequences.  Size of Frame Partitioning Mask, percent Sequences Frame  Tennis  1,1 2,P 3,P 4,P 5,P 6,P 7,P 8,P 9,P 10, P  0 78 % 68 % 69% 62 % 51% 66% 62 % 65% 71 %  Garden 0 63 % 58 % 60% 58 % .. 55 % 59 % 59% 57 % 56%  Bus  Bicycle  0 45 % 46% 45% 46% 47% 47% 42% 46% 45 %  0 45 % 42% 44% 45% 47% 45 % 46% 43% 46%  .  v  Table 2.1. Size of the Frame Partitioning Mask in per cent, out of the total of 1350 macroblocks.  To illustrate the bit-rate improvement of the Motion Compensation with Frame Partitioning over that of the traditional MPEG Motion Compensation, we compare the bit-rate performance of both algorithms for the same quality of the reconstructed images. To ensure that the image quality of both encoded video sequences is indeed the same, we use encoding  40  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSATION BASED: ON FRAME  PARTITIONING  with a fixed quantization factor (mquant) and keep the quantization factor unchanged for all macroblocks. We compare the performance of our algorithm with the standard MPEG-2 solution (again, the value of mquant is kept the same in both algorithms). Due to the same level of quantization for all macroblocks, the perceived quality corresponding to the level of quantization artifacts in both streams would be the same. Table 2.2 shows side by side stream bit-rates yielded by both algorithms in several tests and with different levels of quantization.  Tennis Tennis Tennis  6.4 5.1 3.1  Bil-rale, Mbit/s Encoding v\ilh Motion Compensation with Frame Parti lionine 3.3 2.6 1.8  Garden Garden Garden Garden  6.1 4.7 3.9 2.9  4.7 3.7 3.1 2.4  23 % 22 % 21 % 17%  Bicycle Bicycle Bicycle  6.6 4.4 3.5  5.3 3.7 3.1  20% 16 % 12%  Bus Bus Bus  8.9 5.9 445  7.3 5.1 3.9  18 % 14% 12 %  Traditional MPEG encoding Sequences  '  Degree of improvement. per cent 53 /c 49% 44 % c  Table 2.2. Bit-rate improvement achieved by Motion Compensation with Frame Partitioning.  The results of the tests show that Motion Compensation with Frame Partitioning on average improved the encoding efficiency by 50 per cent for the "Tennis" sequence, by 21 per cent  41  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  for the "garden" sequence, followed by a 16 and 14 percent improvement for sequences "Bicycle" and "Bus" accordingly.  ;  ii) Quality improvement To illustrate the benefits of using Motion Compensation with Frame Partitioning in a compression algorithm, we compare the visual quality of reconstructed sequences that have , been encoded at the same bit-rate using both our technique and standard MPEG. In order to separate the encoding influence of a Frame Partitioning method and a bit allocation and rate control algorithms, we perform encoding with flat quantization, i.e. the same quantization factor (mquant) is used for all macroblocks in all frames of the same sequence. Different values of the flat quantization factor are used in a traditional MPEG algorithm and in oiir technique, so as to match the resulting bit-rates. Fifty frames, making up five GOPs, are encoded by both methods. Samples of the visual quality of the standard and Frame Partitioning streams are shown in Figures 2.12 and 2.13.  42  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Figure 2.12 illustrates the improvement in quality for the sequence "Tennis", encoded at the bit-rate of 2.5 Mbit/s. In this example, the use of Frame Partitioning significantly reduced the level of the artifacts, especially in the background, by allowing a 37 percent de- • crease in the level of quantization (value of mquant) while keeping the bit-rate unchanged.  Figure 2.12. Quality improvement achieved with Comparison between the standard MPEG encoding and MPEG encoding with Frame Partitioning, example "Tennis", bit-rate 2.5 Mbit/s.  (a) Original Frame.  .43  44  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  (g) Original Frame.  II  (h) Reconstructed Frame, traditional MPEG encoding.  (i) Reconstructed Frame, MPEG encoding with Frame Partitioning.  45  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Figure 2.13 provides another example of significant quality improvement that can be achieved using our adaptive Motion Compensation algorithm (sequence "Ballet", encoded at the bit-rate of 2.0 Mbit/, 35 percent drop in mquant). In the example, the use of Frame Partitioning almost eliminates the artifacts of blockiness in the background areas and improves the sharpness of the details (35 percent decrease in the mquant for the same bit-rate).  Figure 2.13. Comparison between the standard MPEG encoding and encoding with Frame Partitioning, example "Ballet", bit-rate 2.0 Mbit/s.  (a) Original Frame.  46  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  (c) Original Frame.  (e) Reconstructed Frame, traditional MPEG encoding.  (f) Reconstructed Frame, MPEG encoding with Frame Partitioning  48  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  For more complex images, as the one in Figure 2.14, the decrease in the quantization factor is lower ( 22 percent for the sequence "Garden"). Even thought the level of quantization does not change as dramatically as in the previous example, it still results in a visibly better image quality. In Figure 2.13, the improvement in quality is evident in the flat areas of the background (blue sky) and in the area that changes from one frame to the next (tree).  49  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  (d) Original Frame.  (e) Reconstructed Frame, traditional MPEG encoding.  (f) Reconstructed Frame, encoding with Frame Partitioning.  mm  i^i V  •>•I-  r  r i ^ t ; - : - . IlliSili  (g) Original Frame.  (j) Original Frame.  ft"  ^H^^^^^^^^^^^^l  (h) Reconstructed Frame, traditional MPEG encoding.  (k) Reconstructed Frame, traditional MPEG encoding,  51  (i) Reconstructed Frame, encoding with Frame Partitioning.  (1) Reconstructed Frame, encoding with Frame Partitioning.  Chapter 2  PERCEPTUALLY  ADAPTIVE MOTION COMPENSATION BASED ON FRAME  PARTITIONING  As we have illustrated in the above examples, the MPEG encoding with Frame Partitioning provides better encoding results that the traditional MPEG encoding even with very complex images. The outmost degree of improvement will be achieved with images that have large areas of background, either moving or still, that contains some changes that are not perceived by the human vision. The benefits of Frame Partitioning will be somewhat less apparent for the sequences that contain large areas with very significant changes, as well as for the sequences that do not have a lot of details (have a lot of flat areas). In the first case, due to a significant amount of changes between the frames, the size of the Frame Partitioning Mask (number of Insignificant Macroblocks) will tend be minor and most of the image information will be encoded following the traditional MPEG approach. For the sequences that contain large flat areas, even though the size of the Frame Partitioning Mask in these frames will often be large, the decrease of the quantization level will not result in a significant bit-rate reduction due to the prevalence of the low detail areas. Another type of images whose encoding is more difficult to improve with Frame Partitioning is that of computer generated graphics. A distinct feature of these images is a frequent and precise repetition of parts of the consecutive frames. Such duplications will produce only areas that are exactly the same from one frame to the next and not the changes that were described in Section 2.2.B as the "non-visible" changes (changes that.do exist on a pixel level, but are not visually apparent). Despite the fact that Frame Partitioning will result in a very large number of Insignificant Macroblocks, the actual degree of lossiness that can be introduced into Motion Compensation is very limited and, thus, the improvement in the encoding efficiency will be lower than average.  52  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  . Our tests have shown, that for most of the video sequences that can be encountered as a regular TV and movie material, Frame Partitioning provides a significant improvement in the encoding efficiency, which in turn can be translated into quality improvement if the bandwidth is not constraint.  .53  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  2.6. SUMMARY In this Chapter, we developed a new method of Frame Partitioning, that significantly improves the use of temporal redundancies in video encoding. Frame Partitioning allows the encoder to distinguish between visible and non-visible temporal changes that occur in the video stream. Separating each frame into Significant and Insignificant areas, Frame Partitioning carries an analogy to MPEG-4 video objects. However, being designed to work within MPEG-2 syntax, our technique is block based, and thus, unlike the MPEG-4 video encoding , it does not require any additional overhead. One way to employ the idea of Frame Partitioning is to use it as an extra stage prior to the traditional M P E G Motion Compensation. This results in a new two level Motion Compensation with Frame Partitioning algorithm. This novel approach changes the traditional M P E G Motion Compensation from being a lossless stage in the compression to a stage that can be either lossless or lossy, depending on the complexity of the video stream and the in-, put encoding parameters. '•Our new Motion Compensation strategy gives an encoding algorithm the ability to use . different temporal resolutions for different parts of a single frame, which can be highly beneficial for encoding. The possibility of reducing the temporal resolution for some part of the frame can prevent visible artifacts from appearing elsewhere and hence can result in big improvement in picture quality. . Motion Compensation with Frame Partitioning represent just one application of our Frame Partitioning technique. In video encoding, Frame Partitioning can also be used as a preprocessing step prior to the traditional M P E G encoding.. Used in this fashion, Frame  Chapter 2  PERCEPTUALLY ADAPTIVE MOTION COMPENSATION BASED ON FRAME PARTITIONING  Partitioning will provide the same benefits as in Motion Compensation with Frame Partitioning, but to a lesser extent, since Frame Partitioning can not impact directly the content of the compressed bit-stream. Finally, another application of our technique can be found in the area of noise reduction for TV and video broadcasting, where Frame Partitioning can help diminish the appearance of artifacts caused by noise.  55  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  Chapter 3 BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.1 INTRODUCTION In the MPEG compression algorithm, motion compensation is followed by the Direct Cosine Transform (DCT) transform and then, quantization. Quantization of the DCT coefficients reduces the spatial redundancies in the image and allows for most of the compression. During encoding, the specific parameters of quantization are determined by a bit allocation (BA) and rate control (RC) algorithm. The BA and RC algorithm redistributes the bits between the frames and modifies the quantization levels for different parts of a single frame, and allows the matching of the output to a given bit-rate. In addition to complying with the required bit-rate, the objectives of bit allocation and rate control algorithm are 1) to maximize the visual quality of the encoded video stream and 2) to provide consistent visual quality for different frames in a video sequence and for different areas in a single image. The problem of quality consistency is typically separated into two levels: the frame level and the macroblock level. In most existing BA and RC algorithms, the problem of consistent quality encoding on a frame level is exclusively handled by bit allocation, while the rate control is solely responsible for providing uniform quality encoding on a macroblock leveh In the previous chapter, we described a new Frame Partitioning technique, which, through a unique approach to exploiting temporal redundancies, significantly improves the encoding  56  \  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  efficiency and quality. When Frame Partitioning is employed, the algorithm substitutes specific macroblocks in a current frame with their corresponding predicted macroblocks from the previous frame. In view of such possible interchangeability between parts of the neighboring frames, it is especially important to guarantee the consistency of quality throughout the frames in a sequence and also throughout all macroblocks within a single frame. In this chapter, we develop a novel bit allocation and rate control algorithm that is designed specifically with the emphasis on providing encoding with perceptually uniform quality. Our technique uses a unified approach to the problem of quality consistency, without separating its solutions into frames and macroblocks level;. Similar to all B A algorithms, our method yields best results when implemented as a two-pass algorithm, but is also successful when implemented for real time encoding (i.e., as a one-pass method). The organization of this chapter is as follows. First, in Section 3.2 we examine some of the proposed solutions for the problem of bit allocation and rate control in MPEG. Then, in Section 3.3 we introduce our bit allocation and rate control algorithm. In Section 3.4 we describe how our algorithm can be applied to different encoding scenarios and, in Section 3.5, we present test results that illustrate the efficiency of our method.  57  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.2. E X I S I N G B I T A L L O C A T I O N A N D R A T E T E C H N I Q U E S The existing MPEG algorithms handle the two main problems of encoding, i.e. complying with the bit-rate requirements and providing quality consistency, in two steps. First, the algorithm determines a desired number of bits for each frame within a GOP (or a larger group of frames), a Frame Target. This process constitutes bit allocation. Then, this target is realized by adjusting a quantization factor (mquant) of each macroblock (MB) or a group of macroblocks in the image. This process is known as rate control. To solve the problem of quality consistency, first, by virtue of determining the target number of bits for each picture, the bit allocation algorithm is responsible for providing consistent quality encoding for the frames within a video stream. Then, by assigning appropriate quantization parameters to macroblocks, the rate control algorithm attempts to achieve uniform quality for all macroblocks within a frame. The MPEG compression standard allows much flexibility in the implementation of both bit allocation and rate control. A great deal of research has been done and is still going on in these areas. In this section we will look at some of the existing BA and RC algorithms that were proposed in recent years.  3.2.A. B i t allocation The most popular approach to bit allocation is based on tracking some global complexity measures for the images in a video sequence and, then, using these measures to redistribute  58  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA •  f  '  bits between the frames by assigning Frame Targets. The exact definitions of image complexity and its role in bit allocation differ from one algorithm to the other. The well known Test Model 5 (TM5) algorithm [4] defines image complexity as the product of the average frame quantization factor and its resulting number of bits. The TM5 method uses the complexity measures of previously encoded I, P or B frames to allocate bits for the current frames of corresponding types. Another algorithm, found in [5], determines the image complexity as the sum of spatial mean absolute differences or the mean absolute prediction differences of Intra and Inter macroblocks respectively. A method described in [3] and [6], arranges video sequences into complexity classes and then uses the precalculated experimental I, P and B frame bit-counts as the Frame Targets for all frames of the corresponding type. A considerable problem with the above bit allocation techniques is that they use statistics of the previously encoded groups of pictures (GOPs) or of previously defined complexity classes to make assumptions about current frames. The drawback of using past-based statistics, given the non-stationary characteristics of video sequence, is unavoidable in real time encoding or in encoding with no delay, where the algorithm is not able to assess the properties of the frames in a current GOP before making its bit allocation decisions. In order to increase the precision of bit allocation, it is necessary to rely on the characteristics of the presently encoded frames instead of those encoded in the past. An example of the latter approach can be found in [2]. Using a two-pass encoding approach, this method modifies the TM5 algorithm so that it works with complexity measures based on current pictures instead of previous ones. By allowing re-encoding, the algorithm in [2] is able to significantly increase the accuracy of the bit allocation process.  59  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.2.B. Rate control In the traditional encoding setup, bit allocation is followed by rate control, the procedure that realizes the Frame Targets by determining the values of the quantization factors for frame macroblocks (mquant). In the MPEG syntax, the mquant must be specified at the beginning of each slice (at the beginning of each row of pixels) in a frame and can also be specified for each 16x16 macroblock (MB). The quantization of the DCT coefficients Cy is performed separately on each of the 8x8 blocks that comprise a macroblock. The process of quantization can be separated into two stages, described below by equation (3.1) and (3.2). First, the DCT coefficients in an 8x8 block are quantized with their corresponding values of the predetermined quantization tables (3.1). QiJ]  CpartQfij] = - p ^ Q[i,j]  i,j = l,...,8  (3.1)  , where C[i,j] is one of the 64 original DCT coefficients; CpartQ[i,j] is one of 64 partially quantized DCT coefficients; Q[i,j] is one of the 64 preset values in a quantization table. The MPEG uses two different quantization tables: one for quantizing Intra macroblocks and another for Inter macroblocks. The encoder can either use the standardized quantization matrices, or custom defined ones (but not both). In the standard quantization matrices, for Intra macroblocks, the higher DCT frequency coefficients are assigned a larger Q[i,j]'s than the lower frequency coefficients, whereas for the Inter macroblocks, the value of Qij is kept the same for all of the DCT coefficients.  60  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  The second stage of quantization determines the final values of the DCT coefficients using a single quantization factor, mquant, for all blocks in a macroblock (3.2). CQfiJ] = integer  Cpart[i, j] mquant  k 2  i,j = l,...,8  (3.2)  , where CQ[i,j] is the final value of the quantized DCT coefficients; mquant is a macroblock quantization factor; k can be either 1 or 0, depending on the rounding rules. The rate control algorithm manipulates the values of mquant in order to carry out the required target number of bits for each frame. A good rate control algorithm must 1) meet exactly the allocated bit rate and 2) maintain a consistent picture quality throughout the frame, i.e. assign the quantization factors for the frame macroblocks in such a way, that all macroblocks are encoded at the same level of quality. In order to achieve the above goals, the RC algorithm has to be able to model two very important relationships: 1) between rate and quantization, i.e. for all values of mquant, to assess how many bits are expected after a macroblock is quantized; 2) between quality and quantization or quality and bit-rate, i.e. for either every level of quantization or every bit-rate, to assess the visual quality of the reconstructed image (or a macroblock). From so on, we will refer to the process of estimating the relationship between rate and quantization as the Rate-Quantization or RQ modeling, between quality and quantization as Quality-Quantization or QQ modeling and between rate and distortion (i.e., quality) as Rate-Distortion or RD modeling.  61  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Possible techniques to model one or both of these relationships, have been proposed in many studies and documents.  f  i) Rate-Distortions and Rate-Quantization modeling in the Test Model 5 algorithm The RD modeling approach of the Test Model 5 (TM5) algorithm [4] is based on the uniform bit distribution over all macroblocks within a frame. The uniformity of the bit distribution results in a coarser quantization in macroblocks with more activity and a finer quantization for macroblocks that have a flat texture. By allowing more artifacts in the highly textured areas, the TM5 approach to Rate-Distortion exploits a so called spatial masking effect, i.e., the ability of the human eye to be more tolerant to artifacts that appear in areas with a lot of activity. In the TM5 uniform bit distribution, all macroblocks should have the same number of bits. To maintain a bit distribution close to the uniform one, the algorithm modifies the quantization parameter (mquant) of each macroblock with accordance to the current value of bit-count (3.3). FrameTARGET \ ( mquant(k) ~ BitCount(k -1) - (k -1) * yv V  (3.3)  ,where N is the number of macroblocks in a frame; k is the number of the current macroblock that is being encoded; FrameTARGET is the target number of bits for the current frame; BitCount (k-1) is the sum of the bits that resulted from encoding all previous (k-1) macroblocks in the current frame.  62  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Because the algorithm uses results of the previous MB bit count to make decisions about the level of quantization for the current macroblock, there is a one MB delay in adjusting the quantization parameters. This delay can result in an uneven quality from one macroblock to the next, especially of the neighboring MBs are different in texture. As an example, let us look at a frame composed of only 4 macroblocks, with the macroblock activity alternating from high to low ( see Figure 3.1).  2 (a) Frame macroblocks  3  4  (b) Uniform (1) and real (2) bit distributions  Figure 3.1. Rate control strategy of the Test Model 5 algorithm.  Suppose that the quantization factor mquant 1 chosen for MB1 was such that its resulting number of bits exceeded the value expected by the uniform distribution. As a result, following (3.3), the algorithm increases the value of the quantization factor chosen for MB2, mquant2. Because of the resulting coarse quantization for MB2 and its flat texture, the perceived quality of the second MB will be much lower than the quality of the first MB. The decision on mquant3 will be again made based on the bit-count of the two previous macroblocks. Because MB2 has a low level of activity, its bit count may be lower than expected. Consequently, mquant3 will be lower and the bit-count of high activity MB3 may  63  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  exceed its expected value, in turn resulting in coarser quantization for MB4. As we have seen from the example of Figure 3.1, the delayed adjustment of the quantization parameters in TM5 can cause high fluctuations in the image quality. The Rate-Quantization model of the TM5 algorithm represents an example of a class of feed-forward rate modeling methods [4], [7]. These techniques rely on the past statistics to predict the relationship between bit-count and quantization in current encoding. While some of these methods allows the encoder to keep the complexity of computation under control, it can also result in a significant error in bit estimation. The problem of the erroneous Rate-Quantization modeling of feed-forward techniques can be solved by re-encoding. The advantage of re-encoding techniques is that, instead of the past statistics, the measurements from the current picture are used to guide the quantization of the macroblocks in this frame, thus providing a greater precision in realizing the exact target number of bits for a frame [2]. The disadvantage of re-encoding techniques is that they can not be used in real-time encoding and are usually computationally complex. Instead of re-encoding, another way to increase the precision of the traditional feedforward techniques is to do pre-encoding or trial encoding. With these techniques, the results of the trial encoding are processed and stored in look up tables, providing references to the relationship between quantization and bit-count for either frames or macroblocks.  ii) Rate-Quantization modeling on a macroblock level with pre-encoding An example of the Rate-Quantization modeling with pre-encoding can be found in [3], which is also a good study of the RQ modeling on a macroblock level. A technique presented in this publication uses the assumption that the variance of a macroblock can be used  64  . Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  as a parameter in determining the relationship between the macroblock quantization factor mquant and its resulting number of bits. In this technique, the range of all possible image variances (from 0 to 100 000) is partitioned into 16 intervals, producing 16 so called Variance-Model Activity Classes (VMAC) or, just, variance classes (VC).  A set of predetermined thresholds defines the boundaries of  the intervals, with the boundaries stretching further apart as the variance increases. The algorithm uses a training set of images to estimate the Rate-Quantization relationship. To produce the RQ estimation, first, all of the training images are analyzed to classify their macroblocks into particular variance classes. Then, the macroblocks of each class are repeatedly encoded using different values of mquant and the resulting numbers of bits for each encoding are tabulated. The mquant values are chosen so as to cover the allowed quantization range, from 1 to 31 in MPEG-2. At the end, the bit-counts for all macroblocks in each VMAC class are averaged over the corresponding numbers of macroblocks to produce the final bit-count/mquant table for each VMAC class in a form of a look up table. During encoding, each of the macroblocks in a frame is first identified as belonging to one of the VMAC classes. Then; the number of bits needed to encode the macroblock is estimated from the look up table of the corresponding variance class, obtained during the training phase. While the method in [3] can provide good results in the estimation of Rate-Quantization for some macroblocks, for others the actual number of bits generated can be significantly different from its predicted value. The reason of such mismatch lies, again, in the statistical approach toward the estimation . Because the algorithm uses training sequences, as in any technique that uses pre-encoding, the results of the bit-count tabulation heavily depend on  65  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  the actual images chosen for the preliminary encoding. If the textures of the training images are far off from the textures that are present in the video stream, there is an increased chance of low precision of the Rate-Quantization estimation. Even if all possible textures are used in the training sequence, the averaging of their statistical results prevents the algorithm from guaranteeing a high precision estimation for any of these textures separately. In addition to the texture problem, the preset boundaries of the variance classes will cause a predetermined uneven precision of estimation between the classes. Because of the difference in the width of the VMAC intervals, macroblocks with smaller variances will have a more accurately estimated Rate-Quantization relationship than the macroblocks with greater variances.  3.2.C.  D r a w b a c k s of the traditional approach  As was pointed out in the beginning of this chapter, the objective of bit allocation and rate control is to encode the video stream at the best as well as consistent level of quality, while satisfying the constraints of the required bit-rate. Consistent quality encoding is equally important to maintain amongst the frames within a stream as well as amongst the macroblocks within a single frame. Existing MPEG compression algorithms view the question of bit allocation and rate control as two independent problems. First, based on some global complexity measures ,the bit allocation process assigns the target number of bits for each frame, the Frame Target. Then, the rate control process attempts to distribute this predetermined bit number amongst the macroblocks that compose the image. In this traditional top down approach, the issue of frame to frame quality control is separated from the issue of quality control within a single frame. The issue of maintaining consistent visual quality from one frame to the other is  66  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  regulated solely by bit allocation. The larger the number of bits that is targeted for a particular frame, the greater the overall quality expected for that image. The question of quality fluctuations amongst the macroblocks in a frame is handled exclusively by rate control algorithm, which uses adaptive quantization techniques to achieve a uniform level of quality for all parts of a single picture. Most of the research on the quality management in MPEG encoding has been focused on rate control techniques, specifically, on making them adaptive to the properties of the human vision. The problem of bit allocation, however, received much less attention and, so far, has not been rendered perceptually adaptive. By simply relying on the global complexity measures, traditional bit allocation algorithms fail to recognize that the highly textured frames should not always require a significantly larger bit budget than the images with smooth textures. Not taking into account the increased sensitivity of the human eye to the artifacts in the smooth images, the existing BA algorithms will allocate a much greater number of bits to the highly textured frames than to the less textured images. In this situation, even if the rate control were to always achieve a consistent quality encoding for all macroblocks within a single frame, similar macroblocks in the textured and the smooth frames will be encoded at different quality levels. Because the existing BA and RC algorithms have different approaches to the problem of consistent quality encoding on the frame and on the macroblock levels, the macroblocks in neighboring images will not be guaranteed to maintain consistent quality, thus making the video stream perceptually unacceptable. In an attempt to resolve this problem, we propose an integrated approach to bit allocation and rate control. In our method, the quality management at both the frame to frame and the macroblock to macroblock levels is governed by the same principles of perceptual adapt-  67  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  ability. Our algorithm does not base the bit allocation decisions on any global measures, but instead, on estimating of the future quality at the macroblock. Our approach to bit allocation and rate control can be described as "rate control first, bit allocation second" or a bottom up approach. Our algorithm is able to significantly increase the consistency in quality from one frame to the other in both one and two pass encoding without introducing an unreasonable increase in the complexity of computation.  68  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  3.3. A N I N T E G R A T E D A P P R O A C H T O B I T A L L O C A T I O N AND R A T E CONTROL BASED ON PERCEPTIAL QUALITY CRITERIA We propose a perceptually adaptive bit allocation and rate control scheme that attempts to encode all macroblocks in all neighboring frames at the same quality level. In the favorable case of the variable bit rate (VBR) encoding, the objective of our algorithm is to achieve a consistent quality encoding for all macroblocks in all frames in a sequence. A more challenging situation for the encoder is presented for the case of the constant bit rate (CBR). In contrast to the VBR encoding, where the bit allocation is determined only by the video information itself, the CBR imposes limitations on how much the bit rate can fluctuate from one frame to the other. It is evident, that for the bit rate to be truly constant, each frame in a video sequence has to be encoded by the same number of bits. However, without the possibility of adjusting the frame bit rate, the video compression algorithm will lose its ability to use temporal redundancies. In order to match the requirements of the constant bit rate channel with the necessary variations in the instantaneous bit rate, the encoder uses a Virtual bit-rate Verification (VBV) buffer. The VBV buffer serves as a connecting link between the output of the encoder and the channel and allows a degree of fluctuation in the local bit-rate, where the range of the allowed fluctuations depends on the size of the buffer. The bigger the buffer, the more is the freedom to adjust the local bit rate. Figure 3.2 illustrates the relationship between the instantaneous frame bit rate and the size of the VBV buffer. In Figure 3.2, the resulting bit count for each frame must lie within the indicated  69  Chapter 3  BIT ALLOCATION  AND RATE CONTROLBASED  ON MACROBLOCK  QUALITY  CRITERIA  boundaries. The upper boundary correspond to a VBV buffer underflow - a state where, for a particular frame the encoder produces too many bits and, thus, the picture will not completely stored in the receiving buffer at the intended decoding time. The lower boundary is related to the opposite effect - VBV buffer overflow - too few bits for a frame.  Buffer underflow ^  BitCount, bits  '  9 *BitRate  1  FrameRate <Size of the V B V buffer  boundary  r  /  ^  /  Acceptable bit • distribution  f  ^<r-  Buffer overflow ^  boundary  <  BitRate FrameRate  i  0 I  1 P  !  2 P  •  3 P  4 P  5 P  6 '. P  1 P  Frame number Frame type  8. P  Figure 3.2. Buffer strategies in constant bit rate encoding.  In the strict sense, the sequence i=0, 1 ... j ... is said to violate the VBV constraint at index j if either of the following conditions holds [8]: ^TARGET[i]>SIZE_VBV  + BITS(j)  -> VBV underflow (3.4)  i=0  ^TARGET[i]<BITS(j + l)  -> VBV overflow (3.5)  ;=o  ,where TARGET(i) is the number of bits allocated to the i-th frame; BITS(j) is the number of bits that is transmitted via the CBR channel for the  70,  Chapter 3 . BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  duration of j frames, i.e., during the interval of j* — seconds for a frame rate of 30 frames per second (note, that BLTS(O) is assumed to be equal 0); SIZE_VBV is the size of the VBV buffer in bits. In both underflow and overflow situations, it is said that the sequence violates the VBV constraints and the visual results can be detrimental, such as a total loss of some parts of an image. '  •  .  •  \  , In the case of the constant bit rate, the objective of our algorithm is to achieve the most consistent quality for all macroblocks in all frames within a GOP, while satisfying the constraints of the VBV buffer. The best encoding results for the given bit-rate and buffer size can be accomplished with a two pass approach. In this case, the algorithm will have the chance to examine the image content of all images that constitute a group of pictures (GOP), before making the final bit allocation decisions.  In what follows, we will first discuss the basis of our bit allocation and rate control algorithm with no reference to either VBR or CBR encoding. Then, in Section 3.4, we will show how our method can be used in a VBR scenario and in a CBR two pass algorithm and, how our method can be adapted td real time encoding and stillfulfill its objective.  71  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.3.A. Bit allocation based on macroblock quality We propose a bit allocation algorithm that determines the Frame Targets based on the maximum (and consistent) quality level that can be achieved for all frames within a sequence (or a GOP) for the required bit-rate. We designed our BA method to work in a bottom up approach, i.e., from a macroblock level up to a frame level. Specifically, our bit allocation technique 1) takes the BA to a macroblock level, by determining the number of bits required to encode a macroblock at different quality levels; 2) determines, for each quality level, the Frame Target as a sum of the macroblock bit targets (where all the macroblock have the same quality level); 3) finds a quality level, that can be accommodated within the constraints of the required bit-rate and outputs the Frame Targets that have been determined for the chosen level of quality. Figure 3.3. illustrates the stated above steps of our bit allocation algorithm. In order implement such an algorithm, we must be able to: 1) measure a perceptual quality of a reconstructed image and a reconstructed macroblock; 2) design an efficient techniques for Rate-Quantization and Quality-Quantization modeling.  72  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  The resulting bitrate is greater or smaller than the required bitrate  Rate-Quantization and Quality-Quantization models  Output the Frame Targets  Figure 3.3. Bit allocation based on a macroblock quality: block scheme of an algorithm.  Measuring perceptual image quality Several assumptions on how the human visual system detects quality are well acknowledged. These assumptions relate to such properties of the human vision as texture or spatial masking, contrast masking and motion masking. These masking effects correspond to the  73  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  increased tolerance of the human visual system to distortion in areas in highly textured areas, extremely bright or dark areas, and in areas with a lot of motion. Significant amount of research has been done in the area of quantitative assessment of visual quality and these techniques can be very complex. Most of the algorithms measure different kinds of artifacts, such as blockiness, blurred edges, fuzziness, and then use their weighted sum as the final quality grade. To limit the computational complexity of our implementation, we will use the signal to noise ratio (SNR) measure to assess the visual quality. However, if complexity of computation is not a concern, our algorithm can be easily modified to accommodate a more advanced measures of visual perceived quality. We define the signal to noise ratio of a macroblock S N R m b as: SNR = 10 * loglO ( mb  where, V A R  mb  VAR /MSE ) mb  mb  (3.6)  is the variance of the intensity of a macroblock in the original frame;  MSE b is a measure of a macroblock mean squared error and is the function of a m  pixel by pixel difference'between the reconstructed and the original macroblocks. MSE  mb  is calculated as the average of the mean square errors of the four 8 x 8 lumi-  nance blocks that make up a macroblock: (3.7) ,where each of the mean square errors of 8x8 blocks M S E b i is determined as 1  (3.8)  74  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  ,with org[i.j] and recjT.j] corresponding to the pixels of the original and the reconstructed blocks. Now we can restate the general objective of our bit allocation algorithm in a more concrete term: to achieve the same SNRmh for all reconstructed macroblocks throughout the video sequence.  Modeling Rate-Quantization and Quality-Quantization relationships In order to fulfill the objective of our algorithm, we must be able to estimate the following: 1) QQ model: the relationship between the reconstructed quality of a macroblock, SNR , and the quantization factor mquant that has to be used to quantize a macmb  roblock so as to result in quality SNR b; m  2) RQ model: the relationship between the quantization factor mquant and the number of bits, BITS , that is produced when a macroblock is quantized with mquant and then mb  coded. Figure 3.4 illustrates how the RQ and QQ models are used in our bit allocation algorithm to determine the number of bits required to encode a macroblock at a specific quality level. Our technique for RQ and QQ modeling is described in detail in the following section.  75  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  set the target quality for aa macroblock,  Using the QQ model, choose the mquant that results in the target quality, SNR i>  Using the RQ model, - resulting find the bit count BITS b number that is produced when a mac- of bits for a roblock is encoded with the macroblock target mquant m  m  Target mquant  BITS b m  SNR  mb  BITSmb  mquant  •  mquant Figure 3.4. Rate-Quantization and Quality-Quantization modeling in a bit allocation algorithm based on macroblock quality criteria.  3.3.B. Rate control The key to a successful quality performance of our algorithm lies in its ability to model the Quality-Quantization and the Rate-Quantization relationships. Previously, in Figure 3.4, we showed how the results of this estimation-will allow us to achieve a quality based bit allocation. Now, we will look closely at how we propose to do the RQ and QQ modeling.  i) Rate-Quantization modeling We designed a technique that models the" Rate-Quantization relationship based on the evaluation of the local macroblock image textures, which we identify based on macroblock variances and group into variance classes. The goal of our RQ modeling technique is to distinguish different macroblock variance classes in an image and to determine the relationship  76  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  between the macroblock quantization factor mquant and the resulting number of bits for macroblocks of each of these classes. Our algorithm is based on the idea, initially proposed in [3], to use variance as the parameter for the RQ modeling and adds several significant improvements to the algorithm in [3] (described in Section 3.2.C.ii) so as to improve the precision of the estimation. In contrast to [3], our method does not use the global statistics as the reference for encoding macroblocks in the current frame. Instead, each frame or some of the frames, depending on the allowed complexity level, undergoes preliminary encoding to produce the local macroblock estimate of the Rate-Quantization model. The localization of the training phase to the current image, allows us to base the RQ modeling on the textures that are present in the current image. If no complexity restrictions exist, our technique employs a twopass encoding scheme for each frame in a video sequence. During the preliminary estimation pass, the goal of our algorithm is to produce the Rate-Quantization model, and, in the second pass, to use the estimate of the RQ relationship for the bit allocation and rate control, as was illustrated in Figure 3.4. The steps in our Rate-Quantization estimation algorithm are as follows.  Step 1. Determination of the Variance Classes. During quantization, the MPEG compression algorithm does not works with the original images, but with the values of the prediction error frame. As we describe in Section 2.2.A, for Inter macroblocks, these values are equal to the difference between the predicted and the original macroblock, and for Intra macroblocks, they are equal to the original image values taken with respect to the value of 128. Our algorithm uses the variance of macroblocks from the prediction error frame, to determine the variance classes.  77  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  First, the algorithm analyzes all macroblocks in a prediction error frame, sorts them in the increasing order based on their variances and separates the MBs into variance classes (VC). Unlike in [3], instead of the variance classes having fixed.boundaries, they are organized so as to contain the same number of macroblocks in each class. With this organization, illustrated in Figure 3.5, macroblocks with variances that are more prevalent in the image will have a greater precision of RQ estimation.  Distribution of the MB variances  I .  500  1000  1500  Distribution of the macroblock variances  2000 0  500  1000  2000  1500  variance  variance Variance bands  Variance bands  v  v c 500  v  c  1000  i fe P  1500  r  2000 0 variance  500  c  v.  v  c  c  K" P  w  |c  6  r  1000  1500  r 2000 variance  Figure 3.5. Determination of the macroblock variance classes in an image.  Because the MPEG algorithm uses different quantization matrices to quantize Intra and Inter macroblocks (3.1), or algorithm determines two sets of Variance Classes: one for Intra MBs and one for Inter MBs.  78  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Step 2. Preliminary encoding of the Variance Classes. i  After all macroblocks in a frame have been assigned to their variance classes, the macroblocks are encoded at different quantization levels; The quantization levels for macroblocks of each variance class are determined so as to cover the allowed quantization range (from 1 to 31 in MPEG-2) and to guarantee that a sufficient number of macroblocks will be quantized with the same mquant so as to minimize the estimation error. We have found that as little as 6 values for mquant, spread through the allowed range of quantization (in our implementation 1, 2, 4, 8,16 and 31) and the minimum of 5 macroblocks for each mquant, provide satisfactory RQ estimation for a single variance band.  Step 3. Calculation of the final RO model for each of the Variance Classes. After the preliminary encoding, the algorithm averages the resulting bit-count values for each mquant in each of the Variance Classes, so as to produce two final RQ estimation models: Intra RQ model and Inter RQ model. Each RQ model consists of several RQ curves (one RQ curve for each variance class), where an RQ curve depicts the relationship between bit-count and mquant for a single variance classes. The algorithm constructs an actual RQ curve through a piece wise approximation of the bit-count values of each mquant used in the preliminary encoding. As can be'expected, macroblocks belonging to a variance class whose average variance is larger, produce a greater number of bits. In addition, if the standard MPEG quantization matrices (3.1) are used, Inter macroblocks will produce a greater number of bits than Intra macroblocks that have the same average variance. An example of a Rate-Quantization relationship for Inter and Intra macroblock is shown on Figure 3.6.  79  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  Macroblock bits 4000  Intra variance class, Average variance =1386  3000  Inter variance class, Average variance = 200  2000  Intra variance class, Average variance = 200  1000  0 02 4  8  12  16  20  24  28 32 mquant  Figure 3.6. Example of the Rate-Quantization curves for Intra and Inter variance classes.  ii) Quality-Quantization modeling The goal of our Quality-Quantization estimation technique is to determine the relationship between the quantization factor of a macroblock, mquant, and the reconstructed quality level of a macroblock, measured using the MB signal to noise ratio, SNR b , defined in m  (3.6).  80  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Our technique for the QQ modeling, follows the same steps as the RQ estimation, described in the previous section, but measures quality of the reconstructed macroblocks instead of the bit-count. The steps of in our Quality-Quantization estimation algorithm are: 1) determination of the variance classes, 2) preliminary encoding; 3) quality assessment and calculation of the final Quality-Quantization model. The process of the measuring quality is approached differently for macroblocks of Intra and Inter variance classes. Quality-Quantization estimation for Intra V a r i a n c e Classes.  The estimation of Quality-Quantization curves for the macroblocks of Intra VCs, does not present any complications. For each of the variance classes, the algorithm computes the values of the S N R m b of each macroblock and then averages these values over all MBs encoded with the same mquant to produce the final Intra QQ estimation. Quality-Quantization estimation for Inter Variance Classes.  For the macroblocks of Inter Variance Classes, the problem of estimation the relationship between Quality and Quantization is more complex. Because all Inter macroblocks contain information that describes error (difference) between the predicted and the original images, the reconstructed quality of Intra MBs will depend on the accuracy of the prediction. For example, if the difference between the original and the predicted macroblocks is very small, then the final SNR b for such macroblock may m  81  Chapters BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  still be high, even with coarse quantization. However, if that difference is large, then coarse quantization will result in significantly lower reconstructed macroblock quality. For Inter macroblocks, instead of estimating the relationship between mquant and the,actual signal to noise ratio, our algorithm estimates the relationship between mquant the signal to noise ratio gain of a macroblock, GSNR  mb  . The SNR gain indicates the improve-  ment in the SNR after the quantized prediction error is added to the predicted macroblock and is defined as: GSNR  mb  where, SNR  mb  = SNR - predSNR mb  (3.9)  mb  is the signal to noise ratio of the reconstructed macroblock and has been  already defined in (3.6); predSNR is the signal to noise ratio of the predicted macroblock and is similarly mb  defined as:  predSNR = 10* log mb  (3.10)  predMSE ) mb  ,where predMSE b is the mean square of the difference between the predicted macm  roblock and the original macroblock. Similar to M S E  mb  (3.7), predMSE b can be m  defined as the average of the prediction mean square errors of four 8x8 corresponding blocks: predMSE = ^ llpredMSE [k]  (3.10)  k  mb  k  block  ,where each of the predMSEbi k is calculated as 0C  predMSE  block  = ^ X ^ ' ^JoiorgtfJ]-  82  pred[i,j])  2  (3.11)  Chapter 3  BIT ALLOCATION  AND RATE  CONTROL  BASED  ON MACROBLOCK  QUALITY  CRITERIA  ,with org[i,j] and pred[i,j] corresponding to the pixels of the original and the predicted 8x8 blocks. Quality-Quantization estimation produces two final QQ models: Intra QQ model and Inter QQ model. An Intra QQ model consists of QQ curves that describe the relationship between the macroblock mquant and the macroblock signal to noise ratio. An Inter QQ model contains QQ curves showing the relationship between the macroblock mquant and the SNR gain, the increase in the SNR of the reconstructed MB relative to the SNR of its predicted MB. An example of Quality-Quantization relationship for Inter and Intra macroblock is shown on Figure 3.7.  Macroblock SNR or GSNR 40  Intra variance class, ^ArVerage variance =1386  30  Intra variance class, Average variance = 200  20  Inter variance class, Average variance = 200  10  0 02 4  8  . 12 , 16  20  24  28 32 mquant  Figure 3.7. Example of the Quality-Quantization curves for Intra and Inter variance classes.  83  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.4. A N O V E L B I T A L L O C A T I O N A N D R A T E C O N T R O L A L G O R I T H M WITH T H E EMPHASIS O N CONSISTENCY OF RECONSTRUCTED PICTURE  QUALITY  • In the previous section, we demonstrated a technique that allows an algorithm to estimate the relationships between the level of quantization, the number of bits and the reconstructed quality for all macroblocks in a frame. The results of the Rate-Quantization and QualityQuantization estimations are used to predict the number of bits needed to encode each macroblock at a certain quality level and the value of the quantization factor that would produce these bit results^ Having developed the mechanisms to successfully anticipate the encoding quality and td control the bit-rate, we are now able to carry out the objective of our algorithm: to achieve a flat (same) signal to noise ratio for all reconstructed macroblocks in all frames throughout a video sequence. The degree to which we will be able to fulfill this goal and the exact sequence of steps that will be performed by our algorithm depend on the particular restrictions of each encoding situation. We will discuss in detail three encoding scenarios: variable bit rate (VBR), constant bit rate (CBR) with two passes over a GOP and a real time CBR encoding.  3.4.A. B i t allocation a n d rate control algorithm based on macroblock quality criteria i n variable bit rate encoding In the beginning of this chapter, we examined some of the distinct features of VBR and CBR encoding ( see Figure 3.4). Being free from the constraints of the limited VBV buffer size, the objective of a variable bit rate encoder is to encode a video sequence at the best and  84  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  consistent level of quality, while yielding the preset number of bits allocated for the whole sequence. In any encoding scenario, the best quality of the reconstructed video sequence can be achieved with two passes. In case of a variable bit rate, the two passes should be ideally performed over all frames in a sequence to guarantee the most efficient bit allocation. To achieve a maximum and constant level of SNR for all macroblocks in the video stream, our algorithm has to undergo three major steps: 1) the first pass Rate-Quantization (RQ) and Quality-Quantization (QQ) estimation, 2)  the bit allocation;  3) the second pass encoding. During the first pass, the encoder will estimate the Rate-Quantization and QualityQuantization relationships for all frames in a video sequence. To decrease the complexity of computation and the memory requirements, we can limit the number of the estimated ratecontrol curves from one set of Intra and one set of Inter RQ and QQ curves for each frame to one set of Intra and one set of Inter RQ and QQ curves for each scene in a video stream. Using the results of the RQ and QQ estimation from the first pass encoding, our bit allocation algorithm will assess the target SNR, i.e., highest value of the SNR that can be uniformly achieved over all macroblocks in a sequence while meeting the stream bit target requirement. Finally, using the RQ and QQ models and the target SNR, the bit allocation algorithm will compute Frame Targets, the target number of bits for each frame in a video sequence.  \  In the second pass, the algorithm will encode each frame based on the corresponding Frame Targets and the RQ and QQ models.  85  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY CRITERIA  Figure 3.8. explains the general structure of our algorithm for a VBR encoding and Figure 3.9 illustrates the steps of the first pass RQ and QQ estimation.  Pass 1  Estimation of the RateQuantizationand QualityQuantization relationships one Intra QR and QQ model and one Inter QR and QQ model for each scene in the input video sequence  Bit Allocation  Using the RQ and QQ models, assess the highest value of] the.MB_SNR that can be uniformly achieved in a [sequence, while) meeting the total number of bits closest to the stream bit target  Pass 2  Using the predetermined frame [bit targets and| the target SNR for each of the macroblocks, encode the video stream so that the requirements of |MB_SNR and frame targets are met  /Output\ the\ 'com-.! MPEG |  video \ stream/  Stram^t target allowed total number of bits toi the whole video .stream  Figure 3.8. Bit allocation and rate control algorithm based on macroblock quality criteria in variable bit rate two-pass encoding.  86  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Motion compensation  Separate macroblocks into Variance Classes  Quantize the macroblocks and estimate the RQ and QQ relationship  Separate Intra macroblocks into Variance Classes  Quantize the Intra macroblocks and estimate the RQ and QQ relationship  Separate Inter macroblocks into Variance Classes  Motion compensation  Quantize the Inter macroblocks and estimate the RQ and QQ relationships  Continue until the end of a GOP  Figure 3.9. Bit allocation and rate control algorithm base on macroblock quality criteria in constant bit rate two-pass encoding: first pass.  87  Chapter 3  BIT ALLOCATION  AND RATE CONTROL BASED ON MACROBLOCK  QUALITY  CRITERIA  3.4.B. B i t allocation a n d rate control algorithm based on macroblock quality criteria i n constant bit rate two pass encoding In the constant bit rate (CBR) situation, the encoder is limited to a preset constant bit-rate for all frames. In a CBR encoding, although every frame is allocated a certain fixed number of bits, a certain degree of flexibility in this bit-rate is provided by the Virtual Bitrate Verification (VBV) buffer, which allows us to redistribute the bits between the frames to attain consistent quality throughout a GOP. Due to the fixed value of the bit-rate, there is a preset target number of bits for all Groups of Pictures in the video sequence. Therefore, it is not necessary to evaluate the content of the whole stream, before making the bit allocation decisions for a GOP and its final encoding. A two pass encoding scheme over one GOP will be able to guarantee the best and consistent quality for the frames included in that GOP under the given constraints of the VBV buffer size and the value of the bit-rate. The structure of our CBR encoding process is analogous to that of our VBR encoding of Figure 3.8. and Figure 3.9., except that for the CBR case, the two passes are not performed over the complete video stream, but only over one GOP. Similar to the VBV encoding, our algorithm will go through 3 stages to encode one GOP: a first pass preliminary encoding to estimate the RQ and QQ relationship, the bit allocation for the GOP and the final (second pass) encoding. The following information will provide a more detailed explanation.of each stage of the encoding process.  88  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Preliminary encoding, first pass. For each frame in a GOP do the following: • For each macroblock in a frame, determine the position of the MB on a variance scale according to the values of their variances. • Group the macroblocks of each frame into Intra and Inter variance classes ( depending on type of the corresponding macroblock) following the rules described in Section 3.3.B. • Quantize the macroblocks of each variance class as was described in 3.3.B. • After the quantization, tabulate the bit-count of each macroblock. For each Intra macroblock tabulate its SNR, for each Inter macroblock get its SNR_GAIN. • After the frame had been encoded, estimate the Rate-Quantization model and the QualityQuantization models as was described in 3.3.A. and 3.3.B. A s the results of the RQ and QQ estimation, each frame is associated with an Intra Rate-Quantization and QualityQuantization estimation and, in addition to that, all P frames are associated with an Inter Rate-Quantization and Quality-Quantization estimation. After all frames in a GOP have been encoded, their corresponding RQ and QQ estimations > may be combined to make up a single set of Intra RQ and QQ estimations and a single set of Inter RQ and QQ.  Bit allocation based on macroblock quality criteria, after the first pass. After the preliminary encoding, the results of the first pass are evaluated to estimate the frame bit targets and the target SNR as follows: • Set a first try SNR to be used as a possible target SNR. For the first GOP in a video sequence use a reasonable arbitrary value for SNR (for example 20). For the other groups of pictures, use the target SNR of the previous GOP as the initial target SNR.  89  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  • Do the following three steps for each frame in a GOP. • For each macroblock, determine the bit-count that will correspond to the target SNR. Use either Intra or Inter QRQ curves to do the estimation (see Figure 3.4 and 3.9 for more details). If possible, use two sets of QRQ curves for the current macroblock estimation: rate control curves of the variance band with its average variance above and below the variance of the current macroblock. After all macroblocks have been encoded, determine the local macroblock bit distribution model for a frame and calculate the total frame,bit-count as the sum of bit-counts of all macroblocks. • Check if the total number of bits for a frame exceeds the buffer size requirements or if it is too small (VBV buffer overflow, VBV buffer underflow) or if the total GOP bit-count does not conform with the value of the CBR. If any of the above is correct, respectively decrease or increase the target SNR and repeat the GOP bit allocation for the current frame from the start. Otherwise, go to the next step. • Tabulate the final target SNR and the values of the target number of bits for each frame in a GOP. Because the GOP is being encoded with a constant bit rate, it might not be possible to achieve a flat SNR for all frames in a GOP. In this situation, our bit allocation approach will yield the bit allocation that provides the smallest level of SNR fluctuations.  As a result of the bit allocation, we obtain the following pieces of information: 1) the target SNR for each frame, a target quality level that will be used to encode each macroblock of every frame in a GOP;  90  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  2) a target number of bits for each frame within a GOP and the global frame bit distribution for a GOP; 3) a local macroblock bit distribution for each frame in a GOP;  Encoding, second pass.  '  The results of the bit allocation and the QRQ estimation curves, generated in the first pass, are used to guide the final encoding. For each macroblock of every frame in a GOP: • Determine the place on a variance scale of each macroblock in a frame.  v  • Group the macroblocks into Intra and Inter (depending on type of the corresponding macroblock) variance bands. • Using the corresponding Rate-Quantization estimation and the target number of bits for a macroblock, find the quantization factor (mquant), that will produce the required bitcount. • Using the. corresponding Quality-Quantization estimation to find the quantization factor that will result in the MB_SNR equal to the target SNR. • After the encoding of each frame is complete, compare the bit-count and the expected target number of bits. Depending on the results of the comparison, modify the targets for the rest of the frames using the global frame bit distribution.  91  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  3.4.C. B i t allocation a n d rate control algorithm based on macroblock quality encoding i n constant bit rate real-time encoding In a real time encoding scenario, all bit allocation and rate control decisions for a GOP have to be made without any prior knowledge about the frames that actually compose the GOP. An additional constraint, that may or may not be imposed on the encoder is a causality constraint within a single frame. The causality constraint means that a macroblock quantization factor must be determined without assessing future macroblocks of the same frame. To condense our discussion we will omit the macroblocks causality constraints and assume that all of the frame information is known before the start of quantization. All real time video compression algorithms use statistics of the previously encountered frames to make current encoding choices. Usually, these are the complexities of the I and P frames from the previous GOP that will be carried to the bit allocation decision-making process of the current GOP. In our algorithm, instead of using complexity measures, we base the current bit allocation and rate control decisions on the target macroblock signal to noise ratio, SNR b , and the RQ and QQ estimations of the previously encoded GOP. Note, that m  only the RQ and QQ models and the expected quality, defined by SNR b, and not the rem  quired numbers of bits are carried through from the previous GOP. The most challenging part of the real time encoding algorithm, is the bit allocation for the frames of the new GOP. Because the encoder relies on past statistics, the mismatch bit allocation error can be very significant. Our method estimates the frame bit targets using the RQ and QQ estimation curves determined for the previous GOP and its average SNR as the tar-  92  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  get level of quality. Because the actual GOP bit allocation is performed during the encoding of its first frame, we will use the contents of this frame to improve the precision of the bit allocation.  Determination of the preliminary frame targets for a GOP. In order to determine the target number of bits for the first frame in a GOP and the preliminary targets for other frames, our algorithm will do the following: • Read the first frame, determine the place of each of its macroblocks on a variance scale and group the macroblocks into variance bands. • Using the last Intra RQ and QQ estimation models of the previous. GOP, determine the number of bits that is necessary to encode this frame at the level of quality equal to the average SNR of the previous GOP. Instead of using the estimations models of the previous GOP, the Intra QRQ curves can also be generated by the preliminary encoding of the first frame in the current GOP, which would improve the precision of estimation. • If the resulting number of bits conforms with the required bit-rate and the VBV buffer constraints, go to the next step. Otherwise, decrease the target SNR and repeat the assessment of bit-count. • Using the SNR of the first frame, evaluate the number of bits needed to encode the rest of the frames in a GOP. To do so, use the last Intra/Inter RQ and QQ models and assume that the macroblocks in all P frames of the current GOP have their variance distribution equal to the average macroblock variance distribution of P frames in the previous GOP. • Compute the predicted total number of bits for the current GOP. If this number does not exceed the total number of bits allocated for a GOP, commence the encoding of the cur-  93  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  rent GOP starting with its first frame. Otherwise, decrease the target SNR and repeat the GOP bit-count estimation (the previous three steps). After the target number of bits for the frames have been identified, the rate control will go through the same steps for the quantization and rate control on a macroblock level as it did in the case of two passes over a GOP (see Section 3.4.B). The Quality-Quantization and Rate-Quantization models used here, will either come from the previously encoded GOP or can be generated by the preliminary encoding of current frames.  94  Chapter 3  3.7.  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  RESULTS  In this chapter we presented a novel bit allocation and rate control algorithm, that is specifically designed to minimize the fluctuations of quality between macroblocks within a single frame and between frames with a GOP. To test the performance of our algorithm, we compare the signal to noise ratio (SNR) statistics rendered by our method and the SNR statistics of the Test Model 5 algorithm [4]. We present separate measurements of SNR fluctuations on a macroblock level for a single frame and of quality fluctuation on a frame by frame basis for several groups of pictures.  i) Macroblock level statistics To illustrate the performance of our algorithm in reducing the fluctuations in quality of macroblocks within a single frame, we measure the macroblock level statistics of several I and P frames encoded with our BA and RC algorithm and compare these measurements with statistics rendered by the TM5 algorithm. Using both of the algorithms, we encode 5 GOPs, each containing 10 frames for sequences "Tennis" and "Garden" and compare the average SNR macroblock level statistics for I and P frames. To provide the same conditions for both algorithms, the resulting number of bits for the corresponding encoded streams was kept the same. The results of our comparison are shown in Tables 3.1. The variables in the table show the minimum, the maximum and the average signal to noise ratios for a macroblock as defined in (3.6), as well as the variance of the SNR as a measure of SNR fluctuations.  95  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  Encoding algorithm Sequence "Tennis", bit-rate 2Mhits/s  TM5 algorithm  Sequence "1 emus'". bit rate 4Mbits/s  TM5 algorithm Our BA and RC algorithm Sequence "Tennis". bit-rate 6Mbits/s TM5 algorithm Our BA and RC algorithm Sequence "Garden". bit-rate 3Mbits/s TM5 algorithm Our BA and RC algorithm Sequence "Garden", bit-rate 4Mbits/.s  TM5 algorithm Our BA and RC algorithm Sequence "Garden". bit-rate 6Mhils/s TM5 algorithm Our BA and RC algorithm  Snr statistics for I frame min max avg var SNR SNR SNR SNR  Snr. statistics for P frame min max avg var SNR SNR SNR SNR  -0 7 2.0  19.2 5.1  3.5 3.9  11 9 2.5  -3.0 2.0  19.0 7.0  4.0 3.9  10.2 3.4  -0.3 4.0  18.1 10 1  5.0 6.1  13.0 2.9  -1.6 3.1  20.9 8.9  54 5.9  II 6  -0.1 5.0  22.8 9.4  6.4 7.2  I3.S 3.9  -1.3 3.8  23 2 15.1  7.0 7.1  129 4.9  -0.4 4.0  19.7 10.1  6.4. . 6.4  1 50  -(>.5 1.9  22.2 12.S  5.J 5.Q  17 0 4.9  -0.3 5.1  21.5 13.3  8.1 S3  21.4 5.4  -5 0 2.3  24 7 13 4  6.8 79  19 7 5.0  -0.2 5.9  22.0 15.4  y.5 9.7  26 6 5.8  -5 4 4.1  26.4 15.1  8.1 9.6  25.7 7.2  1  3.7 ^  .  Table 3'. 1. Comparison of macroblock level SNR statistics for I and P frames encoded with our bit allocation algorithm and with the Test Model 5 algorithm.  As can be seen from Table 3.1, our bit allocation and rate control algorithm provides improvement in the average macroblock signal to noise ratio, as well as significantly reduce the fluctuations of quality from one macroblock to the next.  ii) Frame level statistics The performance of our algorithm in reducing the fluctuations of quality between the frames is illustrated in Table 3.2. Five groups of pictures, each containing ,10 frames, were encoded by both our algorithm and the T M 5 algorithm for sequences "Tennis" and "Gar-  96  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  den". The assessment of quality fluctuation has been obtained as the average of the corresponding SNR measurements of minimum, maximum, average signal to noise ratio (defined similarly to (3.6) but for a frame instead of a macroblock) and the variance of the SNR on a frame level. Encoding algorithm Sequence "Tennis", , bit-rate 2Mbits/s TM5 algorithm Our BA and RC algorithm Sequence "Tennis", bit-rate 4Mbits/> TM5 algorithm Our BA and RC algorithm Sequence "Tennis", bit-rate 6Mbits/s TM5 algorithm Our B A and RC algoi i thm  Snr statistics for I- frame max avg SNR SNR .  min SNR  var SNR  7.0 7.9  8.0 7.9  7.6 7.9  0.18 __0.001_  8.3 9.5  9.4 9.8  9.3 9.6  0.23 0.08  10 0 11 9  11.7 12.2  11.3 12.0  0.28 0.12  13.7  14.3  14.6 14.6  14.0 14.5  0.12 0.06  14.6 15.6  16.8 16.0  15.5 15.9  0.42 0.15  16.2 18.0  18.3 18.5  17.7 18.4  0.46 0.18  Sequence '"Garden". bit-rate  3Mbits/s  TM5 algorithm Our BA and RC algorithm Sequence "Garden", bit-rate 4Mbits/s TM5 algorithm Our BA and RC algorithm Sequence "Garden", bit-rate 6Mbits/s TM5 algorithm Our BA and RC algorithm  Table 3.2. Comparison of frame level SNR statistics for frames encoded with our bit allocation algorithm and with the Test Model 5 algorithm.  By comparing the encoding results presented in Table 3.2, we can see that our bit allocation and rate control algorithm improves the problem of fluctuations of quality between the frames, as well as improves the average quality of the reconstructed images.  97  Chapter 3  3.8.  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  SUMMARY  In this chapter, we presented a new bit allocation and rate control algorithm that specifically focuses on attaining the best and the most consistent quality for the frames within a GOP and for the macroblocks within a single frame. In case of the variable bit rate encoding situation, where the bit allocation and rate control algorithm almost exclusively determines the consistency of quality on both a macroblock and a frame level, the goal of our algorithm is to encode the input video stream so that the highest and consistent quality level that is obtained for the given total bit-rate.. In a constant bit rate encoding scenario, the size of the VBV buffer can limit the ability of the algorithm to achieve consistent quality throughout all frames. In this situation, the objective of the algorithm is to keep the fluctuation of quality between the frames to the minimum and, on the macroblock level, to guarantee consistent quality encoding. In order to achieve the above goals, we developed a unique Rate-Quantization and Quantization -Quality modeling technique. This technique estimates the relationships amongst the bit-count, quantization and the reconstructed image quality on a macroblock level. Knowing the relationships between mquant, bit-count and SNR, which we use to measure quality, our algorithm can assess the number of bits necessary to encode a macroblock, a frame, a GOP or a whole stream at a certain quality level. A highly precise rate control mechanism guarantees the execution of the bit allocation decisions. Our macroblock quality based approach to bit allocation and rate control allows the encoder to significantly reduce the fluctuations in quality on both the frame and the macroblock levels and in both two-pass and one-pass encoding. In case of a VBR encoding, our  98  Chapter 3  BIT ALLOCATION AND RATE CONTROL BASED ON MACROBLOCK QUALITY CRITERIA  algorithm can guarantee consistent quality encoding throughout all macroblocks in a video stream. For the CBR encoding, our method can significantly improve the consistency in quality of the reconstructed images and, in many situations, can provide the constant SNR level for all macroblocks in all frames belonging to one group of pictures.  99  Chapter 4  CONCLUSIONS  Chapter 4 CONCLUSIONS . The two most challenging goals of video compression are, first, meeting the required bit rate and second, preserving picture quality so that the quality reaches its highest level for a given bit-rate and remains consistent throughout each frame and the entire video stream. Traditionally, both problems have been handled almost exclusively by the bit allocation (BA) and rate control (RC) algorithms. In their attempts to attain better quality, different BA and RC techniques are limited to their abilities of only redistributing bits between the frames in a stream and the macroblocks within a frame. In order to achieve significant improvement in the encoding efficiency and thus quality, an algorithm has to employ additional innovative techniques outside of the traditional solutions, such as taking greater advantage of the redundancies in the input data. We developed a new Frame Partitioning technique that significantly improves the use of temporal redundancies in encoding by taking the exploitation of temporal redundancies to a new level - a level of perceptual adaptability. Based on the properties of the.human visual system, Frame Partitioning distinguished the visible and the non-visible temporal changes in the video stream and uses this information to modify the traditional MPEG motion compensation, consequently resulting in the superior encoding efficiency. To further improve picture quality, we developed an original bit allocation and rate control algorithm. The objective of our technique is to minimize the fluctuations in the macroblock quality within a single frame, a GOP and a video stream. The distinctive trait of our method lies in an integrated approach toward the bit allocation and rate control problems, as  100  Chapter 4  CONCLUSIONS  opposed to a traditional top down approach where the bit allocation always precedes arid is separated from rate control. Our algorithm makes a macroblock level estimation of the Rate-Quantization (RQ) and Quality-Quantization (QQ) relationships and uses the found RQ and QQ models for the bit allocation and rate control. Our bit allocation and rate control approach allows the encoder to greatly reduce the fluctuation in the macroblock quality measures and, thus, to achieve a more consistent quality throughout a frame, a GOP and a video stream.  101  Chapter 4 CONCLUSIONS  Bibliography  [1] Leonardo Chiariglione, "MPEG and multimedia communications", IEEE Trans. Circuits Syst. Video Technol., vol. 7, no. 1, pp. 12-19, Feb. 1997. [2] Wei Ding and Bede Liu, "Rate control of MPEG video coding and recording by ratequantization modeling", IEEE Trans. Circuits Syst. Video Technol, vol. 6, no. 1, pp. 12-19, Feb. 1996. [3] Atul Puri and R. Aravind, "Motion-compensated video coding with adaptive perceptual quantization", IEEE Trans, Circuits Syst. Video Technol., vol. 4, no; 4, pp: 351-361, Dec. 1991. [4] ISO-IEC/JTC1/SC29/WG11, "Test Model 5", Draft, Apr. 1993. [5] E. Viscito and C. Gonzales, "A video compression algorithm with adaptive bit allocation and quantization", SPIE Visual Commun. Image Processing, vol. 1605-1, pp. 58-72, Nov. 1991 [6] Atul Puri and R. Aravind, "On comparing motion-interpolation structures for video coding", Visual Commun. Image Processing '90, vol. 1360, pp. 1560-1571, Oct. 1990. [7] E. Viscito and C. Gonzales, "Motion video adaptive quantization in the transform domain", IEEE Trans. Circuits Syst. Video Technol.,-vol. 1, no. 4, pp. 374-378, Dec. 1991. [8] Prasoon Tiwari and Eric Viscito , "A parallel MPEG-2 encoder with a look-ahead rate control", 0-7803-3192-3/96, IEEE. "  102  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0065211/manifest

Comment

Related Items