Image Processing for Embedded Devices

Sebastiano Battiato was born in Catania, Italy, in 1972. He received the degree in Computer Science (summa cum laude) in 1995 and his Ph.D. in Computer Science and Applied Mathematics in 1999. From 1999 to 2003 he has leaded the ”Imaging” team c/o STMicroelectronics in Catania. Since 2004 he has been a researcher at Department of Mathematics and Computer Science of the University of Catania. His research interests include image enhancement and processing, image coding and camera imaging technology. He published more than 90 papers in international journals, conference proceedings and book chapters. He has authored 2 books and is a co-inventor of about 15 international patents. He is a reviewer for several international journals and he has been regularly a member of numerous international conference committees. He has participated in many international and national research projects. He is an Associate Editor of the SPIE Journal of Electronic Imaging (Specialty: digital photography and image compression). He is a director (and cofounder) of the International Computer Vision Summer School. He is a Senior Member of the IEEE. For more details see (http://www.dmi.unict.it/ battiato)

Arcangelo R. Bruna

Arcangelo R. Bruna received the degree in Electronic Engineering (summa cum laude) in 1998 at the University of Palermo. First he worked in a telecommunication company in Rome. He joined STMicroelectronics in 1999 where he works in the Advanced System Technology (AST) Catania Lab - Italy. Today he leads the Image Generation Pipeline and Codecs group and his research interests are in the field of image acquisition, processing and enhancement. He published several patents and papers in international conferences and journals.

Giuseppe Messina

Giuseppe Messina was born in Crhange, France, in 1972. He received his MS degree in Computer Science in 2000 at the University of Catania doing a thesis about Statistical Methods for Textures Discrimination. Since March 2001 he has been working at STMicroelectronics in the Advanced System Technology (AST) Imaging Group as Software Design Senior Engineer II / PL. Since 2007 he is Ph.D. student in Computer Science at the University of Catania accomplishing a research in Information Forensic by Image/ Video Analysis. He is member of the Image Processing Laboratory, at the University of Catania. His research interests are in the field of Image Analysis e Image Quality Enhancement. He is author of about several papers and patents in Image Processing field. He is a reviewer for several international journals and international conferences. He is an IEEE member.

Giovanni Puglisi

Giovanni Puglisi was born in Acireale, Italy, in 1980. He received his degree in Computer Science Engineering (summa cum laude) from Catania University in 2005 and his Ph.D. in Computer Science in 2009. He is currently contract researcher at the Department of Mathematics and Computer Science and member of IPLab (Image Processing Laboratory) at the University of Catania. His research interests include video stabilization, artificial mosaic generation, animal behavior and raster-to-vector conversion techniques. He is the author of several papers on these activities.

Image Processing Lab (http://iplab.dmi.unict.it)

IPLab research group is located at Dipartimento di Matematica ed Informatica in Catania. The scientific knowledge of the group is on Computer Graphics, Multimedia, Image processing, Pattern Recognition and Computer Vision. The group has a good expertise in the overall digital camera pipeline (e.g., acquisition and post acquisition processing) as well as a good and in-depth knowledge of the recognition of scene categorization field. This is confirmed by the numerously research paper, within the area of image processing in single sensor domain (in acquisition and post acquisition time) as well as different works relatively the semantic analysis of images content, to drive some image processing tasks such as image enhancement. Moreover, the collaboration between members of the Catania unit and industrial company leaders in single sensor imaging (e.g., STMicroelectronics) has already done the possibility of transferring to the industry (pre-competitive research) the knowledge acquired in academic research facilitating the industry in producing new advanced products and patents. A joint research lab IPLab-STMicroelectronics, has been recently created where researchers coming from both partners work together on imaging research topics. More specifically, 2 Ph.D. students in Computer Science (XXIII Ciclo Dottorato in Informatica - Universit`a di Catania) have received financial support by STMicroelectronics to investigate about ”Methodologies and Algorithms for Image Quality Enhancement for Embedded Systems”. The group published more than 100 papers on topics related to the previous mentioned disciplines. Moreover the IPLab group established a number of international relationships with academic/industrial partners for research purpose. In the last years the group organized the ”Fourth Conference Eurographics Italian Chapter 2006” and the ”International Computer Vision Summer School 2007, 2008, 2009, 2010” (http://www.dmi.unict.it/icvss).

Advanced System Technology - Catania Lab - STMicroelectronics (http://www.st.com)

Advanced System Technology (AST) is the STMicroelectronics organization in charge of system level research and innovation. Active since 1998, AST responds to the need to strengthen the position of STMicroelectronics as a leading-edge system on chip company. The AST Catania Lab and, in particular, the Imaging Group, works on research and innovation in the field of imaging processing. Its mission is to acquire digital pictures with superior Performance/Cost using advanced image processing methodologies, to extend the acquisition capability of imaging devices through the development of new applications and to determine the computational power, the required bandwidth, the flexibility and the whole imaging engine. Its members have long experience in image algorithms, documented also by many patents and scientific publications. Primarily, through active contacts and collaborations with several universities and a dedicated joint lab with the IPLab of Catania University, they have concretized and made effective the link between academic and industrial R&D.

Full text available

We would like to take this opportunity to thank all contributors of this book, and all people working into the two involved research groups: Image Processing Lab (Catania University) and Advanced System Technology, Catania Lab (STMicroelectronics). A special thanks to Massimo Mancuso for having contributed with his extraordinary tenacity and competence to establish an imaging R&D group in Catania. We also thank prof. Giovanni Gallo for his invaluable help and support

The main goal of this Chapter is devoted to provide all the fundamental basis related to the involved technological issues relative to the single-sensor imaging devices. A rough understanding of the overall ingredients of a typical imaging pipeline is important also to consider the performance of any imaging devices, from low to high level, as the result of several components that run together to compose a complex system. The final image/video quality is the result of a certain number of design choices, that involve, in almost all cases, all aspects of the hardware and software technology. As briefly stated in the preface, the book aims to cover all aspects of algorithms and methods for the processing of digital images acquired by imaging consumer devices. More specifically, we will introduce the fundamental basis of specific processing into CFA (Color Filter Array) domain such as demosaicing, enhancement, denoising, compression together with ad-hoc matrixing, color balancing and exposure correction techniques devoted to preprocess input data coming from the sensor. We conclude the Chapter just including some related issues related to the intrinsic modularity of the pipeline together with a brief description of the hardware/software partitioning design phase.

This Chapter gives information about the optics and the sensor in an image acquisition system. Optic is the first stage of the image acquisition system and is composed by one or more lenses aiming to concentrate the light in the physical sensor obtaining an in focus image. It is one of the most expensive parts of an imaging system. In mobile cameras usually a lens is compound of a system of plastic and glass lenses stacked together, while in single lens reflex cameras (SLR) a group of several glass lenses system is employed to reduce image artifacts. In this Chapter an overview of the lenses will be discussed. Moreover, some typical artifacts will be introduced (e.g., cross talk and chromatic aberration). The sensor is the part that converts the optical image (light) to an electric signal. There are several kinds of sensors depending on the technology (CCD - Charge Coupled Device and CMOS - Complementary MetalOxideSemiconductor), on the color filter array (Bayer , Foveon, 3CCD, Panchromatic), on the transducer function (LDR - low dynamic range, WDR - wide dynamic range and HDR - high dynamic range sensors).

The problem of the proper exposure settings for image acquisition is of course strictly related with the dynamic range of the real scene. In many cases some useful insights can be achieved by implementing ad-hoc metering strategies. Alternatively, it is possible to apply some tone correction methods that enhance the overall contrast of the most salient regions of the picture. The limited dynamic range of the imaging sensors doesn’t allow to recover the dynamic of the real world. In this Chapter we present a brief review of automatic digital exposure correction methods trying to report the specific peculiarities of each solution. Starting from exposure metering techniques, which are used to establish the correct exposition settings, we describe automatic methods to extract relevant features and perform corrections.

In this Chapter different auto focusing techniques are analyzed. First Auto-focus (AF) techniques were implemented into Single Lens Reflex (SLR) cameras. They use a dedicated system to focus a scene which is independent from the acquisition part mainly based on a phase detection system. Digital still cameras (DSC) instead use the acquisition sensor also to focus the scene. An image processing system based on contrast analysis finds the in-focus position and is employed in DSC due to its compactness and cheapness. Both the SLR and the DSC optics need a moving lens to focus the scene. Very low cost and ultra small cameras, such as those integrated into mid-low cost Personal Digital Assistant (PDA) (i.e., smart-phones), don’t have any moving part. In this case to accomplish a further extension of the Depth of Field (DoF) these modules implement a digital auto-focus technique known as Extended Depth of Field (EDoF). Nowadays most of the camera systems implement sophisticated content dependent AF models: they are capable to optimize their behavior for still and video acquisition, to detect and prioritize focus on faces and to predict new lens position when moving object are being focused.

The color reproduction accuracy of digital imaging embedded devices is a key factor of the overall perceived image quality. The human visual system, under most conditions, is able to compensate for the effects of the scene illuminant on the perceived image. Therefore on digital imaging devices, some processes have to be performed across the image generation pipeline in order to obtain an effective color accuracy regardless of the scene illuminant and the sensor response features. In this chapter we describe the most common color processing algorithms performed across the image generation pipeline: white balancing algorithm, which is aimed to compensate the effects of the illuminant power spectral distribution and the color correction process, which compensates the mismatch between the color filters array transmittance and the color response of the human visual system to different wavelengths. The first one, on embedded devices, is usually performed through an image statistical analysis to obtain an estimation of the scene illuminant and is often based on strong assumptions on scene spectral reflectance distribution. The second relies on the characterization of the sensor color filters spectral transmittance. We illustrate the role of such algorithms on the overall perceived color image quality and describe typical methods for white balancing performance and sensor characterization benchmarking. We describe also some additional algorithms which, frequently on consumer devices, can be used to improve the visual appearance of common colored objects (e.g., skin tones, vegetation and sky).

Among the many factors contributing to image quality degradation, noise is one of the most recurrent and difficult elements to deal with. Smart filters capable to remove noise without affecting the tiny details of a digital image are of primary importance to produce pleasant pictures. Different noise sources, having different characteristics, are superimposed to the image signal; consequently, the design of effective filters capable to discriminate and remove unwanted signal from useful data, requires analysis and understanding of the whole image formation process. This Chapter is devoted to the analysis of the main noise sources that contaminate the ideal image signal, providing an overview of noise estimation and filtering techniques.

Acquisition of color images requires the presence of different sensors for different color channels. Manufacturers reduce the cost and complexity by placing a color filter array (CFA) on top of a single image sensor, which is basically a monochromatic device, to acquire color information of the true visual scene. Since each image sensing element can detect only one color of illumination, the missing information must be filled in. The color interpolation process (also called demosaicing) aims to reconstruct the full resolution image acquired by the sensor, by calculating the missing components. Picture quality is strictly related to the peculiarity of demosaicing process. Due to the aliasing phenomenon, such as false colors and zipper effects, the color interpolation has to guarantee the rendering of high quality pictures avoiding artifacts. In this chapter we review some solutions devoted to demosaicing and antialiasing. Demosaicing algorithms can be basically divided into two main categories: spatial-domain and frequency-domain. Demosaicing solutions are not always able to completely eliminate false colors and zipper effects, thus imaging pipelines often include a post-processing module, with the aim of removing residual artifacts. Some of these techniques are also described.

Since the large diffusion of mobile devices with embedded camera and flashgun, the red eye artifacts have de-facto become a critical problem. Red eyes are caused by the flash light reflected off the blood vessels of the human retina. This effect is more pronounced when the flash light is closer to the camera lens, which often occurs in compact imaging devices. To reduce these artifacts, most cameras have a red-eye flash mode which fires a series of pre-flashes prior picture acquisition. The biggest disadvantage of the pre-flash approach is power consumption (flash is the most power-consuming part of imaging devices), and thus it is not suitable for power-constrained systems (e.g., mobile devices). Moreover, this approach does not guarantee total prevention of red eye artifacts. Red eye removal must then be performed in post-processing, through the use of automatic correction algorithms. The aim of this Chapter is to depict the state of the art of automatic detection and correction of red eyes, taking into account strong points and drawbacks of the most well-known techniques, with particular emphasis on the image degradation risk associated to false positives in red eye detection and to wrong correction of red eyes. Furthermore the problem of estimating the quality of the final result, without reference image, is examined.

To make a high quality video with a hand-held camera is a very difficult task. The unwanted movements of our hands typically blur and introduce disturbing jerkiness in the recorded video. Moreover this problem is amplified when a zoom lens or a digital zoom is used. To solve this problem many video stabilization techniques have been developed. Optical based approaches measure camera shake and control the jitter acting on lens or on the CCD/CMOS sensor. On the other hand digital video stabilization techniques make use only of information drawn from images and do not need any additional hardware tools. This Chapter describes the algorithms typically involved in the video stabilization pipeline (motion estimation, unwanted movement detection, frame warping) highlighting their issues and weak points.

Vision is perhaps the most important sense for humans. Among the different complex tasks accomplished by the Human Visual System, the categorization is a fundamental process that allows humans to effectively interpret their surroundings efficiently and rapidly. Computer Vision researchers are increasingly using algorithms from Machine Learning to build robust and reusable machine vision systems that act taking into account the visual content of images. Since learning is a key component of biological vision systems, the design of artificial vision systems that learn and adapt represent one of the most important trend in modern Computer Vision research. Despite the advances in the context of single sensor imaging devices, this technology is still quite far from the ability of automatically categorize and exploit the visual content of the scene during (or after) acquisition time. Different constraints should be considered in order to transfer the ability of inferring the category of a scene in imaging devices domain. Indeed, these devices have limited resources in terms of memory and computational power, and the image data format change over time through the imaging pipeline (i.e., from Bayer Pattern at acquisition time to JPEG format after acquisition time). This Chapter presents Computer Vision and Machine Learning techniques within the application contexts of scene recognition and red-eye detection. The techniques introduced here could be used in building complex imaging pipeline in which image categorization (e.g., scene recognition, red-eye detection) is exploited to drive other tasks (e.g., white balance, red eye removal).

The data acquired by the sensor have to be processed by the coprocessor or the host microprocessor, so both the systems have to share the same communication protocol and data format. Moreover, at the end of the image generation pipeline the image must be coded in a standard format in order to be read by every external device. Usually the sensor provides the image in the Bayer format. In the past the Bayer data were stored and transmitted using proprietary format and protocol; such solution has the drawback that every designer had to use the same proprietary interface to manage the sensor data. In the latest years the majority of companies making, buying or specifying imagine devices proposed a new standard called Standard Mobile Imaging Architecture (SMIA). It allows interconnecting sensors and hosts of different vendors. Concerning the output of the coprocessor, several standard formats are available. For still images the most frequently used are the Joint Picture Expert Group (JPEG) with a lossy compression, the Targa Interchange Format (TIF) with a lossless compression. In the top level imaging devices the output of the sensors can also be stored directly by making use of a proprietary file format, such as the Nikon Electronic Image Format (NEF), the Canon RAW File Format (CRW), etc. For videos the most used are MJPEG, MPEG-4, H.263 and H264 standards. This chapter besides presenting the main data formats gives also a short description to the next JPEG XR Image Coding Standard. Moreover some techniques concerning the compression factor control and the error detection and concealment are introduced.

The image quality depends on compromises made in the design of the algorithms and devices for image capture, transfer, storage, and display. Quality assessment plays, consequently, a decisive role to successfully promote image processing algorithms and system performances. The evaluation of image quality is basically a human issue, but subjective metrics are computationally expensive and not practical for real-time applications. So, in the last decades, a great deal of efforts has been put into the development of objective quality metrics able to automatically predict perceived quality. To design an objective measure capable to be in close agreement with subjective test is a difficult task because the human visual system has a multifaceted structure and is not totally known. The tracked problem is to emulate human vision which is a cognitive activity and not a pure image sensor process. In this Chapter a classification of quality indexes, from classical to recent approaches is reported. For a better understanding of quality assessment topic, some of the principal human phenomena involved in the development of objective perceptual metrics are also explained.

In the recent years, technology advances in sensors and image processing have allowed significant improvements that encompass not only quality, but also image content analysis and understanding. Consumer devices are now being equipped with sophisticated algorithms allowing complex tasks such as face recognition, smile detection, automatic red eye removal, etc. In addition, topics related to image forensics oriented to verify the authenticity of digital images, such as camera identification and image manipulation detection are now available. This Chapter provides an overview of the main topics representing a challenge to innovation both from hardware and software point of view: Super Resolution, Temporal Demosaicing, Bracketing or High Dynamic Range Reconstruction. Also some aspects of Computational Photography and Forensics Camera Identification are briefly described. These topics represent the current trends and novel solutions for the next generation of imaging devices.

Full text available