Real-time image processin is a resource-intensive activity that frequently calls for specialized technology. Let’s examine processors made especially for photo and video applications now that we have that in mind.
Digital signal processors (DSPs) and microcontrollers are the two broad categories into which embedded processors fall. Even though it’s getting harder to distinguish between microcontrollers and DSPs, it’s generally not too difficult to classify a component as either.
In contrast to a microcontroller, a DSP is designed for high-speed mathematical and data-transfer tasks, making it ideal for use in applications like artificial intelligence (AI), software-defined radio, and real-time audio processing. The processing ICs known as image signal processors (ISPs) will be examined in this article. It should be noted that while it would make sense to include ISPs as a subgroup under the DSP category, I believe it makes more sense to give them their own category as real-time image processing applications become more common and ISPs grow in strength and versatility.
The overall goal of this essay is to provide some instances of image, image signal, and media processors so that readers can grasp the subtle differences in corporate terminology and how difficult it is to distinguish between those several product categories.
Turn into the hub of image processing for mobile terminals: ISP’s industrial shift
The industrial transformation of ISP refers to the evolution of ISP chips from traditional image signal processing to AI visual processing to achieve smarter and more advanced image analysis and applications34. For example, ISP chips can support face recognition, target detection, scene understanding and other functions, providing more possibilities for smartphones, security, automobiles and other fields.
The industrial transformation of ISP is driven by many factors, including the following:
Image sensor development. As the resolution, frame rate, dynamic range and other parameters of the image sensor increase, the data volume and complexity of the image signal also increase, which requires the ISP chip to have stronger processing capabilities and higher efficiency.
Advances in AI technology. Artificial intelligence technology is increasingly used in the field of image processing, providing more value and meaning to images. AI technology can help ISP chips achieve more precise image optimization, richer image functions, and more flexible image configuration.
User needs. Users have increasingly higher requirements for image quality and experience. They not only require images to be clear, real, and beautiful, but also require images to reflect more information and content. Users also hope to achieve more interactions and applications through images, such as taking photos, live broadcasts, games, social networking, etc. 3 .
To sum up, the industrial transformation of ISP is an inevitable trend and an opportunity for innovation. As the core of image processing in mobile terminals, ISP chips will play a more important role in the future visual field.
The Requirement of Specific Hardware for Image Processing
Image processing is a resource-intensive task that often requires specialized hardware to perform. The image processor (ISP) is a processor specially designed for image and video applications. It can provide high-speed mathematical and data transfer operations, as well as some dedicated hardware modules, such as automatic exposure control, automatic focus, and automatic white balance. , object detection, image stabilization and noise suppression functions.
The demand for image processors mainly comes from the huge amount of image data. For example, a 24-megapixel SLR camera needs to process more than 24 million bytes of data to take a photo every second. Some image processing algorithms are also very complex, and the efficiency will be very low if only the central processing unit (CPU) is relied on to execute customized code. Therefore, image processors typically require architectures with parallel processing capabilities or multi-core SoCs.
The Introduction of the Image Signal Processor
Image Signal Processor (ISP for short) is a media processor or dedicated digital signal processor (Digital Signal Processor (DSP for short)) used for image processing in digital cameras or other devices.
The role of the image signal processor is to convert raw data (RAW) captured by an image sensor (such as CMOS) into an image format (such as JPEG) suitable for display or storage. The image signal processor can also perform some enhancements and optimizations on the image, such as automatic exposure, automatic white balance, noise reduction, color correction, contrast adjustment, edge enhancement, etc.
The image signal processor can be an independent chip or a module integrated in a system on chip (SoC). Different image signal processors may have different performance and functions, such as supported resolutions, frame rates, algorithms, interfaces, etc. Image signal processors are widely used in different markets, including high-end smartphones, security monitoring, games, automobiles, and medical fields.
Structure, Operation, and Terminology of Image Processors
The structure, working principle and terminology of the image processor are as follows:
Structure: The main components of the image processor are image sensor, image processor and image output device. An image sensor is a device that converts optical signals into electrical signals. There are two common types: CMOS and CCD. An image processor is a device that converts electrical signals into digital image signals. There are two common types: ISP and DSP. An image output device is a device that converts digital image signals into visual images. There are two common types: monitors and printers.
Working principle: The working principle of the image processor is through the following steps:
Acquisition: The image sensor receives the light reflected by the object, converts it into an electrical signal, and transmits it to the image processor.
Preprocessing: The image processor performs some basic processing on electrical signals, such as enhancement, filtering, correction, etc., to improve the quality and usability of the image.
Analysis: The image processor performs some advanced processing on the preprocessed image, such as segmentation, feature extraction, recognition, etc., to extract the information and meaning of the image.
Output: The image processor converts the analyzed image into a format suitable for the output device, such as RGB, YUV, etc., and transmits it to the image outputter.
Terminology: Image processors involve some professional terms, as follows:
Pixel: The basic unit of an image, which represents a point on the image. It usually consists of three color components (red, green, and blue). The value range of each component is 0-255.
Resolution: The size of the image, indicating how many pixels the image consists of, usually expressed in terms of the number of pixels in width and height, such as 1920×1080.
Color space: The color representation of an image, indicating which color components each pixel in the image is composed of. Common ones include RGB, YUV, HSV, etc.
Image format: The storage method of the image, indicating what file type the image is saved in. Common ones include JPEG, PNG, BMP, etc.
Image compression: The optimization method of images, indicating how images are used to reduce storage space. There are two common types: lossy compression and lossless compression.
Recap of Image Signal Processor and Further Resources
An image signal processor is a digital or mixed-signal integrated circuit designed to analyze and modify visual data. They are commonly used in devices such as cameras, mobile phones, tablets, etc. to improve image quality and performance1. ISP can implement and enhance many automatic and user-friendly functions, such as automatic exposure control, autofocus, automatic white balance, object detection, image stabilization and noise suppression.
The working principle of ISP is that it first receives raw color and brightness data from the image sensor, then processes the data according to different algorithms and parameters, and finally outputs an optimized image. The ISP processing process includes multiple steps, such as demosaicing, color correction, tone mapping, sharpening, noise reduction, etc.
There are many methods for the design and implementation of ISP, one of which is model-based ISP, which utilizes a learnable dictionary to represent the structural and texture information of images, thereby improving the efficiency and accuracy of ISP4. The other is ISP based on deep neural networks, which utilizes end-to-end training to learn the mapping from raw data to optimized images, thereby improving the expressive power and flexibility of ISP.
