How do smartphone cameras work?

The first camera-equipped cellphone reached the United States in 2002. Since then, smartphone manufacturers have competed to improve the quality and capabilities of mobile cameras. When smartphones became mainstream around 2008 and social media rapidly expanded, having a capable phone camera became increasingly important for people who wanted to capture and share everyday moments.

Before smartphone cameras became responsible for a huge portion of the world's photographs, taking pictures usually required devices that were considerably larger than the phones that eventually replaced them. So how can such a small device produce photographs with resolutions reaching 200 megapixels?

Every smartphone camera is built around three fundamental components. The first is the lens, which directs incoming light toward the camera. The second is the image sensor, which transforms those focused photons into electrical signals. The third is the software, which processes those electrical signals and turns them into a finished photograph that can be shared on social media. Each component plays an important role in creating the final image.

Lenses

Before light reaches the image sensor, it has to travel through the camera lens. Before entering the lens itself, the light first passes through a small opening in the phone's body. This opening is known as the aperture, and its size determines how much light can enter the camera and reach the sensor.

In general, a larger aperture is beneficial for smartphone photography because it allows the sensor to receive more light.

Aperture is expressed using f-stops. The f-stop represents the relationship between the focal length of the lens and the physical diameter of its aperture. As a result, an f/1.7 aperture is wider than an f/2 aperture.

After entering the camera module, light from the scene passes through the lens system, which gathers and focuses it onto the image sensor. Smartphone cameras generally use multiple small plastic lens elements rather than a single lens.

Light behaves differently depending on its wavelength. Different wavelengths, which correspond to different colors, bend at different angles as they pass through a lens. This can cause the various colors in a scene to reach the sensor at slightly different positions.

Multiple lens elements are therefore used to help correct this and other optical effects and ensure that a clearer image reaches the sensor.

Focus

One of the important jobs performed by the camera's lens system is focusing. Although this process is usually handled automatically, some smartphone camera applications also allow users to manually adjust focus.

In most cases, however, the phone determines focus automatically through software, additional hardware such as a laser distance sensor, or a combination of both approaches.

Software-based autofocus, often referred to as passive autofocus, uses information from the image sensor to determine whether the scene is properly focused. The system then adjusts the position of the lenses accordingly.

A common passive autofocus technique relies on image contrast. The camera changes the lens position until the detected contrast reaches its maximum. Because this approach primarily depends on software, it is relatively inexpensive to implement. However, it can be slower and less effective when lighting conditions are poor.

Another autofocus technique is phase-detection autofocus. This method works by directing comparable amounts of light toward two closely positioned sensing areas on the image sensor.

Traditional PDAF systems use dedicated photosites, which account for roughly 5% of the sensor's total photosites. These specialized sites measure light arriving from different sides of the lens. When the light intensity measured by the corresponding sites is the same, the image is considered properly focused.

If the two measurements differ, the camera can determine how far the image is from being focused and adjust the lens accordingly. This allows phase detection to operate more quickly than contrast-based autofocus.

Modern PDAF systems can make use of all of the photosites on an image sensor. For example, newer systems can analyze photosites facing both the top and bottom of the sensor in addition to those positioned toward the left and right. This provides more information for determining focus.

Active autofocus takes a different approach by using additional hardware to estimate the distance between the phone and the subject. Early active autofocus systems used sonar-like technology, similar to the Soli system found in some Google Pixel phones.

Later developments introduced low-power infrared lasers for distance estimation. However, these systems can sometimes be affected by other infrared sources, including flames or nearby smartphones that emit infrared light.

Sensor

The image sensor is a thin piece of silicon whose primary function is to transform incoming photons into electrons, which become electrical signals.

This conversion takes place across millions of photosites distributed over the small surface of the sensor. When little or no light reaches a photosite, the corresponding pixel is recorded as black. When a large amount of light reaches it, the pixel becomes white.

The number of different shades between black and white that a sensor can distinguish is related to its bit depth.

So how does a sensor produce a color photograph if its basic light measurements are essentially monochromatic?

A color filter is placed over each photosite. These filters allow specific wavelengths of light to pass through while blocking others. The Bayer color filter array is the most widely used arrangement.

In a standard Bayer CFA, every 2×2 group of photosites contains one red filter, one blue filter, and two green filters. This configuration is commonly represented as RGGB.

Huawei's SuperSpectrum sensors provide a notable alternative to the traditional RGGB arrangement. Instead of using two green filters, Huawei's RYYB design uses yellow filters. This allows more light to reach the sensor and can provide additional luminance information.

The sensor therefore collects a mosaic of color and brightness measurements rather than a complete full-color image. Sophisticated demosaicing algorithms are later used to transform this raw information into a finished color photograph.

Two particularly important specifications when evaluating an image sensor are the overall sensor size and the physical size of its individual pixels or photosites.

Sensor dimensions are commonly expressed as fractions of an inch. Generally, a larger sensor can produce better image results because it provides more physical space for larger photosites.

The photosites themselves are measured in micrometers, or µm. The total number of photosites determines the potential megapixel resolution of the camera.

Larger photosites are generally more effective at collecting light, which makes them especially useful in dark or low-light environments.

Smaller photosites do not automatically mean that a camera will produce poor photographs. However, their size is worth considering if you expect to take many pictures in challenging lighting conditions.

Samsung addresses the challenge of fitting extremely large numbers of small photosites onto a sensor through a technique called pixel binning. On some of its high-resolution sensors, groups of 16 photosites arranged in a 4×4 pattern can be combined and treated as one larger pixel. This allows the system to collect more luminance information from the combined area.

Image stabilization

A steady camera is one of the most important requirements for producing a sharp photograph. Since smartphone users are unlikely to carry a tripod everywhere, phone manufacturers have developed various technologies to reduce the effects of hand movement.

Image stabilization generally falls into two main categories: optical image stabilization and electronic image stabilization.

Optical image stabilization, or OIS, uses a gyroscope to detect movement of the phone. Small motors or electromagnetic mechanisms then shift the lens or image sensor in the opposite direction to compensate for that movement.

OIS is particularly useful when taking pictures in low-light environments because the camera may need a longer exposure to collect enough light.

Electronic image stabilization, known as EIS, uses the phone's accelerometer to identify movement. Rather than physically moving the lens or sensor, the system adjusts the captured frames or exposures to compensate for motion.

Because EIS aligns exposures according to the content within the image rather than the physical boundaries of the sensor frame, the resulting photo or video can have lower resolution.

Many modern smartphones combine optical and electronic stabilization in a hybrid system. By using both approaches simultaneously, phones can achieve improved stabilization and particularly benefit video recording.

Software

Once the lens has directed light onto the sensor and the sensor has converted that light into electrical information, the image signal processor takes over.

The ISP's role is to transform the raw numerical data generated by the sensor into a finished image suitable for viewing and sharing.

The information initially delivered to the ISP is essentially a monochrome image. Photography enthusiasts often refer to this type of original sensor data as a RAW image.

One of the first jobs of the ISP is to reconstruct the color information according to the known arrangement of the camera's color filter array. At this stage, the image contains pixels representing different intensities of red, green, and blue, or red, yellow, and blue in the case of certain Huawei sensor designs.

The next major stage is known as demosaicing. During this process, the ISP analyzes the color information of neighboring pixels and adjusts their values to create a complete color image.

For example, if an area contains a large amount of red and green information but relatively little blue, the demosaicing algorithm can infer that the area is likely to contain yellow and adjust the resulting pixel colors accordingly.