Vertical Divider
Pixelworks Delivers Video and Display Solutions
March 29, 2020
As smartphone screens improve in visual quality, including color accuracy, color gamut, and brightness, many of the technical advancements in even the top-tier displays are now imperceptible or unnoticed. Panel vendors and OEMs are constantly trying to come up with new ways, besides “just make it bigger” and form factor, to make the most engaging part of your phone even more attractive. Pixelworks and their Iris visual processor aim to improve the smartphone display experience by integrating unique display features and adaptive elements based on human visual perception.
Pixelworks has mostly been low-profile in the smartphone space, but the company has been working with video and display solutions for about twenty years. They debuted their first partnership with a smartphone maker in 2016 with ASUS on the ZenFone 3 Ultra, integrating an early version of their Iris visual processor. Their most noteworthy smartphone collaborations to date include the Nokia 6.2/7.2, the ASUS ROG Phone, and just recently, the Black Shark 3 and the OPPO Find X2. The latter two phones include the newest fifth generation of Pixelwork’s Iris processor. The Iris 5, along with the company’s software which they coin “Soft Iris”, are responsible for the company’s display-facing features. The Iris chip is a display processor that sits between the device SoC and the display driver IC, connecting to both via the MIPI DSI, supporting dual MIPI. However, for me, this raises immediate concerns for display latency and video hardware acceleration: any bandwidth/stream compression requires additional decoding on the Iris chip so that pixel data can be processed, then it’s re-encoded and sent to the DDIC where it must be decoded again.
Figure 1: Pixelworks Processor and Customers
March 29, 2020
As smartphone screens improve in visual quality, including color accuracy, color gamut, and brightness, many of the technical advancements in even the top-tier displays are now imperceptible or unnoticed. Panel vendors and OEMs are constantly trying to come up with new ways, besides “just make it bigger” and form factor, to make the most engaging part of your phone even more attractive. Pixelworks and their Iris visual processor aim to improve the smartphone display experience by integrating unique display features and adaptive elements based on human visual perception.
Pixelworks has mostly been low-profile in the smartphone space, but the company has been working with video and display solutions for about twenty years. They debuted their first partnership with a smartphone maker in 2016 with ASUS on the ZenFone 3 Ultra, integrating an early version of their Iris visual processor. Their most noteworthy smartphone collaborations to date include the Nokia 6.2/7.2, the ASUS ROG Phone, and just recently, the Black Shark 3 and the OPPO Find X2. The latter two phones include the newest fifth generation of Pixelwork’s Iris processor. The Iris 5, along with the company’s software which they coin “Soft Iris”, are responsible for the company’s display-facing features. The Iris chip is a display processor that sits between the device SoC and the display driver IC, connecting to both via the MIPI DSI, supporting dual MIPI. However, for me, this raises immediate concerns for display latency and video hardware acceleration: any bandwidth/stream compression requires additional decoding on the Iris chip so that pixel data can be processed, then it’s re-encoded and sent to the DDIC where it must be decoded again.
Figure 1: Pixelworks Processor and Customers
|
|
Source: Pixelworks Promo
Figure 2: Pixelworks' Key Customers and Partners
Source: Pixelworks Promo
Furthermore, many video apps utilize display acceleration by directly rendering a chroma-subsampled pixel format through MIPI; I imagine that some of the Iris 5’s frame operations may require subsampled display data to be first converted to RGB before processing and decomposition. Brief preliminary findings of ours also hint that some of the Iris 5’s features don’t work with full hardware-accelerated video playback.
Many display “enhancements” that OEMs provide generally involve artificial and arbitrary picture adjustments that may skew the artistic intent of content. One common way that smartphone OEMs help their displays to stand out is by using a very vibrant color profile that expands all the colors on the screen to appear more saturated than originally intended. This is also usually accompanied by a blue-ish white point, which most consumers find more appealing than the standard white point known as D65. Both of these characteristics were originally artifacts of crude color calibration and the lack of color management in past displays, but improvements to both have been poorly received by many; accurate colors have a common perception as being the constraint of display color, while the standard white point appears warmer than what most people had been used to. Smartphone makers have had to continue to artificially oversaturate their screen color and to use a colder white point. This is done by many OEMs even to this day. Samsung had been notorious for shipping all their OLED phones with oversaturated displays, but they stopped this practice with the Galaxy S10 and they now ship them with an accurate color profile in most parts of the world (with exception of their in-store demo units, which understandingly defaults to that vibrant color profile to compare with the others that do, too). It takes more than a “well-calibrated” display for an accurate viewing experience. Lighting conditions in a display’s viewing environment can significantly alter the look of content on the display. For content to look right, it should ideally be viewed in the environment that it was mastered for. And for this reason, color standards that displays conform to also impose a reference viewing environment for which those colors appear accurate. When viewing the display in other environments, the colors on the display may appear incorrect. Thus, a “well-calibrated” display should also be calibrated for its viewing environment.
Smartphones, however, are used in all sorts of viewing environments: outside in the bright sunlight, in bed at night, or perhaps in a venue with multi-colored lights. All these different environments change the perceived look of the content on your smartphone’s screen. Pixelworks focuses on improving display accuracy in these real-world conditions and recreating the artistic intent of the content creator. Instead of frivolous boosts to image contrast/quality/saturation, Pixelwork’s solutions are based on improving content fidelity by adjusting the display and its contents to adapt to ambient conditions. Pixelworks and their Iris 5 chip does the following:
Many display “enhancements” that OEMs provide generally involve artificial and arbitrary picture adjustments that may skew the artistic intent of content. One common way that smartphone OEMs help their displays to stand out is by using a very vibrant color profile that expands all the colors on the screen to appear more saturated than originally intended. This is also usually accompanied by a blue-ish white point, which most consumers find more appealing than the standard white point known as D65. Both of these characteristics were originally artifacts of crude color calibration and the lack of color management in past displays, but improvements to both have been poorly received by many; accurate colors have a common perception as being the constraint of display color, while the standard white point appears warmer than what most people had been used to. Smartphone makers have had to continue to artificially oversaturate their screen color and to use a colder white point. This is done by many OEMs even to this day. Samsung had been notorious for shipping all their OLED phones with oversaturated displays, but they stopped this practice with the Galaxy S10 and they now ship them with an accurate color profile in most parts of the world (with exception of their in-store demo units, which understandingly defaults to that vibrant color profile to compare with the others that do, too). It takes more than a “well-calibrated” display for an accurate viewing experience. Lighting conditions in a display’s viewing environment can significantly alter the look of content on the display. For content to look right, it should ideally be viewed in the environment that it was mastered for. And for this reason, color standards that displays conform to also impose a reference viewing environment for which those colors appear accurate. When viewing the display in other environments, the colors on the display may appear incorrect. Thus, a “well-calibrated” display should also be calibrated for its viewing environment.
Smartphones, however, are used in all sorts of viewing environments: outside in the bright sunlight, in bed at night, or perhaps in a venue with multi-colored lights. All these different environments change the perceived look of the content on your smartphone’s screen. Pixelworks focuses on improving display accuracy in these real-world conditions and recreating the artistic intent of the content creator. Instead of frivolous boosts to image contrast/quality/saturation, Pixelwork’s solutions are based on improving content fidelity by adjusting the display and its contents to adapt to ambient conditions. Pixelworks and their Iris 5 chip does the following:
- Factory display calibration
- Real-time motion processing on the Iris
Source: Pixelworks
- Motion processing, when done right, is a key component in reducing judder and in dealing with framerate mismatch. Pixelworks emphasizes that this is not to be confused with generic motion interpolation, which results in the nefarious “soap opera effect“. We’re told that their motion processing preserves the intended motion appearance of content and adapts it to the type of content and to the environment. This is important since many films are not intended to have super-smooth motion, while perhaps a sports stream should. Additionally, the perception of judder increases with the contrast of the content, and contrast is further affected by the viewing environment. Pixelworks claims that they compensate for both these factors in their motion processing. This is especially important for HDR content, which has the potential for very high contrast. The motion processing is also said to work for mobile games. Pixelworks has previously won the Hollywood Professional Association (HPA) Award and the Award for their TrueCut motion grading video platform used in cinema films.
- Automatic display white balance & contrast adjustments
The appearance of content on a display changes depending on the viewing environment. Pixelworks deals with the effects that ambient brightness and color have that changes your perception of colors on a display. A warmer viewing environment will make a display’s white balance appear relatively colder — and vice versa– an effect known as chromatic adaptation. To compensate, a display should adjust its colors towards the color of the ambient light so that the display appears perceptually similar under different color lighting. Many phones now provide this feature, arguably popularized by Apple’s True Tone which was introduced in their 9.7-inch iPad Pro. However, True Tone only adapts to ambient color, while ambient brightness has a further effect on the contrast of a screen. The brighter the ambient light (relative to the display), the darker the colors on the screen appear, compressing towards black. And the brighter the display (relative to the ambient light), the lighter the colors on the screen appear, compressing towards white. This is known as the Bartleson-Breneman effect, and Pixelworks is capable of compensating for this by adjusting the system gamma and using local contrast enhancement. However, the details of correctly implementing this are intensely complicated, from perceptual measurements to the mapped display pixel values. Samsung is one other smartphone OEM that considers this, though only for sunlight high brightness mode. - DC dimming to prevent OLED flicker
In most phones with an OLED screen, the display brightness is adjusted by quickly flickering the screen on-and-off, a method called pulse-width modulation (PWM). For most phones, this flickering happens at about 240 times per second, and the resulting display brightness depends on how long the display is in the “on” state. However, some consumers claim to have induced headaches from this flickering, and it’s exacerbated at lower display brightness levels. DC dimming attempts to remedy this by instead adjusting the display brightness via traditional analog control. While this eliminates the flickering, display calibration and uniformity can be negatively impacted since changes in an OLED’s voltage can significantly alter its output characteristics. However, we may be seeing OLEDs with higher DAC bit-depths, which can rely more on the current to adjust the amplitude of the individual LEDs. - SDR-to-HDR conversion for videos and games
The Iris 5 is capable of converting standard content into HDR. Pixelworks declined to state how exactly they’re doing this, but it’s done in real-time on the Iris 5 DSP. We’re told that the HDR output format is based on HLG and that that conversion works alongside their other adaptive features. I wrote a whole precursor about arbitrary display enhancements, so this feature sort of feels like a slap to the face. Pixelworks rationalizes that there is a drought of HDR content that’s available and that this feature allows us to take advantage of our display hardware’s current capabilities. However, I’ll reserve harsher judgment until I can demo this feature.
Figure 3: Pixelworks’ Auto-Adaptive Displays
Source: Pixelworks
- HDR10 for low- and mid-range devices
While HDR10 is typically reserved for more-premium devices, Pixelworks can collaborate with mid-range and budget devices to bring “certified HDR” to the masses. The Iris 5 supports native 10-bit processing, which most budget SoCs don’t support. Cheap mobile displays are now capable of at least the 95% DCI-P3 and about 400 nits with 1000:1 static contrast, which can serve a passable mobile HDR experience for the price. In addition, for LCDs the Iris 5 can manage the display backlight, employing dynamic backlight control for reduced power consumption and improved dynamic contrast. Most LCDs are already capable of this, however.