[Interview] Making Your Screen Seamless: Experts Explain the Engineering Behind Galaxy Z Fold4’s UDC

Selfie cameras are a vital part of any smartphone — but finding the perfect place for the front-facing camera can often be an obstacle to seamless viewing experience. However, thanks to the Under Display Camera (UDC), users can say goodbye to the little black dot on the front of their screens. Samsung Electronics has given the UDC a serious upgrade in the newly-released Galaxy Z Fold4, offering users an even more immersive and uninterrupted full-screen experience.

Like so many great innovations, the latest UDC was a group effort — driven by Samsung’s experts and consumers’ valuable feedback. To learn more about the enhanced UDC for Galaxy Z Fold4, Samsung Newsroom spoke with the experts at Samsung Research,¹ who collaborated on the technology.

Read on to see what John Seok-Jun Lee, Head of Computational Camera Part at Samsung Research America (SRA), and Alok Shukla, Head of AI Vision Solutions Part at Samsung R&D Institute India-Bangalore (SRI-B), had to say about their collaboration behind bringing enhanced UDC technology.

Q: What is an Under Display Camera (UDC)?

Seok-Jun: An Under Display Camera (UDC) is an innovative camera solution that enables an uninterrupted viewing experience on a mobile device by hiding the camera under the display and dedicating the whole screen to users while applications are running. It not only requires hardware innovation by placing a camera under a display panel but also requires algorithm innovation for restoring image quality — one of the most complex image restoration problems.

Q: How does the UDC work?

Alok: In UDC, a front-facing camera sits underneath the display. The screen pixel density is reduced in this area, allowing the screen to display imagery unhindered — but at the same time, not block light from reaching the camera when it’s in use. As the camera is placed underneath the display, UDC can suffer from poor image quality caused by diffraction artifacts, which results in flare, saturation, blur and haze. Therefore, while the UDC brings a better display experience, it also affects camera image quality and other downstream vision tasks. These complex and diverse distortions make restoring UDC images extremely challenging.

Q: Why is the UDC an important technology when it comes to Galaxy Z Fold4? What value does it provide to customers?

Seok-Jun: Samsung’s foldables are designed to be new and different, combining smartphone and immersive viewing experiences into one mobile device via breakthrough display technology. When folded, it provides smartphone-like use, and when unfolded, the display provides a more immersive and productive experience. Our consumer research showed that Galaxy Z Fold series users prefer the Cover Screen camera for capturing photos and selfies and the immersive Main Display for more productive tasks like watching videos, playing games and working on documents. Therefore, we thought UDC technology would be the perfect fit to meet their needs for an uninterrupted screen while still being capable of tasks like opening video calls. Samsung has launched the first-ever foldable UDC solution in Galaxy Z Fold3 and enhanced its quality further in the Galaxy Z Fold4.

Alok: Galaxy Z Fold4 truly unfolds a world of immersive display. UDC technology is at the core of the pursuit of a true full-screen display. By combining the leading innovation in display technology with advanced AI techniques for image restoration, Galaxy Z Fold4 provides the perfect balance between screen quality and camera image quality.

Q: How has the camera’s image quality improved since Galaxy Z Fold3?

Alok: Samsung Galaxy Z Fold3 showcased the world’s first UDC on a foldable. After the market release of Galaxy Z Fold3, one of the key feedbacks from consumers was to make the UDC less visible. On the Galaxy Z Fold4, sub-pixel arrangement was optimized to further hide the UDC.

As mentioned previously, when the camera is placed underneath the display, this can result in poor image quality caused by diffraction artifacts, which cause flare, saturation, blur and haze.

That’s why, in Galaxy Z Fold4, we have leveraged the power of AI, backed with multi-frame processing, to develop a novel computational imaging pipeline for UDC image restoration.

SRI-B, along with Visual SW R&D Group at MX Business, focused on developing AI Image Restoration for photo capture and preview to compensate for optical diffraction in UDC. At the same time, SRA’s Mobile Processor Innovation (MPI) Lab improved the multi-frame computational photography algorithm. Galaxy Z Fold4’s UDC solves key technical challenges in image quality to bring a unique experience to end customers.

Q: Walk us through the journey of the SRA-SRI partnership project. Why did your two teams collaborate?

Seok-Jun: The SRA MPI Lab has been working with SRI-B throughout MPI’s entire history at Samsung, from 2014’s Galaxy Note4 camera commercialization all the way to 2022’s Galaxy Z Fold4 and Z Flip4. SRA MPI has been leading MX multi-frame processing (MFP) technology R&D for UDC, Night Mode, Expert RAW and other technology. SRI-B has contributed to all commercialized SRA MPI’s MFP by providing amazing camera software, image quality tuning and resources toward commercialization as well as AI camera R&D. Our 3-way collaboration with MX R&D group, SRA MPI and SRI-B has been a success — especially when the worldwide team used time zone differences effectively to complete milestones on time for such a complicated project. It’s always a pleasure to work with them and deliver Galaxy devices’ camera successes together to accomplish our mutual goal of being the world’s best smartphone camera experience.

Q: Please explain your job role and your team’s contributions to the UDC. How did you contribute to making the best possible user experience?

Alok: Samsung Galaxy Z Fold4 UDC was a complex R&D project, and SRI-B’s role was to think in terms of end-to-end for the best consumer experience. This required multi-domain expertise and deep collaboration among multiple teams. Experts from SRI-B, SRA-MPI Lab and MX Visual SW R&D Group worked seamlessly to realize an end-to-end UDC experience.

SRI-B also played a key role in utilizing a Neural Processing Unit (NPU) to execute the AI model, achieving real-time performance in previews and significantly faster capture processing time than in Samsung Galaxy Z Fold3 UDC. Camera previews, video recordings and video calling scenarios need real-time performance without losing any frames. AI-based UDC restoration accelerated with NPU for preview processing was applied for the first time in Samsung Galaxy Z Fold4. Although real-time processing was not required, as camera capture requires higher image quality than a preview, achieving better image quality with less processing time was desired for user experience.

Q: What were some challenges you faced while developing this technology, and how did you overcome them?

Seok-Jun: Low light transmittance and light diffraction are the main challenges for UDC camera technology. Still, one more challenge, which is different from a punch hole camera image pipeline, is that UDC capture quality is affected by not only the camera module but also the display panel. SRA MPI’s AI-based 3D ISP algorithm was designed with the needed flexibility to tune the output image quality with the different display characteristics. Having such flexibility in 3D ISP helped to overcome such challenges.

Alok: SRI-B’s key focus was developing an AI image restoration module for Samsung Galaxy Z Fold4 UDC. These restorations are required for photo capture mode as well as video mode. Video recording and video calling scenarios have strict requirements for real-time processing within the constraints of current consumption.

AI-based UDC restoration was applied for the first time in Samsung Galaxy Z Fold4 UDC for video. To achieve a better video quality experience than Galaxy Z Fold3, we had to make the AI model lighter while still achieving good restoration quality. Multiple experiments were conducted in network design and training to achieve good quality in real-time. SRI-B extensively utilized NPU and advanced quantization techniques to prepare a lighter, highly optimized AI model that achieves real-time performance with low current consumption.

Q: What is your vision for future UDC iterations at Samsung?

Alok: Samsung Galaxy Z Fold4’s UDC has opened up a new dimension to the smartphone camera form factor. UDCs in the future will enable placing larger camera sensors and multiple cameras at flexible positions without compromising the viewing experience. Another big challenge is towards mass adoption of UDC technology in more products as the main front camera. For this to happen, there will need to be a significant improvement in camera image quality, processing time and display experience.

Seok-Jun: Since the Galaxy Z Fold4 UDC provides a much better experience in terms of hiding the camera under the display and improving image quality, the R&D direction for the upcoming flagship devices will work towards making the UDC completely invisible. This will provide the best uninterrupted immersive mobile display experience while achieving the same image quality as a punch-hole camera.

¹ Samsung Research, acting as Samsung Electronics’ advanced R&D hub, leads the development of future technologies for the company’s Device eXperience (DX) Division.

Source: Samsung Mobile Blog
—