Why Mobile Could Be The Primary Driver of 4K Experiences

As I begin my journey to this year’s Mobile World Congress in Barcelona, I am thinking through the major trends and inflections that will most impact mobile technology and usage models going forward. One of those I wrote about last week was quick charging. What I would like to talk about now is 4K capabilities on mobile devices and how mobile could be the primary driver for 4K. Let me start with explaining what 4K is and how it applies to mobile devices. I also wanted to touch on some interesting things Qualcomm is doing in this arena.

What is 4K?

Most of today’s TVs, many tablets, and some phones today display at what is called “1080p”. For a phone, this would mean that it displays 1,080 horizontal pixels by 1,920 vertical pixels, combining for around 2 million pixels (2M).  “4K” devices display 4X the amount of pixels, 8 million (8M), driven by a resolution of 3,840 vertical and 2,160 horizontal pixels. Another related and arguably more important measure of merit has become “pixels-per-inch” or PPI as this describes how dense the pixels are in each square inch. My 5.7”, 1,920 x 1,080 Samsung Galaxy Note 3 has 386 PPI while my 9.7”, 2,048 x 1,536 iPad Air has a PPI of 264. Net-net, higher resolution does not necessarily mean higher picture quality, as you need to factor in varying display sizes and viewing distance

Why 4K-Derived PPI matters

My first reaction when I heard of 4K displays on phones and tablets was skepticism, until I started researched the topic. First of all, many in the industry generally agree that humans can perceive differences in max PPI ranging roughly from 350 to 575 PPI at a ten inch distance. The better the vision, the higher thevisual acuity, and my Note 3, at nearly half the maximum, has a ways to go. Personally, I am painfully near-sighted and I can see pixels on a phone or tablet where others cannot. Others in the industry (Sharp and Qualcomm) and some researchers are talking about vernier acuity, or the ability to see the differences among two line segments. This suggests that human’s visual system is much more complex, indicating that humans can assess differences 5-10X the detail of visual acuity.

LG Pro 2Regardless of what theory you buy into, I’m very safe saying that many consumers will be able to distinguish the difference between a 386 PPI device, a 441 PPI tablet (10”, 4K) and a 774 PPI phone (5.7”, 4K). In the end, you have to determine if it matters to you. If you are at Mobile World Congress, you can see this in action at Qualcomm’s booth. Now let’s talk about content.

Where’s all the 4K Content?

Ironically, the biggest generator of 4K content is probably in your pocket already. Most all modern smartphones take 8 mega-pixel (8MP) pictures so you are most likely taking 4K pictures already. The problem is that your phone or tablet display is lower resolution than your pictures and therefore displays them at that lower resolution. Some smartphones also take 4K videos, too like my Note 3, and LG just announced their new G2 phones will have the same capability. Watching content captured in 4K is best viewed on a 4K display because it eliminates what is called “scaling artifacts”, image defects created by downscaling content from 4K to 1080P.

4K paid video content is a much easier story to tell in North America after this year’s CES:

  • Netflix: announced they would stream in 4K to devices supporting HEVC (H.265) in 2014
  • Amazon: announced they would provide Prime Instant subscribers access to 4K content
  • YouTube: announced they will support 4K using their VP9 video format. You can see 4K YouTube videos here.

I will be anxiously awaiting their specific mobile app announcements.

Other benefits for 4K exist, namely improved text legibility and games.

The English alphabet is pretty easy to render on devices, but that is not the case with other alphabets. In particular, Asian characters look very blocky and jaggy (aka aliased) on a low PPI display and having more lifelike text as you are reading an e-book on a 4K tablet could be a big benefit. I urge you to try 4K mobile devices to see if you see a difference.

Let’s talk games. Even the latest game consoles only render internally to 1080p (out to 4K) and mobile games would have the capability to upscale to the native resolution of a 4K tablet or phone. To me, the benefit is simple; the higher the resolution and PPI, the more detailed the game is and more enjoyable the experience.

Finally, I want to point out that everything on the 4K mobile display could be at 4K, providing the opportunity for OEMs and ISVs to create much more immersive environments.  Apple already proved the power of this with their Retina Display.

But is it a Good 4K Experience?

There will be many different quality levels of mobile, 4K experiences delivered at MWC. 4K isn’t new as vendors like NVIDIA have been shipping 4K out of Tegra 4 devices for a while. What’s new at MWC will be the implementations.

Qualcomm has taken a holistic approach to mobile 4K on the Snapdragon 805, one they are calling a “full 4K experience”. It enables consumers to view 4K on their mobile device display and also connect to a 4K HDTV. Additionally, all of the content is 4K and “post-processed” for quality; every video, text, overlay, UI, game, and picture.

Adding new and desired capabilities to any product is a good thing unless it is counter-balanced by something negative. Adding a rocket engine to the family mini-van is a great thing for speed to pick up groceries, but you’ll need to upgrade your brakes, fuel system, dashboard, steering wheel, air bags and you may need to add wings and a crash cage. The same thing goes for adding 4K to a phone or tablet. Just adding 4K to a mobile device without considerations for the holistic experience could result in a bad experience with jaggy text, stuttering video or limited battery life. As you are running 4X the data through the mobile SoC, something must change.

Technically, Qualcomm’s taking a mobile heterogeneous computing approach with 4K. I’ve written a lot on heterogeneous computing and it sounds complex, but I like to explain it through a golfing analogy. Golfers use different clubs for different strokes and lies just like mobile designers can use different processing engines for specific workloads. You wouldn’t putt with your driver nor would you decode 4K video with your CPU. Qualcomm has used different, custom-designed heterogeneous compute “engines” to tackle specific jobs, then integrated them together to deliver a holistic experience. I am most familiar with what they have done on the Snapdragon 805, so let me outline what they have done to optimize for 4K:

  • 4K Multimedia engines: 4K video runs (or is decoded) on a new and specialized, “fixed function” silicon block and supports H.265 video playback at 30fps. H.265 is the new standard for lower bandwidth-higher quality video. Running 4K video entirely on a CPU is not the right approach for mobile with limited power and thermal envelopes. A newly introduced engine also runs very sophisticated video quality enhancements based on technology from the recently acquired HQV (Hollywood Quality Video) company. Finally, to actually display an image at 4K on the phone or tablet’s display, Qualcomm uses a special display engine.
  • 1.2 GPixel/s camera engineKnown in the industry as an ISP (image signal processor), the 805 enables sharp, higher-quality, higher-resolution photos with pretty advanced pre and post-processing noise reduction features for low-light conditions in both snapshot and video modes. It has what is called a “dual pipeline”, which provides support for image resolution up to 55 million (50M) pixels, picture-in-picture from dual cameras, 4K video encode for taking videos and adding a high resolution snapshot.
  • GPU: The Adreno 420 delivers support for up-scaled games for 4K displays as well as advanced 3D graphics capabilities like “hardware tessellation” and “displacement mapping” found in the latest game consoles. Hardware tessellation enables greater detail for more visually realistic scenes in a manner that requires less memory bandwidth and lower power consumption.
  • Security engines: With new 4K content comes an increased vigor to protect it. The 805 uses a pre-integrated security solution called “Snapdragon StudioAccess” which protects both the compressed and uncompressed premium video. Hardware-based access control is enforced on each component that touches 4K content like the video decoder and the cryptographic engine. The chipset use what Qualcomm calls the SecureMSM architecture which provides fundamental security services at boot-time and while playing content.
  • Memory: 4K data density is 4X larger than 1080P, so you need a wide highway to move it back and forth between “engines”. The Snapdragon 805 added a memory controller that is as wide as many notebooks on the market today and does it in a way that maximizes battery life for mobile use cases. Qualcomm runs the memory controller like a smart, HOV Autobahn, only opening the lanes it needs and directing cars to pack closely together.  This results in less power needed to run the highway.

As you can hopefully gleam from this, getting to a much better mobile 4K experience isn’t about slapping on the jet engine, but architecting it from the start.

Mobile as the Primary 4K Driver?

Getting back to the headline of this column, I ask, “could mobile be the primary 4K driver for the industry?” I have personally participated in many content transitions and a few characteristics come to mind as “drivers”:

  • Content: plain and simple, no content, no transition. Mobile arguably has more content today with photos and video than PCs or TVs and the first 4K video services, OTT, are devoured on mobile devices. Yes, premium content with TVs are first out of the gate, but mobile apps will come shortly afterward.
  • Volume: content producers want high volume to spread their expensive content across. With TVs getting upgraded every 8-10 years and selling in the tens of millions, PCs upgraded every 5 years selling in the 100s of millions, and smart phones upgraded every 2-3 years selling in the billions, phones have the 4K volume potential advantage.
  • Demonstrable improvement: On large TV and PC displays, there is no doubt that 4K looks better, particularly at anything less than 10 feet (TV) and 2 feet (PC). For mobility, industry experts peg the max demonstrable PPI between from around 350 to 575 and some researchers say we can perceive 5-10X this through “vernier acuity”. All of this is well within the range of 4K. I urge you to see it yourself and decide on your own.
  • Marketing Ease: Anytime consumers are faced with too many “if-thens” or contingencies on that big new feature for that “next big thing”, they don’t buy. They are paralyzed. 3D TV was a great example; “if” I have the right TV, “if” I have the right content player, “if” I’m the 90% who doesn’t get sick, “if” I have enough glasses for everyone in the room, “if” I have the right content, etc. A 4K tablet has 4X the display resolution and every decent smartphone takes photos in 4K (8MP) already. How simple is that?

When I factor all these things in, I have to conclude that there is a very high probability that mobile will be the primary driver to 4K. I hope to see you in Barcelona at Mobile World Congress.  Let me know what you think below.