Huawei Kirin 980 Looks Good On Paper But AnandTech Benchmarking Accusations Could Indicate Otherwise

By Patrick Moorhead - September 5, 2018

Update: See Huawei's prepared statement at the end of the article.

Huawei has been on an up and down roller coaster related to smartphones as of late. On one hand, it achieved the number two global smartphone unit market share position surpassing Apple, only second to Samsung. Its premium smartphone market share is up, too, so it’s more than Huawei selling just cheap phones, and based on Moor Insights & Strategy analyst Anshel Sag’s analysis, the company has the best smartphone camera at this moment with the P20 Pro.  I love the P20 Pro's 3X physical zoom lens. Last week at IFA, Huawei announced the Kirin 980 processor and characterized it in its press release as having “absolute superiority” and on the big IFA stage, displayed a bunch of superior benchmarks versus the chip competition like Qualcomm and Apple. On the other hand, Huawei’s smartphones were essentially banned in the U.S. over security concerns while other China-based smartphone makers remained on U.S. carrier shelves, and now Australia and Japan are reportedly investigating potential security issues with Huawei products. And just yesterday, Anandtech ran a piece that accused Huawei of “cheating” on benchmarks and laid out an ugly case where the test phones with the Kirin 970 went into a special benchmark mode only when running benchmarks that a user would not ever achieve, and without warning to the reviewers. So, you can imagine, the initial excitement over the Kirin 980 is waning as it’s hard to know which claims made at the announcement were real or only applicable when running in a special, undocumented “benchmark mode.” Let’s start with Huawei’s claims at IFA… At IFA, Huawei made some exciting claims about the new Kirin 980. In the press release, Huawei characterized the 980 as having “absolute superiority” and was the “ultimate engine to power next-generation productivity and entertainment applications” delivering “75% more powerful” CPU and “46% greater graphics processing.”  All of this made a bit of sense to me as the Kirin 980 featured Arm’s new Cortex A76 CPU and Mali G76 GPU has dedicated math matrix silicon for ML inference, was fabbed on 7nm and supported LPDDR4X. Finally, Huawei said the Kirin 980 will pair up with the 5G Balong modem sometime in 2019. But the story seemed to fall apart at IFA as comparisons were made to the Qualcomm Snapdragon 845 and the Apple A11. Huawei said the Kirin 980 outperformed the Snapdragon 845 with 37% better CPU performance with 32% higher power efficiency. Anandtech reported in an earlier article that the numbers used for CPU power efficiency were Dhrystone, which the publication characterized as “putting doubt into the above estimates.” This was even before Huawei was caught what was characterized as “cheating” by Anandtech. Huawei showed that the Kirin 980 had nearly 2X the image inference performance of the Snapdragon 845 and 3X that of the Apple A11. Finally, Huawei bashes Qualcomm on WiFi performance showing 2X the throughput. And then AnandTech’s Ian Cutress and Andrei Frumusanu researched and ran a piece entitled, “Huawei & Honor’s Recent Benchmarking Behavior: A Cheating Headache”, a bombshell of a piece laying out evidence that Huawei “cheats” on the way its recent smartphones perform benchmarks. I urge you to read the article in entirety, but here are a few snippets:
  • “newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom”
  • “We see a stark difference between the resulting scores – with our internal versions of the benchmark performing significantly worse than the publicly available versions.”
  • “What Huawei instead is doing is boosting benchmark scores by coming at it from the other direction – the benchmarking applications are the only use-cases where the SoC actually performs to its advertised speeds. Meanwhile every other real-world application is throttled to a significant degree below that state due to the thermal limitations of the hardware.”
  • “using the 'cheating' variants of the benchmarks completely explodes the power budget”
  • “This means that the 'true performance' figures aren’t actually stable.”
  • “Looking back in hindsight, it is pretty obvious there’s been some less than honest communications with Huawei.”
Anandtech reran the tests with the “cheats” off and concluded that “overall, the graphs are very much self-explanatory. The Kirin 960 and Kirin 970 are lacking in both performance and efficiency compared almost every device in our small test here.” These were comparisons that included many different flavors of Qualcomm Snapdragons, Samsung Exynos and A-series Apple SoCs. The tests showed that at the extreme, the OnePlus 6 with the Snapdragon 845 outperformed the Honor Play with the Kirin 970 by 3X on the sustained 3DMark Sling Shot 3.1 Extreme Unlimited. 3X the sustained performance! AnandTech tests showed that “cheating” vs “non-cheating” scores could vary by 50%.  ANANDTECH Hopefully, you can see why I’m a bit skeptical of the claims made at IFA on the Huawei Kirin 980 SOC. Huawei must feel immense pressure at this time and is doing its best to paint a good picture headed into the western holidays with the upcoming Apple launch and its Android competitors who rely on Qualcomm, who I believe will shortly launch its new 7nm SOC for 2019. Huawei's biggestpressure could be that its Android competitors will likely ship 5G handsets before Huawei. Qualcomm is on record saying every Snapdragon 845 customer is doing a 5G handset for launch in 2019, has shown off tiny digital chips and mmWave antennas and even a smartphone design with mmWave 5G capabilities. Huawei has talked about the Balong 5G modem for 2019 but the only 5G silicon I have seen from Huawei was the size of a desktop processor, looking as if it were for a base station, nor have I seen smartphone test vehicles. I have not seen Intel 5G smartphone silicon yet either, but at least the company showed off 5G in a PC form factor in a detachable kickstand at Computex. Anyways, I believe being late with 5G is likely Huawei’s biggest risk versus benchmark controversies but it could answer why the company is being so aggressive. We'll have to wait for the credible reviewers who run benchmarks and do evaluations to say whether Huawei delivered on its claims made at IFA.  For the sake of consumers, I hope the Kirin 980 performs well and its phones deliver a good experience. I’d like for Huawei to also pop up a notification during benchmark mode saying it has entered that mode to be more transparent. I reached out to Huawei today for comment but did not receive a response. UPDATE: here is Huawei's response in full: Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.  When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed. In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to. Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.
Patrick Moorhead
+ posts

Patrick founded the firm based on his real-world world technology experiences with the understanding of what he wasn’t getting from analysts and consultants. Ten years later, Patrick is ranked #1 among technology industry analysts in terms of “power” (ARInsights)  in “press citations” (Apollo Research). Moorhead is a contributor at Forbes and frequently appears on CNBC. He is a broad-based analyst covering a wide variety of topics including the cloud, enterprise SaaS, collaboration, client computing, and semiconductors. He has 30 years of experience including 15 years of executive experience at high tech companies (NCR, AT&T, Compaq, now HP, and AMD) leading strategy, product management, product marketing, and corporate marketing, including three industry board appointments.