The world of smartphone SoCs is extremely cutthroat and everyone wants to say they deliver the fastest possible performance while also having the lowest power. These two goals are always at odds with each other because, higher performance typically, all things equal, means more power consumption and worse battery life. Because a phone generally has a limited source of power in a non-removable battery, short battery life is generally not an accepted compromise for higher performance smartphones, not to mention thermal limitations. This is what makes smartphone SoC design so difficult and why so few companies even bother building their own smartphone chip. It’s hard.
Earlier this month, one of the industry leading technology news and review websites, Anandtech, said that MediaTek was cheating on their benchmarks. This wasn’t the first time that Anandtech had caught an SoC vendor “cheating” on benchmarks. In the past, Andrei tipped off Brian Klug and Anand Shimpi who were at Anandtech at the time to Samsung’s Exynos SoC GPU optimizing for GPU benchmarks. That eventually blew up into an industry-wide article that called out almost the entire smartphone industry for cheating. This will be important later, because of how MediaTek has chosen to address Anandtech’s claims about their cheating.
So, what did Anandtech’s Andrei Frumusanu find, exactly? He found that MediaTek was cheating on benchmarks by creating a whitelist of applications that would run the device in a special ‘Sports Mode’ that significantly boosts the performance of a device beyond it’s real world day to day capabilities and would be unsustainable in day to day usage. His discovery started by getting two different phones, the OPPO Reno 3 Pro which ships with a MediaTek Helio P95 in Europe and The Reno 3 which ships with MediaTek’s latest and most powerful Dimensity 1000 chip in China. But for some reason, the Chinese Reno 3 Pro with the faster SoC got worse benchmark scores than the European version with the Helio P95 and then he went deeper. Andrei found that with the cheating turned on and off that the performance difference in benchmarks was 30% and up to 75% in some subtests. Anandtech even reached out to UL to confirm their findings and they gave them an anonymized version of their PCMark benchmark to validate their results. Andrei also broke out a Chinese model of the Reno3 Pro with a Snapdragon 765G, but even though it was from the same manufacturer and the same model as the Helio P95 device, it did not exhibit the same behavior.
Eventually, Andrei discovered that the cheating mechanism had been hiding in plain sight for years in some innocuous looking code. Hidden in the firmware there is an .xml file with a list of popular applications with various power management tweaks, including a list of popular benchmarks that would kick off the MediaTek ‘Sports Mode’. Andrei found that many MediaTek devices employed this benchmark cheating scheme, but not always on the same applications. They found this behavior went as far back as the Helio P20 on a Sony XA1 which came out in 2017. Anandtech reached out to MediaTek regarding their findings and got this response:
MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.
Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.
We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.
I have been a smartphone reviewer and have been running benchmarks on phones for over a decade and have seen lots of bad behavior. But I have never seen an SoC vendor effectively punt its responsibility to their OEMs like Huawei, LG, Motorola, Nokia, Oppo, Realme, Samsung, Vivo and Xiaomi. and effectively shirk any responsibility for enabling their OEMs to “cheat” on the benchmarks of their choice. MediaTek created the defeat device (to use the VW emissions cheating scandal terminology) and enabled their OEMs to decide which benchmarks they wanted certain devices to run in ‘Sports Mode’. Other SoC vendors and smartphone vendors that got caught “cheating” in the past have owned up to their behavior and cleaned up their acts. Hopefully, MediaTek’s OEMs that used this defeat device ‘Sports Mode’ will do the same, but MediaTek’s doesn’t seem to back down from this at all. In fact, the company published a blog titled ‘Why MediaTek Stands Behind Our Benchmarking Practices’ on their website shortly after the Anandtech article went live.
MediaTek’s blog starts off talking about being clear about their benchmarking practices and being transparent about how they approach benchmarking, but there was very little transparency shown. Anandtech’s own article went into more depth of how MediaTek’s ‘Sports Mode’ works and what applications were on the list. MediaTek’s blog merely accuses the OEMs of implementing it of their own volition and that their practices are industry standard. Not only this, MediaTek also accuses unnamed competitors of doing the same thing, which we can only assume is Qualcomm. And of course, Qualcomm just came out with a response to MediaTek’s claims. In a statement to Android Authority, Qualcomm stated that, “Whitelisting refers to the technique of using the app name to determine whether to put the device into performance enhancement mode,” the company explained. “The action of whitelisting a benchmark app is generally considered by the industry as cheating since it defeats the purpose of a benchmark, which is to reflect user experience for day to day use. Qualcomm does not whitelist.”
This sounds cut and dry to me from Qualcomm, and I don’t think Qualcomm needs to cheat to provide good performance per watt, so that really leaves MediaTek in a very peculiar position. I agree with Qualcomm’s stance that whitelisting benchmarking apps is absolutely cheating and is not an industry accepted method of benchmarking. I believe that MediaTek is putting themselves in a potential bad spot in terms of reputation and they should back off this stance that having their ‘Sports Mode’ is okay because the OEMs are implementing it. That isn’t a good excuse for cheating and there’s a very high chance that this could end in a lawsuit, like Samsung’s cheating scandal did. MediaTek knows better than to do this, but I believe the decision to double down on this is going to hurt the brand’s reputation and any time they make any performance claims in the future.
Note: Moor Insights & Strategy writers and editors may have contributed to this article.