The saga of MediaTek’s benchmark score scandal continues where AnandTech accused the company of ‘cheating’ via whitelisting certain benchmarks into a special unsustainable ‘Sports Mode’ high-performance mode. This was written in detail by Anandtech and covered by yours truly with some of my own analysis based on over a decade of benchmarking phones. One of my key takeaways was that I believe MediaTek should apologize and own up to its mistakes rather than double down on them like it seems to have done with its blog, which has pointed out to me was published before Anandtech’s article went live.
UL Benchmarks was previously known as Futuremark prior to UL’s acquisition of Futuremark in 2014 and the name change came in 2018. The company itself prior to acquisition has been around since 1997 and was based in Espoo Finland. It’s been the expert in benchmarking PCs and smartphones pretty much from the beginning and have been the leaders in the space for as long as I have been in the industry (early 2000s). So, this is the standard by which many reviewers and companies go by and its leaderboards are a common battleground for some of the world’s PC overclockers as well.
UL Benchmarks has a long track history of delisting certain devices and manufacturers from its leaderboards and benchmark results in general as a result of cheating. As recently as 2018, UL Benchmarks delisted Huawei phones (P20, P20 Pro, Nova 3 and Honor Play) with some help from none other than Anandtech. Following that, with Tech2’s help UL Benchmarks also delisted OPPO’s Find X and F7 from its benchmark results list due to the two versions of the 3DMark app having hugely different performance results. As far back as 2013, UL Benchmarks (Futuremark at the time) delisted a whole host of Android phones from HTC and Samsung for cheating on its benchmarks. However, because of Futuremark/UL Benchmark’s actions, Samsung updated its phones to remove this cheating feature and admitted guilt. This resulted in Samsung’s phone being re-listed by UL Benchmarks for as long as it rain the correct version of the Android operating system with Samsung’s fixes. Redemption is possible for smartphone manufacturers that cheat. Samsung eventually also ended up settling a class action lawsuit regarding the benchmark cheating as well.
This leads us to UL’s decision to delist 50 devices from 25 different OEMs. It has temporarily delisted devices featuring MediaTek’s Helio G90, G70, P95, P90, P65, P60, P20 and A22. This is a significant amount of chipsets from MediaTek’s product lineup. Its decision is based on two things, the discrepancy in performance numbers found by AnandTech and confirmed by UL Benchmarks, but also its own benchmarking rules and MediaTek’s own statements about benchmarking practices. In fact, it quotes MediaTek’s own statement regarding industry standards and call it misleading,
We contacted MediaTek with our findings, but did not receive a response. Shortly after AnandTech published its article, however, MediaTek responded with a post on its website.
While the response is mostly a point of view, we would specifically call out the following misleading claim,
“MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets.”
Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an “accepted industry standard.” It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device’s true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.
MediaTek’s case is a little bit more complex than the others before it. First, the company doesn’t fully control how smartphone manufacturers configure its devices. Second, the company provided the tool to the smartphone OEMs that enabled them to select which benchmarks it wanted to ‘cheat’ on. Third, and arguably most important, MediaTek doesn’t believe there has been any wrongdoing. The company chooses to stand by its benchmarking practices and doesn’t believe that it’s done anything wrong, even though it has been almost universally accepted that whitelisting applications for performance boosting is considered wrong.
Until MediaTek reverses its decision to double-down on these practices, I believe the company’s products will forever have a cloud above, especially when the company releases new ones and makes performance claims. Credibility in benchmarks is everything and if you ruin the sanctity of that, you have an exceptionally long road to travel until your credibility is repaired. I hope that MediaTek reconsiders its ‘accepted industry standards’ and takes the path towards owning this and making things right.
Note: Moor Insights & Strategy writers and editors may have contributed to this article.