Apple hosted its iPhone 12 launch event on Tuesday and gave more details on its newest Apple A14 Bionic SoC. The A14 SoC is the heart of Apple’s latest iPad Air, and all four versions of the new 5G enabled iPhone 12. During the new iPhone 12 and A14 Bionic chip’s unveiling, the company made some significant performance claims without qualifying them or backing them up. None of Apple’s claims are supported anywhere in the company’s hour-long presentation or that I can find on the product pages or news release. The complete and utter lack of evidentiary support is something that even Apple has not done a lot in the past. Apple usually shows at least a bar graph comparing themselves to the stated competition and mentioning the performance measurement and the application tested. A simple benchmark bar graph would be what is expected in the silicon industry at a very minimum. I reached out to Apple’s analyst relations department for details on Tuesday and did not receive a response. I will update this story if I receive anything.
Challenging the performance of laptops?
Apple’s first, unsubstantiated claim was that the Bionic was “challenging the performance of laptops.” There were absolutely zero details on the claim in the presentation or on the company’s website. Here is what I would expect from Apple:
- Which smartphone?
- What laptops?
- What were the configurations?
- What workloads?
- What benchmarks?
While I appreciate the marketing weasel wording “challenging” and not “beating,” it still begs the question of what exactly was tested. Were dirt-cheap two-year-old Chromebooks tested that deliver a limited experience, or were the latest and greatest laptops chosen with the latest and greatest processors and discrete graphics chosen? What games were tested, and what resolutions? Was the video transcoding done on the CPU or GPU? What was the video density? Were benchmarks cherry-picked that just favor the iPhone? After about 30 years of benchmarking- 10 years for a PC OEM, 10 years for a processor company, and 10 years as an analyst I will tell you that this matters.
CPU and GPU claims
Apple claimed that the A14 has both the fastest GPU and CPU by up to 50% compared to the competition. Flat out, no competitors named, benchmarks used, conditions explained. Nothing. We have just got to take Apple’s word for it. In the CPU and GPU world, the likes of Intel, AMD, Qualcomm, and NVIDIA would never make such claims without at least running a benchmark and showing a graph and then explaining at what settings. If you go and look at some of Intel and AMD’s performance claims from its latest launch events, dozens of footnotes generate multiple pages of explanations of specific performance claims and how it validated those claims. Apple expects that consumers will take them at its word and that reviewers will either trust them or be the ones to validate those claims if it can figure out how. After all, Apple did not tell anybody how it reached these numbers.
AI performance claims
When it comes to AI, Apple claims that the A14 Bionic features a 16-core Neural Engine with an 80% percent increase in performance in specific workloads. Apple states that the new Neural Engine is capable of 11 Trillion operations per second but does not communicate if that is concurrently or in aggregate. Apple also claims that the CPU’s ML accelerators are 70% faster, but we do not know what Apple is comparing to. For all we know, Apple could be comparing the A14 Bionic to the A12 or A11; there are no footnotes, no explanations, just significant percentages. We don’t know what workloads Apple is running or whether it favor Apple’s architecture, or even how the chip stacks up in industry-accepted benchmarks like MLPerf. If you go on Apple’s website, you cannot find any qualifications of Apple’s claims or how it arrived at them. The technical specs for the iPhone 12 merely state that it features the A14 Bionic with 2nd Generation Neural Engine.
Apple also loves to talk about console-quality games but does not qualify what that means or how it is defined. If you consider where the Xbox Series X and PS5 are in terms of performance, I do not think anyone believes that Apple can compete with that. It would be helpful if Apple explained what console it are referring to because consoles are getting quite the performance boost this year with the new SoC. Perhaps Apple is comparing the A14 to the Switch, but that features an NVIDIA SoC from 2015, and I’m not sure that’s an excellent comparison.
If Apple is getting these performance numbers between 50 and 80% that it is quoting for the A14, I would love to know what applications it is using to get this performance. Realistically, performance improvements based on theoretical raw performance capability exist, but without actual benchmarks or applications running on them, it is not very meaningful to users and developers. We see numbers like FLOPS thrown around all the time when talking about the theoretical peak performance of a GPU from AMD or NVIDIA. Still, that number does not consider actual real-world performance once you account for the driver, operating system, API overhead, and how applications utilize that potential energy. We need Apple to be more transparent about performance claims like the rest of the industry and use those performance claims to explain what I can’t do with other phones that I can only do with the iPhone 12’s new A14 Bionic processor.
New Mac performance? Trust us
This alarming trend of Apple making claims with thin to no evidence matters to iPhone buyers, developers and critically for the new, Arm-based Macs. I expect when the rumored new Macs are announced next month, the company will make many unsubstantiated claims like it did at Tuesday’s launch. The risk is a lot bigger for developers in the new Mac case as developers and consumers are making such a huge leap of faith. More on that later.
I always liked Apple’s “think different” claim, but when it comes to undocumented and unsubstantiated processor performance claims, I wish it would be more open and forthright.
Note: Moor Insights & Strategy senior analyst significantly contributed to this article.