Following the publication of controversial — and categorically flawed — competitive benchmarks pitting Intel’s new Core i9-9900K CPU against various AMD Ryzen processors, the company Intel commissioned for the study has issued a response to accusations of bias from tech press and gaming enthusiasts.
Over the past 15 years Principled Technologies has carried out extensive product testing for companies like Intel, AMD, HP, Dell and Google. Intel hired the company to, in its own words, “create as level a playing field as possible for comparing the AMD and Intel processors as the majority of the gaming market would likely use them.”
The resulting benchmarks set off red flags in the tech press due to various inconsistencies in the company’s testing methodology, specifically the use of AMD’s “Game Mode” which reduces the cores by half and is intended onlyfor use in Threadripper processors, and the implementation of XMP profiles with tight memory timings in Intel systems but not in AMD ones.
That being said, Steve Burke from Gamers Nexus detailed a laundry list of problems (above) with the benchmarks that go well beyond these major ones, calling out a number of critical issues the majority of tech press didn’t pick up on.
The Response From Principled Technologies
Credit where credit is due, though. Principled Technologies co-founder Bill Catchings took the time to hear and process the various complaints, and issued a fairly lengthy response. He’s even pledged to do some level of retesting to make it right.
The letter opens with a brief overview of the company’s history, and a firm denial of any bias or intentional handicapping. “Before going further, we […] must categorically deny any dishonesty in our work on this project for Intel or in any of our other projects,” it reads.
So the burning question I have is why Ryzen Game Mode was used on the 2700X system? Here’s the answer:
“Based on AMD’s recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling the Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode.”
While I appreciate that tests will be conducted properly next time around, there’s a serious red flag with this response. AMD itself only recommends that Game Mode be enabled with Threadripper CPUs. So I’m not sure what kind of “recommendations” AMD provided, but my best guess is that Principled Technologies either failed to read the Game Mode documentation or ignored it.
The response also includes technical details surrounding their implementation of system memory settings across Intel and AMD systems, the highlight being that D.O.C.P was selected for “AMD equivalent memory settings to XMP.” I’m still puzzled why XMP wasn’t actually switched on and memory sub-timings dialed in, as that makes a notable difference in performance.
All things being equal, the fact that Principled Technologies is engaging in a dialogue and communicating with its detractors is refreshing, especially when the response from Intel PR was the equivalent of “nothing to see here, but we love our reviewer community.”
Note that Principled Technologies considers this response a living document,so you may want to keep an eye out for changes. In closing, Catchings says “We are confident in our test methodology and results. We welcome questions and are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up.”
You can contact Principled Technologies on Twitter at @PrincipledTech, and via email at firstname.lastname@example.org.
Since 2005 I’ve been entrenched in the video game and consumer tech industries, and fascinated with the rapid evolution of the technology surrounding them. In addition to Forbes, I’ve contributed to gaming and technology features on PCWorld and Computer Shopper, and spent…