We've maintained at NDTV Gadgets that synthetic benchmark scores used by gadget reviewers for gauging smartphone performance are not as reliable as some make them out to be. They are not helpful in determining the phone's day to day performance, which is why our reviews focus on informing our readers the overall experience of the phone, and stay away from any benchmarks. While we do run benchmarks on the phones and tablets we receive for review, we like to keep the results to ourselves, as they are indicative at best, and unreliable at worst. We've also heard chatter that manufacturers have been optimising their phones to perform better in benchmarks than real world usage, and now there's evidence that the most high-profile smartphone manufacturer has been indulging in these under-handed tactics.
Popular website Anandtech has
revealed that Samsung has been optimising the Exynos 5 Octa powered Samsung Galaxy S4 for benchmarks. A
forum post by
@AndreiF on the Beyond3D forums in June, that alleged that Samsung was over-clocking the phone's graphics processor to 533MHz for certain benchmarks, while other apps/ games were limited to 480MHz, inspired the folks over at Anandtech to investigate the issue.
The website informs that Samsung is over-clocking the Galaxy S4's Cortex A15 cores (the Octa SoC features two Cortex quad core processors) to 1.2GHz while running the GLBenchmark 2.5.1, AnTuTu, Linpack, and Quadrant apps for benchmarking. However, the phone's processor ran at 500MHz while using the GFXBench 2 app, for which Samsung has not optimised its hardware.
Samsung has also tinkered the GPU (Graphics Processor) of the Galaxy S4's Exynos 5 Octa variant to run at a higher frequency while benchmarks are being run. The GPU is set to run at 533MHz while performing benchmarking tests. It runs at 480MHz, otherwise.
The report also mentions that Samsung includes a code referred to as "BenchmarkBooster" that essentially trigger the phone's hardware to run at a higher clock speed when benchmark tests are run.
The whole expose reveals that benchmarks cannot be relied upon to gauge the everyday performance of a smartphone or a tablet, and hardware makers are tricking reviewers and end users who rely only on benchmarks while recommending or buying a device.
Update: Samsung has
responded to the report, saying it did not use any specific tools to achieve higher benchmarks. However, its statement completely ignores the damning bits Anandtech found in the code.
Here's Samsung's response as quoted by The Verge:
"[We] did not use a specific tool on purpose to achieve higher benchmark scores."
Samsung adds that, "under normal conditions, the Galaxy S4 operates up to 533MHz at its best performance." The Korean manufacturer says certain "full screen apps" (any app in which the status bar isn't present) such as the camera, browser, video player, and benchmarking tools, are classified as requiring the highest performance available. Many games don't require the maximum clock speed to run, the company notes. Samsung doesn't address AnandTech's discovery of strings of code that implied specific benchmark apps were being targeted for higher clock speeds, but the site did note that other benchmark apps that are not explicitly mentioned in code were also behaving the same way.