Galaxy S4 Benchmark “Optimization” Drama Hits Samsung

There’s a new wave of drama hitting the Samsung Galaxy S4 today. After running some tests tech site Anandtech has noticed that Samsung was allowing at least a version of the S4 to run its graphics processor (GPU) at higher frequency (533MHz) in some benchmarks, while games yielded a lower frequency of 480MHz. In other words, Samsung would be inflating its graphics horsepower by 11.04%. The reviewers were able to get accurate frequency numbers thanks to a built-in command available to anyone who can log as an administrator (root) in Android.

Upon further inspection of the app responsible for changing the clocks (TwDVFSApp.apk), the team discovered the names of the various benchmarks. This is a good indicator that those programs are being singled-out as “special” and would benefit from a special treatment. Upon launching them, TwDVFSApp.apk will set the frequencies to a higher frequency in order to push the score higher. It’s simple enough, and probably a bit too simple for their own good since this was easily discovered and criticized.

This is not new. Remember Quack?

We have seen this movie before: back in the days, there was “scandals” similar to this one. For example, back in 2011, ATI was caught optimizing for a specific application: Quake III. It was discovered that the driver was looking for the current executable name to trigger specific optimizations, and that by renaming “quake.exe” to “quack.exe”, the optimizations were no longer in effect. The world shouted “foul!” and ATI defended itself by saying that its optimizations did improve speed, without affecting image quality. Unfortunately, independent tests showed that texture quality was affected and all this eventually went away as the whole “optimization” crashed in flames.

What you see may be the tip of the iceberg

The Samsung “optimization” is not affecting image quality here, but the parallel with the ATI story is that the engineers behind TwDVFSApp.apk have been naive enough to think that nobody would look. This is a lesson that has learned the hard way in the PC world and there has not been any such case as of late (unless I missed something?), and it’s not really because no one is walking the line anymore, but because everyone is much more careful to not be obviously caught crossing the line.

A more recent case: Intel vs. ARM

That fact that no “quack” scandal happened does not mean that vendors have stopped trying to push the limit. Recently, there was another benchmark scandal when it was discovered that a claim about Intel’s latest system on chip was beating those of Samsung, NVIDIA and Qualcomm turned out to be backed by bogus data. This time, the benchmark itself was at fault, since its programming made it possible for the compiler to skip huge chunks of work, without affecting the final outcome. After a fix, Samsung, NVIDIA and Qualcomm came back on top again. Ooops.

There are smarter ways to identify a benchmark and switch to a new code path

The reason why we should always take benchmarks with a grain of salt is that there are ways to make it much harder to discover such “special code paths”. For example, graphics drivers could easily detect a particular application by looking for a specific sequence of geometry and shaders before switching to a different code path. Each game or graphics benchmark has its own signature if you look for those attributes, and it is much harder to detect than looking around for clear-string in an HEX editor.

Optimizations: legit or not?

The context

That is the real question and we should come back to the Galaxy S4 drama here. If Samsung had built a larger list of apps and shown that many non-benchmark apps could benefit from the extra frequency boost, they would have had a compelling argument for those who accuse them of “cheating”. I think that Samsung is clamping down on the frequency for non-benchmark apps because they don’t know how long games will run, while we all know that benchmarks only run for a few minutes at best. This basically avoid overheating and possible device damage.

Since Samsung seems to only allow the higher frequencies to be used on benchmarks, there is a good argument for saying that these optimizations do not benefit the regular users, and therefore they show a distorted view of the device’s performance. On the other hand, some may say that as long as the benchmark runs on the said hardware and doesn’t crash or fail, this is fair game — I take it that most users will side with the first argument.

Samsung’s answer:

Under ordinary conditions, the GALAXY S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode. Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.

The maximum GPU frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.

Samsung Electronics remains committed to providing our customers with the best possible user experience.

As I said earlier, this could be a compelling argument if the list of apps hitting 533MHz included a lot of “real apps” (or even 4 or 5 for that matter). Maybe Samsung will extend it, but right now it’s not enough to convince.

Going forward and what you should take away

This type of frequency boost optimizations will most likely solve itself in the next generations of chips. In you look at the PC, Intel chips have a built-in overclock capability which is mainly regulated by thermal conditions. It will go as fast as it can, for as long as it can tolerate – and it does work for all applications. In essence, TwDVFSApp.apk exists today only because the device is not yet capable of overclocking itself in a safe way, or not in an “aggressive enough” way.

Secondly, Samsung needs to show that whatever it does actually benefits users and preferably all apps. This is why this whole thing blew up to start with. Also, you should always assume that the industry is trying to push the boundaries and walk the line. Someone may stray a bit too far and get called out by the community, but in the grand scheme of things, it’s fair to say that successful chip companies are where they are because they have been doing a great job. Benchmarks can only carry one so far.

Finally, we all have to use and trust benchmarks with caution. While performance indicators are very interesting, they do not tell the whole story. Performance alone doesn’t translate into user experience, and a product review is much more than a product benchmark.

Hubert Nguyen
Categories: Cellphones, Featured
Tags: benchmarks, Galaxy S4, Samsung