Android Benchmarks: Why it shouldn’t matter that much

A lot of people, tech-heads specifically, would look at the benchmark scores of a specific Android device before making any purchase decisions. We don’t normally do this when buying other devices with different mobile operating systems. Which brings us to the question — do Android benchmarks matter?

Introduction to Benchmarks

First of all, let us explain what benchmarking is. Benchmarks are usually done to measure the performance of a certain device. It’s a numerical value we use to compare with other Android devices. In essence, it’s the measure of how fast a specific hardware can perform a set of tasks.

The Google Play Store is loaded with a lot of apps for such a purpose. Each benchmark too has a specific parameter to measure. The performance of your device is represented by these major components — the processor, RAM, graphics, internal storage.

While Android has a lot of benchmark tools in the Play Store compared to any other platform, some are also cross-platform.

Antutu is very consistent with its measurements of CPU speed, GPU rendering, storage write speeds and etc., while Vellamo does the same with slight differences, plus HTML5 benchmarking. Quadrant is also somewhat similar.

For the graphics, the gaming performance and all of that, there’s 3DMark, Epic Citadel and Nenamark 2. 3DMark just became available for Android, and it’s a heavy duty tool since you need to download nearly 300MB of data. It’ pretty comprehensive and is also available for iOS and Windows RT/8.

Nenamark 2, which we also use as a standard in our reviews, focuses on OpenGL|ES 2.0 benchmarking. Epic Citadel is also quite heavy, as it puts your device through Unreal Engine 3 with intensive graphics and measure frame rates.

Benchmarks can be inconsistent and unreliable. There’s also the fact that it can be manipulated. Here’s why:

People don’t usually go through all of that suite of benchmarks just to judge a specific device. A few tests or so will probably suffice but we can’t really let that be the main basis for the device’s performance, and we want to explain the logic behind this.

Here are the Antutu scores of devices running on a 1.5GHz dual-core Qualcomm MSM8260 Snapdragon, with the same GPU and RAM (1GB):

• Sony Xperia S: 8418
• Samsung GS2 Skyrocket (US variant): 6,336

Through that, you can see there’s discrepancy of around 2,000. Benchmarks can be quite inconsistent. The Xperia S features a higher resolution display so that’s a factor, though it doesn’t really make significant discrepancies. For reference, the HTC Sensation on a lower 1.2GHz and lower 768MB RAM scores in at 3,932 (that being qHD).

A bigger flaw is noticeable when you look at this next example. The Samsung Galaxy S2 has been very known for its Exynos processor, which raged through the benchmark charts in the past, and that pretty much started the Exynos craze. Due to network incompatibilities with US carriers, Samsung had to switch the Exynos with a Snapdragon (the SGS2 Skyrocket). A lot of people found this disappointing due to difference in benchmark scores of the variant.

Turns out, they were wrong. When the Skyrocket was tested, it got scores near to the dual-core Exynos for Quadrant, Antutu annd others. The interesting thing to note here is that HTC devices such as the Amaze 4G, which ran exactly similar specs to the Skyrocket safe for the qHD display, never went near the scores of the Skyrocket.

You can check out the specs of the devices here for confirmation.

Here are the scores of the devices from the reviews of GSMArena (other sites’ benchmarks are approximately the same):

• Amaze 4G on Quadrant: 2,793
• Skyrocket on Quadrant: 3,224
• Amaze 4G on Antutu: 3,641
• Skyrocket on Antutu: 5,881

It’s very suspicious to see here that Samsung has a huge lead with the benchmarks when the configurations are almost the same. Samsung may have tinkered with a few elements to keep its title as the benchmark king. Plus, if that’s not enough, you should see another anomaly in our Starmobile Diamond review, where the Quadrant scores were suspiciously high.

Oh, and more on Quadrant; Android Central has an article and a video on how to play with the software to make your device give out higher scores than usual, without having to tinker with the CPU and all that.

Benchmark scores may also be affected by a lot of external factors — OS versions can greatly affect it (Jellybean has improved a lot of scores of older devices), background services and apps can also affect it, CPU throttling of some chipsets (like Atom Z-series) will also reflect erratic results.

Conclusion

There are still a number of people out there who would look at benchmark scores and immediately conclude that they should get that device after seeing superior numbers. We’d like to advise caution and give pause. From low-end phones to high-end ones, benchmark scores can be manipulated. After all, these are just apps, and hardware can be optimized to make sure they get good results on these benchmarks.

Even if you’re looking at the high-scorer out there (at the time of this writing) like the Samsung Galaxy S4, you can’t really tell the difference anymore as high-end Android phones have already eliminated the lag. Probably, the only thing worth looking at now is how well it works with software (as Windows Phone devices run smoothly in the UI even with low specs), and how great it is in handling graphics and power efficiency.

In the end, we still think people will still look at benchmarks. We advise that you don’t base your decisions solely from the numbers. Read reviews, watch hands-on videos, and ask people around you who have experienced the device. Real world experiences can never equal any glowing spec sheets and benchmark numbers.

{source}