In previous years, 3DMark established itself as de facto standard for benchmark tests of our VGA cards in the PC realm. As smartphones and tablets become more and more powerful, there were several benchmark programs around to test them, but all of them had some shortcomings. Some have results varying too much after several consecutive runs, some were made by hardware vendors (not independent) and some wasn’t simple and/or easy to use. Futuremark recognized its opportunity, and released 3DMark to test and compare our mobile devices.
3dmark logo color cover

The test content, settings and rendering resolution are the same on all platforms and scores can be compared across Windows, Windows RT, Android and iOS. Ice Storm includes two graphics tests focusing on GPU performance and a physics test targeting CPU performance. On Android and iOS, Ice Storm uses OpenGL ES 2.0. Each test gives a score, which you can use to compare similar devices and systems. Unlike previous versions, there is no overall 3DMark score, only individual test scores. 3DMark works by running intensive graphical and computational tests on your hardware. The more powerful your hardware, the smoother the tests will run. As usual, don't be surprised if your frame rates are low as 3DMark tests are very demanding. That said, owners of latest flagships like Nexus 4, Xperia Z and HTC One, or upcoming ones like Galaxy S 4, should be safe.

2013-04-01 13.49.28-s 2013-04-01 13.49.46-s


Ice Storm is rendered at a fixed resolution of 1280x720 (720p) or for Extreme version in 1920x1080 (1080p) and then scaled to the native resolution of the display. This is commonly called "off-screen rendering" and is the best approach for ensuring that devices can be compared fairly in terms of raw power. We appreciate these mods, but we would like to see one more, where rendering is done in device native resolution. It would give us opportunity to quantize our subjective feeling of device speed, be it for lower level device with screen resolution of 800x480 and lower or for FullHD+ like Nexus 10 with its 2560x1600 resolution, or not so standard 2048 by 1536 resolution for iPad.

2013-04-01 13.50.12-s 2013-04-01 13.50.53-s


Ice Storm default settings are: rendering resolution set to 1280x720, texture resolution set to normal, post-processing quality set to normal. Ice Storm Extreme settings are: rendering resolution set to 1920x1080, texture resolution set to high, post-processing quality set to high. To be able to successfully run 3DMark test, devices need to meet several criteria. For Android platform, minimum OS version is 3.1, device needs to have 1 GB of RAM, to be OpenGL ES 2.0 compatible and to have at least 300 MB of storage. For Apple devices, minimum iOS is 5.0 iPhone for iPhone 4, iPad 2 for iPads and iPod Touch (5th gen), along with 300 MB free storage.

2013-04-01 13.54.36-s 2013-04-01 13.54.55-s


Ice Storm benchmark is running three different tests, Graphics test 1, Graphics test 2 and Physics Test. Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects. In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame. Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.

2013-04-01 13.55.20-s 2013-04-01 13.56.02-s


Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets. On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur. In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.

2013-03-28 18.01.20-s screenshot 2013-03-31-22-53-36-s screenshot 2013-03-31-23-06-00-s


The purpose of the Physics test is to benchmark the hardware’s ability to do game play physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed. This test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load. The Physics test uses the Bullet Open Source Physics Library.

We have done our first batch of test on three different smartphones: Sony Xperia Z, Galaxy SIII and Galaxy SII. Xperia Z is most powerful Android smartphone we have had at hand and with its Full HD resolution a perfect subject to this test. Galaxy SIII with its quad core CPU and HD ready resolutions is recently dethroned flagship which still should show some muscle. Galaxy SII is now two years old Samsung flagship, but still a solid represent how powerful mid-range smartphones are. As previous benchmarks taught us, version of Android OS can have a great impact on test results - that’s why we used same version on all three phones, 4.1.2. If you look at the results, several things can be concluded.

screenshot 2013-03-31-23-23-17-s screenshot 2013-03-31-19-56-32-s 2013-03-31 23.33.58-s


First, on the CPU side, just as expected, Physics test doesn’t show big difference regardless if 720p or 1080p resolution is used. Performance difference here is as expected, SIII is 2x SII + 15%, just in line with bump from dual core to quad core architecture as well as frequency speed increase. Furthermore, in Physics test Xperia Z is some 15% faster than SIII, benefiting from slightly higher CPU speed and architectural difference.

screenshot 2013-03-31-23-23-34-s screenshot 2013-03-31-20-02-39-s 2013-03-31 23.34.22-s


Much, much different story is to be told with GPU performance in mind. Galaxy SII and SIII share same GPU foundation with former benefiting on some design improvements as well as from higher clock speed. Unlike CPU side where performance difference was 115% in favor of SIII, on GPU side it is more moderate 60% faster than SII on all benchmarks. Sony Xperia Z, the star of this test literally obliterates these two phones in 3D performance. Xperia Z is 4 to 10 times faster than Galaxy SII depending of the test, making it obsolete in terms of absolute performance. A bit more modest difference is between Xperia Z and SIII phones, but it is still 2 to 6.5 times faster, depending of the particular test.

Smartphone Sony Xperia Z C6603 Galaxy SIII I9300 Samsung I9100 Galaxy SII
CPU Quadcore Qualcomm S4 Pro APQ8064 @ 1.5 GHz (ARM Cortex-A9+) Quadcore 1.4 GHz Exynos 4412 (ARM Cortex-A9) Dualcore 1.2 GHz Exynos 4210 (ARM Cortex-A9)
GPU Adreno 320 Mali-400 MP4 Mali-400 MP
Memory 2 GB RAM Dual Channel 1GB RAM Dual Channel 1GB RAM
OS Android 4.1.2 Android 4.1.2 Android 4.1.2
Ice Storm 1280x720 pix
Score 10547 2739 1602
Graphics test 1 (FPS) 46.5 7.2 4.3
Graphics test 2 (FPS) 45.2 16 9.8
Physics test (FPS) 33.5 28.9 11.8
Ice Storm Extreme 1920x1080 pix
Score 6065 2091 1236
Graphics test 1 (FPS) 27.2 6 3.6
Graphics test 2 (FPS) 20.8 9.7 6
Physics test (FPS) 32.7 28.8 13.1

However, this is not the full story. While the performance difference is nothing short than spectacular, real life usage will not show the same level of speed increase. Main culprit here is screen resolution, as Xperia Z needs to render around 2 Mpix (1920x1080), Galaxy SIII 0.9 Mpix (1280x720) and SII just 0.4 Mpix (800x480). Impressive performance difference in graphics test between Xperia Z and SII suddenly drops from 4-10 times range, to 0.8 - 2 times, making it much more comparable in real life. Similar story is to be told when comparing SIII and Xperia Z where former instead to be 2-6.5 times faster drops down to less spectacular 0.9-3 times. This is exactly the reason why we would like to see another variant of Ice Storm test, which will render benchmark in native screen resolution.

That aside, 3DMark is great tool. UI is pleasant, adapting itself to screen estate, it is easy to use, fast to execute, with results that can easily be verified in real life. Online base where you can instantly see how your device stands comparing to others in same series is a nice addition. If you are experimenting with overclocking, different OS version or just different settings it can come handy with its nice graph representing results in timeline. Online base also shows hardware specification of your device, which can come handy. Additional plus is cross platform compatibility, enabling us to test and compare iOS, Android and Windows RT devices. Quite important is that 3DMark is independent toll developed in cooperation with Broadcom, Imagination Technologies, Intel, NVIDIA, Qualcomm, and many other world-class companies, but not sponsored by any of them. First iteration is good, and we will continue to use it in our tests. Keep up the good work Futuremark and continue to polish 3DMark in line with users/reviewers needs, you will soon become industry standard for benchmarks.