AITuTu Is AnTuTu’s New Benchmark to Test Smartphones’ AI Capabilities

Chinese benchmark firm AnTuTu has released a public beta version of a new benchmarking app designed to test the AI capabilities of modern smartphones. Called AITuTu, the new app is available for download as an APK file from the company's own website, but is not yet widely released. In a blog post announcing the new app, AnTuTu states that it might not actually be possible to quantify AI performance as a single score thanks to the current state of the smartphone industry, in which multiple SoC manufacturers, phonemakers and software developers are all working on their own implementations of AI. Also, some companies have not released SDKs to allow apps like this to tap into their AI capabilities, which will negatively affect the scores that some smartphones can achieve.

However, AI is growing in importance not only as a buzzword but also as a set of features that people now live with, which is why the firm decided to go ahead and release its new AI benchmark. The company says that it has established a unified standard for testing AI capabilities in cooperation with several manufacturers including Qualcomm, Samsung, MediaTek, and Nvidia and is working towards developing a way to quantify AI performance across all of their various implementations.

AnTuTu's blog post (translated from Chinese) goes on to state clearly that there will be differences between devices based on different hardware platforms. Also, manufacturers have different approaches to releasing SDKs, or tools that allow third parties to address their hardware capabilities. Samsung and Huawei in particular have not released any SDK yet, which will affect scores on their devices. The version of Android used will affect the score as well, since Android 9 has new AI optimisations.

The benchmark in its present form tests two AI scenarios – image recognition and object classification. The image classification test uses a set of 200 photos that need to be classified using a neural network called Inception V3. Object recognition parses a 600-frame video using a different neural network called Mobilenet SSD. These need to be translated into a format that each device can work with. If the benchmark is run on a device without AI acceleration the tests will fall back to using the CPU through a translation layer called TensorFlow Lite, which is not efficient and will not deliver a good score.

AnTuTu also says that it has built in guards to prevent phones from cheating. The score is determined by speed and accuracy, and the latter cannot be sacrificed for the former in order to get a high speed score. The company hopes that it will help the industry reach a common standard for AI, and has stated that it will release a tool for developers to address hardware AI functions on different hardware platforms directly in order to help them target their efforts.

Gadgets 360 downloaded the AITuTu public beta and ran it on a Google Pixel 3 (Review). The APK download itself is just 33MB but the test data needs to be downloaded within the app, and it prompted us to enable Wi-Fi to do so. The image classification test showed 200 photos being classified into six categories: transportation, electonric machinery, food, animals and plants, sports scenery, and other. Although some of these categories are vague, the test seemed to do a good job of sorting the photos by content, and took about one second per photo. The app then began loading its object detection but then exited to its loading screen without any error message. This could be because the app is still a public beta release.

We also tried running the AITuTu test on a recent Samsung phone, and while the image classification test took longer to run, the object recognition test did in fact run to completion. We were shown individual scores for each test and a combined overall score. With nothing to compare these scores against yet, it was not possible to determine how much Samsung's lack of an SDK affected its showing in the benchmark.