The inference phase of the ML lifecycle is where we get to witness the machine thinking like a human: a trained model scores actionable outputs by processing the input data. This is essentially where we interact with AI.

ML Commons is an open-engineering consortium that works to provide industry benchmarks and metrics through a comparison of machine learning systems, software, and solutions. The initiative seeks to promote transparency in the ML industry. Their benchmark suite, known as the MLPerf, was first unveiled in 2018. The MLPerf Inference v1.0, released in April 2021, measures the inference performance of neural networks. Essentially, it ascertains how quickly a trained NN can process new data for a wide range of applications on a variety of form factors.

For the first time, the latest benchmark round has also introduced a new metric that understands system efficiency. This implies that the MLPerf Inference v1.0 suite includes power measurement techniques, tools, and metrics to complement the performance benchmarks. The power measurement was developed in partnership with Standard Performance Evaluation Corp. (SPEC), the leading provider of standardised benchmarks and tools for evaluating the performance of today’s computing systems.

“We wanted to add a metric that could showcase the power and energy cost from different levels of ML performance across workloads. MLPerf Power v1.0 is a monumental step toward this goal and will help drive the creation of more energy-efficient algorithms and systems across the industry,” said Arun Tejusve, Chair of the MLPerf Power Working Group, in the official statement.

• Fastest AI systems

The latest listing ranks 1,994 AI systems, spanning from edge devices to data centre servers. Entries came from 17 organisations, including Alibaba, Centaur Technology, Dell Technologies, EdgeCortix, Fujitsu, Gigabyte, HPE, Inspur, Intel, Lenovo, Krai, Moblint, Neuchips, NVIDIA, Qualcomm Technologies, Supermicro, and Xilinx.

Datacentre (commercially available systems, ranked by server condition)

(Image source: IEEE Spectrum)

Edge (commercially available, ranked by single-stream latency)

(Image source: IEEE Spectrum)

• Most efficient AI systems

For the new power measurement round, a total of 862 results were released. Here are the tops in each category.

Datacentre

(Image source: IEEE Spectrum)

Edge (commercially available, ranked by single-stream latency)

(Image source: IEEE Spectrum)

The detailed results of the MLPerf Inference v1.0, along with a background on how these benchmarks work, can be accessed here.

• Read also | Which are the top 10 supercomputers in the world?

Image from Nvidia