FP8 Format | Standardized Specification for AI

Artificial intelligence processing requires collaborative innovation across hardware and software platforms to meet the growing computational demands of neural networks. A key area of efficiency improvement is the use of lower-precision digital formats to increase computational efficiency, reduce memory usage, and optimize interconnect bandwidth.

To realize these benefits, the industry has converted from 32-bit precision to 16-bit and now even 8-bit precision formats. The transformer network is one of the most important innovations in artificial intelligence, particularly from 8-bit floating-point precision. 

We believe that having a common interchange format will enable rapid development and interoperability of hardware and software platforms that drive computing.

NVIDIA, Arm and Intel have jointly authored a white paper, FP8 Formats for Deep Learning, which describes the 8-bit floating point (FP8) specification. It provides a common format to accelerate AI development by optimizing memory usage for AI training and reasoning. This FP8 specification is available in two variants, E5M2 and E4M3.

The format is implemented natively in the NVIDIA hopper architecture and has shown excellent results in initial testing. It will immediately benefit from the work done by the broader ecosystem, including the AI framework, to implement it for developers.

Compatibility and flexibility

FP8 minimizes deviations from the existing IEEE 754 floating-point format through a good balance of hardware and software to leverage existing implementations, speed adoption, and increase developer productivity.

E5M2 uses five bits for exponents and two for the mantissa, a truncated IEEE FP16 format. In cases where higher accuracy is required at the expense of some numerical ranges, the E4M3 format has been adapted somewhat to extend the range expressed in four digits for exponents and three digits for the mantissa.

The new format saves additional computation cycles because it uses only 8 bits. It can be used for artificial intelligence training and inference without any recasting between accuracies. Moreover, minimizing deviations from the existing floating-point format provides maximum freedom for future AI innovations while still adhering to current conventions.

High-precision training and inference

Testing the proposed FP8 format shows accuracy equivalent to 16-bit precision across a wide range of use cases, architectures, and networks. Results from the transformer, computer vision, and GAN networks all show that FP8 training accuracy is similar to 16-bit accuracy but can be significantly faster. For more information on accuracy studies, see the FP8 Formats for Deep Learning white paper.

Language Model AI Training

Different networks use different precision metrics (PPL and Loss)

Language Model AI Inference

In MLPerf Inference v2.1, the industry-leading benchmark for AI, NVIDIA Hopper leverages this new FP8 format to achieve a 4.5x speedup on BERT high-precision models, gaining throughput without sacrificing precision.

Towards Standardization

NVIDIA, Arm and Intel have released this specification in an open, unlicensed format to encourage widespread industry adoption. They will also submit the proposal to the IEEE.

By adopting an interchangeable format that maintains accuracy, AI models will continue to run efficiently on all hardware platforms and help drive the development of AI.

Aktie:

Vorherige: How to make an RF transmitter and receiver with HT12E and HT12D

Nächste: The Security MCU for IoT Applications