Large language models (LLMs) like ChatGPT and Llama are popular for their natural language processing capabilities, which enable text generation and code completion.

Although these models are beneficial, their expensive operational costs have been a significant obstacle, leading academics to explore creative strategies to improve their efficiency and scalability. 

Decoding process

The cost of generating a single response is, on average, $0.01. However, when these models are scaled to service billions of users, each with several daily interactions, the expenses can quickly become significant. The costs might increase rapidly, especially in intricate jobs such as code auto-completion, where the model remains active throughout the coding process. To improve the decoding process, researchers have investigated methods to simplify and expedite attention operation, which is a vital element in producing coherent and contextually appropriate text. 

FlashAttention v2

Tokens are generated incrementally during LLM inference, also known as decoding, and the time it takes to do so depends heavily on the attention action. While developments such as FlashAttention v2 and FasterTransformer have optimized memory bandwidth and processing resources to improve the training process, difficulties during the inference phase persist. Scalability of the attention action with extended contexts is a significant limitation experienced during decoding. 

As LLMs are increasingly charged with managing more large documents, discussions, and codebases, the attention operation can occupy a substantial portion of inference time, limiting the overall efficiency of the model.

Flash-Decoding

The researchers implemented a revolutionary method termed Flash-Decoding to tackle these difficulties, expanding on the groundwork laid by previous approaches. The primary innovation of Flash-Decoding resides in its unique strategy for parallelization, which revolves around the length of sequences of keys and values. The approach enables optimal exploitation of the GPU by dividing keys and values into smaller fragments, resulting in great efficiency even with smaller batch sizes and prolonged contexts.   

Flash-decoding uses parallelized attention computations and the log-sum-exp function to reduce GPU RAM and optimize model architecture computation. 

Evaluation

Benchmark experiments were done to assess the efficacy of Flash-Decoding on the CodeLLaMa-34b model, which is known for its sturdy design and advanced capabilities. The results demonstrated a significant 8-fold improvement in decoding speeds for longer sequences compared to current methods. Furthermore, the effectiveness of Flash-Decoding was confirmed using micro-benchmarks conducted on the scaled multi-head attention, which involved different sequence lengths and batch sizes. 

These benchmarks consistently demonstrated the performance of Flash-Decoding, even when the sequence length was increased to 64k. This outstanding achievement has been crucial in significantly improving the effectiveness and expandability of LLMs, representing considerable progress in large language model inference technology. 

Conclusion

To summarize, Flash-Decoding has become a revolutionary approach for tackling the difficulties related to attention operation when decoding big language models. It has the potential to significantly decrease operational expenses and increase the accessibility of models in many applications by improving GPU utilization and enhancing overall model performance.   

Furthermore, this innovative method signifies a noteworthy achievement in the inference of extensive language models, facilitating improved efficiency and expedited progress in natural language processing technologies.

Image source: Unsplash