The artificial intelligence community is buzzing with excitement over the latest addition to its arsenal, Falcon 180B—an open-source large language model (LLM) equipped with a whopping 180 billion parameters, meticulously trained on a colossal dataset. This formidable newcomer has notched several remarkable achievements, setting a new benchmark for open-source LLMs.
As disclosed in a blog post by the Hugging Face AI community, Falcon 180B has officially made its debut on the Hugging Face Hub. This state-of-the-art model builds upon the foundations laid by the previous Falcon series of open-source LLMs, incorporating innovative features such as multiquery attention to scale up to its remarkable 180 billion parameters, all trained on a staggering 3.5 trillion tokens.
One particularly noteworthy feat is Falcon 180B’s single-epoch pretraining, which is the longest of its kind in the realm of open-source models. This milestone was achieved through the collective power of 4,096 GPUs, working tirelessly for approximately 7 million GPU hours, with Amazon SageMaker serving as the platform for training and fine-tuning.
To put the sheer magnitude of Falcon 180B into perspective, its parameter count dwarfs that of Meta’s LLaMA 2 model, measuring a staggering 2.5 times larger. LLaMA 2, previously hailed as one of the most capable open-source LLMs following its launch earlier this year, boasted a mere 70 billion parameters trained on 2 trillion tokens.
Falcon 180B eclipses not only LLaMA 2 but also other models in terms of both scale and benchmark performance across various natural language processing (NLP) tasks. It proudly claims a position on the leaderboard for open-access models, boasting a remarkable 68.74 points, nearly rivaling commercial giants like Google’s PaLM-2, as evidenced by its performance on the HellaSwag benchmark.
In fact, Falcon 180B not only matches but often surpasses Google’s PaLM-2 Medium in performance across commonly utilized benchmarks such as HellaSwag, LAMBADA, WebQuestions, Winogrande, and more. It stands shoulder-to-shoulder with Google’s PaLM-2 Large, a testament to its remarkable capabilities as an open-source model, even when compared to industry behemoths.
When pitted against ChatGPT, Falcon 180B emerges as a more robust alternative to the free version, albeit slightly less capable than the premium “plus” service.
According to the blog post, “Falcon 180B typically sits somewhere between GPT 3.5 and GPT4 depending on the evaluation benchmark, and further fine-tuning from the community will be very interesting to follow now that it’s openly released.”
The release of Falcon 180B marks a significant leap in the rapid advancement of large language models (LLMs). Beyond mere parameter scaling, innovative techniques like LoRAs, weight randomization, and Nvidia’s Perfusion have revolutionized the efficiency of training these massive AI models.
With Falcon 180B now freely accessible on Hugging Face, researchers anticipate that the model will continue to evolve and improve as the community collaboratively enhances its capabilities. Its immediate showcase of advanced natural language prowess heralds an exciting chapter in the world of open-source