Elon Musk’s xAI has announced the open-sourcing of Grok, its highly advanced language model. Grok-1, with its staggering 314 billion parameters and 25% of weights active per token, is now available to the public allowing developers and researchers worldwide to dive into its advanced neural network architecture.
Grok’s journey began with a vision to create a Mixture-of-Experts (MoE) model, leading to a language model with unprecedented scale. The model’s pre-training phase wrapped up in October 2023, unfettered by fine-tuning for specific tasks, thus opening a world of possibilities for its application across various domains. The team behind Grok has taken inspiration from the ‘Hitchhiker’s Guide to the Galaxy,’ aiming to create an AI that can answer a broad spectrum of questions and, more ambitiously, suggest queries itself.
With two out of eight active MoE models, it boasts around 86 billion active parameters, surpassing even the largest models like Meta’s Llama 2. Its development leveraged a unique training stack, utilizing JAX and Rust.
Grok is available under the Apache 2.0 license, which enables businesses to use the model for commercial purposes. However, it requires that the license be included in any redistribution.
While the model is currently available in its raw form, it offers a foundation for developers to build upon, allowing for customization and fine-tuning for specific applications. Access to Grok is provided via GitHub, inviting developers to explore its potential.
The release of Grok follows Elon Musk’s legal action against OpenAI, alleging that its partnership with Microsoft contravened the original non-profit ethos. Musk’s move with Grok reflects his belief in the importance of open research for AI safety.
