Summary: Elon Musk and xAI have just released the weights and architecture of their massive 314B parametric language model Grok-1 under the open source Apache 2.0 license.
We have released the weights and architecture of our 314 billion parameter Mixture-of-Experts model Grok-1. This is the original base model checkpoint for Grok-1 in the October 2023 pre-training phase and is not fine-tuned for a specific application. The model was trained on large amounts of text data, using custom training stacks on JAX and Rust. The weights and architecture are released under the Apache 2.0 license. To use the model, you can find the github.com/xai-org/grok View instructions on.
Details.
Grok-1 is a hybrid expert model where only 25% weights are active for any given input token for more efficient computation.
The published model is an untrained checkpoint from October 2023 and has not been fine-tuned for any particular task.
xAI provides a guide for developers to get started on its GitHub repository, and has published the model on Hugging Face.
Importance: by open-sourcing one of the world's largest LLMs, xAI is getting out in front of what Musk sees as a moral battle against OpenAI's closed model. While Grok's features haven't broken any new barriers yet, the move is another major victory for collaborative and transparent AI development.
