Ling-V2 - The MoE Architecture Language Model Series of Ant Centurion Open Source

What is Ling-V2?

Ling-V2 is a family of large-scale language models based on MoE architecture introduced by Ant-Belling team. The first version Ling-mini-2.0 has 16 billion total parameters, with only 1.4 billion parameters activated per input token. The model is trained on 20 trillion high-quality data tokens, enhanced by multi-stage supervised fine-tuning and reinforcement learning, with powerful complex inference and instruction adherence, based on the 1/32 activation ratio of the MoE architecture, which achieves 7x equivalent dense performance leverage, fast generation, efficient training and inference, and open-sourced with an FP8 efficient training solution, which provides a variety of pre-training checkpoints and supports continuous training. MoE is an ideal starting point for MoE research, widely used in natural language processing, intelligent customer service, content creation, education, healthcare and other fields.

Functional Features of Ling-V2

• Highly effective reasoning skills: excels in complex reasoning tasks, including coding, math, and cross-domain knowledge-intensive tasks, providing accurate solutions that outperform both partially dense models and larger-scale MoE models.
• Excellent performance efficiency: Based on the MoE architecture with 1/32 activation ratio, only 1.4 billion parameters can be activated to achieve the performance of 700-800 million dense models, and the generation speed can be up to 300+ tokens/s, which is a significant improvement in the efficiency when dealing with long text.
• Advanced training techniques: The whole training process is mixed precision with FP8, and the open source FP8 training scheme further optimizes the memory usage and significantly improves the training throughput.
• Open Source Strategy: A trained version of the model is provided, and five pre-training checkpoints are open-sourced to facilitate continuous training and in-depth study by researchers.

Core Benefits of Ling-V2

• Balancing high performance and efficiencyWith a MoE architecture with a 1/32 activation ratio, Ling-V2 significantly improves computational efficiency while maintaining high performance.
• Efficient training solutions: Ling-V2 is trained with FP8 mixed-precision during the training process. The open-source FP8 training scheme further optimizes the memory usage, which significantly improves the training throughput, reduces the consumption of computational resources, and makes the training of the model more efficient and economical.
• Open Source Strategy: The availability of trained versions of the model and the open-sourcing of five pre-training checkpoints provide researchers with more flexibility and convenience, supporting them in continuous training and in-depth research, and driving the development of inclusive technology.
• Wide range of application scenarios: Ling-V2 is suitable for a variety of application scenarios, including natural language processing, intelligent customer service, content creation, education, healthcare and other fields, and can meet the needs of different industries and users with high practicality and adaptability.

What is Ling-V2's official website?

• GitHub repository:: https://github.com/inclusionAI/Ling-V2
• HuggingFace Model Library:: https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86

People for whom Ling-V2 is indicated

• natural language processing (NLP) researcher: Ling-V2 provides powerful inference capabilities and efficient training solutions for professionals engaged in natural language processing research, helping users make breakthroughs in the fields of text categorization, sentiment analysis, and machine translation.
• Corporate Technical Team: For enterprise technology teams that need to process text data efficiently, the model is integrated into intelligent customer service, content creation, and knowledge management systems to improve enterprise operational efficiency and user experience.
• Educators and students: In education, personalized educational support for educators and students.
• Medical professionals: Assist doctors in case analysis, medical literature search, etc., to improve the accuracy and efficiency of medical decision-making, applicable to professionals in the field of healthcare.
• content creator: Ling-V2 aids in content creation, helping creators improve the efficiency and quality of their creations and inspire more creativity.