On April 18, Meta officially released its latest open source model Llama3, claiming it to be the "most powerful open source large model" currently available.

What impact will the open source of Llama 3 series have on the industry? Will it support open source? Or closed source? Or both?

Ruoyu Technology, a company invested by Kunzhong, focuses on the multi-modal general robot brain track. Now Ruoyu Technology has gradually formed a closed-source model framework with the best performance in the field of embodied intelligent robots. Let's see what our CEO Sun Teng said~

Image source: Internet

Regarding the recent hot topics, we interviewed Sun Teng, co-founder and CEO of Ruoyu Technology. Let's take a look at his views below~

1.What are the technical iterations of Llama 3 ?

Llama 3 continues the Transformer structure of the previous generation in terms of overall architecture, with the following major improvements:

A. The Token dictionary is expanded from 32K to 128K to enhance coding efficiency

B. Supports contextual input up to 8K tokens, but still inferior to competitors

C. Introducing Grouped Query Attention (GQA) to improve reasoning efficiency

According to the evaluation on MMLU, GPQA, HumanEval and other datasets, Llama 3-70B scored 82.0, 39.5 and 81.7 respectively, outperforming models of the same level such as Claude-Sonnet and Mistral-Medium, and basically reaching the level of GPT-3.5+, and approaching GPT-4. The subsequent Llama-3-400B+ version is expected to further narrow the gap with GPT-4 and benchmark models such as Gemini Ultra and Claude3.

2. What new breakthroughs does Llama 3 have in model training data ?

The training data scale of the 8B and 70B versions of Llama 3 is as high as 15T and 50T tokens respectively, far exceeding the optimal data volume of 160B and 70B (1.4T) predicted by the Chinchilla law for the 8B scale, which basically overturns the industry's understanding of the Chinchilla law.

In other words, this means that even a small model of a fixed size can achieve a logarithmic linear improvement in performance as long as it is continuously fed with high-quality data. This opens up a new way of thinking for cost-effectiveness optimization and the development of an open source ecosystem, that is, through the model of small models + massive data, it is also possible to achieve a balance between performance and efficiency. Under the premise of sufficient high-quality data feeding, the upper limit of small and medium-sized models in the future may far exceed expectations.

3. What impact will the open source of the Llama 3 series have on the industry?

The open source of the Llama 3 series will provide more powerful basic model capabilities for enterprises and entrepreneurs in the fields of big models and AI, on which they can develop various value-added services and products and commercialize them, such as customizing special models for specific industries. In addition, Llama 3 also provides a new window of opportunity for AI startups, which is expected to launch solutions that are equivalent to or even better than existing products in certain vertical fields.

On the other hand, Llama 3 will also pose a huge impact and challenge to existing artificial intelligence companies, intensify competition in the field of artificial intelligence, and determine the survival of the fittest.

4. What do you think of the increasing number of technology giants entering the open source big model market?

Indeed, many technology giants have joined the ranks of open source big models. The participation of technology giants in the open source big model has the benefits of promoting technological innovation, lowering R&D thresholds, building a developer ecosystem, and innovating applications. At the same time, it will also build new business models, such as driving the sales of cloud computing related products.

However, we also see that many major domestic Internet companies actually launch closed-source big models: on the one hand, their application scenarios are mostly based on their own businesses, such as office, conference, entertainment, productivity tools, etc., and they use the capabilities of big models to enhance the competitiveness of their own products; on the other hand, this also involves efficiency tuning of vertical field-specific applications and industry data privacy protection issues, which are not convenient for open source.

Therefore, we believe that open source and closed source will coexist for a long time, which will help accelerate the maturity and application of large model technology, but at the same time it will also trigger discussions on issues such as technology control, data privacy and market monopoly. This trend requires the joint attention and management of industry participants, government agencies and regulators.

5. Support open source? or closed source? or both?

We believe that this is not an either-or question, but one of mutual promotion and common development.

When making actual choices, it is more about choosing the most appropriate technology to solve practical problems in what scenario. For the open source model, it can accelerate prototype verification and build a developer ecosystem in some innovative fields and early product stages to improve innovation productivity. For example, the Linux open source community and RISC-V open source chip architecture have actually greatly promoted the huge development of domestic operating systems, chip architecture design and other industries. However, open source will also bring problems such as high maintenance costs and non-optimal vertical performance.

Closed-source large models are more suitable for applications in special scenarios. QoS and performance are guaranteed, privacy, policy, and abuse risks are avoided, and they help to create top-performance and exciting products. We have noticed that, especially in the field of software and hardware integration, software and hardware performance integration and tuning are required to achieve top performance. The development of mobile phones, drones, and self-driving cars has all confirmed this.

There are also many companies that adopt a hybrid strategy, such as open-sourcing some of their technologies to build communities and standards, while keeping core products or advanced features closed source to maintain commercial advantages. This model can protect key intellectual property and business interests while building brand reputation and community participation. For many companies, adopting a hybrid strategy may be the best way to take advantage of the benefits of open source while protecting core competitiveness through closed source.

Ruoyu Technology was founded in 2023. We focus on the multimodal general robot brain track. After long-term tracking of industry applications, we have accumulated a large amount of data and model design in some specific scenarios to provide intelligent human-computer interaction, task planning and action execution for embodied intelligent robots. This not only involves visual models, perception models, semantic big models, as well as the action execution part of the robot body introduced for this purpose, and the model compression technology introduced to ensure the execution efficiency of the big model on the terminal side, combined with the robot's unique industry big data, Ruoyu Technology has gradually formed a closed-source model framework with the best performance in the field of embodied intelligent robots.

Llama 3 model download link: https://llama.meta.com/llama-downloads/

Llama 3 GitHub project address: https://github.com/meta-llama/llama3