MicrosoftChief Technology OfficerCTOKevin Scott said in an interview with Sequoia Capital’s podcast last week:He reiterated his beliefLarge Language Models (LLM)’s view that the “law of scale” will continue to drive progress in AI, even as some in the field suspect that progress has stalled. Scott played a key role in pushing Microsoft to reach a $13 billion technology-sharing agreement with OpenAI.
“Others may disagree, but I don’t think we’ve reached a point of diminishing returns with scale,” Scott said. “I want people to understand that there is an exponential process here, and unfortunately,You only see it every few years because it takes time to build supercomputers and then train models on them.. "
In 2020, OpenAI researchers explored the “scaling law” of LLM, which states thatLanguage model performance tends to improve predictably as models become larger (more parameters), more training data is available, and more computing power is available.This law means that simply increasing the size of models and the amount of training data can significantly improve AI capabilities without requiring fundamental algorithmic breakthroughs.
However, other researchers have since questioned the long-term validity of the "law of scale." Still, the concept remains a cornerstone of OpenAI's philosophy on artificial intelligence research and development. Scott's optimism contrasts with the views of some critics in the field of artificial intelligence, who believe that progress in large language models has stagnated at the level of models like GPT-4. This view is based primarily on informal observations and benchmark results of the latest models, such as Google's Gemini 1.5 Pro, Anthropic's Claude Opus, and OpenAI's GPT-4o. Some believe that these models have not made the same leaps and bounds as previous generations of models,The development of large language models may be approaching a stage of “diminishing marginal returns”.
Gary Marcus, a prominent critic of artificial intelligence, wrote in April: “GPT-3 is clearly better than GPT-2, and GPT-4 (released 13 months ago) is clearly better than GPT-3. But what happens next?”
Scott's stance suggests that tech giants like Microsoft still believe it's reasonable to invest in large AI models, betting on continued breakthroughs. Given Microsoft's investment in OpenAI and its heavy marketing of its own AI collaboration tool, Microsoft Copilot, the company has a strong desire to maintain the public perception of continued progress in the field of AI, even if the technology itself may hit a wall.
Ed Zitron, another prominent AI critic, recently wrote on his blog that one argument some people make for continued investment in generative AI is that “OpenAI has some technology we don’t know about, a powerful and mysterious technology that will completely crush all skeptics.” He wrote, “But that’s not the case.”
The public perception of the slowdown in the capabilities of large language models, as well as the results of the benchmarks, may be partly due to the fact that AI has only recently entered the public eye, while in fact, large language models have been in development for many years. OpenAI continued to develop large language models for three years after the release of GPT-3 in 2020, until the release of GPT-4 in 2023. Many people may not have realized the power of models like GPT-3 until the launch of ChatGPT, a chatbot developed using GPT-3.5, in late 2022, and therefore felt that the capabilities were greatly improved when GPT-4 was released in 2023.
In the interview, Scott refuted the view that progress in artificial intelligence has stagnated, but he also admitted that the data points in the field are indeed updated slowly because new models often take years to develop. Nevertheless, Scott is still confident that future versions will improve, especially in areas where current models perform poorly.
“The next breakthrough is coming, and I can't predict exactly when it will occur."We don't know how much progress it will make, but it will almost certainly improve the current imperfect aspects, such as the model is too expensive or too fragile to be used with confidence," Scott said in an interview. "All of these aspects will be improved, the cost will be reduced, and the model will become more stable. By then, we will be able to achieve more complex functions. This is exactly what each generation of large language models has achieved through scale."