Silicon Valley, November 14, 2024 — OpenAI’s upcoming flagship model, Orion, has reportedly shown smaller performance improvements than its predecessor, GPT-4, reigniting industry debates on the limits of AI advancements. According to insiders familiar with Orion’s development, the leap from GPT-4 to Orion has been notably less significant than previous transitions, particularly in coding tasks, as reported by Information.
This apparent slowdown suggests that rapid progression in generative AI models seen over recent years may be tapering off, leading some experts to question the future trajectory of the industry. As AI development becomes increasingly costly and challenging, OpenAI and others are left re-evaluating principles of scaling laws— an ethical framework that suggests AI models improve as they grow in size and data availability.
Scaling Laws in Question
OpenAI CEO Sam Altman has long been an advocate of scaling laws, which posit that AI models become more capable as they are fed with more data and computing power. However, as Orion reportedly shows only modest improvements, Altman and his technical team are contending with a harder reality: practical challenges of scaling may no longer guarantee significant gains previously observed.
Scaling laws, Altman explained earlier this year, are “decided by god; constants are determined by members of technical staff.” Yet recent evidence raises doubts about the extent to which the laws can drive further AI enhancements. Following Information’s report, discussions within Silicon Valley have intensified, with industry experts questioning if current leading models are reaching the limits of what scaling can achieve.
Orion’s Training Adjustments
While Orion’s development is still ongoing, OpenAI has had to resort to additional measures such as human feedback post-training to enhance its performance. Despite initial testing, the model is said to perform below expectations in certain complex tasks. This has led some in the field to wonder whether forthcoming AI iterations will lack the transformative impact that earlier models boasted.
AI researchers have pointed to two significant limitations affecting model scalability: data availability and computing power. Companies have scraped vast amounts of human-generated data, from online text to video content, yet such resources are finite. Data scarcity has led firms to consider synthetic data generated by AI itself, but experts like Ion Stoica of Databricks caution against relying on synthetic data alone. “For general-knowledge questions, you could argue that for now, we are seeing a plateau in the performance of LLMs,” Stoica told Information, stressing that factual data still holds more value than synthetic data.
The availability of computing power is also becoming a concern. In a recent Reddit AMA, Altman admitted that OpenAI faces tough decisions regarding the allocation of computational resources, signaling that scaling costs could become unsustainable. Estimates for Orion’s training run are already anticipated to exceed $100 million, a figure that industry experts like Anthropic CEO Dario Amodei predict could reach upwards of $100 billion for future models.
“Diminishing Returns” and AI Skepticism
Prominent AI critics, such as New York University professor emeritus Gary Marcus, argue that diminishing returns are becoming evident across the field. Following reports of Orion’s smaller-than-expected performance leap, Marcus published a post titled “CONFIRMED: LLMs have indeed reached a point of diminishing returns,” expressing doubt over the ongoing push for exponential model growth. Marcus has previously voiced concerns about AI’s limits, contending that recent models, including Anthropic’s Claude 3.5, offer only incremental improvements over existing tools.
Ilya Sutskever, OpenAI cofounder, has acknowledged that results from scaling pretraining models have begun to plateau. “Scaling right thing matters more now than ever,” Sutskever told Reuters, suggesting that refining existing methodologies may now be as important as expanding model size.
Optimism Amid Scaling Debate
Despite concerns, some leaders within the AI industry remain optimistic. Altman himself has not fully conceded that scaling is no longer the way forward. Microsoft CTO Kevin Scott dismissed plateau concerns in a July interview, saying that “we’re not at diminishing marginal returns on scale-up.” Industry optimists point to advancements in inference— the stage where models refine their outputs based on new, unseen data. OpenAI’s recent model OpenAI o1, launched in September, focused on inference improvements, achieving benchmark results that rivaled Ph.D. students in scientific tasks.
Future of Generative AI and Scaling Challenges
As Orion’s development continues, industry faces a pivotal moment. Scaling laws may be recalibrated, and companies are likely to explore alternative methods to maintain progress in AI performance. For now, OpenAI and its competitors appear committed to pushing boundaries of what AI can accomplish, even if the path forward looks more uncertain.
If future models like Orion underwhelm, the generative AI boom could face a re-evaluation. Industry, though driven by ambitious goals, may increasingly need to temper expectations, investing as much in the refinement of current models as in pursuit of the next groundbreaking AI innovation.
