Elon Musk concurs with different AI specialists that there’s little real-world information left to coach AI fashions on.
“We’ve now exhausted principally the cumulative sum of human data …. in AI coaching,” Musk mentioned throughout a live-streamed dialog with Stagwell chairman Mark Penn streamed on X late Wednesday. “That occurred principally final yr.”
Musk, who owns AI firm xAI, echoed themes former OpenAI chief scientist Ilya Sutskever touched on at NeurIPS, the machine studying convention, throughout an tackle in December. Sutskever, who mentioned the AI trade had reached what he known as “peak information,” predicted an absence of coaching information will power a shift away from the best way fashions are developed at this time.
Certainly, Musk advised that artificial information — information generated by AI fashions themselves — is the trail ahead. “The one strategy to complement [real-world data] is with artificial information, the place the AI creates [training data],” he mentioned. “With artificial information … [AI] will form of grade itself and undergo this means of self-learning.”
Different firms, together with tech giants like Microsoft, Meta, OpenAI, and Anthropic, are already utilizing artificial information to coach flagship AI fashions. Gartner estimates 60% of the info used for AI and analytics tasks in 2024 have been synthetically generated.
Microsoft’s Phi-4, which was open-sourced early Wednesday, was educated on artificial information alongside real-world information. So have been Google’s Gemma fashions. Anthropic used some artificial information to develop considered one of its most performant methods, Claude 3.5 Sonnet. And Meta fine-tuned its most up-to-date Llama sequence of fashions utilizing AI-generated information.
Coaching on artificial information has different benefits, like value financial savings. AI startup Author claims its Palmyra X 004 mannequin, which was developed utilizing virtually fully artificial sources, value simply $700,000 to develop — in contrast to estimates of $4.6 million for a comparably-sized OpenAI mannequin.
However there as disadvantages as properly. Some analysis means that artificial information can result in mannequin collapse, the place a mannequin turns into much less “artistic” — and extra biased — in its outputs, finally critically compromising its performance. As a result of fashions create artificial information, if the info used to coach these fashions has biases and limitations, their outputs will likely be equally tainted.
