John Schulman, the co-founder of OpenAI, has shared his insights on the performance of GPT-4 and the potential challenges it faces due to a limited amount of training data being available. Schulman played a key role in the launch of ChatGPT in November 2022.
What Happened: In a recent episode of Dwarkesh Patel's podcast, Schulman addressed the potential limitations of GPT-4 and the future of AI models.
When asked if the AI community is approaching a “data wall,” where the benefits of memorizing a vast amount of pre-training data may not lead to significantly smarter models than GPT-4, Schulman admitted that there are a few challenges in that regard, but companies like OpenAI and others will have to change their training methods to overcome these challenges.
"There are definitely some challenges from the limited amount of data, but I wouldn’t expect us to immediately hit the ...