The Weekender - Generative AI is Hitting a Wall. Or the Side of a Barn.
Edition 19 - The Buzz is Generative AI has reached a point where bigger won’t always be better.
Listen to the podcast above or on Spotify.
1. Tell Me What’s a Happening
It’s the latest buzz. For the past year. It’s still buzzing today - Generative AI has reached a point where bigger won’t necessarily be better. Chips are scarce which is unhelpful at best.
We are collectively distracted from some of the limitations and realities by the many incredible visual and chat user experiences and creative output never before available.
OpenAI CEO Suggests That ChatGPT And Generative AI Have Hit The Wall And Getting Bigger Won’t Be The Way Up, Raising Eyebrows By AI Ethics And AI Law
Quite a title there. This from Forbes:
Part of this wall-hitting could be due to the lack of added benefits of making generative AI larger and larger. Likewise, investors that put money into generative AI startups are anxiously wondering the same thing.
continued …
The brouhaha was launched last week when various reporting about a talk by the CEO of OpenAI, Sam Altman, indicated that he said this: “I think we're at the end of the era where it's going to be there, like, giant, giant models.” Furthermore, he reportedly stated, “We'll make them better in other ways.”
2. Other Influences
Progress in AI may hit a wall — unless we can push it past the ‘imitation barrier’
Programs like ChatGPT are built on a computing model that doesn’t encode ingenuity.
Boston Globe Updated July 10, 2023, 3:00 a.m.
As of today, AI, even “generative AI,” uses a binary language of ones and zeros to produce seemingly creative outcomes. But these outputs are not truly original; they mimic styles and data provided by humans. This is the imitation barrier.
For example, large language models such as ChatGPT and GPT-4 transform previously learned words and knowledge into sentences that emulate text written by humans. Despite the facade of intelligence, there is a hitch. The resulting text is often biased, blatantly unethical, or simply incorrect because GPT-4 is akin to a mechanical parrot — able to mimic the right sounds but lacking the understanding and creative spark that fuels genuine thought, problem solving, and communication.
My One Big Tech-Fueled Prediction for 2024: AI ‘Vanishes’
So, here we go: In the year ahead, artificial intelligence will become ever more pervasive. So pervasive, in fact, that it will start to become invisible — to actually begin to vanish, from the hype cycles, even from our consciousness.
Only 4% of businesses currently use AI. In the future, it may be invisible to users, an analysis from NBC shows. "Few businesses will work with AI directly. Instead, they’ll use applications built on top of AI that look familiar and accessible."
In other words, awareness and hype around AI will follow the path of other foundational technologies.
AI processing could hit a wall in 2024.
Limited chip availability will force us to keep our feet on the ground regarding AI expectations.
The successive models in the GPT (Generative Pre-trained Transformer) lineage have seen a staggering increase in the number of parameters, reflecting the rapid advancement and scaling up of this language model architecture. According to published reports, the parameter counts for the GPT models are as follows:
GPT-1: Introduced with around 110 million parameters, a scale measured in the millions.
GPT-2: A significant leap to approximately 1.5 billion parameters, entering the low billions range.
GPT-3: A massive jump to a staggering 175 billion parameters, propelling the model into the realm of hundreds of billions of parameters.
This exponential growth in parameter count highlights the underlying trend of increasing model complexity and size, facilitated by the ever-growing computational resources and advancements in training methodologies.
As we build bigger data models, there are no guarantees that bigger will mean better. But it doesn’t mean it won’t be, either.
Regulators will have GenAI in their sights.
As generative AI like ChatGPT proliferates, regulators will intensely scrutinize privacy and data protection issues. Companies mishandling personally identifiable information through genAI apps risk major fines from authorities. An app developer improperly collecting or exposing user data while integrating ChatGPT could face hefty penalties under laws like GDPR or CCPA.
Unfortunately, data breaches have become normalized. How often do you receive a letter regarding a number of personal data breaches that draw an apology and a year of “FREE” credit monitoring.
3. A Word From the Boss
In a surprising admission, Microsoft co-founder Bill Gates expressed doubt that OpenAI's GPT technology can advance much further. The 68-year-old billionaire told Handelsblatt that GPT has likely hit a ceiling, contrary to OpenAI's beliefs about GPT-5. While acknowledging he could be wrong, Gates stated bluntly: "With GPT-4, the company has reached a limit."
However, Gates remains bullish on AI's near-term potential overall. He cited opportunities for AI to become more reliable and comprehensible through new research. The tech titan also highlighted AI's promise for developing nations, like providing health advice via smartphones.
Though skeptical of GPT's future leaps, Gates clearly sees AI's broader impact just beginning to be realized globally. His sobering assessment of GPT's limitations stands in stark contrast to the boundless optimism surrounding the technology.
4. Conclusion
As the world continues to be dazzled by the luster of generative AI systems like ChatGPT, it's clear we are entering a pivotal moment in the evolution of artificial intelligence. While the rapid progress has been freaking nuts, the roadblocks and limitations outlined here suggest the path forward will not be as simple as just making models bigger and bigger.
Chip constraints, lack of true understanding, and concerns around ethics and regulation represent significant hurdles that will slow AI’s continued evolvement. The AI community now faces a crossroads - either find innovative ways to push past the imitation barrier and imbue systems with more robust reasoning abilities, or risk stagnation. The “more robust reasoning” will be a hardcore tipping point.
Achieving more reliable, trustworthy, and impactful AI breakthroughs will require moving beyond the narrow pursuit of scale toward developing fundamentally new architectures and training paradigms. The next phase promises to be an immense challenge, but also an opportunity to ensure artificial intelligence continues advancing while acknowledging that we’re still in-charge .
And besides, what else do you have going on other than animating Midjourney birds in Runway?
Stay tuned.
The Weekender Trivia question: Which Broadway musical do I reference in this post? Answer below in the comments or email theadstack@adverstuff.com. If I ever get any merch I’ll send something to the first person who answers correctly.
Have a great weekend!