The Five-Year Silence: Why AI Chatbots Took So Long to Arrive

If the Transformer was invented in 2017, why did it take five years for the chatbots to appear in late 2022 and 2023? There were a number of reasons for this, but it wasn’t due to inactivity. Once the Transformer concept was in place, work did continue, but it was largely out of sight.

The first order of business was to address scaling laws. Trying to create more economical algorithms was no longer the answer. AI would now depend on increasing the number of parameters, the internal variables the model adjusts as it is trained on mountains of text. Success also depended on vastly increasing the amount of data and the computing power to process it. Brute force became the solution, rather than just elegant code.

Unfortunately, simply increasing the number of servers was insufficient; new computer chips had to be designed. Without them, the models would remain theoretical, stuck in small-scale trials. It wasn't until 2020, with the release of NVIDIA’s A100 GPUs and their “Tensor Core” optimizations, that large-scale Transformer training became commercially viable. After about two years of integrating this hardware, the AI chatbots we recognize today finally began to emerge.

This technological leap was impressive, but it immediately hit the wall of legal liability. Tech giants like Google had a comfortable status quo as the world's information gatekeepers. This new AI, if it went off the rails, could irreparably harm their core businesses. Executives remembered Microsoft’s 2016 "Tay" debacle, where a chatbot learned to spew offensive invective within 24 hours. Combined with the persistent issue of "hallucinations,” the industry faced loss of trust.

Another concern, particularly for Google, was the cannibalization of its search engine. If an AI gives you a direct answer, you don't click on blue links, and Google doesn't get ad revenue. I have wondered about this myself, since I haven’t used traditional Google Search in six months after incorporating AI into my work life. I believe Google must have figured out that AI is here to stay, and they must now work out how to replace the revenue lost by a search model that suddenly feels archaic. I never thought I would ever see the day that happened!

So, the five-year gap between the invention of the Transformer and the rise of ChatGPT wasn't a delay, it was a gestation period. It took that time for the hardware to catch up to the math, and for corporate giants to get used to the inevitable AI disruption to come. The era of scrolling through "blue links” is fading, and the era of just asking what you want has officially begun.