Happy Sunday and welcome to InsideAI! I'm Rob May, Founder and CTO at Dianthus. If you like the newsletter, please forward it to a friend so they can sign up!
This week I want to talk about foundational models. This is the name the industry is giving to big mega models like GPT3. There are two key forces driving the rise of foundational models. The first is that more data equals a better performing model. In fact, the improvements don't seem to be slowing so far, as these models get bigger. The second is that these large models are expensive to build. You have to gather a ton of data, and the training itself of a model like GPT3 is rumored to be in the millions of dollars.
The question I want to ask today is - as businesses, are they defensible long term? I won't look at the other issues of whether we should build them, and whether or not they might cause various problems in the industry. This is just a quick analysis of - can they be good businesses?
At the moment there are only a handful of companies with the talent, tech, and budget to build and train these models. It doesn't mean a startup can't do it (and there are a few trying) but, they might have to raise a lot of money or, skimp on data and building something not-quite-as-big. In general, businesses with high barriers to entry that require large amounts of capital are very very defensible and we can expect them to earn good returns on capital over time.
But there are trends that are quickly eroding these advantages. First, new AI chips are coming to market. These will outperform GPUs for many use cases, and will change the compute costs of training foundational models. Second, there is work being done to patch together open source competitors using distributed computing and donated cloud compute credits. And third, there are attempts to move away from neural networks and look at more small data AI. How fast will these happen? If they happen rapidly, the value of owning a foundational model business may diminish.
But wait, there are soooo many use cases for foundational models, the industry could verticalize by use case. That could be more defensible. Or, maybe as GPT-3 becomes trainable by a cheaper AI chip in a matter of minutes or hours, OpenAI and other groups have moved on to GPT 6/7/8, and they are much much more valuable than GPT3.
I think the answer to whether or not these businesses are valuable really depends on the marginal benefits of additional training data. At the moment, it seems to be growing, not shriking. That would tell us that owning a foundational model, and the capacity to make one, it only going to get more valuable. At some point though, we would expect the marginal value of new data to slow. Maybe in a few years when we are at GTP- 9, it isn't worth moving to, from GPT-8 for all but the most extreme use cases.
There are many interesting forces at play in this world. If you have an opinion on the economic value of foundational models long term, I'd love to hear it.
Thanks for reading.
@robmay
Comments
Post a Comment