Keynote | Still Training: A Decade of Learning the Hard Way

AI Research

Rohan Anil on eleven years of babysitting training runs, from Shampoo to one model trained across GPU clusters 9,000 miles apart.

Keynote | Still Training: A Decade of Learning the Hard Way

Keynote | Still Training: A Decade of Learning the Hard Way

AI Research

Rohan Anil on eleven years of babysitting training runs, from Shampoo to one model trained across GPU clusters 9,000 miles apart.

Thank you! Your recording is ready.

Your submission was successful. The video will appear in a moment.

Rohan Anil
Co-Founder, Core Automation
Core Automation

About This Talk

Rohan Anil has spent over a decade training models, and he is still babysitting runs. He retraces years of distributed training, from Google’s Sibyl and parameter servers to Shampoo and codistillation, showing how each bottleneck forced the next idea. The same trade-off between local work and communication returns at frontier scale, where power and land, not chips, now cap how big one cluster can get. That leads to Core Automation’s work on compute islands, where one model trained across two GPU clusters about 9,000 miles apart, landing slightly ahead of single-cluster training at the same compute. The next problem is building models that keep learning after deployment.

‍

About The Speaker

Rohan Anil

Transcript

Show transcriptHide transcript

00:00

I think in the past two talks, we learned about understanding and dreaming. This talk is about love. So this is a quote from one of my favorite movies, "Love transcends dimensions of time and space." About more like 11 years ago, I was at Google, and I was working on training systems all the way back before we had neural networks in production. And I fell in love with training algorithms, and I started working on it. And this talk is going to be a whirlwind tour of all the work that I was part of and I have seen, and to where we are today. And this is going to be a lot of math, and I don't expect all the math to be grokked, but it should be, I think, pretty exciting to see. But the thing is, the run hasn't really finished. I'm still working on training, and I'll explain why. So this talk is going to be about large scale ML for folks you may be aware of, like 10 plus years ago, and how we moved a large scale logistic regression to neural networks that powered search and ads, and then building TPUs to serving large scale transformers on them, all the way to what we have today, which is scaled up pre-training and RL that gave us agents that can now do a lot of things. So one of the things that I'm working at with Jerry Turek and many others at Core Animation is agents today can act in the world, can observe, but it cannot learn. There is this notion that we are not able to give feedback to the models for the models to be able to learn at test time. And I think it might be an experience for many of us that sometimes these agents work brilliantly and sometimes it does some really dumb stuff.

02:06

And this is why we have this debate whether models are actual-- people have different experiences using agents. Some people rage at agents, some people think it's AGI already. And this problem will continue to be the case until we have training algorithms that can learn and let the models learn at test time. This is one of the other things that we notice when we work with agents, is that the human expertise matter a lot. Who uses these agents. The outcome that we see from who's using the agents differ, and it seems like almost a high variance where you get outcomes that are in log scale for folks who understand what they want to do with these agents, and they can harness the power of agents. Now, going back to the talk, the first thing that I'll talk about is back in 2014, we used to train models that were in neural networks, and I'll connect it all the way back to the final part of the talk, which is about possibly training large scale neural networks across geographical regions and possibly in space one day. So Sybil was Google's large scale supervised learning system. This is the empirical risk minimization objective there. And all it did was count for each feature whether to take a positive gradient step or a negative gradient step. You calculate it over billions of examples, you run a very large MapReduce, and then you make an update. And this algorithm was called parallel boosting, and this worked quite well. And they implemented the system, and then they soon found out what we saw in theory did not hold in practice, like every training algorithm ever created. You had to modify a few things. And the first observation someone made was, well, it's actually quite slow to converge, and what you should just do is that if a feature moved in one direction in the previous step, and you're also again pointing out that it should go in the same direction in the next step, just increase the learning rate for that coordinate so that you can go much faster. This one tweak made the convergence much faster, and you could scale to large scale, lot of data, and lot of features.

04:43

And the next one is, well, we had these systems that now could converge, but it worked for problems that had lot of sparse features but did not work when you had a lot of dense features. So these things would be like if you were doing risk prediction, fraud detection, you have a lot of time-based features that you're accounting for, and it would not converge. So there was another set of algorithm that changed the optimization function on its head. Instead of optimizing for the weights, you optimize per example, you have a parameter we call the dual, and then the final set of weights is a weighted combination of all your examples. Now you have just to solve for this alpha parameter here, and then everything works out. And this had really nice theory. You could use it to come up with a notion of how far you are from convergence. And this is one of those most beautiful algorithms that'll actually tell you how far you are to the solution, so you can actually continue to put more compute till you get to the solution. But the thing was, this would work on one single machine, and you could not distribute it across many machines. So some folks made an observation. Well, you can by looking at the objective function and thinking about how to decompose it into different subproblems that can be solved in each machine. And they had two key observations. One, compute updates in local ... steps on each of these machines, aggregate the update, and then add the update across many machines. And prior to this work, it was all about averaging the updates, which would make you lose progress. And here they had a mechanism that could additively combine these updates to make much faster convergence. Then there was this resurgence of this algorithm back from 2011 called AdaGrad for scaling neural networks, because now the neural networks that came out after logistic regression was these models that had very large embedding tables, were extremely sparse, were used for recommender systems.

07:04

So you had this property that the features were sparse, and you had to learn embeddings that occurred very few number of times. It had a different count of number of times it appears in the training data. So then there was this beautiful algorithm called AdaGrad, and there was this version that was asynchronous and lock-free on a large set of parameter servers. You would put the embeddings on CPUs on a lot of parameter servers, and then you would use CPUs to asynchronously train them across tens of thousands of CPUs. And then we got the model neural networks, the transformers. You apply something like AdaGrad on those problems, and then you soon realize that it's actually not feasible. You would need to keep a covariance of 53 trillion parameters. We don't have as much RAM even today to keep them. So then you had to think about how to come up with clever approximations that made it feasible to come up with some update rule that is much stronger but allows us to train modern neural networks. And that's where the Shampoo algorithm comes from. Since this audience probably is not aware of Shampoo, the name comes from it being a pre-conditioner. So if you take a bath, before you use a conditioner, you would use shampoo. And this is why the algorithm is called Shampoo, it has nothing to do with anything else. But this joke has caught on for the last eight years. Now we have algorithms like Lather, Soap, and these are things that make the agents of today. So then what is the Shampoo algorithm at a very high level? Skip the math, but think of factorizing your neural network into rows and columns, and then coming up with covariances for those rows and covariances for those columns, and then use those to compute update rules. It turns out this is an extremely strong state-of-the-art method, and if you use versions of this, you get extremely strong models.

09:19

Interestingly, when the original algorithm was published in 2018, this is the pre-LLM era where we were all trying to convince our peers that the algorithms are great, this is amazing. And then we ran into some bottlenecks. There's always this predominant thing that everyone's chasing after. At that time, it was Adam, and there was this whole community of people was like, "Adam is all you need to get to the next thing." And we had a bunch of people at Google like, "Hey, look, there are these second-order methods that are useful. We've been using it in our production systems. Maybe we should use it to scale." It took about something like six years for most of the world to accept that something like this actually could make your models converge faster. We even organized a competition to figure out what is the best thing. And then in 2024, a group at Meta submitted an entry on distributed Shampoo and won the competition with some speed-up. But this also tells you mostly the research is out there, but it's mostly often not looked because most of our attention, both the researcher attention, funding, and everything else, goes to the most in thing today. And this becomes more clear in later part of my talk when I talk about how to distribute training across many, many clusters. So then another problem when training neural network is backprop. When you train a neural network, you go forward, you take a bunch of steps, and then you have this sequential set of steps to compute gradients for each of those layers. It would be nice if we could parallelize it across many more GPUs. So we did this cool idea back at Google called LocalProp. It's called LocalProp because it's local updates. And the idea there was to compute at least one backward pass to compute these targets for each layer, and then you can independently run the updates for each of the layers across many GPUs.

11:30

And you can use just SGD or some first-order method. And what we found was that if you do that, you get actually the performance of second-order methods by just using first-order methods. And I think this idea is now starting to come back into the world because now we have models that are much larger that cannot fit a single machine, and these things are starting to work even more. And then the other set of ideas that were out there was this idea of we want to do distributed training, but you have a limit on how much parallelism you can use. So I said, "If you give me 10,000 GPUs, I can train you a model. But if you give me 100,000 GPUs, I actually don't know how to effectively utilize those GPUs." Because fundamentally, using more data per batch has a limit because the optimization algorithms of today have a limit on how effective you can use those examples. So now the question is, are there other ways you can train models that allows us to use more parallelism. One cool idea that we realized was instead of training one model, train two models. So you can 2X the amount of compute, but make it effective to train these models by having them exchange predictions on each other's batch as a teacher. So think of it as you and your peer are working on an assignment, you're teaching each other, and you're making progress. And this type of acceleration, this is not gradient-based, it's in function space, allows you to use 2X more resources and converge 2X faster. So this gives you more parallelism. Unfortunately, this idea would have been amazing if two could have become three, three could have become 10. Like anything, it stops working after two.

13:34

So you get a factor of two, and it does not continue to increase. Although I think people who worked on this are continuing to think about ways to make this much more effective, one of them being me. And the last part, just as a preview, is this idea of asynchronous training. That is what we used to do when we were training these large-scale models for production systems. You would have these lot of CPUs, and they would keep all the parameters, and then you would have these workers and gradient updates to those workers. And then we moved to this synchronous training where you had to aggregate the gradients across many GPUs. So we had this primitive all-reduce, which sums all the gradients across many GPUs in a tree fashion in one step. And this is what modern neural network training uses, and this is how we train models. And the last bit is the scaling era, around 2020, like the scaling paper from Tom Brown and many others, and Dario and many others, showed that you can scale both data and parameters and then continue to improve training loss. Fundamentally, this was a really simple but at the same time very transformative idea because you could just get better parameters by just spending more computation, and at least at that time, we didn't think that you needed clever algorithms other than the transformer. And then a group at DeepMind figured out, well, how to proportionally set the amount of parameters that you give per token, and then that was the Chinchilla law. And then the last three years were scaling this technology all the way up in terms of flops. We went from clusters of size 100, now we have clusters of size 100,000. And then the last bit I wanted to mention was that it allowed us to get this in-context learning as part of the transformer that gave us the agents of today. Instead of having fine-tuning, you can prompt the model, give it examples, and because attention mechanisms can implement gradient descent as part of a forward pass, we were able to make a learning machine that can meta-learn things as part of the forward pass itself.

15:58

Now, the more interesting part of this talk is this blog post that I wrote while we were at Corodo. It's this idea of training on compute islands. The timing of this blog post actually coincided with all the data center news of this week, but it was not timed that way. We actually had a practical use case, and we wanted to push training algorithms. So the top-line result that we have is that we trained a state-of-the-art scaling law with a reasonably large mixtures of experts model trained over two continents. So we got GPUs at two continents, one in Asia, one in the US, and it's about 9,000 miles apart, and we trained a model that just got as good or slightly better than a regular training on a single cluster. I think the result here is cool because for long, we've been bottlenecked by training algorithms, and this is also for folks who have been following the literature, it's probably not new. This was all built up over the last 15 years. We are getting to training algorithms that can allow us to do really distributed learning, and this is the scaling law for that. Now, as we've all read, it's not just the GPUs. The getting the power, getting cooling, getting land, getting approval, everything else has become harder. This is not my domain of expertise, but at least as a consumer of these GPUs, I see how fragmented the ecosystem is, how long it takes to get GPUs, how unpredictable the market is. And so, if you're working in a world where you cannot take for granted if you are not one of the big players to have hundreds of thousands of GPUs at the same place. So the question is, and the blog post is arguing, could you actually get to frontier training runs by cobbling together smaller data centers? And the third experiment here is, is that actually better? Because you could then maybe build smaller data centers at many more places much more quickly because perhaps that is easier to get power, for example.

18:31

And then Elon had this tweet that because of this power and cooling, there's going to be a lot of chips that are not going to be powered on. Clearly, no training algorithm can solve that problem, but we focused on the other problem. If they were powered on and they were not at full capacity, could you come up with a training algorithm that can use multiple clusters together and get to frontier? So the algorithm essentially is, the way we've proposed it is in three ways. Within a compute island, use all the aggressive interconnect that we have, the NVLinks, the InfiniBands, across clusters, built a data transport layer that works with the training algorithm. So there's a few details around how we fragment the network weights and how often we communicate them, and how we calculate the delta. And then measure performance in terms of both quality to the results as well as the whole overall wall time. And for folks who are new to the compute market, essentially, I see it as these three different blocks. You get these providers that can get us big clusters, and they're great, and they come in nice shapes and sizes and well-managed. Then there are providers that give us smaller clusters, and then the folks who have fragmented set of compute across many islands. So this is how the compute comes today. And then the question for us was how to use them. And this is lines of work going back all... This is during the scaling era of folks trying to push on algorithms that actually allow us to do distributed training. I think DeLoco being one of the famous ones from DeepMind. There's been work by Prime Intellect on Intellect One, and there's been multiple teams trying to come up with ways to train and get high-quality results.

20:39

And in our case, we split this into two clusters. We came up with a modification of distributed shampoo, some version of DeLoco, and then we engineered a network layer that can work at focused on improving the tail latency and connected them together. And on top of that, we also did this experiment, which seems to also give us similar quality models. We call it butterfly mixing, this idea from very long ago. It turns out you don't actually have to synchronize all the GPUs together. You can do something even cooler. The idea is to keep pairs of GPUs synchronized and mix them in a butterfly manner. So which means that if you had eight GPUs, you would synchronize two of them, the other two, and then they exchange over many steps. After, say, three steps for something like eight GPUs, the weights are still very slightly different across all those GPUs, and then you continue training. This idea is something people had explored back in the day before neural networks. And it turns out the modern neural networks of today, like a reasonably standard mixtures expert model, do work with this type of approach. And here is a cool animation of what something could be possible. You could train now across many more geographical regions. You could train with more delay. You can actually fungibly move compute as the model is training. This gives us new operational capability in training models. And now the question is, if we can do this, there's been talks about putting data centers in space. Well, if such a thing is possible, what we're saying is that the training algorithms exist. Now we just need to figure out how to get data centers into space and how to deal with all the physical limitations of what space may provide or may not provide. So this is one of our most ambitious thing. Could you actually train models in space? Could you get to a trillion-parameter model in the next five years? I don't know, but I know that Google did put TPUs in space, got inference working for Gemma 4.

23:08

So I think we are taking steps there. And if we play this out and the constraints that exist get solved, this could be one avenue of having a lot more compute. Thank you for entertaining my talk. I hope you learned a little bit about training algorithms. I'll welcome back our MC, Brian. Thank you.

Related recordings

Welcome: What Will AI Dream Next?

Chang She
•
Keynote
opening-keynote-what-will-ai-dream-next

AI Research

Multimodal Data

AI Infrastructure

Keynote | Still Training: A Decade of Learning the Hard Way

Rohan Anil
•
Keynote
still-training-a-decade-of-learning-the-hard-way

AI Research