Connect with us

Climate News Live

Deep Learning’s Climate Change Problem


Climate Strike

Deep Learning’s Climate Change Problem

The human brain is an incredibly efficient source of intelligence. Today’s AI is not. Getty Images/iStockphoto Earlier this month, OpenAI announced it had built the biggest AI model in history. This astonishingly large model, known as GPT-3, is an impressive technical achievement. Yet it highlights a troubling and harmful trend in the field of artificial…

Deep Learning’s Climate Change Problem

Data center.

The human brain is an incredibly efficient source of intelligence. Today’s AI is not.


Getty Images/iStockphoto

Earlier this month, OpenAI announced it had built the biggest AI model in history. This astonishingly large model, known as GPT-3, is an impressive technical achievement. Yet it highlights a troubling and harmful trend in the field of artificial intelligence—one that has not gotten enough mainstream attention.

Modern AI models consume a massive amount of energy, and these energy requirements are growing at a breathtaking rate. In the deep learning era, the computational resources needed to produce a best-in-class AI model has on average doubled every 3.4 months; this translates to a 300,000x increase between 2012 and 2018. GPT-3 is just the latest embodiment of this exponential trajectory.

The bottom line: AI has a meaningful carbon footprint today, and if industry trends continue it will soon become much worse. Unless we are willing to reassess and reform today’s AI research agenda, the field of artificial intelligence could become an antagonist in the fight against climate change in the years ahead.

Bigger Is Not Always Better

In today’s deep learning-centric research paradigm, advances in artificial intelligence are primarily achieved through sheer scale: bigger datasets, larger models, more compute.

GPT-3 illustrates this phenomenon well. The model consists of a whopping 175 billion parameters. To put this figure in perspective, its predecessor model GPT-2—which was considered state-of-the-art when it was released last year—had only 1.5 billion parameters. While last year’s GPT-2 took a few dozen petaflop-days to train—already a massive amount of computational input—GPT-3 required several thousand.

The problem with relying on ever-larger models to drive progress in AI is that building and deploying these models entails a tremendous amount of energy expenditure and thus carbon emissions.

In a widely discussed 2019 study, a group of researchers led by Emma Strubell estimated that training a single deep learning model can generate up to 626,155 pounds of CO2 emissions—roughly equal to the total lifetime carbon footprint of five cars. As a point of comparison, the average American generates 36,156 pounds of CO2 emissions in a year.

To be sure, this estimate is for a particularly energy-intensive model. Training an average-sized machine learning model today generates far less than 626,155 pounds of carbon output.

At the same time, it is worth keeping in mind that when this analysis was conducted, GPT-2 was the largest model available for study and was treated by the researchers as an upper bound on model size. Just a year later, GPT-2 looks tiny—one hundred times smaller, in fact—compared to its successor.

Why exactly do machine learning models consume so much energy?

The first reason is that the datasets used to train these models continue to balloon in size. In 2018, the BERT model achieved best-in-class NLP performance after it was trained on a dataset of 3 billion words. XLNet outperformed BERT based on a training set of 32 billion words. Shortly thereafter, GPT-2 was trained on a dataset of 40 billion words. Dwarfing all these previous efforts, a weighted dataset of roughly 500 billion words was used to train GPT-3.

Neural networks carry out a lengthy set of mathematical operations (both forward propagation and back propagation) for each piece of data they are fed during training, updating their parameters in complex ways. Larger datasets therefore translate to soaring compute and energy requirements.

Another factor driving AI’s massive energy draw is the extensive experimentation and tuning required to develop a model. Machine learning today remains largely an exercise in trial and error. Practitioners will often build hundreds of versions of a given model during training, experimenting with different neural architectures and hyperparameters before identifying an optimal design.

The 2019 paper mentioned above includes a telling case study. The researchers picked an average-sized model—much smaller than headline-grabbing behemoths like GPT-3—and examined not just the energy required to train the final version, but the total number of trial runs that went into producing that final version.

Over the course of six months, 4,789 different versions of the model were trained, requiring 9,998 total days’ worth of GPU time (more than 27 years). Taking all these runs into account, the researchers estimated that building this model generated over 78,000 pounds of CO2 emissions in total—more than the average American adult will produce in two years.

To this point, the discussion has only addressed training machine learning models. But training is just the beginning of a model’s lifecycle. After a model is trained, it is then put to use in the real world.

Deploying AI models to take action in real-world settings—a process known as inference—consumes even more energy than training does. Indeed, Nvidia estimates that 80% to 90% of the cost of a neural network is in inference rather than training.

As an example, consider the AI underlying an autonomous vehicle. The neural networks first must be trained upfront to learn to drive. After training is completed and the autonomous vehicle is deployed, the model then performs inference on a continuous basis in order to navigate its environment—nonstop, day after day, for as long as the vehicle is in use.

Needless to say, the more parameters the model has, the steeper the energy requirements are for this ongoing inference.

Energy Use and Carbon Emissions

An assumption at the heart of this topic is the relationship between AI’s energy use and carbon emissions. What is the best way to think about this relationship?

According to the EPA, one kilowatt-hour of energy consumption generates 0.954 pounds of CO2 emissions on average in the United States. This average reflects the varying carbon footprints and relative proportions of different electricity sources across the U.S. energy grid (e.g., renewables, nuclear, natural gas, coal).

Strubell’s analysis, mentioned above, applies this U.S.-wide average in order to calculate the carbon emissions of various AI models based on their energy needs. It is a reasonable assumption. The power source mix for Amazon Web Services, for instance, roughly mirrors that of the U.S. as a whole, and most AI models are trained in the cloud.

Of course, if an AI model were trained using electricity generated primarily from renewables, then its carbon footprint would be correspondingly lower. For instance, Google Cloud Platform’s power mix is more heavily weighted toward renewables than is AWS’ (56% v. 17%, according to the Strubell paper).

Or, to give another example, a model that is trained on hardware located in the Pacific Northwest would generate less carbon output than the national average because of the abundant clean hydropower in that region. And it is worth mentioning that every cloud provider touts its investments in carbon offsets.

But taken in the aggregate, applying the overall U.S. power mix as Strubell does should yield a roughly accurate approximation of AI models’ carbon footprints.

Diminishing Returns

The problem with chasing progress in AI via ever-larger models is highlighted by the relationship between model size and model performance. The data here is clear: increases in model size eventually exhibit sharply diminishing returns to performance.

An illustrative example will help make this point. ResNet was a well-known computer vision model released in 2015. An improved iteration of the model, known as ResNeXt, came out in 2017. ResNeXt required 35% more computational resources to train than did ResNet (measured by total floating point operations)—and achieved a 0.5% improvement in accuracy.

A 2019 paper from the Allen Institute for AI provides detailed data documenting the diminishing returns to model size across different tasks, models, and AI subfields. The most recent mega-model released, GPT-3, displays clear signs of diminishing returns compared to GPT-2.

If the AI community continues down its current path, greater and greater amounts of energy will need to be expended to build larger and larger models—in order to achieve ever-smaller incremental improvements in performance. Any cost/benefit analysis becomes increasingly disproportionate.

Given these diminishing returns, what motivates the continued development of ever-larger models? One major reason is the AI community’s current fixation on achieving “state-of-the-art” results on performance benchmarks. Building a model that sets a new accuracy record on a well-known benchmark—even if the improvement is only a fraction of a percent—can earn researchers recognition and acclaim.

As UCLA professor Guy Van den Broeck put it: “I think the best analogy is with some oil-rich country being able to build a very tall skyscraper. Sure, a lot of money and engineering effort goes into building these things. And you do get the ‘state of the art’ in building tall buildings. But…there is no scientific advancement per se.”

The “bigger is better” ethos that currently dominates the AI research agenda threatens to inflict major environmental damage in the years ahead. Thoughtful, bold change is needed to set the field of artificial intelligence on a more sustainable and productive trajectory.

Looking Forward

To start, there are near-term “quick wins” that every AI practitioner should consider to mitigate the carbon impact of their research.

An important first step is to increase transparency and measurement on this issue. When AI researchers publish results for new models, they should include—alongside performance and accuracy metrics—data on how much energy was expended in model development.

In a thoughtful analysis, the team from the Allen Institute for AI proposed floating point operations as the most universal and useful energy efficiency metric for researchers to track. Another group created a Maching Learning Emissions Calculator that practitioners can use to estimate the carbon footprints of the models they build (based on factors including hardware, cloud provider, and geographical region).

Along these lines, it should become best practice for researchers to plot energy costs against performance gains when training models. Explicitly quantifying this tradeoff will prompt researchers to make more informed, balanced decisions about resource allocation in light of diminishing returns.

Hopefully, as sustainable AI practices spread, the community will begin to take efficiency metrics like these into account when evaluating AI research, just as it does today with traditional performance metrics like accuracy: for conference paper submissions, speaking opportunities, academic roles and so forth.

There is other low-hanging fruit that can help reduce AI’s carbon footprint in the near term: using more efficient hyperparameter search methods, reducing the number of unnecessary experiments during training, employing more energy-efficient hardware.

But by themselves, these remedial actions are not enough to solve the problem. A more fundamental long-term shift is needed in the field of artificial intelligence.

We need to take a step back and acknowledge that simply building ever-larger neural networks is not the right path to generalized intelligence. From first principles, we need to push ourselves to discover more elegant, efficient ways to model intelligence in machines. Our ongoing battle with climate change, and thus the future of our planet, depend on it.

To quote AI legend Geoff Hinton, the godfather of deep learning: “The future depends on some graduate student who is deeply suspicious of everything I have said….My view is throw it all away and start again.”

The AI community must begin to work toward new paradigms in artificial intelligence that do not require exponentially growing datasets nor outrageously vast energy expenditures. Emerging research areas like few-shot learning are promising avenues.

The human brain—that original source of intelligence—provides important inspiration here. Our brains are incredibly efficient relative to today’s deep learning methods. They weigh a few pounds and require about 20 watts of energy, barely enough to power a dim lightbulb. Yet they represent the most powerful form of intelligence in the known universe.

“Human brains can do amazing things with little power consumption,” as AI researcher Siva Reddy put it. “The question is how can we build such machines.”

Real Life. Real News. Real Voices

Help us tell more of the stories that matter

Become a founding member

Subscribe to the newsletter news

We hate SPAM and promise to keep your email address safe

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Top Stories

To Top