Deep Thunder: The Rise of Big Meteorology

Today has been predicted 26 billion times. The same could be said for tomorrow and the foreseeable days to follow. This prodigious divination is the work of just one entity—IBM’s The Weather Company. These 26 billion daily forecasts of IBM likely represent only a small fraction of the models and projections to which any particular day is subjected (the financial and military sectors are equally ambitious prognosticators). Backed by IBM’s computational juggernaut, The Weather Company is burning through terabytes at a brow-furrowing velocity in its effort to fit the world into a forecast.

A metropolitan skyline is set against a sunset sky of yellow, pink, orange, and purple. A thermometer on one of the buildings reads 59°.

IBM has spent the last decade swallowing up meteorological services such as The Weather Channel (and all its digital emanations), Weather Underground, and Intellicast, among others. Today, The Weather Company describes itself as the world’s foremost “weather provider”—a rather bold confusion of actual environmental conditions (weather, as I understand the word) with scientifically produced predictions about environmental conditions (weather, as understood by IBM). It is not incidental that it happens to be IBM that is sweeping up meteorology services. This is not just some random tech giant diversifying its financial portfolio. IBM’s specific interest in weather is not simply improving the precision of thermally motivated wardrobe choices. Rather, the company’s meteorological flirtations seem to be devoted to sharpening IBM’s artificial intelligence capacities. The Weather Company is actually a part of IBM’s Consumer A.I. branch.

Quick Bytes for $400, Alex

Perhaps best known for its Jeopardy! winning A.I. Watson, IBM’s artificial intelligence helps Wal-Mart source its produce, Delta coordinate its flights, and ExxonMobil find new sources of energy (Rometty 2019). This manner of A.I. (or machine learning) is less about engaging in humanoid forms of communication or empathy than sorting and calculating data at intrepid velocities. One thing you can say about the weather is it makes a lot of data. The copious meteorological metrics produced by national weather services and personal weather stations keep IBM’s forecasting algorithms quite well-fed. The artificial intelligence system employed in The Weather Company’s forecasts from 2016 to 2019 was known as Deep Thunder (presumably named in affinity with IBM’s chess-playing Deep Blue). Deep Thunder increased “data-handling capacity tenfold and handles 400 terabytes of data every day generating tens of millions of forecasts around the globe within microseconds at 15-minute intervals every day” (personal communication 2019). This capacity is set to be upped to 3.5 petabytes in 2019 with the launch of IBM GRAF.

More than this computational muscle however, the real utility of Deep Thunder seems laid bare in this patter from the Company’s website:

IBM Deep Thunder can also analyze weather for targeted areas retrospectively, and use machine learning-based weather impact models to help businesses more precisely predict how even modest variations in temperature could potentially have an impact on their business, from consumer buying behavior to how retailers should manage their supply chains and stock shelves; how insurance companies can analyze the impact of past weather events to assess the validity of insurance claims related to weather damage; or how utility companies can mine and model historical data of damage caused to power lines or telephone poles and couple that information with a hyper-local forecast to better plan for how many repair crews would be needed, and where (IBM 2016).

Here meteorology appears to be about much more than weather. The weather forecasting prowess honed by IBM is resold as logistical forecasting prowess to Wal-Mart, Delta, ExxonMobil. Those 3.5 petabytes of weather data are fuel for the hypothetical future of projections and models, and it is in this hypothetical future where profits lie.

It has been observed that the disciplines of meteorology and climatology have undergone a shift from focusing on earth and atmospheric physics toward computational and mathematical physics (Palmer 2016). Is the idea behind this shift that we can learn more about weather by studying computation than we can by studying clouds (not the digital kind)? Or is the word weather simply evolving, such that it has more to do with the future than the present? Implicit in the word weather is fore-knowledge of its attributes? What does such a connotative slide suggest about the utility of climatological research? (See Adam Bobbette’s recent Platypus post on forecasting for some ideas.)

Given the complexity of the calculations involved in meteorological forecasting today, no human mind is capable of monitoring all the output of Deep Thunder’s algorithms. However, IBM’s technicians lament that 100% end-to-end automation of forecasts remains about five years away (Rose et al. 2015). Much like Watson’s Jeopardy! blunder (in which they mistakenly identified Toronto as a U.S. city), Deep Thunder is still capable of making a few mistakes that are fairly obvious to meteorologists.

Out of the Loop

To meet this challenge, The Weather Company has developed a concept they refer to as human-over-the-loop (HOTL) modeling. This is a spin-off of the widely used computer science term human-in-the-loop (HITL), which describes models that employ active human participation. The “over” in the HOTL system denotes that no human intervention is necessary in the production of the model, simply that the running of the model is overseen by a human to prevent glaring gaffes. The Weather Company’s HOTL method debuted in 2014 in an effort to decrease lag time in the previous HITL systems. The HOTL reduced forecasting time by two hours. As an indication of how far we’ve come, the earliest efforts at next-day computational forecasting took 24 hours to run—making them completely useless (see Paul Edwards’ A Vast Machine).

This method of mass producing futures is a far cry from efforts at divination and prediction that human societies have always practiced. “Traditional” methods of prognostication, such as oracle bones are more invested in being able to foretell the outcome of events. The predictive practices of big meteorology are more about processing data to construct increasing probabilistic confidence in models—producing as many tomorrows as possible. Big meteorology is more concerned with transforming reality into a hypothetical projection (again, note the perversity of the concept “weather provider”). The Weather Company is only incidentally concerned with weather, insofar as it abets its computational forecasting knowledge for commercial applications.

The neoliberal temperature-industrial-complex.

Anthropogenic Weather Change

The Weather Company’s only sizeable rival in U.S. meteorology is AccuWeather. AccuWeather lacks the computational capacity to compete with IBM (AccuWeather processes a mere twelve terabytes of meteorological data per day), and has subsequently attempted to attract consumers by constructing deep forecasts of up to ninety days. This offering has been criticized by some as a gross misrepresentation of meteorology’s function and abilities. Most meteorologists are skeptical that their methods have much predictive capacity beyond ten days, and certainly not for over a month. Despite having trademarked the phrase “Superior Accuracy,” those that have tracked AccuWeather’s long range forecasts have found them to be no better than guesswork (ninety days from this writing AccuWeather predicts “a morning t-storm in spots”). This shlocky meteorology denigrates the science, much like the muckraking of The New York Post or CNN denigrates journalism.

Finally, on the matter of denigration, it is impossible to escape the politicization of weather and climate today. Below the radar of megastorms and other foreboding environmental calamities, lies the seedy politics of for-profit weather. AccuWeather’s owner Barry Lee Myers was a significant donor to former republican senator Rick Santorum, who in 2005 introduced a bill attempting to restrict public access to the publicly funded information produced by The National Weather Service (a division of NOAA—National Oceanic and Atmospheric Administration). Such a measure seems designed to make consumers of meteorological information reliant on entities like AccuWeather or The Weather Company. Most distressingly though, Myers was nominated by President Trump to head NOAA in 2017. Mr. Myers’ twenty-year effort to undermine NOAA and his B.A. in business would make him peculiarly qualified among his predecessors. (While Myers has awaited congressional confirmation some of AccuWeather’s Best Practices have been revealed.) Such efforts by AccuWeather to infiltrate government and IBM to construct the future illustrate that there is big money in big weather (and where there is big money there is big ethical equivocation).

Photos by author.