Big data’s big impact
The Internet is, of course, old hat. We are all getting used to social media, too — your grandmother probably has a Facebook account, and every CEO worth his salt, along with all the world’s would-be revolutionaries, is on Twitter. Mobile, once the new thing, is now taken for granted as part of the world’s hardware. In 2010, more than 4 billion people, or 60 percent of the world’s population, were using mobile phones. Twelve percent of them were smartphones, whose presence is increasing more than 20 percent a year.
But don’t get complacent. A new wave of the technology revolution is cresting and, like its predecessors, will again change the way we work and live. This latest transformation is being called “big data” — a term for the vast amount of digital data we now create and have an increasing ability to store and manipulate.
If wonks were fashionistas, big data would be this season’s hot new color. When I interviewed him before a university audience a few weeks ago, Lawrence H. Summers, the Harvard professor and former Treasury secretary, named big data as one of the three ideas he was most excited about (the others were biology and the rise of the emerging markets). The McKinsey Global Institute, the management consultancy’s research arm and the closest the corporate world comes to having an ivory tower, published a 143-page report last year on big data, trumpeting it as “the next frontier for innovation, competition and productivity.”
To understand how much data is now at our fingertips, consider a few striking facts from the McKinsey report. One is that it costs less than $600 to buy a disk drive with the capacity to store all of the world’s music. Another is that in 2010 people around the world collectively stored more than 6 exabytes of new data on devices like PCs and notebook computers; each exabyte contains more than 4,000 times the information stored in the Library of Congress.
McKinsey believes that the transformative power of all of this data will amount to a fifth wave in the technology revolution, building on the first four: the mainframe era; the PC era; the Internet and Web 1.0 era; and most recently, the mobile and Web 2.0 era.
Like the four previous stages of the technology revolution, McKinsey predicts big data will lead to a surge in productivity. In the U.S. retail sector alone, for example, the consultants calculate that big data could increase a retailer’s operating margin more than 60 percent.
Those improvements are an almost unalloyed good for the consumer — who doesn’t like Amazon’s recommendations or Walmart’s low prices, two innovations facilitated by pioneering use of big data?
But, as with the rest of the technology revolution, big data is likely to have a more mixed impact on us in our role as workers. David Autor, an economist at the Massachusetts Institute of Technology, has done groundbreaking research on the “polarizing” effect of technology on the labor market: his bottom line is that it has been good for people at the top and not had much of an effect on people doing hands-on jobs at the bottom. But it has hollowed out what used to be the middle. Studying the same phenomenon in the United Kingdom, the economists Maarten Goos and Alan Manning have come up with an evocative term for what is happening — the division of work into “lousy and lovely jobs.”
The lovely jobs are why we should all enroll our children immediately in statistics courses. Big data can only be unlocked by shamans with tremendous mathematical aptitude and training. McKinsey estimates that by 2018 in the United States alone, there will be a shortfall of between 140,000 and 190,000 such graduates with “deep analytical talent.” If you are one of them, you will surely have a “lovely” job, and one that is well-paying to boot.
The work of Autor, Goos and Manning on the impact of the technology revolution so far shows that it has eroded what used to be the middle class by replacing a lot of white-collar and relatively well-paid blue-collar jobs — a travel agent, for example, or many factory positions — with computers and robots. What’s left are the “lousy” jobs — washing floors or wiping tables — that can’t be outsourced or automated.
That trend will surely be exacerbated by big data. Some of those improved margins in the retail sector, for instance, will come from what McKinsey delicately terms “the optimization of labor inputs.” That’s another way of saying that if you use big data to track your sales more precisely, you need fewer salespeople. That’s good news for us as customers, but bad news if you need a job.
This division of the world into lovely and lousy jobs is a pressing political problem. Big data will make it worse. The good news is that it might also offer a partial solution. McKinsey argues that big data could make both healthcare and the provision of government services cheaper and more effective. As an example, it points to the German Federal Labor Agency, a vast bureaucracy with 120,000 employees. Big data techniques helped it to save around €10 billion, or $12.7 billion, in recent years.
This hints at what is the truly revolutionary potential of big data. Inevitably, this latest wave of the technology revolution — like its four predecessors — will transform our lives as consumers and as workers. But so far the technology revolution has lagged in its impact on us as citizens. If our governments can begin to close that gap, then, as societies, we might just have a chance to bridge the growing divide between lousy and lovely jobs.