Big data’s big impact

By Chrystia Freeland
January 12, 2012

The Internet is, of course, old hat. We are all getting used to social media, too — your grandmother probably has a Facebook account, and every CEO worth his salt, along with all the world’s would-be revolutionaries, is on Twitter. Mobile, once the new thing, is now taken for granted as part of the world’s hardware. In 2010, more than 4 billion people, or 60 percent of the world’s population, were using mobile phones. Twelve percent of them were smartphones, whose presence is increasing more than 20 percent a year.

But don’t get complacent. A new wave of the technology revolution is cresting and, like its predecessors, will again change the way we work and live. This latest transformation is being called “big data” — a term for the vast amount of digital data we now create and have an increasing ability to store and manipulate.

If wonks were fashionistas, big data would be this season’s hot new color. When I interviewed him before a university audience a few weeks ago, Lawrence H. Summers, the Harvard professor and former Treasury secretary, named big data as one of the three ideas he was most excited about (the others were biology and the rise of the emerging markets). The McKinsey Global Institute, the management consultancy’s research arm and the closest the corporate world comes to having an ivory tower, published a 143-page report last year on big data, trumpeting it as “the next frontier for innovation, competition and productivity.”

To understand how much data is now at our fingertips, consider a few striking facts from the McKinsey report. One is that it costs less than $600 to buy a disk drive with the capacity to store all of the world’s music. Another is that in 2010 people around the world collectively stored more than 6 exabytes of new data on devices like PCs and notebook computers; each exabyte contains more than 4,000 times the information stored in the Library of Congress.

McKinsey believes that the transformative power of all of this data will amount to a fifth wave in the technology revolution, building on the first four: the mainframe era; the PC era; the Internet and Web 1.0 era; and most recently, the mobile and Web 2.0 era.

Like the four previous stages of the technology revolution, McKinsey predicts big data will lead to a surge in productivity. In the U.S. retail sector alone, for example, the consultants calculate that big data could increase a retailer’s operating margin more than 60 percent.

Those improvements are an almost unalloyed good for the consumer — who doesn’t like Amazon’s recommendations or Walmart’s low prices, two innovations facilitated by pioneering use of big data?

But, as with the rest of the technology revolution, big data is likely to have a more mixed impact on us in our role as workers. David Autor, an economist at the Massachusetts Institute of Technology, has done groundbreaking research on the “polarizing” effect of technology on the labor market: his bottom line is that it has been good for people at the top and not had much of an effect on people doing hands-on jobs at the bottom. But it has hollowed out what used to be the middle. Studying the same phenomenon in the United Kingdom, the economists Maarten Goos and Alan Manning have come up with an evocative term for what is happening — the division of work into “lousy and lovely jobs.”

The lovely jobs are why we should all enroll our children immediately in statistics courses. Big data can only be unlocked by shamans with tremendous mathematical aptitude and training. McKinsey estimates that by 2018 in the United States alone, there will be a shortfall of between 140,000 and 190,000 such graduates with “deep analytical talent.” If you are one of them, you will surely have a “lovely” job, and one that is well-paying to boot.

The work of Autor, Goos and Manning on the impact of the technology revolution so far shows that it has eroded what used to be the middle class by replacing a lot of white-collar and relatively well-paid blue-collar jobs — a travel agent, for example, or many factory positions — with computers and robots. What’s left are the “lousy” jobs — washing floors or wiping tables — that can’t be outsourced or automated.

That trend will surely be exacerbated by big data. Some of those improved margins in the retail sector, for instance, will come from what McKinsey delicately terms “the optimization of labor inputs.” That’s another way of saying that if you use big data to track your sales more precisely, you need fewer salespeople. That’s good news for us as customers, but bad news if you need a job.

This division of the world into lovely and lousy jobs is a pressing political problem. Big data will make it worse. The good news is that it might also offer a partial solution. McKinsey argues that big data could make both healthcare and the provision of government services cheaper and more effective. As an example, it points to the German Federal Labor Agency, a vast bureaucracy with 120,000 employees. Big data techniques helped it to save around €10 billion, or $12.7 billion, in recent years.

This hints at what is the truly revolutionary potential of big data. Inevitably, this latest wave of the technology revolution — like its four predecessors — will transform our lives as consumers and as workers. But so far the technology revolution has lagged in its impact on us as citizens. If our governments can begin to close that gap, then, as societies, we might just have a chance to bridge the growing divide between lousy and lovely jobs.

17 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

Ms. Freeland,

Yes, “big data” absolutely will continue to replace
“…a lot of white-collar and relatively well-paid blue-collar jobs…” with computers and robots. As Willie Sutton (bank robber) famously said: “That’s where the money is”.

Computers and robots can be “on duty” 24/7 as desired. They don’t commute, carpool, park, have “moods”, bicker among themselves, offend customers, complain, need time off unexpectedly for “emergencies”, join unions, strike, or get raises, bonuses, vacations, pensions, sick or pregnant. There are HUGE savings/incentives for the imaginative and flexible employer/purchaser.

And yet computers and robots are quickly clueless when it comes to taking collected voluminous data. None yet have sufficient “perception undefined” to progressively organize and evaluate it so as to distill it into something with practical/commercial use.

It is not yet clear if we should cheer or fear the day when “big data” leads to one or more algorithms capable of accurately determining when a bureau or division is understaffed or overstaffed for it’s purpose, when a function costs more than that function is deemed worth, and whether or not a politician’s actions and inactions are “in the public interest” or otherwise.

Posted by OneOfTheSheep | Report as abusive

The challenge will be to find “meaningful” data…the rest of the big data will be a mere distraction, albeit a profitable one for some.

Posted by DrPatil | Report as abusive

Ever wonder as you become more dependent and driven by this (big) digital media data, what if a large solar storm (with some sun activity) wipes out all the data over, including the backup media data sitting in caverns, with cosmic particles/radiation?

Posted by Mott | Report as abusive

The comparison of the amount of data stored by all people with the holdings of the Library of Congress is misleading. Much of the data people download is identical to what other people download, so it is not a measure of the information content. Importantly, material in a library is organized for useful access; the millions of separate data storage devices on everyone’s computer are not.
It is hard to understand what is new in what is being called “big data”. The trends of automation and computerization of work functions have been going on for at least 50 years, and these trends will continue. Some jobs, both good and bad, are much harder to automate than others. Some of the worst (most repetitive, most dangerous) were among the first to be automated. Even with the biggest data, much remains unpredictable.

Posted by m11213 | Report as abusive

“A gigabyte here, a gigabyte there, and pretty soon we’re talking about real data”, Sen. Everett Dirksen might have said. The proverbial GIGO (garbage in/garbage out) comes to mind leading to the need for a new GIGOOLE search engine perhaps. More seriously, big data may prove more interesting at the metadata level, as it would appear impossible to make sense out of all the details at the exabyte level of data having spent 30 years in the Decision Support/B.I./Analytics market.

Posted by Mazzini | Report as abusive

@OneOfTheSheep: “It is not yet clear if we should cheer or fear the day when “big data” leads to one or more algorithms capable of accurately determining when a bureau or division is understaffed…”

It’s clear … and the answer is “fear”. The day is already here, even though the algorithms don’t yet exist. Problem is: too many executives believe they exist, and this permits said executives to make ‘objective’ decisions from those algorithms, in an attempt to relieve themselves from responsibility for either the decision or the duty to think about the problem. Of course, such executives are accustomed to doing that; they’ve been doing it for centuries, but up to now they have been able to blame their staffs (even when the staffs were deliberately composed of yes-men.) Now they don’t even need to hire a staff. … Have I examples? Yep. Trading programs. Loan approval algorithms. Underwriting algorithms. Automatically calculated management metrics. Point to the generated numbers or graphs and decide … without concern for the quality of the data, the measurable rate of error, or the history of success of such decisions. No actual brains involved … and ‘blame’ neatly avoided.

Posted by tejh | Report as abusive

Interesting that L Summers considers it as one of the top 3 contemporary ideas (along with Biology (should be it Health 2.0?) and Emerging Market (no longer emerging?)…but the view that statistics/(traditional) mining/bottom-up analysis/throwing lots-of-computing is the primary strategy to find insights and efficiencies sounds a bit too “traditional” for me. Think how our brain processes massive amounts of information it receives from senses– it also utilizes background knowledge and experiences. Believe we’re making good progress in approximating that with semantic technologies (as recently argued in “semantics scales up:… Web 3.0″ article): http://bit.ly/ic-s

Posted by Aalag | Report as abusive

Decades ago people imagined a world in which machines would do much of the work. Everyone thought how pleasant life might be. No one seemed to think about the problem of jobless people having no money. But ideally this Brave New World could benefit everyone if everyone did their fair share of work that truly needed doing.

Posted by 123456951 | Report as abusive

I agree with DrPatil. What comprises meaningful data and what exactly does it mean? There are many governments and corporations that have a vested interest in presenting data that indicates what they wish it to indicate.

However, if the “Big Data” concept means that accurate numerical information will be available to the masses and identifiable from the myriad sources of bogus data, that would allow statically astute citizens/investors to draw their own conclusions regarding the meaning of the data.

I am wholeheartedly in favor of that, but doubtful of a future where uncompromised, poignant data will be available for downloads at no cost or for a modest fee.

Posted by breezinthru | Report as abusive

How could this be more out-of-date? Around 1980, many business leaders and digital data experts were concerned with problems related to the tsunami of digital data. As an executive at Citicorp, now Citigroup, one of my biggest challenges in interpreting financial transaction trends for other executives was sifting through the reams of data.

Even then, issues of “information overload” were a concern to the designers and users of computerized displays in fighter aircraft.

Since then, the volume has increased over a thousand fold. Much has been done to consolidate and store these massive volumes of data. Also, Business Intelligence software has been applied to glean information and even knowledge from massive data bases. But,nothing particularly significant happened within the past two years that hasn’t been known for thirty years.

How do you pronounce DOS? Are you Ms Rip Vank Winkle?

Posted by ptiffany | Report as abusive

@ptiffany,

It’s your personal perspective that is out of date”. The dramatically dropping cost of computing power, capability and storage combined with access to an ever-expanding internet are increasingly leveraged by both employed and private individuals to ever more rapidly expand human knowledge. Today information can be in the “cloud”.

Consumers will soon have eyeglasses with “heads up data” (HUD) displays and “steroscope-like” image capability to replace today’s flat-screen monitors. Existing earphone/headsets will increasingly be used for speech-based computer control interface.

People interested in the most esoteric subject now have a way to study, communicate and combine knowledge. While most discoveries originate with an individual whose understanding “stands on the shoulders of those who came before”, current conditions allow and encourage knowledge to expand almost immediately and to a much greater extent between an ever-increasing number of participants.

Such advances have made possible the sequencing of the human genome. Machines that can “do” an individual’s DNA sorting for about $1,000 are today available.

Custom genetic treatments that attack disease with the accuracy and effectiveness of a rifle as opposed to a shotgun are being engineered and tested. Medical knowledge and treatments are expanding at a rate much faster than our social system can pay for them.

It won’t be long before medical interfaces between our hearing and speech centers replace headsets and CPUs implanted between our eye and brain neural networks replace desktops, laptops, iPhones and iPads? The first person to live to 1000 may have already been born!

It’s been a long time since “big bucks”, Cobol, Fortran and DOS were keys necessary to meaningful human access to processing power. How do you pronounce Macintosh OSX and Windows?

Posted by OneOfTheSheep | Report as abusive

There is an ad for an auto manufacturer that advertises that it makes a superior product because it’s virtual crash test dummy uses millions of data points versus only the few strategic points used when they were using the physical crash mannequin.

I suspect most of the points are meaningless and don’t add much to the designers ability to improve the car’s safety or reliability. It cost almost $1000 more to repair a failed chip in my car (that I was not permitted to even see let alone test) to have a disinterested test that it was at fault for the car’s failure to start even after it had made a 600 mile trip. That chip could have also been programmed to fail by the manufacturer when the manufacturer wanted it to. You might want to reconsider having chips implanted in your brain (OOTS). You would be forced to rely upon the scrupulous disinterested honesty of those who control you chip, remember?

The cost of automobiles has still risen enormously. The information may or may not be useful, just as so much of what the Library of Congress stores may be redundant, out of date and outright inaccurate. It is an archive of human knowledge but also of human error.

It may be a wild flight of fantasy to think that increased information is going to lead to an improved quality of life. It can also lead to a vastly improved knowledge of how much is going wrong or can go wrong.

The basics, like food, clean water, affordable energy and affordable health care seem to be inert to vast amounts of knowledge. But some simple means and the right application of human knowledge can work even better in some cases. In fact, technology seems to be contributing to the massive cost increases or those services. Somehow, all those technological goodies have to be paid for and using the technology or the product, inappropriately or unnecessarily, is a way to do that. And from personal experience, I can’t say that the quality of health care care for basic issues has improved at all. In fact, I’m sure it is worse. The Doctors can still make misdiagnoses and fail to use their own judgment. Vast amounts of “information” will also lead to vast amounts of disinformation.

Posted by paintcan | Report as abusive

Chrystia Freeland: “If our governments can begin to close that gap, then, as societies, we might just have a chance…”
This is a scary and delusional premise that asserts a government should provide the job market with solutions to close a natural gap, created by innovation between the “lovely” jobs with shiny new toys and the average menial “lousy” jobs without such fancy new toys. What a pitiful construct and a pitiful analysis. Technological innovation and creative destruction are a natural occurrence in a competitive and innovative economy.

Such paradigm shifts lead to gaps in skills and displace individuals in the labor force, forcing them to adapt, learn new skills, and sometimes require relocation. To not understand this, to find it alarming that it occurs, and to suggest government intervention, so “societies…might have a chance,” is ignorance of the marketplace of the highest magnitude.

It is scary that Ms. Freeland may have a college education (multiple!) and does not realize that people “washing floors or wiping tables” will still be “washing floors or wiping tables,” and many other dirty jobs, whether they have an iPhone or not. No amount of technology or government intervention in the marketplace will ever remove the fact that in life there are always going to be, so called “lovely” jobs, and more importantly, “lousy” dirty jobs!!

It is obvious that Ms. Freeland has led a privileged life, according to her bio. Now, maybe, she might consider helping to close the gap by pushing away from the keyboard of her “lovely” job and try getting her hands dirty! Such arrogance! It is simply astonishing!

Posted by GolgoBrone | Report as abusive

@paintcan,

As usual, your inability or unwillingness to “see” blinds your insight. I agree that “…Doctors can still make misdiagnoses and fail to use their own judgment. Vast amounts of “information”…also lead to vast amounts of disinformation.

So it is still YOUR responsibility to decide if the doctor YOU use is competent and has no malice towards YOU and YOU still have to sift through bulk “information” to separate the wheat from the chaff. That doesn’t change the fact that those capable and willing enjoy greater opportunities and longer lives even if YOU don’t.

Would you refuse a pacemaker because you fear someone may program it to fail? If so, maybe you should change your “ways”, or get help with your paranoia. “…vastly improved knowledge of how much is going wrong or can go wrong” is what allows “fly by wire” computer systems to make a fighter inherently unstable that is infinitely maneuverable controllable by the pilot and useful in battle. The cup half empty is also half full.

You “…can’t say that the quality of health care care for basic issues has improved at all. In fact, I’m sure it is worse.” Oh, really?

In just the last 80 years people in America don’t often die of appendicitis, don’t get smallpox or polio. Antibiotics cure conditions once fatal. Prosthesis have come a long way from the hook for a hand and wooden “peg” leg. At 70+ I still have every one of the teeth I was born with. We have flu, pneumonia and shingles shots.

“The cost of automobiles has still risen enormously.” That depends on how you look at “cost”. Governments routinely allow their currency to inflate, betraying those on fixed pensions. It takes 4.83 dollars today to buy what a dollar bought in 1970. I can’t change that, and neither can you. The age of the average car in service is now over ten years. Back then it was half that.

That 240Z Datsun I bought new for $3600 (added air conditioning later) would today cost $21,000. My 1999 Chevy Metro coupe, bought new for 11,000 came with air conditioning and averages 33 MPG in town or cruising at 70 mph. I get 40,000 miles from a set of Michelin tires and hope to enjoy it another ten plus years.

Life may be no longer worth living for you, but for those who still greet each new day with enthusiasm and anticipation it has never been better!

Posted by OneOfTheSheep | Report as abusive

@OOTS – I didn’t bother reading your comment except for the first and last lines. You haven’t the foggiest idea how I greet the new day so don’t try to guess, you old fraud.

I spent his morning trying to get the old stove to heat my small house at below freezing outdoor temps. I’ve never had the cash to bribe my way into plum contracts. You stinking fraud.

I hope some day soon you’re broke and living on handouts. Then I will believe in the depth of your enthusiasm and good spirits. Some road to an enthusiastic start to a new day you offer. Your homilies are fake and cheep shots and delivered by a liar and a bit of a con man.

Posted by paintcan | Report as abusive

Thank you for creating awareness that “A new wave of the technology revolution is cresting and, like its predecessors, will again change the way we work and live.” It melds with your comments on last Saturday’s “Your Money” panel on Wall Street. Individuals need to bypass Wall Street and direct money to the businesses of the future, so that income from investments may supplement and replace the jobs that are going away. I’ve referred to your blog in bypasswallstreet.com.

Posted by drewfield | Report as abusive

@ paintcan — Thou puking idle-headed dewberry!
Perhaps you would like to take a second effort at understanding, or at least reading, OOTSheep’s point?
Thy bones are hollow; impiety has made a feast of thee.

Posted by EPB | Report as abusive