How the tech boom is bad for innovation

By Felix Salmon
October 14, 2011
Jon Stokes has a fascinating column making a credible case that the VC and tech bubble is hampering development of the cloud.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

Jon Stokes has a fascinating column making a credible case that the VC and tech bubble is hampering development of the cloud.

I recently had a sit-down chat with Ping Li, a venture capitalist at Accel Partners who does investments across the layers of the cloud stack… he explained that the talent shortage is stifling fundamental innovation in the cloud space.

To do really fundamental engineering innovation of the kind that was done, say, in the early days of Google and VMware, you need to hire and retain teams of talented engineers. But in today’s go-go funding environment, top engineers are being enticed with truckloads of money to break off and form two- and three-person startups. This phenomenon, explains Li, is why “many of the really big innovations happen in less frothy times.” He did go on to clarify that “some great companies do get created in these times (like Amazon in the last bubble). It’s just harder given talent shortage.”

I asked Jon if this meant that real innovation in the cloud requires pretty big teams, and can’t be done by smaller startups. And the answer there is absolutely yes — if you’re looking for something huge like Amazon’s AWS, which required the full focus of Amazon’s large technical staff over a multi-year timeframe. Scalable websites can, thanks to Amazon’s cloud, now be launched with a handful of employees. But to develop the cloud itself takes serious resources — to the point at which it’s now conventional wisdom that you need to be Amazon, Apple, Google, or Microsoft to even play in that game. And even they’re starving for talent.

There’s the email I got a few months ago from a friend of mine and product manager at Apple, who was wondering if I knew any cloud computing hackers that they could hire. When we get to the point where Apple product managers on the client side are reaching out to their personal networks in search of cloud coding talent for the world’s largest tech company, you know it’s bad out there.

Jon frames the problem as one of supply and demand:

The current crisis in the cloud is the product of too many dollars and transistors chasing too few coders and sysadmins. It will take a while for the latter to catch up with the former… unless, of course, another major downturn strikes. It seems ironic that less money could equal more innovation, but it wouldn’t be the first time that a wave of downsizing and tight money boosted productivity.

I asked him whether looser skilled-immigration policies might help, and he said probably not:

I think that the root problem isn’t one of geography–it’s that this stuff is happening so fast (i.e. Moore’s Law and my cheap transistors argument) that the hardware build-out is outstripping programmer education. And by “programmer education” I mean not only the number of programmers being trained in aggregate across the world, but also programming as a discipline’s ability to empower ordinary mortal to develop and deploy software on these massively parallel systems. The cloud has to be “de-ninjafied”, so to speak. Getting max productivity out of the cloud has to be brought within the grasp of non-ninjas, the way that, say, VisualBasic from MSFT brought building a relatively complex custom relational database application within the grasp of the average local technical college graduate.

This rings true to me. The cloud is not located in any particular country, and if there were great engineers who could be hired to work on it from Beijing or Bangalore, I’m sure that Apple and Amazon are more than capable of doing that. What needs to be done here is basically cloud-development grunt work: taking a young and complex technology, and building the tools which can bring it to the masses. There’s not a lot of glory in that — while companies which live in the cloud, like Airbnb, can find themselves worth billions, the engineers who work on the cloud are more like the utility workers of the internet. And it’s easy to see why they might be finding more attractive opportunities, right now, elsewhere.

Here’s how Jon puts it:

In order to move the cloud itself forward in a major way by solving large batches of related fundamental technical problems you need longer timeframes. You can fiddle around in the guts of the cloud, smoothing out this and optimizing that, and adding features and bells and whistles. But to do the big projects, you need time.

Now, there are shorter-term innovations that can and will get done in the cloud, so VCs have plenty to fund. But to shift the tectonic plates, you need time and resources.

This isn’t like sustainable energy — it’s not something that the government can or should be stepping in to fund. More money pouring into the tech space would only exacerbate the current problems.

There’s a case to be made that AWS is the result of what happened when Amazon, after the dot-com bust, found itself with an unusual degree of access to the time and talent of a large number of engineers. The cloud is young; it could do with a lot more development along those lines. But as Jon says, we’re unlikely to see such fundamental evolution in cloud architecture for a while. Because for the time being, smaller, lighter, and riskier projects look much more attractive.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
16 comments so far

I’m one of those utility workers trying to bring Big Data technology to the masses, and this theory doesn’t ring true to me.

Most of the interesting large-scale data work is being done through open-source projects, the only things that aren’t are those (like AWS) that require large capital expenditure in hardware. Good examples are Cassandra, Hadoop, or OpenStack. These have an interesting history, a lot of the engineers behind them started at big companies like Google and Amazon, and when they left to found startups they recreated and improved on the systems those large companies had built. They often then decided to open-source the results, so you end up with large teams working on batches of related problems. It’s just that they’re not at traditional corporations, they’re scattered through a lot of different startups.

Are these innovations? They’re certainly big, ambitious projects that are eclipsing what’s being produced at large companies:
http://www.wired.com/wiredenterprise/201 1/10/microsoft-and-hadoop/

Maybe it’s my closeness to the subject, but I’m struggling to understand what Jon and Ping’s definition of ‘real innovation’ actually is. Seen through one lens, EC2 is just a fantastic execution of an existing VM hosting model. If you look at the convenience and refinement they offer as the real innovation, then Heroku clearly made an equally major leap beyond them, despite being a young startup.

Posted by petewarden | Report as abusive

Wait, we’re upset now because one bubble is blocking the inflation of another bubble?

First of all, let’s get it straight – the boom in ad-supported websites is not a “tech” bubble. Yes, technology is used to create those sites, but it is used for just about everything these days. Blame FedEX, blame Wal-Mart, blame BankofAmerica (they deserve it anyway) whoever – they all are absorbing web programmers to expand and maintain their sites. But it’s not a tech bubble, it’s the assimilation of the most recent medium for mass communications.

As for the latest dot com bubble stifling the cloud bubble, investors should be happy, for this will save them money. The cloud is just the internet, and those selling the cloud are just using buzzwords and other marketing babble to sell their services. Without the ad-supported website bubble to exhaust the supply of web engineers, cloud companies would be a sinkhole for capital, for once people find out that any savings realized by the scaling of data centers is more than offset by overhead and profit margin, they will revert back to operating their own “cloud”. At least when long distance fiber lines were overbuilt in the 90s, they eventually became useful when all that bandwidth started being utilized. By the time there is an oversupply of cloud data centers, they will be obsolete, and will be turned into efficient, ultra-modern warehouses.

Posted by KenG_CA | Report as abusive

There is some disconnect here as employment in this area only past the level it was at in 2000 in the last 2-3 years. In that time nearly a decade of new grads faced a stagnant market and I imagine many of them left. This is just the industry trying to hire at costs that don’t cover the cost of expanding industry, because as much as they deny it, the work isn’t valuable enough to pay for it. Yes, even I have been contacted for opportunities ten years after I left the field, so there is some desperation but that vanishes when faced with the cost.

Posted by MyLord | Report as abusive

“… wondering if I knew any cloud computing hackers …”

There’s a big chunk of the problem right there. Someone has called up HR and asked for “cloud computing hackers” rather than computing hackers who could develop cloud computing support software. There’s a huge difference, but you might have to wait a few weeks and invest in a little training – some text books, maybe a seminar – before you started getting output from your new hire. This is generally considered unacceptable in the modern corporation, so they suffer from this kind of “shortage”. Every time there is a new buzz word, they create a new shortage, rather than figuring out what skills they really need, hiring appropriately and investing in their staff.

Cloud services, for example, have been around since the mid-70s when the first networked file systems and servers were developed. By the 80s, the basic pieces and concepts of what we now call cloud computing were in place, generally in familiar forms. There are lots and lots of software people who can get up to speed doing cloud computing given a few weeks of start up time, but managers are looking for buzzwords, not skills or knowledge. There is no way that HR would even recognize that an engineer even worked on a cloud computing project since that buzzword might not have existed at the time.

Since the buzzwords change frequently, we get the myth of the outdated hacker. Some of it is age discrimination. It has been completely internalized, partly because more experienced and more productive hackers tend to cost more, and HR considers body count and average salary management more important than getting the job done. There job is to hire, not build systems in the cloud. It isn’t outdated skills that limit the duration of STEM careers, but rather buzzword change. Instead of looking for cloud computing hackers, look for distributed networking hackers or oxygen file system hackers. You can probably find all the old buzz words on the internet these days in a matter of minutes. You can then hire at your leisure, though you might have to convince your targets to quit their post-STEM career and return to the fray.

Posted by Kaleberg | Report as abusive

> “… wondering if I knew any cloud computing hackers”

Apple is on top of the world right now, so we can be pretty sure this is either an isolated misfire or is inaccurately reported. But if someone actually tried to fill job openings this way (cloud computing hackers!?!) their doom would be fairly inevitable not matter what the venture-capital situation looked like.
Felix, you aren’t a techie but are fairly plugged-in; did you not see at once what a howler this is?

Posted by bxg6 | Report as abusive

@Kaleberg: Well stated. I was going to make a similar point but couldn’t have said it better.

Posted by spectre855 | Report as abusive

Felix, you look at the tech job market with a macro view of supply and demand, but that’s a dangerous simplification. The real problem is that we do a terrible job of matching up candidates with jobs, even if there is an exact match. Your Apple product manager may be setting unreasonable or unnecessary qualifications for the job, or HR may not be recruiting in the right places. Incidentally, “hacker” is not a particularly favorable description these days, which makes me wonder about the story in general.

Your use of one or two anecdotes is inappropriate to draw the conclusions you do (even though you commonly do it). That said, I think your opening proposition (that you pretty much neglected after the opening) that there is less pure innovation during boom times may in fact have legs, although for somewhat different reasons.

Last, it doesn’t take large companies/lots of engineers per se as much as it does enough servers to be able to promote a cloud infrastructure standard. You didn’t mention Rackspace, for example, and as someone else mentioned, there continues to be a lot of innovation in open source, although open source tends to lag commercial R&D by a year or two.

Posted by Curmudgeon | Report as abusive

I’ve been an engineer on “the cloud” for longer than it was called that (man, I hate that term). I also disagree with the claims here.

Yes, demand in one layer is going to take engineers away from low-level cloud systems. But the whole point of “the cloud” is that everyone gets to outsource that level of their stack to people who are really good at it, so you should need fewer of these people overall. And if you move further up the stack, startups like MongoDB or Parse are now able to build cloud middleware on top of lower-level cloud systems.

That said, I think money is a little easy for startups these days, and it does create a talent war/drain when talented people go to unsustainable startups. But I don’t think “the cloud” is hurting.

Posted by AndrewNYC | Report as abusive

what Kaleberg said

Posted by arrgh | Report as abusive

So large corporations with record profitability complain that they cannot get enought software engineers for cheap? Pay them more. Supply and demand in a non unionized labor market. Nasty, when capitalism works for the employee for once, isn’t it?

Sorry for sounding sarcastic, but this piece is really asking for it.

Posted by Finster | Report as abusive

It’s the management!

The most common fall-back for senior management is to blame the workers (don’t work hard, not good enough, lack skills etc.). You see it in these claims, teaching in the school systems, the auto industry collapse etc.

Brooks was writing about this with respect to programming 4 decades ago: http://en.wikipedia.org/wiki/Brooks%27_l aw

He also pointed out that the most productive programmers are 10 times as productive as the average programmer. As a result, small projects can be highly efficient if they have one of the good guys. Unfortunately, it is not possible to have an industry filled with above-average programmers, so large projects cannot be as efficient as small projects.

On big projects, the efficiency and innovation have to come out of how the hive is managed. Good managers mean on-time projects, within budget, while identifying innovations that arise. A bad management team could tank the same thing.

Posted by ErnieD | Report as abusive

Kaleberg & AndrewNYC +1

Sorry, the problem is really bad/lazy/stupid HR people and their retarded “applicant tracking systems”.

Also, everyone here in SV knows that Apple is a terrible place to work. There are much better options out there and people are taking them.

Posted by ChrisMaresca | Report as abusive

Sorry Felix, this whole thing reads like buzz word bingo drivel forwarded by the always attention seeking Stokes. Anyone in the industry could probably spot that there is no such thing as a ‘cloud engineer’, there’s even really no such thing as the ‘cloud’, you might as well write an article about ‘multimedia’ or ‘network computing’.

Posted by inboulder | Report as abusive

It’s also difficult thanks to the intellectual property laws surrounded the area right now. It is almost impossible to profit off of performance improvements unless you are a big enough player that they are worth doing just for yourself. Most wide-spread performance-enhancing technologies come out of either the open source community or Microsoft (which has a vested interest in people writing software that runs on Windows). None of the cloud players are big enough to fund it themselves, and the open source efforts are just now starting (OpenStack is probably the most promising at the moment).

If we had corporate environments or intellectual property right laws more friendly to compatibility, open source or parallel development we’d have a better shot at being innovative. Instead, several start-ups and projects have been scuttled by the threat of patent law suits, inter-compatibility is scuttled by the DMCA and open source projects have run into issues building on top of changeable proprietary APIs. The other reason less money would lead to more innovation is that it would result in fewer lawyers chomping at the bit; early programming productivity enhancements profited greatly from not being worth suing over.

Posted by BlakeA | Report as abusive

Looks like you’ve nailed this entire recession, Mr. Salmon!

The industrial realignment that was predicted with irrational exuberance ten years ago during the “dot com” boom and bust; is finally happening; later and slower than its proponents expected, but still happening.

The coming economy is the product of a new industrial revolution – only this time, it’s an information revolution. The industrial revolution amplified the power of Man’s arm. The information revolution amplifies the power of Man’s brain…

A writer in the Scientific American magazine, about 15 years ago, had it right; in suggesting that the economic gap between the knowledge-”haves” and the knowledge-”have nots” would radically widen. Those who out-compete the rest of us by a small margin will be able to “earn” an increasing share (indeed, an increasingly unfair share) of the available economic rewards, in the future; unless something is done to rebalance the system and make it fair again.

There’s only one way to fix this economy. It’s not by reducing taxes. It’s by radically INCREASING taxes on the wealthiest and most privileged, to pay for the perpetual retraining of displaced workers who despite talent and hard work invested heavily in the “wrong” industries. Obsolescence will become an increasing problem in the future, and it will happen multiple times in the average worker’s career…

The question is, will we fix the system BEFORE America becomes a chattel, pawn and playground of a new aristocracy? Will America be a land of mercy and justice, or a land of misery and exploitation?

(from a computer science graduate)

Posted by matthewslyman | Report as abusive

guys cloud is just mainframe on someone elses network connected to the internet. That may be oversimplified but that’s the basic idea. Not a lot different than computing before personal computers. Just bigger.

But you want to know why you can’t find people. Look at the tech industry.

1. Older tech employees are considered dried up and beyond innovation.

2. Entry to Mid level jobs are drying up. Most are being outsourced. Programmers are treated like chained dogs. If they complain it is pointed out that 100 people in India want thier job.

3. Tech jobs require more knowledge, a far bigger part of your income training and keeping up as your career progresses and you will forever be working in “Cost” center instead of a “Profit” center and be treated as a second class citizen as a result.

For 20 years now I’ve been listening to my fellow Techies tell high school students to RUN RUN RUN to anything else but technology.

Also IT is becoming more and more like the construction industry. You have to pack up and move on every few years to stay employeed. In a world where husband and wife need to work to have a decent standard of living that’s a problem.

Look at the Wall street salaries up till the bust and the fact that an MBA is still a more reliable degree to stay employeed than a programming degree and it’s easy to understand why you can’t find enough IT guys.

Lack of people willing to get technical degrees is a problem that’s been build for years and it’ll take many years to fix it.

Posted by samuel_c | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/