The information singularity is coming!

December 19, 2013

“Data! Data! Data!” Sherlock Holmes cried impatiently. “I can’t make bricks without clay.”

The sleuth’s insatiable hunger for petabytes of data presaged that of the National Security Agency’s by a little less than seven decades. Like the NSA, Holmes took a pointillistic view of the truth. Find as many facts as possible, he held, view them from as many angles as possible, turn them inside out or set them aside until you collect more facts, and then, like pouring iron into a mold, cast your most durable image of reality. “It’s an old maxim of mine that once you have eliminated the impossible, whatever remains, however improbable, must be the truth,” as Holmes stated in Arthur Conan Doyle’s “The Adventure of the Beryl Coronet.”

The age of Snowden has made well-known NSA’s demanding data desires. So, too, have we gotten a glimpse of how the agency’s information foundry works, that place where mathematicians massage the metadata of phone records and Web traffic with powerful relational database software to strip away the impossible in pursuit of the “truth.” Whether you believe the collection and analysis of your personal data is trivial or intolerable, the age of Snowden has alerted us all to the coming of the information singularity, where near perfect portraits and detailed biographies of us all can be assembled if enough computer power is thrown at a big enough data set.

“Records that once would have revealed a few scattered tiles of information about a person now reveal an entire mosaic — a vibrant and constantly updating picture of a person’s life,” U.S. District Judge Richard Leon wrote in his Klayman v Obama (pdf) opinion earlier this week, ruling the NSA’s bulk collection of phone data likely unlawful.

Judge Leon picked his mosaic metaphor shrewdly, knowing that the NSA and other agencies that handle classified information, such as the CIA and FBI, are not principled foes of the mosaic metaphor. Far from it. When it suits their purposes — which is often — federal agencies rely on the regulatory power of “mosaic theory” to knock down freedom of information requests from the press and others for seemingly innocuous material. The Code of Federal Regulations states this power succinctly. Agencies can, in some instances, classify compilations of unclassified government information because taken together and analyzed, the end product can be used to pierce the veil of what has been classified. Likewise, the government frequently rejects FOIA requests because it fears that declassified information can be combined with public domain information to reveal classified information. “[T]he ‘mosaic’ approach,” the code states, is “the concept that apparently harmless pieces of information, when assembled together could reveal a damaging picture.”

One of the clearest examples of the mosaic theory run amuck resides in a 1998 case, where workers sought information about the allegedly unlawful handling of hazardous material at “Area 51.” But because foreign powers, using the power of a mosaic, might be able to determine the nature of the classified weapons programs run there, the information request was rejected. Ten years ago, mosaic theory was used (pdf) to dash FOIA requests pertaining to 700-plus people detained after the 9/11 attacks.

“The mosaic theory turns many of the assumptions of liberal democracy inside out and allows the government to make information secret and put it beyond the reach of judicial requests when that information could not properly be classified either under executive order or statute,” wrote Robert M. Pallitto and William G. Weaver in 2007’s Presidential Secrecy and the Law.

The government obviously wants its mosaic theory both ways. It wants to say we shouldn’t worry about it holding mountains of our metadata that, when run through NSA’s mosaic-machine, can reveal our health status, our financial condition, our comings and goings, our political persuasions, and our tastes in vice, etc. But at the same time, it treats itself to a hysterical fit when asked for the release of bland government information because somebody (a terrorist, a spy, or even a journalist) might use it to piece together a state secret and endanger national security.

When the government seeks to protect its own interests, its mosaic theory reasoning “has no logical limit,” as Susan N. Herman, former head of the ACLU, wrote in 2011. Every whole is made up of pieces, therefore every piece can potentially be denied to the public. “Because there is no way for anyone to disprove the possible significance of any piece of information, taking this observation as a defense to any disclosure of any information gives the executive branch a blank check to withhold all information,” she continues.

The impending information singularity is likely to strengthen both the arguments of the civil libertarians, who will warn that rampant data collection combined with growing computer power will make us all naked before the state, and of the national security establishment, which will argue even more ardently that nothing can be known because if anything is known, all will be known!

Soon, as the singularity coalesces, both constituencies might be proved right. But until then, let’s tease out the primary difference between the two kinds of mosaic-building. Denying the government total access to the minutia of its citizens has long been an American legal principle. The practical argument against additional collection was further advanced this week by President Barack Obama’s NSA-review panel and Judge Leon, with both concluding that the government’s massive collection of individual metadata has averted no terrorist attack thus far. Meanwhile, the parallel principle that citizens have a right 1) to know what their government is up to and 2) change government policy when dissatisfied is well enshrined, mosaic theory or no mosaic theory. Even Sherlock Holmes could get behind that.


Send all your metadata to because, you know, you can trust me. When read closely, my Twitter feed reveals secrets on how to build a backyard H-bomb. Sign up for email notifications of new Shafer columns (and other occasional announcements). Subscribe to this RSS feed for new Shafer columns.

PHOTO: Christie’s auctioneers manuscript specialist Thomas Venning looks at manuscripts by the late Sir Arthur Conan Doyle which are due to be auctioned to benefit British charities, in London, October 16, 2003. REUTERS/Stephen Hird


We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see

Great article. And timely. Orwell predicted this. Now it has come to pass.

I doubt there is a way back.

Posted by DaveinKL | Report as abusive

Great piece, I learned something. Thanks.

Posted by enuffaweady | Report as abusive

This reveals the fact that our government is NOT of, for, or by the people. Currently there is no mechanism to change this situation, which was how it was designed.

Posted by boon2247 | Report as abusive

Yesterday a female PR person for a large corporation tweeted from an airplane… “Going to Africa. Hope I don’t get AIDS. Just Kidding. I’m White”. In the past she had tweeted on her account… “I had a sex dream about an autistic kid last night.” Safe to say, she has most-likely lost her job, and will also have to endure the embarrassment of being ridiculed for months.

This is by no means the first time this will happen, nor the last. Millions of people now tweet everything, short of announcing they just took a poop. Actually, I’m sure some people even do that.

It would appear that many people don’t need the NSA’s help to ruin their privacy. They seem to do a fine job all by themselves.

Posted by dd606 | Report as abusive

Jack Shafer – Completely awesome as usual. Love.

Posted by 2Borknot2B | Report as abusive

I think the reference to Sherlock Holmes is misplaced. Holmes let the evidence speak and had no preconception of guilt and morality. Our government has proven able to deny evidence and persist with lies. They do it all of the time. Thus, the NSA may claim they have evidence but we know they will be capable of intentional misrepresentation of evidence to support their lies. We see it constantly in our government and their owners.

Posted by brotherkenny4 | Report as abusive

There is another perspective on wealth now, too.

The familiar quotation, “Behind every great fortune is a great crime.”, will take on added meaning with the NSA collecting every phone call and email on Earth.

Take for instance the Russian Oligarchs, who at the fall of the Soviet Union, with a few sophisticated slight-of-hand moves, ended up owning vast swaths of factories, built by generations of laboring Russian workers, and vast millions of acres of land, gigantic oil fields already developed and pumping oil, and natural resources of every description.

That type of corruption is as old as mankind. But now, the NSA is listening to the oligarch’s phone calls. Are they interested? Of course they are. Who wouldn’t be interested?

It is not the street corner drug dealer capitalist that has the most to fear from the NSA technology singularity; rather it is the world’s nobility families of great wealth, accustomed for a thousand years to secrecy, to country club insider connections, to discretion, to bending the law to their own needs — they, the nobility, will have the most to fear from the NSA technology singularity.

The common quotation mentioned above is thought to have originated from Balzac’s words in 1835: “The secret of great fortunes without apparent cause is a crime forgotten, for it was properly done.”

The NSA singularity will make it harder for the criminal wealthy to make sure their financial crimes were “properly done”. So, today the oligarchs must be always each looking over his shoulder, wondering if the hard, undeniable evidence against them, enshrouded forever in the NSA cloud hardware, will ever be used to call them to account.

Capitalism is changed forever.

Posted by AdamSmith | Report as abusive

Minority Report is here in reality!

And the scary bit is that the Yanks will decide if you’ve committed a “crime” according to THEIR ‘laws’ – you will then be deported from WHATEVER country you live to the US. If anyone thinks that they’ll be safe – you will be very deluded!

1984 just went global – be afraid, be very, very very afraid!!!!!!

Posted by umkomazi | Report as abusive