Production + loss = conflict: Information generation and archival

A recent post on Mashable reminded me why I started a personal project to archive websites in Sri Lanka during and after the ceasefire agreement (2002 – 2006). To even begin to understand the amount of information we are creating is impossible – there’s just too much of it, and the graphic above is based on best available estimates. The reality is that with petabytes of information locked in the servers of corporations, governments and intelligence services and gigabytes of data production whenever a natural disaster occurs, even for 2011 alone, this is a conservative estimate of information production.

No knowledge here. Just information. And even then, there’s great cause for conflict here. Just the generation of some information will lead to it. Increasingly, the loss of information due to hardware failure, software glitches or malicious erasure will itself lead to conflict, or the exacerbation of existing violence.

My TED talk focussed on this, and in Managing the catastrophic loss of information and knowledge, written way back in 2008, poses questions over the information we will produce and lose that are even more pertinent today. Disturbingly, re-reading them, I find that several years after I penned them, we are no closer to any solution to capture this information for posterity, make sense of it and filter it.

Interestingly, EMC – the same outfit that I anchored my 2008 blog post to – is the one that estimates 1.8 zettabytes of information will be produced over the course of this year. Yet in 2008 it noted,

“the amount of information created, captured, or replicated exceeded available storage for the first time in 2007. Not all information created and transmitted gets stored, but by 2011, almost half of the digital universe will not have a permanent home”.

I wonder if the company’s metrics measured whether what they prophesied has actually come true? What kind of world are we living in, when we don’t even know how much information we’ve lost, much less produced? And amongst a myriad of other questions, are we better humans now that we produce this quantum of information than say a century ago?

The full EMC report for 2011 is worth reading, and also embedded below. Other related posts include,