THE BIG DATA MINE

Top story in the New Yorker this week documents how US National Security Agency is not only saving all of the data about those of us living overseas but now (and for most of this decade) also storing and cataloguing data on every US citizen who uses an electronic device (mobile, Internet, GPS, VISA, electronic transaction, telephone).

It is actually against the US constitution however Bush "overruled" the constitution after 9/11 and the bureaucrats (switching jobs back and forth between public service/outsourced contract firms) have run with it... all the way.

Way back in the late nineties an estimated two and a half billion phones in the world and one and a half billion I.P. addresses were passing approximately twenty terabytes of unique information around the world every minute. One of the NSA programmers created a program to capture and map all of this data, however in good faith to the law and US constitution he anonymised/encrypted all of the data relating to US citizens. Flags on suspicious data were raised and then if warranted a subpoena could be issued and then the privacy of the citizen could be legally be invaded.

Somehow, through quite a dodgy set of circumstances, this privacy protection functionality was all unravelled under George W and the NSA became a massive data mine.

This guy, Drake, who used to work there reported some of this to a journalist and is now being prosecuted for espionage against the nation and will likely end up in the gulag for the rest of his life.

But there is a strong chance the court case will rip all of this stuff open even wider, and well, its this week's number one story in the New Yorker magazine...

You are probably already aware that everything you type, talk, track, buy on an electronic device is being stored by these people (hi guys!) and yeah we all heard about the mysterious Echelon years ago, but this New Yorker magazine article outlines a whole bunch of stuff about how it works.

Warning: The New Yorker article is very long, fascinating, and creepy:
http://www.newyorker.com/reporting/2011/05/23/110523fa_fact_mayer?currentPage=all