Big data analytics should be used for more than just financial gain; the power of data combined with good design can be used to convey complex information to the masses, says Jer Thorp, data artist in residence at The New York Times.
Thorp delivered an opening keynote of this year's Microsoft Tech Ed in Auckland last night. He spoke of his experience creating infographics for the Times, and working with scientific organisations to better display their work to the public.
Thorp helped create a system that allows staff at the Times to map the spread of the publication's articles on social networks.
The tool was developed with the help of Columbia University statistics professor Mark Hensen. It tracks information on which Twitter users share links to the Times' 6500 monthly posts, and shows how those users influence others in passing the links on - this cascading effect was the inspiration for the visualisation tool's name, 'Cascade'.
Thorp had earlier worked with the Times' expansive news database to create aggregated data charts of popular news stories, news trends, and the occurrences of names or specific words in almost 20 years worth of articles. For example, tracking the occurrences of the words "communism" and "terrorism" revealed a steady decline in the use of the former term since the 1990s, and a dramatic spike in the occurrence of the latter word since September 2001.
Using the Times' database Thorp was able to create a series of images displaying news trends for each year between 1984 and 2009.
"My intent is not to clarify the news, instead it's to show the chaotic networks present inside the system," says Thorp.
Thorp urged the "technologically powerful" Tech Ed audience to use data for more than a business case.
Thorp is an advocate of what he calls data philanthropy. This is the voluntary donation of user statistics like cellphone usage and GPS tracking information to scientific organisations for research.
OpenPaths.cc (www.openpaths.cc) is a project co-founded by Thorp which helps researchers visualise user data and also connects "data donors" with researchers.
Thorp says consumers are either wary of sharing data, or completely unaware that their data is being shared without their knowledge - because of abuses by some large companies.
There needs to be more respect for data he says.
"I hear people say data is the new oil, this comes to me as a surprise because we did so well with oil," says Thorp.
Tech Ed is Microsoft's educational and promotional conference for developers, partners, and customers. The event is being held at Auckland's SkyCity, and runs until Friday September 7.
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.