The Human Face of Big Data

In the near future,

every object on earth will be generating data,

including our homes, our cars,

even our bodies.

Do you see it?

Yeah, right up there.

- Almost everything we do today

leaves a trail of digital exhaust,

a perpetual stream of texts, location data,

and other information that will live on

well after each of us is long gone.

We are now being exposed to as much information

in a single day as our 15th century ancestors

were exposed to in their entire lifetime.

But we need to be very careful

because in this vast ocean of data

there's a frighteningly complete picture of us,

where we live, where we go,

what we buy, what we say,

it's all being recorded and stored forever.

This is the story of an extraordinary revolution

that's sweeping almost invisibly through our lives

and about how our planet is beginning to develop

a nervous system with each of us acting as human sensors.

This is the human face of big data.

- All these devices and machines and everything

we're building these days, whether it's phones or computers

or cars or refrigerators, are throwing off data.

- Information is being extracted out of toll booths,

out of parking spaces,

out of Internet searches,

out of Facebook, out of your phone,

tablets, photographs, videos.

- Every single thing that you do leaves a digital trace.

The exhaust or evidence of humans

interacting with technology and what side effect that has

and that's literally, it's just this massive amount of data.

- What we're doing is we're measuring things

more than we ever have.

It's that active measurement that produces data.

If you were some omniscient god

and you could look at the footprints of electric devices,

you could kind of see the world.

If the whole world is being recorded in real time,

you could see everything that is going on in the world

through the footprints.

I think it's a lot like written language, right,

it's just at some point they got to the point

where you had to start writing stuff down.

You just got to the point where it wouldn't work

unless we wrote it down, which is making the same point

where well it ain't gonna work unless we write

all the data down and then look at it.

And all that data coming in is big data.

We estimate that by 2020

the data volumes will be at about 40 zigabytes.

Just to put it in perspective,

if you were to add up every single grain of sand

on the planet and multiply that by 75,

that would be 40 zigabytes of information.

- All the data processing we did in the last two years

is more than all the data processing

we did in the last 3,000 years.

And so the more information we get,

the larger the problems will be that we solve.

- Every powerful tool has a dark side, every last one.

Anything that's going to change the world,

by definition has to be able to change it for the worse

as much as for the better.

It doesn't work one way without the other.

- When it comes to big data, a lot of people

are very nervous.

Data can be used in any number of ways

that you're either aware of or you're not.

The less aware of the use of that data that you are,

the less power you have in the coming society

we're going to live.

- Well sort of just in the beginning of this big data thing,

you don't know how it's going to change it,

but you just know it is.

- The first real data set to change everything in the world

was the astronomical data set,

meticulously collected over tens of years by Copernicus

that ultimately revealed, even though the sun seemed to be

moving over the sky every morning and every night,

the sun is not moving, it is we who are moving,

it is we who are spinning.

It happened again when we suddenly could see

beneath the visible level

and the microscope in the 1650s and 60s,

opened up the invisible world

and we for the first time were seeing cells and bacteria

and creatures that we couldn't imagine were there.

It then happened again when we revealed the atomic world,

when we said wait a second, there's a level

below the optical microscope where we could begin

to see things at billionths of a meter at a nanometer scale,

where we imagined the atom and the nucleus

and the electron, where we understood that light

is electromagnetic frequencies.

But now, there's actual a supervisible world

coming into play.

Ironically, big data is a microscope.

We're now collecting exabytes and petabytes of data

and we're looking through that microscope

using incredibly powerful algorithms

to see what we would never see before.

Before what we did was we

thought of things and then we wrote it down

and that became knowledge.

Big data's kind of the opposite.

You have a pile of data that isn't knowledge really

until you start looking at it and noticing wait,

maybe if you shift it this way and you shift it this way,

this turns into this interesting piece of information.

I think that the BDAD moment,

you know, before data, after data moment,

is really Search.

That was the moment at which we got a tool

that was used by hundreds of millions of people

within a few years,

where we could navigate an incredible amount

of information.

We took all of human knowledge that was in text, right,

and we put it on the web

and we thought to ourselves, "Well we're done.

"Wow that was hard."

And now we realize that was the first minute

of the first inning of the game, right,

because that was just the knowledge we already had

and the knowledge that we continue to add to the web

at a relatively slow pace, you know.

But there is so much more information

that we have not digitized and so much more

information that we're about to take advantage of.

- In recent years, our technology has allowed us

to store and process mass quantities of data.

Visualizing that data will allow us to see

complex systems function,

see patterns and meaning in ways

that were previously impossible.

Almost everything is measurable and quantifiable.

- So when I look at data, what's exciting to me

is kind of recontextualizing that data

and taking it and putting it back into a form

that we can perceive, understand, talk about,

think about.

- This is the data for airplane traffic

over North America for a 24-hour period.

When it's visualized, you see everything starts

to fade to black as everyone goes to sleep,

then on the West Coast, planes start moving across

on red-eye flights to the east

and you see everyone waking up on the East Coast,

followed by European flights in the upper right-hand corner.

I think it's one thing to say that there's 140,000 planes

being monitored by the federal government at any one time

and it's another thing to see that system

as it ebbs and flows in front of you.

These are text messages being sent in the city of Amsterdam

on December 31st.

You're seeing the daily flow of text messages

from different parts of the city until we approach midnight,

where everyone says--

Happy New Year!

- It takes people or programs or algorithms

to connect it all together to make sense of it

and that's what's important.

We have every single action that we do in this world

is triggering off some amount of data

and most of that data is meaningless

until someone adds some interpretation of it,

someone adds a narrative around it.

- Often, we sort of think of data as stranded numbers,

but they're tethered to things

and if we follow those tethers in the right ways,

then we can find the real-world objects

and the real-world stories that were there.

So a lot of the work is that kind of work.

It's almost investigative work of trying to follow

that trail from the data to what actually happened.

Sometimes the power of large data sets

isn't immediately obvious.

Google flu trends is a great example

of taking a look at a massive corpus of data

and deriving somewhat tangential information

that can actually be really valuable.

- Until recently, the only way to detect

a flu epidemic was by accumulating information

submitted by doctors about patient visits,

a process that took about two weeks to reach the CDC.

So the researchers turned it around.

They asked themselves if they could predict a flu outbreak

in real time simply using data from online searches.

So they set out to do the near impossible,

searching the searches, billions of them,

spanning five years to see if user queries

could tell them something.

When we do searches on Google,

we all think of it as a one-way street,