Into the Wonderful
Towards a Virtual Institute

you are here

Data

Lots of data

Lots of data, lots of
people

Lots of data, lots of
people, lots of compute

Lots of data, lots of
people, lots of compute,
lots of uses

Lots of data, lots of
people, lots of compute,
lots of uses, lots and lots
and lots and lots...

Trillionics

A platform for science

1 Get
2 Select
3 Work
4 Save

1 Get
2 Select
3 Work
4 Save

1 Get
2 Select
3 Work
4 Save

1 Get
2 Select
3 Work
4 Save

Work is the killer app
get here quickly

Work = publications

Problematic for
complex data

1 Get
2 Select
3 Work
4 Save

1 Get: flat files / databases
2 Select
3 Work
4 Save

1 Get: flat files / databases
2 Select: scripts / directories
3 Work
4 Save

1 Get: flat files / databases
2 Select: scripts / directories
3 Work: interesting
4 Save

1 Get: flat files / databases
2 Select: scripts / directories
3 Work: interesting
4 Save: flat files / databases

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Get Filter Work Save

Filter
Save
Get
Work
Get Filter Work Save

Filter
Save
Work
Get
Get Work
Get Filter Work Save

Filter
Save
Work
Get
Get Work
Get Filter Work Save

Virtualise

Get Save

Data platform
Get Save

Data platform
Get Save
Work

Data platform
Get Save
Work
App platform

Data accessible via
services

Applications accessible
via services

Data platform
Get / Save
Work
Projects / SNP calling
App platform

Distribute

Data platform
Hintxon Get / Save
San Diego
Work
App platform

Distributed storage
Virtualised services
Application programming interfaces
Getters Filters Savers
Work

Distributed storage
Virtualised services
Application programming interfaces
Getters Filters Savers
Work Work Work Work
Work Work Work
Work Work Work Work Work
Work Work
Work Work Work

A distributed mindset

map/reduce

1. map

@a = [ 1, 2, 3 ]
@result = []
for each $value in @a
push @result, map($value)
end
sub map($incoming)
return ($incoming * 10)
end

2. reduce

reduce(@result)
sub reduce($r)
<transform $r>
end

independent

of array size!
independent

of array size!
independent
of ea ch other!

independent
distribute across virtual machines!

Prerequisites

Open data
easy to get a t data

soft ware as a service
Open APIs

Beyond SQL

Accessibility

East coast
24/7
Accessibility
Dow n the co rridor
West coast

Reliability

Build for flux

Authentication

Privacy

Less software

Distribute everything

Replicate everything
Speed. Redundancy.

Will it scale?

Oh yes

New York Times

11 million TIFs

24 hours
$500

Google, Yahoo!
Amazon

We are here

We need to start now

2
X

150
Tb/week

We need to start now
as in, like, ye sterday

Petabyte journal club
foomongers.org.uk

Thank you

GREENISGOOD.CO.UK