The document discusses the need for a distributed platform and virtual institute to handle large amounts of data, people, compute, and uses. It proposes a platform called Trillionics that would allow users to get data from various sources, select scripts and directories, work on the data through interesting work, and save results back out. The platform would utilize distributed storage, virtualized services, and APIs to get, filter, work on, and save data in a distributed manner across various locations. It emphasizes building systems that can scale enormously to handle petabytes of data per week through a distributed mindset and approach.

1. Into the Wonderful
Towards a Virtual Institute

2. you are here

3. Data

4. Lots of data

5. Lots of data, lots of
people

6. Lots of data, lots of
people, lots of compute

7. Lots of data, lots of
people, lots of compute,
lots of uses

8. Lots of data, lots of
people, lots of compute,
lots of uses, lots and lots
and lots and lots...

9. Trillionics

10. A platform for science

12. 1 Get
2 Select
3 Work
4 Save

13. 1 Get
2 Select
3 Work
4 Save

14. 1 Get
2 Select
3 Work
4 Save

15. 1 Get
2 Select
3 Work
4 Save

16. Work is the killer app
get here quickly

17. Work = publications

18. Problematic for
complex data

19. 1 Get
2 Select
3 Work
4 Save

20. 1 Get: flat files / databases
2 Select
3 Work
4 Save

21. 1 Get: flat files / databases
2 Select: scripts / directories
3 Work
4 Save

22. 1 Get: flat files / databases
2 Select: scripts / directories
3 Work: interesting
4 Save

23. 1 Get: flat files / databases
2 Select: scripts / directories
3 Work: interesting
4 Save: flat files / databases

24. Get Filter Work Save

25. Get Filter Work Save

26. Get Filter Work Save

27. Get Filter Work Save

28. Get Filter Work Save

29. Get Filter Work Save

30. Get Filter Work Save

31. Get Filter Work Save

32. Filter
Save
Get
Work
Get Filter Work Save

33. Filter
Save
Work
Get
Get Work
Get Filter Work Save

34. Filter
Save
Work
Get
Get Work
Get Filter Work Save

36. Virtualise

37. Get Save

38. Data platform
Get Save

39. Data platform
Get Save
Work

40. Data platform
Get Save
Work
App platform

41. Data accessible via
services

42. Applications accessible
via services

43. Data platform
Get / Save
Work
Projects / SNP calling
App platform

44. Distribute

45. Data platform
Hintxon Get / Save
San Diego
Work
App platform

46. Distributed storage
Virtualised services
Application programming interfaces
Getters Filters Savers
Work

47. Distributed storage
Virtualised services
Application programming interfaces
Getters Filters Savers
Work Work Work Work
Work Work Work
Work Work Work Work Work
Work Work
Work Work Work

48. A distributed mindset

49. map/reduce

50. 1. map

51. @a = [ 1, 2, 3 ]
@result = []
for each $value in @a
push @result, map($value)
end
sub map($incoming)
return ($incoming * 10)
end

52. 2. reduce

53. reduce(@result)
sub reduce($r)
<transform $r>
end

54. independent

55. of array size!
independent

56. of array size!
independent
of ea ch other!

57. independent
distribute across virtual machines!

58. Prerequisites

59. Open data
easy to get a t data

60. soft ware as a service
Open APIs

61. Beyond SQL

62. Accessibility

63. East coast
24/7
Accessibility
Dow n the co rridor
West coast

64. Reliability

65. Build for flux

66. Authentication

67. Privacy

68. Less software

69. Distribute everything

70. Replicate everything
Speed. Redundancy.

71. Will it scale?

72. Oh yes

73. New York Times

74. 11 million TIFs

75. 24 hours
$500

76. Google, Yahoo!
Amazon

77. We are here

78. We need to start now

79. 2
X

80. 150
Tb/week

81. We need to start now
as in, like, ye sterday

82. Petabyte journal club
foomongers.org.uk

83. Thank you

84. GREENISGOOD.CO.UK