There is no magic, only awesome platforms for data science. Large scale data requires rethinking data and compute architectures to remove constraints through distributed, programmable infrastructure and cloud services. Data science platforms can be built to make data more effective by accepting all data formats, evolving APIs, and optimizing the most valuable resource - data.

1. There is no magic
There is only awesome
Platforms for data science
D e e p a k S i n g h

2. bioinformatics
image: Ethan Hein

3. 3

4. collection

5. curation

6. analysis

7. what’s the big deal?

9. Source: http://www.nature.com/news/specials/bigdata/index.html

10. Image:Yael Fitzpatrick (AAAS)

11. Image:Yael Fitzpatrick (AAAS)

12. lots of data

13. lots of people

14. lots of places

15. constant change

16. we want to make our
data more effective

17. versioning

18. provenance

19. filter

20. aggregate

21. extend

22. mashup

23. human interfaces

25. image: Leo Reynolds

26. hard problem

27. really hard problem

28. so how do
get there?

29. information
platforms

30. Image: Drew Conway

31. dataspaces
Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data

32. the unreasonable
effectiveness of data
Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)

33. accept all data
formats

34. evolve APIs

35. beyond databases and
the data warehouse

36. data as a
programmable
resource

37. data is a
royal garden

38. compute is a
fungible commodity

39. optimizing the most
valuable resource

40. compute, storage,
workflows, memory,
transmission, algorithms,
cost, …

41. people
Credit: Pieter Musterd a CC-BY-NC-ND license

42. Image: Chris Dagdigian

43. my bias

44. cloud services

45. distributed systems

46. scale

47. global

48. consumption
models

49. on-demand

50. what is the value of
your data?

53. Credit: Angel Pizzaro, U. Penn

54. mapreduce for
genomics
http://bowtie-bio.sourceforge.net/crossbow/index.shtml
http://contrail-bio.sourceforge.net
http://bowtie-bio.sourceforge.net/myrna/index.shtml

56. Bioproximity
http://aws.amazon.com/solutions/case-studies/bioproximity/

59. 30,472 cores

60. $1279/hr

61. http://cloudbiolinux.org/

62. http://usegalaxy.org/cloud

63. in summary

64. large scale data
requires a rethink

65. data architecture

66. compute architecture

67. distributed,
programmable
infrastructure

68. cloud services

69. remove constraints

70. can we build data
science platforms?

71. there is no magic
there is only awesome

72. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.com
Inspiration and ideas from
Matt Wood& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license