If you want to design an Analytics solution, you should have attended this workshop. Contact me if you need further information. Moshe Kaplan

1. The VP R&D Open Seminar
Big Data Workshop
mokplan@gmail.com
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

2. Presentation Objectives
http://www.webperformancetoday.com/2010/06/15/everything-you-wanted-to-know-about-web-performance/
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

3. Why Do I Care?
From 0 to 100 (US mass adaptation)
Phone: 100 yrs
Radio: 40 yrs
TV: 30 yrs
Mobile: 20 yrs
Internet: 10 yrs
Facebook: 2 yrs
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

4. The Prime Suspect
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

5. Assumptions…
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

6. Network Effect Solutions
http://www.akamai.com/stateoftheinternet/
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

7. General Architecture
Client Server Database Apps
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

8. The VP R&D Open Seminar
CLIENT SIDE
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

9. It’s a World Made of Pixels
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

10. The VP R&D Open Seminar
SERVER SIDE
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

11. General Strategies
Online
In Memory Databases
Log files
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

12. In Memory Databases
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

13. 700 Inserts/Sec
In Memory Engine
3000 Inserts/Sec
Amazon
AWS InnoDB Engine
Standard 700 Inserts/Sec
Large
Instance
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

14. The VP R&D Open Seminar
General Strategies
DATA SIDE
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

15. Strategy A - Sharding
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

16. Strategy B – MapReduce
http://blogs.microsoft.co.il/blogs/vprnd
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

17. Strategy C - NoSQL
insert
get
multiget
remove
truncate
<Key, Value>
http://wiki.apache.org/cassandra/API
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

18. The VP R&D Open Seminar
COLUMN ORIENTED
DATABASES
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

19. Column Oriented Database Databases
+ INSERT
+ GROUP BY, SUM …
+ Compression
- Join
- DELETE, UPDATE
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

20. Cloud Services
http://www.theregister.co.uk/2012/11/28/amazon_aws_redshift_data_warehousing/
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

21. The VP R&D Open Seminar
SHARDING IN DEPTH
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

22. Sharding Again
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

23. Vertical Sharding
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

24. Horizontal Sharding
Static Hashing
Mod 10 = 0
Complex growth Mod 10 = 1
Simple Mod 10 = 2
Mod 10 = 3
Mod 10 = 4
Mod 10 = 5
Mod 10 = 6
Mod 10 = 7
Mod 10 = 8
Mod 10 = 9
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

25. Horizontal Sharding
Key locations are defined in a directory
Simple growth
Directory is SPOF
The Directory Can
be Very Large
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

26. Horizontal Sharding
Static Hashing with Directory Mapping
Simple Growth
The Small Directory Can be Cached on
Each App Server
Mod 1000 = 4
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

27. Horizontal Sharding
Each key is signed by the DB#
on creation
Simple growth
The Key Store Can be Cached on
Each App Server
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

28. Big Query
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

29. The Bottom Line: Grow ∞
Thank you!
and Keep Performing!
Moshe Kaplan
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com