1. The document discusses various technologies for building big data architectures, including NoSQL databases, distributed file systems, and data partitioning techniques. 2. Key-value stores, document databases, and graph databases are introduced as alternatives to relational databases for large, unstructured data. 3. The document also covers approaches for scaling databases horizontally, such as sharding, replication, and partitioning data across multiple servers.

1. Understanding and Building Big
Data Architectures
Ran.ga.na.than B, ThoughtWorks
@ran_than
Part 1 - Storage and NoSQL

2. Agenda

3. Img-src: http://dev.assets.neo4j.com.s3.amazonaws.com/wp-content/uploads/graph-data-jim-webber-presentation.
png

4. Basics
Story of Sir William Osler, University of Oxford

6. Img-src:http://i.dailymail.co.uk/i/pix/2009/07/01/article-1196775-0543FB03000005DC-763_634x364.jpg

7. Latency
● Hibernia Express
● 3,000-mile fiber-optic
● across the Atlantic Ocean to connect London to New York
● goal for 5ms latency
● To be used by Financial Institutes for trading
Src: http://shop.oreilly.com/product/0636920028048.do

8. Why distributed?

9. HDD Speed
❏ ~122MB per sec
❏ 1TB in 2hr 22 minutes
❏ SSDs are 2-3 times faster
Multiple disks reading parallel:
❏ With 100 HDDs, it takes 6 minutes

10. Data and Compute together

11. DataCenter
Img-src: https://fortunedotcom.files.wordpress.com/2015/06/screen-shot-2015-06-24-at-11-54-41-am.png?w=1024

12. Cluster and nodes
Img-src:https://en.wikipedia.org/wiki/Computer_cluster#/media/File:Cubieboard_HADOOP_cluster.JPG

13. Clusters
1. MultiNode: e.g: Hadoop, each node has some
responsibility.
2. Peer-to-Peer: e.g: Cassandra, all nodes are equal

14. Cluster Coordination
Activity

15. Expectations from
Data Systems
Non-functional parts of the application.

16. Expectations from Architecture
❏ Reliability
❏ Scalability
❏ Maintainability
❏ High Availability
❏ Fault Tolerance
❏ Security
❏ Compliance
❏ Compatibility

17. Components of
Data Systems

18. Components
❏ Storage
❏ Cache
❏ Search
❏ Stream Processing
❏ Batch Processing
❏ Data Exchange Protocols

19. Components
❏ Storage
❏ Cache
❏ Search
❏ Stream Processing
❏ Batch Processing
❏ Data Exchange Protocols

20. NoSQL Databases
“Database Admins walked into a NoSQL bar. A little while later
they walked out because they couldn’t find a table.”

21. Why NoSQLs?
❏ Scalability
❏ Cost
❏ Flexibility
❏ Availability
❏ Migrations

22. CAP theorem
Strong Consistency, High Availability, and Partition-Tolerance
Img-src:http://image.slidesharecdn.com/cap-131117230434-phpapp02/95/dynamo-and-bigtable-in-light-of-the-cap-theorem-12-638.jpg?cb=1384729712

23. CA

24. CP “when your business requirements dictate
atomic reads and writes”
Src: http://robertgreiner.com/2014/08/cap-theorem-revisited/

25. AP “when the system needs to continue to function
in spite of external errors”
Src: http://robertgreiner.com/2014/08/cap-theorem-revisited/

26. CAP story

27. Activity
Design ticket booking with scenarios of
CA, CP, AP

28. ACID
Atomic, Consistent, Isolated, and Durable

29. BASE
● Basically Available: If a single node fails, part of the
data won't be available, but the entire data layer stays
operational.
● Soft state: Soft state means data that is not persisted
on the disk, yet in case of failure it could be possible to
restore it.
● Eventually consistent: indicates that the system will
become consistent over time, given that the system
doesn't receive input during that time.

30. Key-Value DBs
Memcached, Redis, Riak, Voldemort, ...

31. Namespaces, Keys, Values

32. Implementation 1 - Arrays
● Only int as key
● Values are of same type

33. Implementation 2 - Associative Arrays
Key Value
user1 Mike
user2 Mary
user3 Nina
On hotspace

34. Simple Storage Design
- put key value - will add content to file in one line
- get key - will grep for key and return the value from the
file
What are the problems with this?
Activity
How can we improve this?

35. Simple Storage Design
- Add in memory index, with key and value as byte offset.
What are the problems with this?
Activity
How can we improve this?

36. Simple Storage Design
- Segments
- Compaction
What are the problems with this?
Activity
How can we improve this?

37. Simple Storage Design
- Sorted Key-Value
- Sparse index
- SSTable
What are the problems with this?
Activity
How can we improve this?

38. Simple Storage Design
- Memtable
- Segments as SSTable
Img-src: https://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/

39. Simple Storage Design - Overall
- Writes into RedBlack or AVL trees in memory - memtable
=> faster writes
- When memtable is 64MB, write to disk as SSTable and
clean memtable
- First read from memtable and most recent segments in-
memory sparse index (SSTable) => faster reads
- Run a merging and compaction process in the background
=> lesser storage and faster

40. Assignment
Let us do the e-commerce design with
only key value pair

41. Document DBs
CouchDB, MongoDB, MarkLogic, DocumentDB, OrientDB, ...

42. Img-src: http://blog.philipphauer.de/wp-content/uploads/2015/05/Match-OO-Document.png

43. Query - MongoDB
Img-src: http://bicortex.com/introduction-to-mongodb-nosql-database-for-sql-developers-part-3/

44. Query - CouchDB
// emit the first letter of each pokemon's name
var myMapReduceFun = {
map: function (doc) {
emit(doc.name.charAt(0));
},
reduce: '_count'
};
// count the pokemon whose names start with 'P'
pouch.query(myMapReduceFun, {
key: 'P', reduce: true, group: true
}).then(function (result) {
// handle result
}).catch(function (err) {
// handle errors
});

45. What’s cool?
● Flexible schema.
● Embedded docs come in one read.
●

46. Not so cool
● Familiarity
● Needs more space.
● Doesn’t speak SQL.

47. Assignment
Let us do the e-commerce design with
only document database

48. Graph DBs
Neo4j, TitanDB, OrientDB, ...

49. Relationships in relational DBs means joining

50. Why?
❏ Modelling and storing relationships in RDBMS is
complicated
❏ Performance degrades with number and levels of
relationships.
❏ Query complexity grows
❏ Adding new type requires schema redesign

51. With neo4j, you can traverse 4M+ relationships
per second and core

52. Img-src: http://blog.octo.com/wp-content/uploads/2012/07/RequestInSQL.png
Squeezing to table structure

53. Img-src: http://blog.octo.com/wp-content/uploads/2012/07/RequestInGraph.png
Closer to white board model

54. Graph consists of
❏ Vertices
❏ Edges

55. Cypher Query - Neo4j

56. SPARQL - RDF data model

57. Column oriented
DBs
Apache HBase, Cassandra, BigTable, ...

58. Src: https://www.safaribooksonline.com/library/view/hbase-the-definitive/9781449314682/httpatomoreillycomsourceoreillyimages889228.png

60. Src: http://static.oschina.net/uploads/img/201303/12072155_ROPI.gif

62. Img-src: http://www.slideshare.net/romain_jacotin/undestand-google-bigtable-is-as-easy-as-playing-lego-bricks-lecture-by-romain-jacotin

63. When is this better?
❏ Huge number of columns, with queries on few columns
❏ Aggregation
❏ Column level update
❏ Column data is uniform; so better compression

64. Time Series Data
Measurement and Time of measurement done repeatedly
Img src: https://www.safaribooksonline.com/library/view/time-series-databases/9781491920909/images/tsdn_0103.png.jpg

65. Why - Time Series Data
● Trends

66. When - Time Series
Data
● Huge amount of data
● Mostly query based on time
● Stock exchange
● Sensor data. E.g: Trucks
● Cell towers for usage patterns

67. Time Series Data
● IoT
● Logs

68. Replication
This is useful when you have a ncie photo or color-black as a
background. On this slide only, you can put your elements behind
a master element.

69. Journals

70. Master-Follower

75. Masterless

77. Sharding
Which bucket has your data?

78. Master
Slave
- Master for writes and real-time
reads
- Slaves for reads

79. Table
partitioning
- By rows or columns for parallelizing
reads

80. Feature
specific DBs
- Different DB servers for specific
features of application
- Can this scale?

81. Federated
Tables
A Federated Table is a table which points
to a table in another database instance
(mostly on an other server). It can be
seen as a view to this remote database
table.
- Administration overhead
- Security
- Access over network
- Okay for reporting/analytical tasks

82. Federated
Tables
A Federated Table is a table which points
to a table in another database instance
(mostly on an other server). It can be
seen as a view to this remote database
table.
- Administration overhead
- Security
- Access over network
- Okay for reporting/analytical tasks

83. Range
based
Split data based on range condition. Eg:
zip code, region.
- Non-uniform distribution

84. Hash based Take hash of key and modulo operation,
put the data in the server based on
reminder value.
- Uniform distribution
- Range queries may take time

85. Co-ordinators
- Take request, if key is in the request, talk to correct shard
- Co-ordinate across shards to give the result back
- Monitor health
- Take care of rebalancing
- Can be a random node, which will complete the task
- Set of co-ordinators

86. Take care, while sharding
● Balance your shards, with proper shard key
● Choose correct number of shards. E.g: 12
● Give time for rebalancing. In case of increasing capacity of
server, add nodes faster, and give time move your shards.
● Shard on denormalized data.
● Try to have shard key as part of your queries.

87. Where am I?

88. THANK YOU
For questions or suggestions:
Ran.ga.na.than B
@ran_than