Composing and scaling data platforms
•
2 likes•750 views
This document summarizes Rahul Kumar's talk on composing and scaling data platforms. It discusses how the tools software engineers use shape the software that is built. It also explains how databases affect how developers treat state and mutability. The talk highlights how data platforms range in complexity from caching layers to integrated data pipelines. It discusses composing platforms through concepts like data representation, parallelism, and architecture. Sequential data access and streaming are more efficient than random access. Parallelism and distributing work across servers can scale platforms.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Composing and scaling data platforms
Similar to Composing and scaling data platforms (20)
Recently uploaded
Recently uploaded (20)
Composing and scaling data platforms
- 1. Composing and Scaling Data Platforms Rahul Kumar
- 3. As software engineer we are inevitably affected by the tools we surrounded ourself with Process all act to shape the software we build. Language Frameworks
- 4. Likewise database, which have trodden a very specific path, inevitably affect the way we treat mutability and share state in our application.
- 5. 5 Today’s data platforms range greatly in complexity. From simple caching layers or Polyglot Persistence right through to wholly integrated data pipelines. There are many paths. They go to many different places. So the aim for this talk is to explain how and why some of these popular approaches work. http://www.benstopford.com/2015/04/28/elements-‐of-‐scale-‐composing-‐and-‐scaling-‐data-‐platforms/ This talk is based on Ben Stopford’s actual presentation.
- 6. 6 Computer work best with sequential workload When we’re dealing with data, we’re really just arranging locality. Locality to the CPU. Locality to the other data we need.
- 7. 7 Accessing data sequentially is an important component of this. Computers are just good at sequential operations. Sequential operations can be predicted.
- 8. 8 Random vs Sequential Addressing If you’r taking data from disk sequentially it will be pre-‐fetched in to the disk buffer, the page cache and the different levels of CPU caching. But it does little to help the addressing of data at random, be it in main memory, on disk or over the network. In fact pre-fetching actually hinders random workloads as the various caches and frontside bus fill with data which is unlikely to be used.
- 9. 9 Streaming data sequentially from disk can actually outperform randomly addressed main memory. So disk may not always be quite the tortoise we think it is, at least not if we can arrange sequential access.
- 10. 10 We want to keep writes and reads sequential, as it works well with the hardware. We can append writes to the end of the file efficiently. We can read by scanning the the file in its entirety. Any processing we wish to do can happen as the data streams through the CPU. We might filter, aggregate or even do something more complex.
- 13. 13
- 22. 22 Parallelism
- 29. 29 Architecture
- 38. Thank You