This document outlines the key considerations for building a large-scale analytics platform, including the business need, challenges, options for commercial and open-source tools, cloud vs self-hosted approaches, data pipelines for collection, storage and processing, batch processing vs ad-hoc querying, and balancing cost and performance. It provides a diagram of an example platform architecture showing data sources and feeds, batch and real-time processing, cloud storage, querying, and reporting/visualization interfaces.

1. Building large scale Analytics Platform

2. Agenda
• Business context – why we need a big data platform
• Challenges in building large scale Analytics Platform
• COTS options
• Cloud vs Self-Hosting
• Collecting, storing and processing data – the data pipeline
• Batch processing
• AdHoc querying
• Cost vs Performance

3. Data
Feed
Processor
Data
Source1
Data
Source2
Data
Source3
….
Batch
processing –
MapReduce
Cloud
Storage
Distributed
database
-Data (shorter time range)
User Interface
- Reports
- BI features
- Visualisation
Batch
processing
results (in
files)
Data
Source N
AdHoc Querying
Standard BI
tools