Download as PDF, PPTX
Uploaded byDatabricks
Building a Turbo-fast Data Warehousing Platform with Databricks
The document presents a comprehensive overview of building a high-performance data warehousing platform using Databricks, focusing on key aspects such as infrastructure, data ingestion, ETL processes, and performance optimizations. It highlights the advantages of Databricks over traditional data warehousing methods, emphasizing its speed, flexibility, and ease of use powered by Apache Spark. Additionally, the content addresses security measures, integration with AWS services, and real-world use cases of the platform.
More Related Content
Similar to Building a Turbo-fast Data Warehousing Platform with Databricks
Building a Turbo-fast Data Warehousing Platform with Databricks
- 1. Building a Turbo-fastData Warehousing Platform with Databricks Parviz Deyhim
- 2. Agenda • Introduction toDatabricks • Building a end-to-end Data warehouse platform • Infrastructure • Data ingest • ETL • Performance optimizations • Process & Visualize • Securing your platform • Conclusion
- 3. 3 Parviz Deyhim (Speaker) Parvizworks with variety of different customers and helps them with adopting Apache Spark and architecting scalable data processing platform with Databricks Cloud. Previous to joining Databricks, Parviz worked at AWS as a big-data solutions architect. Denny Lee (Moderator) Denny is a Technology Evangelist with Databricks. Previous to joining Databricks, Denny worked as a Senior Director of Data Sciences Engineering at Concur and was part of the incubation team that built Hadoop on Windows and Azure (currently known as HDInsight). About the Speakers
- 4.
- 5. We are Databricks,the company behind Spark • Founded by the creators of Apache Spark • Contributed ~75% of the Spark code in 2014 • Created Databricks cloud, a cloud-based big data platform on top of Spark to make big data simple
- 6. Typical big dataproject is far from ideal Weeks to prepare then explore data, and find insights Import and explore data Months to build, weeks to provision in existing Get a cluster up and running Months of re-engineering to deploy as an application Build and deploy data applications For each new project, it takes months until results
- 7. How Databricks poweredby Spark helps our customers No infrastructure management Interactive workflow Collaboration across the organization Experiment to production instantly 100x faster than MapReduce Spark SQL + ML + Streaming + Graph processing Speed Flexibility Ease-of-use Unified
- 8. Databricks helps youto harness the power of Spark “Light switch” Spark clusters in the cloud 3rd Party Applications Interactive workspace with notebooks Production Pipeline Scheduler
- 9.
- 10. 10 Databricks Internal DWHUse Case Today: Collect logs from deployed customer clusters Our Goal: ○ Understand customers behavior ○ Create reports for various teams (e.g. customer success & support)
- 11. Stages Build & Maintain Infrastructure DataIngest Process & Visualize Transform & Store
- 12. 12 Stages Build & Maintain Infrastructure DataIngest Process & Visualize Transform & Store
- 13. 13 Challenges of Buildinga Data Warehouse Datacenter or Cloud? • Build/rent data center or use a public cloud offering? Picking the right resources • If datacenter: what server sizes and types? Storage? • In cloud: what instance size, how large of a disk/SSD to use? Deployment and Automation • How to automate the deployment process: • Chef, Puppet, Cloudformation and etc
- 14. 14 Maintenance • How toperform seamless upgrades? Securing the platform • How to encrypt datasets? • Controls, Policies, Audits Challenges of Building a Data Warehouse
- 15. 15 Databricks Hosted Platform Managedand automated hosted platform • Fully deployed on AWS • Create resources with a single click • Zero touch maintenance
- 16. 16 Compute Resources Automatic InstanceProvisioning • R3.2xlarge instances • Use SSD for caching • No EBS • Deployed in major regions and more coming
- 17. 17 Networking: VPC Security &Isolation with AWS VPC
- 18. 18 Networking: Enhanced Networking Highperformance node to node connectivity with placement groups
- 19. 19 Integration with AWSservices S3 Kinesis RDS Redshift ...
- 20.
- 21. 21 Stages Build & Maintain Infrastructure DataIngest Process & Visualize Transform & Store
- 22. 22 Customer Data Sources Customerhave variety of different data sources Cloud storage: S3 Databases: MySQL, NoSQL APIs: Facebook, SalesForce and etc Often required to join datasets
- 23. 23 Traditional Approach Traditionally datawarehouses require data to be copied Common Question: How do I move my datasets to Databricks?
- 24. 24 Traditional Approach Required tocreate a schema before data is copied
- 25. 25 Traditional Approach: Challenges MovingData: • Very expensive and time consuming • Creates inconsistency as data gets updated Predefined Schema: • Challenging to change schema for different use-case
- 26. 26 Databricks Approach: DataSources De-coupling compute from storage ● Leverage S3. No HDFS Read directly from data sources ● Eliminate the need to copy data Schema definition on read ● SparkSQL
- 27. 27 Spark Data SourcesSupport
- 28. 28 Databricks Use Case Differentdata sources • Customer metrics on S3 • Internal CRM Need a single view of our customers
- 29. 29 Databricks Use Case Weuse Spark to join datasets
- 30. 30 Databricks Demo 1. Readingdata from external API 2. Reading usage logs data from S3 3. Joining usage and external datasets Link
- 31. 31 Stages Build & Maintain Infrastructure DataIngest Process & Visualize Transform & Store
- 32. 32 Data Transformation Need totransform data before the join operation • Aggregation • Consolidation • Data cleansing
- 33.
- 34. 34 ETL: Common Approaches Twocommon approaches • Offline • Streaming/real-time
- 35. 35 Extract & Transformation OfflineET • Data gets stored in raw format (as is) • Some recurring job perform ET on the dataset • New transformed dataset gets stored for later processing Advantage • Easy and quick to setup Disadvantages • Traditionally slow process
- 36. 36 Databricks Jobs Databricks Jobs •Schedule Production workflows using Notebooks or Jars • Create pipelines • Monitor results
- 37.
- 38.
- 39. 39 Performance Optimizations Storing datain parquet Partitioning dataset Spark caching • JVM • SSD
- 40. 40 Spark allows datato be stored in different data sources
- 41. 41 Parquet: Efficient columnarstorage format for data warehousing use-cases
- 42. 42 Optimization: Parquet Columnar • FasterScans Better compression Optimized for storage • Memory • Disk
- 43. Advantages • Fast memoryaccess Disadvantages • GC pressure • No durability after JVM crash 43 Optimization: Caching (JVM) Spark caching: JVM
- 44. 44 Optimization: Caching (SSD) Sparkcaching: SSD Advantages • Survives JVM and instance crash Disadvantages • Much slower than JVM caching
- 45. 45 Databricks Use Case: Storingaggregate data in Parquet
- 46. 46 Databricks Use Case: Storingaggregate data in Parquet
- 47.
- 48. 48 Stages Build & Maintain Infrastructure DataIngest Process & Visualize Transform & Store
- 49. 49 Databricks Visualizations Notebook Visualizations 1.Built-in graphing capabilities 2. ggplot and matplotlib 3. D3 visualizations
- 50. 50 Databricks Visualizations Notebook Visualizations(DEMO) D3/SVG
- 51.
- 52.
- 53. 53 Secure Platform Encryption 1. Inflight: SSL 2. At rest: S3 Encryption
- 54. 54 Secure Platform User Management:ACLs Notebooks read-write-execute Admin users
- 55. 55 Secure Platform On OurRoadmap S3 KMS encryption Single Sign On (SSO) AD/LDAP support
- 56. 56 Secure Platform On OurRoadmap IAM Roles for Spark nodes
- 57.