Download as PDF, PPTX
Uploaded byWlodek Bielski
Cloud Developer Days - BigQuery
This document provides an overview and agenda for a presentation on Google BigQuery. It includes sections on BigQuery internals and architecture, use cases, and demos. The presentation will cover an overview of Google Cloud Platform and BigQuery, their internals including storage, execution engines, and compute resources, common use cases, and end with a demo in Google Cloud Platform console and BigQuery. It is presented by an expert in BI, analytics, and cloud platforms.
More Related Content
![code lab live Google Cloud Endpoints [DevFest 2015 Bari]](https://cdn.slidesharecdn.com/ss_thumbnails/nickcodelablivegooglecloudendpoints-150509100715-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Cloud Developer Days - BigQuery
![[DSC Europe 25] Ivica Milaric - The Future of Gaming and AI Tools.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/tijgzsmgse2kj2y5pzzp-5-ivica-milaric-the-future-of-gaming-x-ai-tools-260114111931-87c2b3ac-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Mijat Kustudic - Building Financial Intelligence with AI Agen...](https://cdn.slidesharecdn.com/ss_thumbnails/38y2lb5lse6wstegtvas-3-mijat-kustudic-building-financial-intelligence-with-ai-agents-260114111931-1a4783ce-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Srba Markovic - From Pilot to Production: Overcoming AI Deplo...](https://cdn.slidesharecdn.com/ss_thumbnails/yjjmrtytmwbalxlba7px-4-srba-markovic-from-pilot-to-production-overcoming-ai-deployment-blockers-with-260114111931-4a892d44-thumbnail.jpg?width=640&height=640&fit=bounds)
Cloud Developer Days - BigQuery
- 1.
- 2.
- 3. cloud.developerdays.pl@DeveloperDaysPL About me • InBI / Analytics space since 2005 • In Clouds since 2014 • Google Professional Data Architect • Google Professional Data Engineer • MCSD: Azure Solutions Architect • MCSE: Cloud Platform and Infrastructure • MCSE: Data Management and Analytics • AWS Certified Solution Architect - Associate
- 4. cloud.developerdays.pl@DeveloperDaysPL Agenda • GCP overview •BigQuery overview • BigQuery internals • BigQuery use cases • Session on ML with GCP: 15:00-16:00
- 5.
- 6. cloud.developerdays.pl@DeveloperDaysPL Gartner MQ forCloud IaaS, 2018 „Google has been most differentiated on the forward edge of IT, with deep investments in analytics and ML, and many customers who choose Google for strategic adoption have applications that are anchored by BigQuery”
- 7.
- 8.
- 9.
- 10. cloud.developerdays.pl@DeveloperDaysPL What is BigQuery? •Fully managed, No-Ops analytics data warehouse • Highly parallel / distributed processing model • Only pay for actual storage and compute used • Virtually unlimited storage and compute resources • Runs on Google infrastructure (US, EU, Asia) • Multi-tenant architecture
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17. cloud.developerdays.pl@DeveloperDaysPL Storage subsystem: Colossus •Successor to GFS (GFS: 2003, Colossus: 2010) • While GFS – batch, Colossus – real time • Powering most of Google internal services (Gmail, YouTube, GCS) • Reliable and fault-tolerant (Reed-Solomon) • Supports fast table scans (no indexes in BigQuery!) https://cloud.google.com/files/storage_architecture_and_challenges.pdf
- 18. cloud.developerdays.pl@DeveloperDaysPL Storage format: Capacitor •Replaced earlier ColumnIO (base for Parquet and ORC) • Columnar storage format for Colossus • Once imported, all input formats encoded into Capacitor • Able to operate directly on compressed data • Maintains rich metadata on datasets – used by Dremel https://cloud.google.com/files/storage_architecture_and_challenges.pdf
- 19.
- 20. cloud.developerdays.pl@DeveloperDaysPL Networking: Jupyter • Googlepetabit network • Google internal design • Separation of storage and compute
- 21. cloud.developerdays.pl@DeveloperDaysPL Execution engine: Dremel •Scalable, interactive ad-hoc query system for analysis of read-only nested data • Dynamic processing trees, in-memory shuffle component • Dynamic query execution, aided by metadata • No configs/knobs exposed to end user • When idle, running Batch Ingests (for free!) - Poseidon https://ai.google/research/pubs/pub36632
- 22. cloud.developerdays.pl@DeveloperDaysPL Compute: Borg • Container-orientedcluster-management system • Precursor to Kubernetes • E.g. Allocs = Pods https://ai.google/research/pubs/pub43438 https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/
- 23.
- 24.
- 25. cloud.developerdays.pl@DeveloperDaysPL BigQuery use cases •Analyzing online marketing • Google Analytics 360 • Firebase • AdWords • Doubleclick • Hadoop-like processing • BigQuery + DataFlow • Data lake for ML • Archive for cold data (long-term storage pricing)
- 26.
- 27.
- 28.
- 29.
- 30.