7th Athens Big Data Meetup - 2nd Talk - Amazon Redshift Vs Google BigQuery

•

0 likes•81 views

Title: Amazon Redshift Vs Google BigQuery Speaker: Kostas Pardalis (https://gr.linkedin.com/in/kostaspardalis/) Date: Wednesday, May 3, 2017 Event: https://meetup.com/Athens-Big-Data/events/239088154/

Technology

Amazon Redshift
Released on 2012 (beta)
based on ParAccel (PostgreSQL clone)
Designed for OLAP and BIapplications
Relationaland Columnardatabase
Petabyteto Exabytescale (Spectrum)

Google BigQuery
evolution of Dremel(2006)
Initially launched in 2010
WebService on top of Dremel Technology
More of a hybridsystem (columnar + nested data)
Petabytescale

Amazon Redshift Google BigQuery
Build on top of a proven technology
Relational
SQL
Analysts
Build something from scratch
Nested data structures are a first class citizen
NoSQL
Developers
VS

Amazon Redshift Google BigQuery
S3
Kinesis
CSV, Avro, JSON
Google Cloud Storage
Streaming Inserts
Google Analytics Premium
CSV, Avro, JSON
VS

Amazon Redshift Google BigQuery
Schemas
Tables
Datasets
Tables
VS

Data Types
Redshift:Closer to the StandardSQLdatatypes (e.g. INT4,
INT8) but doesnotsupport the full range of PostgreSQL
data types
BigQuery: Smaller set of data types supported. But...

Data Types
Redshift:Very basic support for JSON
BigQuery: Support for Array and STRUCT types. Nesteddata
structures are first class citizens.

Data Manipulation
BigQuery used to be appendonly, now it supportsUpdates
andDeletes (DML). But still limited.
Redshift always had this Supported via SQL but with a
catch (Vacuum)

Table Manipulation
BigQuery: Limited and expensive via standard SQL, or via
HTTP API (but you have to unload and reload the table).
Redshift: Supported via SQL
Both support views but not materialized

Data Consistency
Redshift supports transactions VS BigQuery No
Deduplication harder to be achieved on BigQuery (costly
also).
Even more complex when we go streaming.

Data Consistency
Kinesis: At least once semantics
BigQuery: best-effort deduplication time window +
insertId.

Cluster Management
Here is where BigQueryreallyshines. It is fullymanaged with
supportforHA.
Redshift doesnotabstract completely the hardware from the
user and it is difficult to implement it as a HA service.
This changes with Spectrum.

Connectivity
Redshift:API+ FullJDBC/ODBCsupport.
& access to all the standardPostgreSQL tools.
BigQuery: Mainly through the RESTAPI, JDBC/ODBC drivers
only for queries.

Authentication
Redshift:AWSIAM
BigQuery: OAuth

Amazon Redshift Google BigQuery
Resources capped by your cluster size
No quotas related to inserts/updates
etc
2,000 slots per account
Encourages the append only model with
strict DML quotas
Both have a limit of 50 concurrent Queries
Cluster resizing a pain with Redshift.

Optimization
Redshift:DistributionKeys(Partitioning),SortKeys,Column
Compression
BigQuery: You don’t have to worry about all these, but it
allows you to define time based partitioning.

Optimization
Redshift:DistributionKeys(Partitioning),SortKeys,Column
Compression
With great power comes great responsibility: Vacuuming

Optimization
With BigQueryyou can only optimize your queries on the
statement level and partition over time.

Costs
BigQuery:more difficulttoestimatecosts but costs scale
based on usage. $5 / TB (on demand)
Redshift: you know exactlyhow much you will pay but
you payregardlessofusage.

Performance
BenchmarkbyPanoply
DebunkingMisleadingBenchmarksOfRedshiftvsBigQuery
HNDiscussion
BenchmarkbyPeriscopeData

Amazon Redshift Google BigQueryVS
similarities are greater than the differences

Amazon Redshift Google BigQuery
More predictable costs
More intuitive data modeling (Analysts)
Options for optimizations
Easier & cheaper to start with
Good for nested data
Easier to work with time series
VS

Thanks!kostas@blendo.co
www.blendo.co
Get the free Amazon Redshift guide
http://bit.ly/redshift-guide

What's hot

Deliver Your Modern Data Warehouse (Microsoft Tech Summit Oslo 2018)

Cathrine Wilhelmsen

Azure Big Data Story

Lynn Langit

Introduction to Azure Data Factory

Slava Kokaev

Microsoft Azure Data Warehouse Overview

Justin Munsters

ETL in the Cloud With Microsoft Azure

Mark Kromer

Technical overview of Azure Cosmos DB

Microsoft Tech Community

Azure Data Factory for Azure Data Week

Mark Kromer

Azure data factory

David Giard

Building Data Lakes with Apache Airflow

Gary Stafford

Modern Data architecture Design

Kujambu Murugesan

Building a data warehouse with AWS Redshift, Matillion and Yellowfin

Lynn Langit

Intro to databricks delta lake

Mykola Zerniuk

Gab document db scaling database

MUG Perú

MetricsHub is a monitoring and scalability service for public clouds, allowing companies to continuously gather data from their systems and auto-scale their deployments to optimize service costs. Taking advantage of Cassandra rapid ingestion rates, reliable replication model, and easiness of deployment, Metrics Hub can handle billions of datapoints per day. During this session, you will learn about the architecture supporting this service, which combines the power of the PaaS + IaaS on the Windows Azure platform.

C* Summit 2013: Optimizing the Public Cloud for Cost and Scalability with Cas...

DataStax Academy

Azure Data Factory Data Flow

Mark Kromer

IBM Cloud Day January 2021 - A well architected data lake

Torsten Steinbach

2017 09-27 democratize data products with SQL

Yu Ishikawa

Eugene Polonichko "Architecture of modern data warehouse"

Lviv Startup Club

Big data in Azure

Venkatesh Narayanan

Session 02 data_storage_and_database_services_in_aws_and_azure

Ashish Pandey

What's hot (20)

Deliver Your Modern Data Warehouse (Microsoft Tech Summit Oslo 2018)

Azure Big Data Story

Introduction to Azure Data Factory

Microsoft Azure Data Warehouse Overview

ETL in the Cloud With Microsoft Azure

Technical overview of Azure Cosmos DB

Azure Data Factory for Azure Data Week

Azure data factory

Building Data Lakes with Apache Airflow

Modern Data architecture Design

Building a data warehouse with AWS Redshift, Matillion and Yellowfin

Intro to databricks delta lake

Gab document db scaling database

C* Summit 2013: Optimizing the Public Cloud for Cost and Scalability with Cas...

Azure Data Factory Data Flow

IBM Cloud Day January 2021 - A well architected data lake

2017 09-27 democratize data products with SQL

Eugene Polonichko "Architecture of modern data warehouse"

Big data in Azure

Session 02 data_storage_and_database_services_in_aws_and_azure

Similar to 7th Athens Big Data Meetup - 2nd Talk - Amazon Redshift Vs Google BigQuery

2 years ago if someone had claimed they could stand up a petabyte scale data warehouse in under an hour and then have a non-technical business user querying it live 30 minutes later without knowing any SQL or coding language, they would have been laughed out of the room. These days, that’s called taking advantage of disruptive technology. Amazon Web Services and Tableau Software have shifted the entire paradigm by which organizations not only store and access their data, but ultimately how they innovate with it. The fast, scalable, and inexpensive services that AWS provides for housing data combined with Tableau’s unbelievably flexible and user friendly visual analytic solution means that within hours an organization can securely put the power of their massive data assets into the hands of their domain experts without expensive overhead or lengthy ramp-up time. Attend this webinar to learn how Amazon Web Services and Tableau Software are leveraged together everyday to: • Empower visual ad-hoc data discovery against big data • Revolutionize corporate reporting and dashboards • Promote data driven decision making at every level The presentation will include: • A live demonstration of AWS and Tableau working together • A real customer case study focused on fraud detection and online video metrics • Live Q&A and an opportunity to trial both solutions

AWS Webcast - Tableau Big Data Solution Showcase

Amazon Web Services

Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...

Yahoo Developer Network

Serverless Analytics with Amazon Redshift Spectrum, AWS Glue, and Amazon Quic...

Amazon Web Services

Nonrelational Databases

Udi Bauman

SQL Server 2016 new features

SpanishPASSVC

Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.

Building a modern data warehouse

James Serra

Moving from SQL Server to MongoDB

Nick Court

Build Intelligent Apps with the Microsoft Data & AI Platform

Microsoft Tech Community

Analyze Big Data for Consumer Applications with Looker BI and Amazon Redshift Customizing the customer experience based on user behavior is a constant challenge for today’s consumer apps. Business intelligence helps analyze and model large amounts of data. Looker offers a modern approach to BI leveraging AWS that’s fast, agile, and easy to manage. Join this webinar to learn how MessageMe, which provides emotionally engaging messaging apps to consumers, leverages Looker business intelligence software and the Amazon Redshift data warehouse service to analyze billions of rows of customer data in seconds. Webinar topics include: • How MessageMe turns billions of rows of customer data stored in Amazon Redshift into actionable insights • How Looker connects directly to Amazon Redshift in just a few clicks, enabling MessageMe to build a modern, big data analytics in the cloud. Who should attend • Information or Solution Architects, Data Analysts, BI Directors, DBAs, Development Leads, Developers, or Technical IT Leaders. Presenters: • Justin Rosenthal, CTO, MessageMe • Keenan Rice, VP, Marketing & Alliances, Looker • Tina Adams, Senior Product Manager, AWS

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...

Amazon Web Services

Beyond Relational Databases

Gregory Boissinot

In this session, we will introduce Amazon RedShift, a new petabyte scale data warehouse service. We'll walk through the basics of the Redshift architecture, launching a new cluster and run SQL queries across a large scale, public dataset. After demonstrating how easy it is to get started with RedShift, we will show how to visualize and query large scale datasets, running queries, reports, and analytics against millions of rows of records in just a few seconds.

Getting Started with Amazon Redshift

Amazon Web Services

Amazon-Redshift-dBT-Best-Practices_paper.pdf

Hoang CHi THang

SQL Server 2008 In this event we'll take a look at what's coming down the line for database developers with SQL Server 2008. We'll look at changes to the core engine and the T-SQL language, any changes in the toolset and we'll also take a good look at what's coming with ADO.NET V3.0 in terms of the new Entity Framework and the new Microsoft Synchronisation Framework. If you're a SQL Server developer come along and see what's in store for 2008. For more details and the original slidedeck visit http://www.microsoft.com/uk/msdn/events/new/Detail.aspx?id=322

SQL Server 2008 for Developers

ukdpe

Open Source für den geschäftskritischen Einsatz

MariaDB plc

Data SLA in the public cloud

Liran Zelkha

http://www.hfadeel.com/Blog/?p=151

xlight

Join Johan Andersson, CTO at Severalnines, and Ralf Gebhardt, Product Manager at MariaDB, as they unveil the latest release of ClusterControl, the all-inclusive database management system that lets you easily deploy, monitor, manage and scale highly available open source databases - and load balancers - in any environment: on-premise or in the cloud. We have a particular focus on MariaDB 10.2, thanks to Ralf’s participation, who talk us through the latest features, and give us a sneak preview of what to expect in MariaDB 10.3. ClusterControl now supports the latest versions of MariaDB, MySQL NDB Cluster and PostgreSQL; and introduces a series of new database backup functionalities that range from AWS & Google Cloud integration backup services to automatic backup verifications, making it ever more efficient to run a solid backup strategy for open source database infrastructures. We also look at our new operational reports and email notification features - all in a live demo that you don’t want to miss. AGENDA - MariaDB 10.2: all the new features and a first look at MariaDB 10.3 - ClusterControl 1.5 - What’s new: - MariaDB 10.2 support - AWS & Google Cloud services integration - Enhanced backup functions - New features & support for: - PostgreSQL - MySQL NDB Cluster - ProxySQL - Operational reports - Live demo - Q&A PRESENTERS Johan Andersson, CTO, Severalnines - Johan's technical background and interest are in high performance computing as demonstrated by the work he did on main-memory clustered databases at Ericsson as well as his research on parallel Java Virtual Machines at Trinity College Dublin in Ireland. Prior to co-founding Severalnines, Johan was Principal Consultant and lead of the MySQL Clustering & High Availability consulting group at MySQL / Sun Microsystems / Oracle, where he designed and implemented large-scale MySQL systems for key customers. Johan is a regular speaker at MySQL User Conferences as well as other high profile community gatherings with popular talks and tutorials around architecting and tuning MySQL Clusters. Ralf Gebhardt is Product Manager at MariaDB Corporation. He is responsible for MariaDB Server and MariaDB Connectors. He joined MariaDB/SkySQL in 2011 as Principal Sales Engineer. After 10 years professional experience in Software Development, Support, Training and Consulting, He started working at MySQL GmbH as Sales Engineer in 2002. In the course of the acquisition of Sun Microsystems he joined Oracle, still responsible for MySQL. He holds a masters degree in Computer Engineering from the University of Cooperative Education (in cooperation with IBM Deutschland).

Webinar slides: Severalnines & MariaDB present: Automation & Management of Ma...

Severalnines

[pgday.Seoul 2022] PostgreSQL with Google Cloud

PgDay.Seoul

Aws Summit Berlin 2013 - Understanding database options on AWS

AWS Germany

NoSQL Options Compared

Sergey Bushik

Similar to 7th Athens Big Data Meetup - 2nd Talk - Amazon Redshift Vs Google BigQuery (20)

AWS Webcast - Tableau Big Data Solution Showcase

Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...

Serverless Analytics with Amazon Redshift Spectrum, AWS Glue, and Amazon Quic...

Nonrelational Databases

SQL Server 2016 new features

Building a modern data warehouse

Moving from SQL Server to MongoDB

Build Intelligent Apps with the Microsoft Data & AI Platform

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...

Beyond Relational Databases

Getting Started with Amazon Redshift

Amazon-Redshift-dBT-Best-Practices_paper.pdf

SQL Server 2008 for Developers

Open Source für den geschäftskritischen Einsatz

Data SLA in the public cloud

http://www.hfadeel.com/Blog/?p=151

Webinar slides: Severalnines & MariaDB present: Automation & Management of Ma...

[pgday.Seoul 2022] PostgreSQL with Google Cloud

Aws Summit Berlin 2013 - Understanding database options on AWS

NoSQL Options Compared

More from Athens Big Data

Title: MLOps Workshop: The Full ML Lifecycle - How to Use ML in Production Speakers: Spyros Cavadias (https://www.linkedin.com/in/spyros-cavadias/), Konstantinos Pittas (https://www.linkedin.com/in/konstantinos-pittas-83310270/), Thanos Gkinakos (https://www.linkedin.com/in/thanos-gkinakos-03582a128/) Date: Saturday, December 17, 2022 Event: https://www.meetup.com/athens-big-data/events/289927468/

22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...

Athens Big Data

21st Athens Big Data Meetup - 2nd Talk - Dive into ClickHouse storage system

Athens Big Data

19th Athens Big Data Meetup - 2nd Talk - NLP: From news recommendation to wor...

Athens Big Data

21st Athens Big Data Meetup - 3rd Talk - Dive into ClickHouse query execution

Athens Big Data

21st Athens Big Data Meetup - 1st Talk - Fast and simple data exploration wit...

Athens Big Data

20th Athens Big Data Meetup - 2nd Talk - Druid: under the covers

Athens Big Data

20th Athens Big Data Meetup - 3rd Talk - Message from our sponsor: Velti

Athens Big Data

20th Athens Big Data Meetup - 1st Talk - Druid: the open source, performant, ...

Athens Big Data

19th Athens Big Data Meetup - 1st Talk - NLP understanding

Athens Big Data

18th Athens Big Data Meetup - 2nd Talk - Run Spark and Flink Jobs on Kubernetes

Athens Big Data

18th Athens Big Data Meetup - 1st Talk - Timeseries Forecasting as a Service

Athens Big Data

17th Athens Big Data Meetup - 2nd Talk - Data Flow Building and Calculation P...

Athens Big Data

17th Athens Big Data Meetup - 1st Talk - Speedup Machine Application Learning...

Athens Big Data

16th Athens Big Data Meetup - 2nd Talk - A Focus on Building and Optimizing M...

Athens Big Data

16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...

Athens Big Data

15th Athens Big Data Meetup - 1st Talk - Running Spark On Mesos

Athens Big Data

5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...

Athens Big Data

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Athens Big Data

13th Athens Big Data Meetup - 2nd Talk - Training Neural Networks With Enterp...

Athens Big Data

11th Athens Big Data Meetup - 2nd Talk - Beyond Bitcoin; Blockchain Technolog...

Athens Big Data

More from Athens Big Data (20)