BDA309 Build Your First Big Data Application on AWS

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Allan MacInnis
Solutions Architect, Amazon Web Services
Ryan Nienhuis
Sr. PM, Amazon Kinesis, Amazon Web Services
BDA309
Building Your First Big Data Application
on AWS

Workshop Objectives
 Build an end-to-end analytics application on AWS
 Implement batch and real-time layers in a single
application
 Explore AWS analytics portfolio for other use cases

Why Analytics on AWS?
 Unmatched durability, and availability at
EB scale
 Best security, compliance, and audit
capabilities
 Object-level controls for fine-grain
access
 Fastest performance by retrieving
subsets of data
 The most ways to bring data in
 2x as many integrations with partners
 Analyze with broadest set of analytics &
ML services
Analytics
Data lake
on AWS
Real-time data
movement
On-premises
data movement
Machine
learning

Your Application Architecture
Kinesis Producer UI Amazon Kinesis
Data Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
Generate web logs Collect web logs
and deliver to
Amazon S3
Process & compute
aggregate web log metrics
Deliver processed web
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
Visualize web logs to
discover insights
Amazon S3
Bucket
Interactive analysis of
web logs
Amazon
Athena
Interactive querying of
web logs
AWSGlue
Extract metadata, create
tables and transform
web logs from CSVto
parquet
Amazon S3
Bucket
Transformed web
logs from AWS Glue

What Is qwikLABS?
 Provides access to AWS services for this workshop
 No need to provide a credit card
 Automatically deleted when you’re finished

Sign in and Start the Lab
Once the lab is started, you will see a lab setup progress bar. It takes
~10 min for the lab to be setup.

Navigating qwikLABS
• Student resources:
Scripts for your labs
• Open console: Opens
AWSManagement
Console
• Additional connection
details: Links to different
interfaces

Everything You Need for the Lab
 Open the AWS Management Console, log in, and verify that the following AWS
resources are created:
 One Amazon Kinesis Data Analytics application
 One Kinesis Data Analytics preprocessing Lambda function
 Two Amazon Kinesis Data Firehose delivery streams
 One Amazon EMR cluster
 One Amazon Redshift cluster
 Sign up (later) for:
 Amazon QuickSight

Amazon Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Build custom
applications that process
and analyze streaming
data
Easily process and
analyze streaming data
with standard SQL
Easily load streaming
data into AWS

Amazon Kinesis Data Streams
• Easy administration and low cost
• Build real-time applications with framework of choice
• Secure, durable storage

Amazon Kinesis Data Firehose
 Zero administration and seamless elasticity
 Direct-to-data store integration
 Serverless, continuous data transformations

Amazon Kinesis – Kinesis Data Firehose vs. Kinesis Data
Streams
Amazon Kinesis Data Streams is for use cases that require
custom processing, per incoming record, with sub-1 second
processing latency, and a choice of stream processing
frameworks.
Amazon Kinesis Data Firehose is for use cases that require
zero administration, ability to use existing analytics tools
based on Amazon S3, Amazon Redshift, and Amazon
Elasticsearch, and a data latency of 60 seconds orhigher.

Activity 1
Collect Logs Using a Kinesis Data
Firehose Delivery Stream

Kinesis Producer UI Amazon
Kinesis
Data
Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data
Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to
Amazon S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
Interactive querying
of web logs
AWSGlue
Extract metadata, create
web logs from CSVto
parquet
Amazon S3
Bucket
Transformed web logs
from AWS Glue

 Time: 5 min.
We are going to:
 Write to a Kinesis Firehose delivery stream. We’ll simulate writing
transformed Apache web logs to a Firehose delivery stream that is configured
to deliver data into an S3 bucket.
 There are many different libraries that can be used to write data to a
Kinesis Date Firehose delivery stream. One popular option is called the
Amazon Kinesis Agent.
Collect Logs with a Kinesis Data Firehose Delivery
Stream

 Amazon Kinesis Agent
 Standalone Java application to collect and send data to Kinesis
Data Firehose
 Continuously monitors set of files
 Handles file rotation, check-pointing, and retry on failures
 Emits CloudWatch metrics
 Pre-process records parsed from monitored files
Stream

Stream
For example, the agent can transform an Apache web log to JSON.
From:
To:
{"HOST" : "125.166.52.103",
"IDENT" : null,
"AUTHUSER" : null,
"DATETIME" : "08/Mar/2017:17:06:44 -08:00",
"REQUEST" : "GET /explore",
"RESPONSE" : 200,
"BYTES" : 2503,
"REFERRER" : null,
"AGENT" : "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 5.2;
Trident/5.0)“}
125.166.52.103 - - [08/Mar/2017:17:06:44 -08:00] "GET /explore“
200 2503 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 5.2;
Trident/5.0)“}

So that we don’t have to install or set up software on your machine, we are
going to use a utility called the Kinesis Data Generator to simulate using the
Amazon Kinesis agent. The Kinesis Data Generator can populate a Firehose
delivery stream using a template and is simple to set up.
Let’s get started!
Stream

Activity 1A: Working with the KDG
Qwiklabs has already created and set up the Kinesis Data Firehose delivery
stream for us. All we have to do is start writing data to it.
Go to this Kinesis Data Generator (KDG) Help Section at
http://tinyurl.com/kinesispublisher
Which will redirect to:
https://s3.amazonaws.com/kinesis-data-producer-test/help.html

 Choose Create Amazon Cognito User with AWS CloudFormation.
 This link takes you to a service called AWS CloudFormation. AWS
CloudFormation gives developers and systems administrators an easy way
to create and manage a collection of related AWS resources, provisioning
and updating them in an orderly and predictable fashion.
 We use AWS CloudFormation to create the necessary user credentials to use
the Kinesis Data Generator.

On the next screen, choose Next. Here, we are using a template stored in
an Amazon S3 bucket to create the necessary credentials.

Specify a user name and password (and remember them!), and choose Next.
This user name and password will be used to sign into the Kinesis Data
Generator. The password must be at least 6 alpha-numeric characters, and
contain at least one number.

 We use a service called Amazon Cognito to create these credentials.
 Amazon Cognito lets you easily add a user sign-up and sign-in to your mobile
and web apps. With Amazon Cognito, you also have the options to authenticate
users through social identity providers such as Facebook, Twitter, or Amazon,
with SAML identity solutions, or by using your own identity system.

The next screen has some additional options for the AWS CloudFormation stack,
which are not needed. Choose Next.

This screen is a review screen so you can verify that you have selected the
correct options. When you are ready, choose Acknowledge, and choose
Create.

You are taken to a screen that shows the stack creation process. (You may have
to refresh the page.) In approximately one minute, the stack will complete, and,
under Status, you will see CREATE_COMPLETE. Once this occurs, a) select
the template, b) select the outputs tab, and c) choose the link to navigate to your
very own Kinesis Data Generator (hosted on Amazon S3).

 Log in using your user name and
password.
 Select the region us-west-2” (Oregon).
 Select the delivery stream with name (qls-<somerandomnumber>-
FirehoseDeliveryStream-<somerandomnumber>-11111111111).
 Specify a data rate (Records per second). Choose a number less than 10.

We chose a number less than 10 because everyone is on the same Wi-Fi network, and we
want to be sure we don’t use all the bandwidth. Additionally, if you are not plugged in, you
may run into a battery issues at a higher rate.
For the Record Template, you should use the template Apache Combined Log Template,
in the Student Resources section. The template should look like the following:

Choose Send Data to Kinesis.

Review: Monitoring Your Delivery Stream
 Go to the Amazon CloudWatch Metrics Console and search
IncomingRecords. Choose this metric for your Firehose delivery stream,
and for Sum, choose 1 Minute.
 What are the most important metrics to monitor?

Activity 2
Real-Time Data Processing
Using Kinesis Data Analytics

Data Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data
Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
web logs
AWSGlue
Extract metadata,create
web logs from CSVto
parquet
Amazon S3
Bucket
Transformed web
logs from AWSGlue

Amazon Kinesis Data Analytics
• Powerful real time applications
• Easy to use, fully managed
• Automatic elasticity

Kinesis Data Analytics Applications
Easily write SQLcode to process streamingdata
Connect to streamingsource
Continuously deliver SQLresults

Process Data Using Kinesis Data Analytics
 Time: 20 min.
We are going to:
 Write a SQL query to compute aggregate metrics for an interesting statistic on
the incoming data.
 Write a SQL query using an anomaly detection function.

Activity 2A: Start Kinesis Data Analytics App
 Navigate to the Amazon
Kinesis dashboard.
 Choose the Kinesis analytics
application.

Activity 2A: Start Kinesis App
Choose Go to SQL editor. On the next screen, choose Yes, start
application.

View Sample Records in Kinesis App
Review sample records delivered to the source stream
(SOURCE_SQL_STREAM_001).

Kinesis Analytics Application Metadata
Note that Kinesis adds metadata to each record being sent that was shown in
the formatted record sample:
- The ROWTIME represents the time the application read the record, and is a
special column used for time series analytics. This is also known as a the
process time.
- The APPROXIMATE_ARRIVAL_TIME is the time the delivery stream received
the record. This is also known as ingest time.

Kinesis Data Analytics: How Streaming App Works
 Writing SQL over streaming data using Kinesis Data Analytics follows a
two part model:
1. Create an in-application stream for storing intermediate SQL results. An in-
application stream is like a SQL table, but is continuously updated.
2. Create a PUMP that will continuously read FROM one in-application stream
and INSERT INTO a target in-application stream.
DESTINATION_STREAM
Part 1
AGGREGATE_STREAM
Part 2
SOURCE_SQL_STREAM_001
Source Stream
OUTPUT_PUMP
AGGREGATE_PUMP
Send to Amazon Redshift

Activity 2B: Calculate an Aggregate Metric
Calculate a count using a tumbling window and a GROUP BY clause.
Atumbling window is similar to
a periodic report, where you
specify your query and a time
range, and results are emitted at
the end of a range. (e.g.,COUNT
number of items by key for 10
seconds)

Tumbling
Sliding
Custom
• Fixed size and non-overlapping
• Use FLOOR() or STEP() function (coming soon) in a GROUP BY statement
• Fixed size and overlapping; row boundaries are determined when new
rows enter thewindow
• Use standard OVERand WINDOW clause (i.e., count (col) OVER
(RANGE INTERVAL‘5’ MIN)
• Not fixed size and overlapping; row boundaries are determined by
conditions
• Implementations vary, but typically require two steps (step 1 –
Identify boundaries, step 2 – Perform computation)

 The window is defined by the following statement in the SELECT statement. Note
that the ROWTIME column is implicitly included in every stream query, and
represents the processing time of the application.
 This is known as a tumbling window. Tumbling windows are always included in a
GROUP BY clause and use a STEP function. The STEP function takes an interval
to product the periodic reports.
 You can also use the SQL FLOOR() function to achieve the same thing.
STEP(source_sql_stream_001.ROWTIME BY INTERVAL '10' SECOND)

Create an aggregate_stream using SQL command found in the Kinesis Data
Analytics SQL file in the Student Resources section of your lab. Copy and paste
the SQL in the SQL editor underneath the DESTINATION_SQL_STREAM DDL.
Choose Save and run SQL.

Activity 2C: Anomaly Detection
 Kinesis Data Analytics includes some advanced algorithms that are extensions
to the SQL language. These include approximate count distinct
(hyperloglog), approximate top K (space saving), and anomaly detection
(random cut forest).
 The random cut forest algorithm detects anomalies in real time on multi-
dimensional datasets. You pass the algorithm n number of numeric fields and
the algorithm produces an anomaly score on your stream data. Higher scores
are more anomalous.
 The minimum anomaly score is 0 and the maximum is log2 s, where s is the
subsample size parameter passed to random cut forest (third parameter). You
will have to try the algorithm on your data to get a feel for the anomaly score, as
the score is data-dependent.

Activity 2C: Anomaly Detection
Create an anomaly_stream using the SQL command found in the
Kinesis Data Analytics SQL file in the Student Resources section of
your lab. Append the SQL in your SQL editor. Choose Save and run
SQL.

Review: In-Application SQL streams
 Your application has multiple in-application SQL streams, including
DESTINATION_SQL_STREAM, AGGREGATE_STREAM, and
ANOMALY_STREAM. These in-application streams are like SQL tables that are
continuously updated.
 What else is unique about an in-application stream aside from its continuous
nature?

Activity 3
Deliver Streaming Results to
Amazon Redshift

Data Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data
Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
of web logs
AWSGlue
Web logs from CSVto
Parquet
Amazon S3
Bucket
Transformed web
logs from Glue

Activity 3: Deliver Data to Amazon Redshift using
Amazon Kinesis Data Firehose
 Time: 5 min.
 We are going to:
A. Connect to Amazon Redshift cluster and create a table to hold web
logs data.
B. Update the Kinesis Analytics application to send data to Amazon
Redshift, via the Firehose delivery stream.

Activity 3A: Connect to Amazon Redshift
 Or use any JDBC/ODBC/libpq client
- Aginity Workbench for Amazon
Redshift
- SQL Workbench/J
- DBeaver
- DataGrip
 If you use the above SQL clients, the
user name and password are in your
qwikLABS console at the bottom. You will
find the cluster endpoint, database name
(logs) in the qwikLABS console as well.
 You can connect with pgweb.
 Already installed and configured for the
Amazon Redshift cluster.
 Just navigate to pgweb, and start
interacting.
Note: From the qwikLABS console, open the
pgWeb link in a new window.

Activity 3B: Create a Table in Amazon Redshift
 Create table weblogs to capture the incoming data from a Firehose delivery stream.
Note: You can download Amazon Redshift SQL code from qwikLabs Student Resource
section (choose Open Redshift SQL).

Activity 3C: Deliver Data to Amazon Redshift Using
Firehose
Update Kinesis Analytics application to send data to Firehose delivery stream.
Amazon Kinesis Data Firehose delivers the streaming data to Amazon Redshift.
1. Go to the Kinesis Data Analytics console.
2. Choose the Amazon Redshift delivery stream as destination, and
choose the edit icon (the pencil icon in the figure below).

Activity 3C: Deliver Data to Amazon Redshift Using
Kinesis Data Firehose
 Validate your destination
1. Choose the Kinesis Data Firehose qls-xxxxxxx-
RedshiftDeliveryStream-xxxxxxxx delivery stream.
2. Keep the default for Connect in-application stream.
3. Choose CSV as the Output format.
4. Select Choose from IAM roles that Kinesis Analytics can assume.
5. Click Save and continue.
 It will take about 1–2 minutes for everything to be updated and for data to start
appearing in Amazon Redshift.

Activity 3C: Deliver Data to Amazon Redshift using
Kinesis Data Firehose

Review: Amazon Redshift Test Queries
 Find distribution of response codes over days (copy SQL from Amazon Redshift
SQL file)
 Count the number of 404 response codes

Review: Amazon Redshift Test Queries
Show all requests paths with status “PAGE NOT FOUND"

Extract, Transform, and Load (ETL)
with AWS Glue

Fully managed ETL (Extract, Transform, and Load)
Service
AWS Glue
• Categorize your data, clean it, enrich it, and reliably
move it between various data stores
• Once cataloged, your data is immediately
searchable and queryable across your data silos
• Simple, flexible, and cost-effective
• Serverless; runs on a fully managed, scale-outSpark
environment

AWS Glue Components
• Discover and organize your data in various databases, data
warehouses, and data lakes.
• Focus on the writing transformations.
• Generate code through a wizard.
• Runs jobs in Spark containers. Automatic scaling based
on SLA.
• AWS Glue is serverless. Only pay for the resourcesyou
consume.
Write your own code
Job Authoring
Data Catalog
Job Execution

Activity 4
Transform Web Logs to Parquet
Format Using AWS Glue

Data Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
web logs
AWSGlue
web logs from CSVto
parquet
Amazon S3
Bucket
Transformed web
logs from Glue

Activity: Catalog and Perform ETL on Weblogs
Time: 40 min.
We are going to:
A. Discover and catalog the web log data deposited into the S3 bucket
using AWS Glue Crawler.
B. Transform web logs to parquet format with theAWS Glue ETL
job authoring tool.

Activity 4A: Discover Dataset with AWS Glue
We use the AWS Glue crawler to extract data and metadata. From the
AWS Management Console, select AWS Glue Service, and on the
next screen choose Get Started.

Activity 4A: Add Crawler Using AWS Glue
On the left, choose Crawlers, and choose Add crawler.

Activity 4A: ETL with AWS Glue
 Specify a name for the crawler. Choose the folder icon to select the
data store on Amazon S3.
 Choose Next.

 Provide the S3 path location where the weblogs are deposited.
Navigate to the S3 path that contains the word logs, and select the
raw folder.
 Choose Select.

Choose Next on the next screen to not add another data store.

In the IAM Role section, select Create an IAM role, add default to the
IAM role, and choose Next.

Activity 4A: Add Crawler in AWS Glue
Choose Run on demand to run the crawler now, and choose Next.

Activity 4A: Add Crawler with AWS Glue
On the next screen, choose Add database to add a database (use weblogdb for
the database name).

Review and choose Finish to create a crawler.

Choose the Run it now link to run the crawler.

Crawler shows a Ready status when it is finished running.

Activity 4A: Table creation in AWS Glue
 Observe that the crawler has created a table for your dataset.
 The crawler automatically classified the dataset as combinedapache log
format.
 Click the table to look at the properties.

Activity 4A: Table Creation in AWS Glue
 AWS Glue used the GrokSerDe (Serializer/Deserializer) to correctly
interpret the web logs.
 You can choose the View partitions link to look at the partitions in the
dataset.

Activity 4A: Create ETL Job in AWS Glue
 With the dataset cataloged and table created, we are now ready to convert the
web logs based Apache combined log format to a more optimal parquet format for
querying.
 Choose Add job to begin creating the ETL job.

Activity 4B: ETL Job in AWS Glue
 In the Job properties window, specify
the job name.
 In this example, we will create a new ETL
script.
 AWS Glue automatically chooses the name
of the script file and the path where the
script will be persisted.
 For the Temporary directory, specify an S3
temporary directory in your lab account (use
the s3://<stack>-logs-<account>-us-west-
2/ bucket, and append the folder name,
temp).
 The path should look like:
s3://<stack>-logs-<account>-us-west-
2/temp/

 Do not choose Next yet.
 Copy the temporary path to a text editor, modify the path as follows, and
save it in a file (we will need this path for storing parquet files).
s3://<stack>-logs-<account>-us-west-2/weblogs/processed/parquet
E.g.,
s3: / / qls- 108881- f75177905b7b5b0e- logs- XXXXXXXX3638- us- west-
2/weblogs/processed/parquet

 Expand the Script libraries and job
parameters section, and increase the DPUs to
20.
 Let’s pass a job parameter to send the S3 path
where parquet files will be deposited.
Specify the following values for Key and
Value:
 Key: --parquet_path (notice the 2 hyphens
at the beginning)
 Value: s3://<stack>-logs-<account>-us-
west- 2/weblogs/processed/parquet/
Note: The value is the S3 path we stored in
the previous section.

Choose Next in the following screen, and choose Finish to complete the job
creation.

Activity 4B: ETLJob in AWS Glue
 Close Script Editor tips
window (if it appears).
 In the Glue Script Editor,
copy the ETL code by
Choosing the Open Glue
ETL Code link in
Student Resources.
 Ensure that the database
name (db_name) and table
name reflect the database
and table name created by
the AWS Glue crawler.

Choose Save and then Run job to execute your ETL. Choose Save and run
job in the next window.

 Run job to continue. This might take a few minutes. When the job finishes,
weblogs will be transformed to parquet format.

Interactive Querying with Amazon
Athena

Interactive Query Service
 Query directly from Amazon S3
 Use ANSI SQL
 Serverless
 Multiple data formats
 Cost effective
Amazon Athena

Familiar Technologies Under the Covers
Used for SQL Queries
In-memory distributed query engine
ANSI-SQL compatible with extensions
Used for DDLfunctionality
Complex data types
Multitude of formats
Supports data partitioning

Comparing Performance and Cost Savings for
Compression and Columnar Format
Dataset Size onAmazon Query run time Data scanned Cost
S3
Data stored as text 1 TB
files
236 seconds 1.15 TB $5.75
Data stored in
Apache Parquet 130 GB 6.78 seconds 2.51 GB $0.013
format*
Savings / 87% less with 34x faster 99% less data 99.7% savings
Speed up parquet scanned
(*compressed using Snappy compression)
https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/

Activity 5
Interactive Querying with Amazon
Athena

Kinesis Producer
UI
Amazon Kinesis
Data Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
of web logs
AWSGlue
Web logs from CSVto
Parquet
Amazon S3
Bucket
Transformed web
logs from Glue

Activity: Interactive Querying with Amazon Athena
Time: 15 min.
We are going to:
A. Create a table over the processed weblogs in Amazon S3. These
are the parquet files created by the AWS Glue ETLjob in the
previoussection.
B. Run interactive queries on the parquet weblogs.

Activity 5A: Set Up Amazon Athena
 From the AWS Management Console, search for Athena, and choose the
service.

 In the Analytics section, choose Amazon Athena, and on the next page,
choose Get Started.

 Close the running the Athena tutorial.
 Close any other open tutorial windows.

 We are now ready to create a table in Amazon Athena. But before we do that,
we need to get the S3 bucket location where the AWS Glue job delivered parquet
files. Choose Services, and in the Storage section, choose S3.

Locate the bucket that looks like qls-<stackname>-logs-#####-us-west-2.
Navigate to the parquet folder. Copy the name of the bucket into a text editor.

 Go back to the Athena console.
 Let’s create a table in Athena on the parquet dataset created by AWS Glue.
 In the Athena console, in the database drop-down menu, choose weblogdb.
 Type the SQL command (found by choosing the Open Athena SQL link in the
Student Resources section of qwiklabs) to create a table.
 Make sure to replace the <your-parquet-path> with the bucket location for the
parquet files you copied in the previous step.
Screen shot on next slide.

Activity 5B: Working with Amazon Athena

Activity 5B: Working with Amazon Athena
 The SQL ddl in the
previous step creates a
table in Athena based on
the parquet files we
created with the AWS
Glue ETL job.
 Select weblogdb from
the database section,
and click the three
stacked dots icon to
sample a few rows of the
Amazon S3 data.

Activity 5C: Interactive Querying with Amazon Athena
Run interactive queries (copy SQL queries from Athena SQL in Student
Resources), and see the results on the console.

Activity 5C: Interactive Querying with Athena
 Optionally, you can save the results of a query to CSV by choosing the file icon
on the Results pane.
 You can also view the results of previous queries or queries that may take some
time to complete. Choose History. Then, either search for your query or
choose View or Download to view or download the results of previous
completed queries. This also displays the status of queries that are currently
running.

Review: Amazon Athena Interactive Queries
 Query results are also stored in Amazon S3 in a bucket called aws-athena-
query-results-ACCOUNTID-REGION.
 Where can you change the default location in the console?

Data Processing with Amazon EMR

Amazon EMR Service
Applications
Hive, Pig, Spark SQL/Streaming/ML, Flink, Mahout, Sqoop
HBase/
Phoenix
Presto
MXNet
Batch
MapReduce
Interactive
Tez
In Memory
Spark
Streaming
Flink
YARN
Cluster resource management
Storage
Amazon S3 (EMRFS), HDFS
Ganglia - Monitoring
Hue - SQL
Zeppelin - Notebooks
Livy - Job server
Connectorsto
AWSservices
Amazon EMR
service

On-Cluster UIs
Manage applicationsNotebooks
SQL editor, workflow designer,
metastore browser
Design and execute
queries and workloads
And more using
bootstrap actions!

The Hadoop Ecosystem Can Run in Amazon
EMR

Deep Learning with GPU
Instances
Use P3 instances
with GPUs

Easy to Use Spot Instances
Meet SLA at a predictable cost
On-demand for
core nodes
Standard
Amazon EC2
pricing for
on-demand
capacity
Exceed SLA at a lowercost
Spot Instances
for task nodes
Up to 90% off
Amazon EC2
on-demand
pricing

Amazon S3 as Your Persistent Data Store
 Separate compute and storage
 Resize and shut down Amazon EMR
clusters with no data loss
 Point multiple Amazon EMR clusters
at the same data in Amazon S3
EMR
EMR
Amazon
S3

EMRFS Makes It Easier to Use Amazon S3
 Better performance and error-handling options
 Transparent to applications – Use “s3://”
 Consistent view
- For consistent list and read-after-write for newputs
 Support for Amazon S3 server-side and client-side encryption
 Faster listing using EMRFS metadata

Apache Spark
 Fast, general-purpose engine for large-
scale data processing
 Write applications quickly in Java, Scala,
or Python
 Combine SQL, streaming, and complex
analytics

Apache Zeppelin
 Web-based notebook for interactive
analytics
 Multiple language backend
 Apache Spark integration
 Data visualization
 Collaboration
 https://zeppelin.apache.org/

Activity 6
Interactive Analysis Using Amazon
EMR

Kinesis Producer
UI
Amazon
Kinesis
Data
Firehose
AmazonKinesis
Data Analytics
AmazonKinesis
Data
Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
web logs
AWSGlue
Web logs from CSVto
Parquet
Amazon S3
Bucket
Transformed web
logs from Glue

Activity 6: Process and Query Data with Amazon EMR
 Time: 20 minutes
 We are going to:
A. Use a Zeppelin notebook to interact with an Amazon EMR cluster,
B. Process the data in Amazon S3 using Apache Spark.
C. Query the data processed in the earlier stage, and create simple
charts.

Activity 6A: Open the Zeppelin Interface
 Copy the Zeppelin endpoint in Student
Resources section in qwiklabs.
 In the Student Resources section,
select the Open Zeppelin Notebook
link to open the Zeppelin link in a new
window.
 Download the file (or copy and save it
to file with .json extension).
 Import the notebook using the Import
note link on the Zeppelin interface.
Note: Disconnect from VPN, or the
page will not load.

Activity 6A: Open the Zeppelin Interface
 Use the s3://<stack>-logs-<account>-us-west-
2/processed/parquet bucket where the processed parquet files
weredeposited.
 Run the paragraph.

Activity 6B: Run the Notebook
 Enter the S3 bucket name where the parquet files are stored. The bucket
name begins with <stack>-*-logs-#####-region.
 Execute step 1
- Enter the bucket name (<stack>-*-logs-##########-us-west-2).
 Execute step 2
- Create a data frame with the parquet files from the Glue ETL job.
 Execute step 3
- Sample a few rows.

 Execute step 4 to process the data
• Notice how the AGENT field consists of the BROWSER at the
beginning of the column value. Let’s extract the browser from that.
• Create a UDF that will extract the browser part and add it to the
dataframe.
• Print the new data frame.

 Execute step 6
- Register the data frame as a temporary table.
- Now you can run SQL queries on the temporary tables.
 Execute the next three steps, and observe the charts
created.
- What did you learn about the dataset?

Review : Interactive Analysis Using Amazon EMR
 You just learned how to process and query data using Amazon EMR
with Apache Spark.
 Amazon EMR has many other frameworks available for you to use:
- Hive, Presto, Flink, Pig, MapReduce
- Hue, Oozie, HBase

Optional Exercise:
Data Visualization with Amazon
QuickSight

Amazon QuickSight
 Fast, easy interactive analytics
for anyone, everywhere
 Ease of use targeted at business users
 Blazing fast performance powered by
SPICE
 Broad connectivity with AWS data
services, on-premises data, files,
and business applications
 Cloud-native solution that scales
automatically
 1/10th the cost of traditional BI solutions
 Create, share, and collaborate with
anyone in your organization, on the web
or on mobile

Connect, SPICE, Analyze
Amazon QuickSight enables you to connect to data from a wide variety of AWS,
third-party, and on-premises sources and import it to SPICE or query directly. Users
can then easily explore, analyze, and share their insights with anyone.
Amazon RDS
Amazon S3
Amazon
Redshift

Activity 7
Visualize Results in Amazon
QuickSight

Kinesis Producer
UI
Kinesis
Data
Firehose
KinesisData Analytics
KinesisData Firehose
Amazon
EMR
Amazon
Redshift
Amazon
QuickSight
and deliver to S3
Process & compute
logs to Amazon
Redshift
Raw web logs from
Kinesis Data
Firehose
Run SQL queries on
processed web logs
discover insights
Amazon S3
Bucket
web logs
Amazon
Athena
web logs
AWSGlue
Web logs from CSVto
Parquet
Amazon S3
Bucket
Transformed web
logs from AWSGlue

Activity 7: Visualization with Amazon QuickSight
We are going to:
A. Register for an Amazon QuickSight account.
B. Connect to the Amazon Redshift cluster.
C. Create visualizations for analysis to answer questions like:
A. What are the most common HTTP requests, and how
successful (response code of 200) are they?
B. Which are the most requested URIs?

Activity 7A: Amazon QuickSight Registration
 Go to the AWS Management
Console, and in the
Analytics section, choose
QuickSight.
 On the next screen,
choose Signup.
 Make sure the subscription
type is Standard, and on the
next screen, choose
Continue.

Activity 7A: Amazon QuickSight Registration
 On the Subscription type page, type
in the account name. The QuickSight
account name is the AWS account
number on the qwikLabs console.
 Type in your email address.
 Select US West region.
 Select S3 (all buckets).

Activity 7A: QuickSight Registration
 If a pop box to choose S3 buckets
appears, choose Select buckets.
 Choose Go to Amazon
QuickSight.
 Dismiss the welcome screen.

Activity 7B: Connect to the Data Source
 Choose Manage data and
then choose New dataset to
create a new dataset in
Amazon QuickSight.
 Choose Redshift Auto-
discovered as the data source.
Amazon QuickSight
autodiscovers databases
associated with your AWS
account, in this case the
Amazon Redshift database.

Activity 7B: Connect to Amazon Redshift
Note: Use dbadmin as the user
name. You can get the Amazon
Redshift database password from
qwikLABS by navigating to the
Connection details section (see
below).

Activity 7C: Choose Your Weblogs Amazon Redshift
Table

Activity 7D: Ingest Data into SPICE
 SPICE is Amazon QuickSight's in-
memory optimized calculation
engine, designed specifically for
fast, interactive data visualization.
 You can improve the performance of
database datasets by importing the
data into SPICE instead of using a
direct query to the database.

Activity 7E: Creating Your First Analysis
 What are the most requested
HTTP request types and their
corresponding response codes
for this site?
 Simply select request,
response, and let
AUTOGRAPH create the
optimal visualization

Review – Creating Your Analysis
 Exercise: Add a visual to demonstrate which URI is the most requested.

Submit Session Feedback
1. Tap the Schedule icon. 2. Select the session
you attended.
3. Tap Session
Evaluation to submit your
feedback.

Thank you!

BDA309 Build Your First Big Data Application on AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to BDA309 Build Your First Big Data Application on AWS

Similar to BDA309 Build Your First Big Data Application on AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

BDA309 Build Your First Big Data Application on AWS