AWS has a large and growing portfolio of big data management and analytics services, designed to be integrated into solution architectures that meet the needs of your business. In this session, we look at analytics through the eyes of a business intelligence analyst, a data scientist, and an application developer, and we explore how to quickly leverage Amazon Redshift, Amazon QuickSight, RStudio, and Amazon Machine Learning to create powerful, yet straightforward, business solutions.
2. What to expect from this session
• AWS toolkit for analytics
• Understand stakeholders
• Demo
• Case Study – WagglePractice
• Case Study – Ivy Tech
• Q&A
3. AnalyzeStore
Amazon
Glacier
Amazon
S3
Amazon
DynamoDB
Amazon RDS,
Amazon Aurora
Big data portfolio—but what do I recommend?
AWS Data Pipeline
Amazon
CloudSearch
Amazon
EMR
Amazon EC2
Amazon
Redshift
Amazon
Machine
Learning
Amazon
Elasticsearch
Service
AWS Database
Migration
Amazon
Kinesis
Analytics
Amazon Kinesis
Firehose
AWS Import/Export
AWS Direct
Connect
Collect
Amazon Kinesis
Streams
Amazon
QuickSight
4. Match toolset to right persona
• Business intelligence (BI) analyst
• Primary tool is SQL
• Historical data resides in data warehouse such as
Amazon Redshift
• Data scientist—Uses programmatic languages such as R or
Python
• Application developer—Requires API to integrate with AWS
services
6. BI analyst with existing BI tools
BI Analyst
BI tools
Amazon EC2
Amazon Redshift
QuickSight API
• Primary tool is SQL
• Data is largely structured with well known data sources
• Primary concern is fast, consistent performance
• Need to extend SQL with custom functions
BI tools
Amazon EC2
Amazon QuickSight
Amazon QuickSight
7. Amazon Redshift system architecture
Leader node
• SQL endpoint
• Stores metadata
• Coordinates query execution
Compute nodes
• Local, columnar storage
• Execute queries in parallel
• Load, backup, restore via
Amazon S3; load from
Amazon DynamoDB, Amazon EMR, or SSH
Two hardware platforms
• Optimized for data processing
• DS2: HDD; scale from 2 TB to 2 PB
• DC1: SSD; scale from 160 GB to 356 TB
10 GigE
(HPC)
JDBC/ODBC
8. New SQL functions
We add SQL functions regularly to expand Amazon Redshift’s query capabilities
Added 25+ window and aggregate functions since launch, including:
LISTAGG
[APPROXIMATE] COUNT
DROP IF EXISTS, CREATE IF NOT EXISTS
REGEXP_SUBSTR, _COUNT, _INSTR, _REPLACE
PERCENTILE_CONT, _DISC, MEDIAN
PERCENT_RANK, RATIO_TO_REPORT
We’ll continue iterating but also want to enable you to write your own
Window function examples: http://docs.aws.amazon.com/redshift/latest/dg/r_Window_function_examples.html
9. Scalar user defined functions
You can write UDFs using Python 2.7
• Syntax is largely identical to PostgreSQL UDF
• Python execution is performed in parallel
• System and network calls within UDFs are prohibited
Comes integrated with Pandas, NumPy, SciPy, DateUtil, and
Pytz analytic libraries
• Import your own libraries for even more flexibility
• Take advantage of thousands of functions available through Python
libraries to perform operations not easily expressed in SQL
10. A very fast, cloud-powered, business
intelligence service for 1/10 the cost of
traditional BI software
What is Amazon QuickSight?
13. Data scientist with existing toolsets
Data scientist Toolkits like SAS or
R Studio installed
with Amazon EC2
Unstructured data
Amazon S3
Structured data
Amazon Redshift
• Work with unstructured datasets
• Use existing toolsets to connect to Amazon Redshift
14. Querying Amazon Redshift with R packages
• RJDBC—Supports SQL queries
• dplyr—Uses R code for data
analysis
• RPostgreSQL—R compliant
driver or database Interface (DBI)R User
R Studio
Amazon
EC2
Unstructured data
Amazon S3
User profile
Amazon RDS
Amazon Redshift
Connecting R with Amazon Redshift blog post: https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon-Redshift
21. Smart, responsive practice
Math and ELA (Grades 2-8)
Provides students the right
challenge at the right time
What is Waggle?
22. Right Challenge, Right Time
Waggle looks for more than
correct answers. Waggle
continually analyzes each
student’s decisions and
progress. That way, students get
tougher material right when
they’re ready.
What is Waggle?
23. Productive Struggle
Waggle motivates students to
push themselves forward. How?
Through helpful hints,
supportive feedback, and
achievement badges that build
grit and confidence.
What is Waggle?
24. Constructive Grouping Waggle’s
insights means you can easily
group students together based
on learning needs. All without
sacrificing the quality of
individual instruction.
What is Waggle?
25. Waggle: Product Demo
• Data Creators
Differentiated learning experience
Fun and engaging
• Data Visualizers
Seamless integration with application
Analytics with a Story
Actionable Data
26. Redshift: Data Warehouse Layout
Write Cluster
Compute – dw2.large
Redshift
Read Cluster
Compute – dw2.large
Redshift
History Cluster
Density – dw1.xlarge
Redshift
Initial and
Increment
al
{processed
} data
loads
Periodic Data
Snapshots for
historical analysis
Data
sources
For serving Jaspersoft
reports
APIs
OLTP
S3 COPY
S3 UnLoad
and Load
S3 UnLoad
and Load
Data mart
(aggregations)
NodesNodes
Staging
Datamart
(aggregations)
Nodes
S3 UnLoad and
Load
S3 UnLoad and
Load
+ UPSERTS
27. Results and Lessons Learned
• Performance Metrics
– Millions of records are processed in <1 minute
• LOAD/UNLOAD commands | UPSERTS | S3 COPY Command
– Report queries average < 1 to ~1.5 seconds
– {compression} – gained 20+% efficiencies in data retrieval
• Best Practices
– {sort keys} – lens-based data model: visualize data in variety of ways
– {commit stats} – Redshift is not a transactional system
– {nested loops} – no Cartesian products, ensure joins well managed
– {queries that queue} – tune the WLM configuration
– {query runtimes} – faster query means less queuing
– {stats missing} – analyze and vacuum when possible
– {alerts with tables} – monitor to ensure queries running optimally
31. • Transforming the culture of the College to be more data driven
• Moving from reporting silos to an Integrated Analytics system, we call
this a Data Democracy
• Collecting and analyzing a vast variety of data at a scale that no one
in Higher Ed is doing
• Using machine learning tools to identify students who may need
further assistance
• Starting this fall, we are implementing a one-on-one coaching
initiative for the students we identified with the machine learning tools
What We’re Doing
32. 96% of organizations in the United States
use data in the same way.
…and it’s wrong.
But it’s not just education…
35. Data Regimes
Data Dictatorship: Data is controlled and its use is restricted. There
is asymmetric distribution of information based on your position
Data Aristocracy: Data analysts, scientists and PhDs are needed to
do anything meaningful. Power concentrates in the hands of these
employees and their supervisors
Data Anarchy: Business users feel underserved and take matters into
their own hands. They create “shadow IT” systems and work around
the “unresponsive” IT group
Data Democracy: Everybody gets timely and equitable access to
data. Line of business users are empowered and “own” the data.
Executives and IT get out of the way
1 Shash Hegde, Mariner, “The Rise of Data Regimes”, 9/12/13, http://www.mariner-usa.com/rise-data-regimes/ (image substitution for Mao Zedong)
36. Every organization moves through
increasingly complex stages of data
accessibility.
Data Maturity Model
… very few complete the transition to
Integrated Analytics
37. Stage 1: Report Silos
Request
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
This is what we have had for
decades at Ivy Tech…
38. Request
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
Stage 2: Data Warehousing
This is what
most
companies
do…
but we are
taking this a
step further…
39. Stage 3: Integrated Analytics
Request
Tracker
Banner Blackboard Luminis StarfishSCCM CAS
Authentication
Students by
Financial
Aid
Students
by
Award
Students
by
Term
Students
by
Class
Classes by
Class
Section
Students
These curated collections of
data are designed to enable
direct access to...
…the data you need, regardless of
where it came from. Quickly.
Easily.
40. GPA Graduation—Cumulative
Graduation Grade Point Average (Cumulative) is an indication of a student's academic progress for all
semester credit classes for all registered terms up to and including the selected term. Letter grades are
assigned points (A=4, B=3, C=2, D=1, F=0) and the GPA is calculated by taking the number of grade
points a student earned in a selected period of time divided by the total number of classes taken during
that same period.
GPA Graduation Cumulative = Sum of a student's total grade points earned in credit classes for all
classes for all registered terms up to and including the selected term / Sum of student's total classes
taken during that same period
NOTES ON USING THIS TERM: GPA Graduation - Cumulative does not include grades from remedial
classes.
Related Terms: [GPA Graduation - Term]
In this session, we will start by looking at current trends in data-driven development, then look at how Redshift is evolving from a traditional data warehouse and expanding its capabilities to handle more sophisticated data analytics use cases. We will then discuss how to build advanced data driven applications with Amazon Machine Learning and use Amazon QuickSight for visualization. Finally, Radhika will take over and do a brief demo and then we will open up the session for Q&A. Just to set expectations, this is not a deep dive in any of the services. There are SKO sessions on service deep dives and roadmaps.
This is the AWS Big Data portfolio. We have tools like Direct Connect and Import Export that can bring in a lot of data. We can persist that data into a number of storage services from S3 to DynamoDB to EMR and RedShift for further analysis.
Amazon Redshift provides a fast, fully managed, petabyte-scale data warehouse for less than $1000 per terabyte per year. Amazon Elastic MapReduce provides a managed, easy to use analytics platform built around the powerful Hadoop framework. Recently we announced Amazon Kinesis, a managed service for real-time processing of streaming big data. Amazon Glacier allows you to backup and archive an unlimited amount of data at just 1 cent per GB per month. Automate and schedule big data processing workloads with Data Pipeline.
The tools to support big data collection, computation along with collaboration and sharing are all available in a couple of clicks, with AWS.
These personas are not hard and fast rules but a good grouping of categories of our users. We can expect a single individual could wear multiple hats. We also want to emphasize the flexibility that AWS provides to mat h the use case to a broad, rich set of analytic tools. Contrast this to existing solutions that require a large capital expense that locks in future questions to previous platform decisions
Thank you Greg. Let’s start by looking at the BI Analyst
Again, just keep in mind these personas are general characterizations and may have significant overlap—depending on whatever project you are working on.
What do BI Analysts do?
Just to level set, they spend most of their time analyzing data to figure out market and business trends for companies with a goal to increase profits and efficiency.
For example, they analyze past trends, and current conditions, and then communicate those trends using reports.
A typical use case here is analyzing sales data for a retail company to optimize profits.
For these types of use cases like historical analysis, Amazon Redshift is a great platform for such workloads.
A BI Analysts can connect to Redshift using three options:
They can install their existing BI Tools on EC2 and connect to Redshift
We recently announced Amazon QuickSight, a low cost BI tool that connects to all your data in the cloud including Redshift
We are working with our BI partners like Tableau, TIBCO Jaspersoft to integrate Amazon Quicksight and Greg will discuss QuickSight in more detail.
BI Analysts primarily use SQL to manipulate data. With Redshift, they can extend SQL with new Redshift SQL functions and add custom functions via user defined functions or (UDFs).
Lets look at how Redshift is evolving from a traditional data warehouse and expanding its capabilities to handle more sophisticated data analytics use cases.
Redshift has a massively parallel processing (MPP) data warehouse architecture. Redshift automatically distributes data and query load across all the compute nodes to take advantage of all available resources. This means that all queries are executed in parallel across all the physical resources. You can extend Native SQL with Python UDFs or custom libraries. Python’s compiled byte code also runs in parallel across all your compute nodes.
This horizontal scale out architecture makes it it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows.
Redshift is a MPP—horizontally scalable
Allows paralleization
Native SQL or extension with python
When the leader node receives a SQL query, it generates compiled C++ code. This compiled code is then broadcast to the compute nodes and executed in parallel.
Python and UDFs also get executed at the compute nodes for parallelization. Python is an interpreted language. The generated python byte code is also distributed across the compute nodes and executed in parallel just as a normal SQL query. Keep in mind that compiled code executes faster than interpreted code and uses less compute capacity. When benchmarking your queries, you should always compare the times for the second execution of a query, because the first execution time includes the overhead of compiling the code.
Python is a great language for data manipulation and analysis, but the programs are often a bottleneck when consuming data from large data warehouses. Python is an interpreted language, so for large data sets you may find yourself playing “tricks” with the language to scale across multiple processes in order to distribute the workload. In Amazon Redshift, the Python logic is pushed across the MPP system and all the scaling is handled by AWS. The Python execution in Amazon Redshift is done in parallel just as a normal SQL query, so Amazon Redshift will take advantage of all of the CPU cores in your cluster to execute your UDFs. They may not be as fast as native SQL, which is compiled to machine code, but the automated management of parallelism will alleviate many of the struggles associated with scaling the performance of your own Python code.
Python and UDFs gets executed at the compute nodes for parallelization
C++ code—does the same the same thing
Interpreted and has the ability—scalability to overcome interpretation
but otherwise the full python interpreter is available to you and it runs as compiled byte code in parallel on all the compute nodes.
When benchmarking your queries, you should always compare the times for the second execution of a query, because the first execution time includes the overhead of compiling the code.
Redshift is a columnar, multi-node, shared nothing cluster, database service
There’s a leader node that you connect with SQL clients
Data resides in the compute nodes and that’s where all the work happens
All the work happens in parallel directly through the compute nodes
Since this is advanced Redshift session, we will not focus on the details. There’s a Getting Started guide for Redshift and we will provide a link at the end of the webinar.
We’ve been iterating quickly and we add SQL functions regularly
This is because we want to expand the range of analytics you can do in SQL
We have added over 25 since launch - new aggregate and window functions
List Agg is quite popular with our customers - The idea here is that you can aggregate multiple values into a single string which is quite powerful
We also added the Approximate count function for very large datasets
REGEX capabilities and so on…
We are going to continue iterating here but we also want to enable you to add your own functionality to Redshift, and so we’ve added …
----
Scalar User Defined Functions.
UDFs provide more flexible querying and analysis, enabling a broader range of analytic and data science use cases.
You can write your custom functions in python 2.7
The syntax is very similar to what you can do with PostgreSQL UDF
We do sandbox the UDFs so you can’t make filesystem or network calls, but otherwise the full python interpreter is available to you and it runs as compiled byte code in parallel on all the compute nodes.
The python library in Redshift comes with Pandas, NumPy, Scipy for analytics, so those functions are available to you for use, but you can also bring in your own libraries.
You also have the flexibility to import your own libraries into Redshift
---
Time to first insight from months to minutes
Goes from GBs to Terabytes
Very fast response to queries
Cost down to 1/10th
Fast to get started—sign up and go
The title Data Scientiest is realatively and has been around for only a few years. It was actually coined in 2008 by data and analytics leads at LinkedIn and Facebook.
This title has come to replace data analyst.
A Data Scientist can have formal training in a quantitavie field but they can also emerge from any field that has a strong data and computational focus. A data scientists’ most basic, universal skill is the ability to write code, but they also have a solid foundation in math, statistics, probability, and computer science. They work with complex datasets and they are good at converting masses of unstructured data into structured data, but most importantly, getting it into a form that can be analyzed. Often they are creative in displaying information visually and making the patterns they find clear and compelling.
Data Scientists can use their existing toolsets to access Redshift. Let’s look at some toolsets that data scientists use to analyze data.
Similarly, you can install packages in R to query Redshift. For example, you can install the RJDBC package to load Redshift’s JDBC driver and then send SQL queries to Amazon Redshift.
The difference here between python UDFs and R in this scenario is that you have to take data out of the database to do your analysis.
You can also write R code using the dplyr package to analyze your data in Redshift.
First, you have to create a connection to Redshift via the RPostgreSQL package.
Once connect R to Redshift, you have the flexibility to add predictive modeling using Rs packages.
There’s a big data blog that shows you how to Connect R with Redshift in detail.
Now that we better understand the role of a Data Engineer, lets switch gears and discuss the Application Developer persona.
Yesterday, there was an article on The Wall Street Journal, and Hilary Mason, CEO and founder of Fast Forward Labs, which advises companies on data science and machine learning, observed that many many companies are now built on data and cannot exist without it. “Companies doing well … are building products using unique, proprietary data.
Data and analytics has become a new competency for developers. All developers are doing something with data in addition to their web, mobile or backend development efforts. Developers now treat instrumenting and collecting events as one of the most important parts of the development process. And because of this, applications are logging tremendous amounts of data and so developers are picking up the skills to analyze this data to derive some value from it and to build useful applications on top of the data the application generates.
Beyond historical analysis and knowing what is happening here, organizations are using the data they already have to make accurate, actionable predictions about what will happen in the future. They are building a new breed of smart applications using these predictions. As you can see, Machine learning is becoming an increasingly important tool to build advanced data-driven applications.
The primary emphasis is that these application developers are not experts in machine learning but want to extend their application with ML constructs through an API.
As airline passengers we always want to book flights when the likelihood of flight delays is minimum and with carriers that historically have the least amount of delays .
In this demo we analyze those two use case by using historical flight and weather data to develop predictive model for flight delays using Amazon Analytic services .
Before we take a look at a demo , let’s take a quick look at the architecture behind the solution.
We begin the exercise by downloading publicly accessible flight and weather data and staging it in a s3 bucket . The datasets are then copied into a redshift cluster .We then use ML to create and develop models using redshift as a Datasource. We deploy a sample training model and store predictions in s3 bucket . Then we run batch predictions on the full data set based on that model we just created and finally use Quicksight for visualization.
First step , We create a redshift cluster. This is the redshift cluster creation screen , once you create the cluster you would connect to it and load the raw data from a s3 bucket into it using the copy command.
Here is the schema that I used.
Here we have created a UDF in Redshift that calculates the numbers of days from a public holiday , This is important because we want to know the impact of holiday on the flight delays . We are using that in close_to_holiday column
You will also see an example of copy command that can be used to copy data from s3 into redshift.
Now let’s start with Machine learning . We start with creating a new Data source object and machine learning model using Redshift as a datasource. We need to fill in cluster detail ,as you see it identifies my demo cluster ,we then specify a IAM role , specify a staging location for ML model ,and issue a query against the redshift cluster and the table that contains my training data .
After validation we are presented the schema , you can allow Amazon Machine learning to infer the data types or, you can modify these if needed ,the next step is to select a target which is the attribute that ML is learning to predict which is dep_delay attribute for us and you can optionally select a row identifier to include with your prediction..
we review the setting and create a DS object.
Next we create a machine learning Model , here you see ML automatically detects this to be a binary classification model because the target attribute (flight_delay) is binary . It is attached to the right data source .We now review the settings and create a machine learning model.
you can also control the training process by setting parameters as model size , number of passes you want over your training data . In this case we take the default values and review settings and create the model.
When the model is ready , first thing you want to do is look at how it performed .
For binary classification model , the prediction accuracy is defined using an industry standard area under the curve score . As you can see we received a score of 0.76 which is considered good . The actual output of a ML algorithm is a prediction score ,lets take a deeper look into model performance .so a model such as this is bound to have errors , false positives and false negatives . We want to reduce the number of false positives because that means people are going to miss their flights .We can do that by simply moving the slider to a threshold where it makes sense for our business case.
for example if you look at target distribution report you will notice we have 13K rows corresponding to no delays and 8 k Reports corresponding to delay . Similarly you would also notice how the different attributes correlate t o your target, for example departure hour correlates to delay more than the day of the week. For numerical attributes , Machine learning provides information like mean median values , and neat histograms to understand how the data is distributed .
Once you are done with that , we are ready to create the batch predictions.To create batch predictions we first need the model we want ti deploy. We will use the model we just created. In the second step we need test data , I have already specified a test data here , and finally you want a place where your predictions are stored in this case S3 bucket . You review your setting and you create your batch predictions . When batch predictions are ready ,their output is in s3 bucket that you can download , uncompress and take a peek at it .
As you can see the results consists of a small number of columns including the prediction score given by ML and a best answer which is calculated by comparing the prediction score based on the cutoff value . Once we have the predictions ready to go , we can visualize them using Quicksight .
I copied the predictions back to redshift cluster , connected the quick sight to redshift cluster and created couple of visualizations.
steps are fairly straightforward .I am simply entering my cluster details here and I am on my way to create a connection were I have a graph where time Vs number of flights delayed and the We can see 6 pm is what you want to avoid .
With that we encourage you to create your own analysis and applications with Machine learning , Redshift and Quicksight and I turn it over to Greg to wrap things up.
Thank you !