Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Machine Learning | AWS Public Sector Summit 2016
•Download as PPTX, PDF•
11 likes•7,797 views
AWS has a large and growing portfolio of big data management and analytics services, designed to be integrated into solution architectures that meet the needs of your business. In this session, we look at analytics through the eyes of a business intelligence analyst, a data scientist, and an application developer, and we explore how to quickly leverage Amazon Redshift, Amazon QuickSight, RStudio, and Amazon Machine Learning to create powerful, yet straightforward, business solutions.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Machine Learning | AWS Public Sector Summit 2016
Similar to Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Machine Learning | AWS Public Sector Summit 2016 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Machine Learning | AWS Public Sector Summit 2016
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Greg Khairallah, Business Development Manager, AWS Malini Saxena, Senior Consultant, AWS Raj Chary, VP of Technology / Architecture, WagglePractice Lige Hensley, Chief Technology Officer, Ivy Tech June 20, 2016 Easy Analytics with AWS
- 2. What to expect from this session • AWS toolkit for analytics • Understand stakeholders • Demo • Case Study – WagglePractice • Case Study – Ivy Tech • Q&A
- 3. AnalyzeStore Amazon Glacier Amazon S3 Amazon DynamoDB Amazon RDS, Amazon Aurora Big data portfolio—but what do I recommend? AWS Data Pipeline Amazon CloudSearch Amazon EMR Amazon EC2 Amazon Redshift Amazon Machine Learning Amazon Elasticsearch Service AWS Database Migration Amazon Kinesis Analytics Amazon Kinesis Firehose AWS Import/Export AWS Direct Connect Collect Amazon Kinesis Streams Amazon QuickSight
- 4. Match toolset to right persona • Business intelligence (BI) analyst • Primary tool is SQL • Historical data resides in data warehouse such as Amazon Redshift • Data scientist—Uses programmatic languages such as R or Python • Application developer—Requires API to integrate with AWS services
- 5. BI analyst
- 6. BI analyst with existing BI tools BI Analyst BI tools Amazon EC2 Amazon Redshift QuickSight API • Primary tool is SQL • Data is largely structured with well known data sources • Primary concern is fast, consistent performance • Need to extend SQL with custom functions BI tools Amazon EC2 Amazon QuickSight Amazon QuickSight
- 7. Amazon Redshift system architecture Leader node • SQL endpoint • Stores metadata • Coordinates query execution Compute nodes • Local, columnar storage • Execute queries in parallel • Load, backup, restore via Amazon S3; load from Amazon DynamoDB, Amazon EMR, or SSH Two hardware platforms • Optimized for data processing • DS2: HDD; scale from 2 TB to 2 PB • DC1: SSD; scale from 160 GB to 356 TB 10 GigE (HPC) JDBC/ODBC
- 8. New SQL functions We add SQL functions regularly to expand Amazon Redshift’s query capabilities Added 25+ window and aggregate functions since launch, including: LISTAGG [APPROXIMATE] COUNT DROP IF EXISTS, CREATE IF NOT EXISTS REGEXP_SUBSTR, _COUNT, _INSTR, _REPLACE PERCENTILE_CONT, _DISC, MEDIAN PERCENT_RANK, RATIO_TO_REPORT We’ll continue iterating but also want to enable you to write your own Window function examples: http://docs.aws.amazon.com/redshift/latest/dg/r_Window_function_examples.html
- 9. Scalar user defined functions You can write UDFs using Python 2.7 • Syntax is largely identical to PostgreSQL UDF • Python execution is performed in parallel • System and network calls within UDFs are prohibited Comes integrated with Pandas, NumPy, SciPy, DateUtil, and Pytz analytic libraries • Import your own libraries for even more flexibility • Take advantage of thousands of functions available through Python libraries to perform operations not easily expressed in SQL
- 10. A very fast, cloud-powered, business intelligence service for 1/10 the cost of traditional BI software What is Amazon QuickSight?
- 11. Business User Business User QuickSight APIQuickSight UI Mobile Devices Web Browsers Partner BI Products MetadataData PrepConnectors SuggestionsSPICE Amazon S3 Amazon Kinesis Amazon DynamoDB Amazon EMRAmazon Redshift Amazon RDSFiles Third-party
- 12. Data scientist
- 13. Data scientist with existing toolsets Data scientist Toolkits like SAS or R Studio installed with Amazon EC2 Unstructured data Amazon S3 Structured data Amazon Redshift • Work with unstructured datasets • Use existing toolsets to connect to Amazon Redshift
- 14. Querying Amazon Redshift with R packages • RJDBC—Supports SQL queries • dplyr—Uses R code for data analysis • RPostgreSQL—R compliant driver or database Interface (DBI)R User R Studio Amazon EC2 Unstructured data Amazon S3 User profile Amazon RDS Amazon Redshift Connecting R with Amazon Redshift blog post: https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon-Redshift
- 15. Querying Amazon Redshift with R packages example
- 17. Application developers can build smart applications using Amazon Machine Learning Structured data/predictions Amazon Redshift Generate/query predictions Amazon QuickSight Application Amazon Machine Learning Visualize • All skill levels • Amazon Machine Learning technology is accessed through APIs and SDKs • Embed visualizations in applications
- 18. Demo
- 20. Raj Chary, WagglePractice Vice President of Technology/Architecture
- 21. Smart, responsive practice Math and ELA (Grades 2-8) Provides students the right challenge at the right time What is Waggle?
- 22. Right Challenge, Right Time Waggle looks for more than correct answers. Waggle continually analyzes each student’s decisions and progress. That way, students get tougher material right when they’re ready. What is Waggle?
- 23. Productive Struggle Waggle motivates students to push themselves forward. How? Through helpful hints, supportive feedback, and achievement badges that build grit and confidence. What is Waggle?
- 24. Constructive Grouping Waggle’s insights means you can easily group students together based on learning needs. All without sacrificing the quality of individual instruction. What is Waggle?
- 25. Waggle: Product Demo • Data Creators Differentiated learning experience Fun and engaging • Data Visualizers Seamless integration with application Analytics with a Story Actionable Data
- 26. Redshift: Data Warehouse Layout Write Cluster Compute – dw2.large Redshift Read Cluster Compute – dw2.large Redshift History Cluster Density – dw1.xlarge Redshift Initial and Increment al {processed } data loads Periodic Data Snapshots for historical analysis Data sources For serving Jaspersoft reports APIs OLTP S3 COPY S3 UnLoad and Load S3 UnLoad and Load Data mart (aggregations) NodesNodes Staging Datamart (aggregations) Nodes S3 UnLoad and Load S3 UnLoad and Load + UPSERTS
- 27. Results and Lessons Learned • Performance Metrics – Millions of records are processed in <1 minute • LOAD/UNLOAD commands | UPSERTS | S3 COPY Command – Report queries average < 1 to ~1.5 seconds – {compression} – gained 20+% efficiencies in data retrieval • Best Practices – {sort keys} – lens-based data model: visualize data in variety of ways – {commit stats} – Redshift is not a transactional system – {nested loops} – no Cartesian products, ensure joins well managed – {queries that queue} – tune the WLM configuration – {query runtimes} – faster query means less queuing – {stats missing} – analyze and vacuum when possible – {alerts with tables} – monitor to ensure queries running optimally
- 29. Thank You
- 30. Ivy Tech & Amazon Redshift May 25, 2016
- 31. • Transforming the culture of the College to be more data driven • Moving from reporting silos to an Integrated Analytics system, we call this a Data Democracy • Collecting and analyzing a vast variety of data at a scale that no one in Higher Ed is doing • Using machine learning tools to identify students who may need further assistance • Starting this fall, we are implementing a one-on-one coaching initiative for the students we identified with the machine learning tools What We’re Doing
- 32. 96% of organizations in the United States use data in the same way. …and it’s wrong. But it’s not just education…
- 34. Relevant Data for Everyone
- 35. Data Regimes Data Dictatorship: Data is controlled and its use is restricted. There is asymmetric distribution of information based on your position Data Aristocracy: Data analysts, scientists and PhDs are needed to do anything meaningful. Power concentrates in the hands of these employees and their supervisors Data Anarchy: Business users feel underserved and take matters into their own hands. They create “shadow IT” systems and work around the “unresponsive” IT group Data Democracy: Everybody gets timely and equitable access to data. Line of business users are empowered and “own” the data. Executives and IT get out of the way 1 Shash Hegde, Mariner, “The Rise of Data Regimes”, 9/12/13, http://www.mariner-usa.com/rise-data-regimes/ (image substitution for Mao Zedong)
- 36. Every organization moves through increasingly complex stages of data accessibility. Data Maturity Model … very few complete the transition to Integrated Analytics
- 37. Stage 1: Report Silos Request Tracker Banner Blackboard Luminis StarfishSCCM CAS Authentication This is what we have had for decades at Ivy Tech…
- 38. Request Tracker Banner Blackboard Luminis StarfishSCCM CAS Authentication Stage 2: Data Warehousing This is what most companies do… but we are taking this a step further…
- 39. Stage 3: Integrated Analytics Request Tracker Banner Blackboard Luminis StarfishSCCM CAS Authentication Students by Financial Aid Students by Award Students by Term Students by Class Classes by Class Section Students These curated collections of data are designed to enable direct access to... …the data you need, regardless of where it came from. Quickly. Easily.
- 40. GPA Graduation—Cumulative Graduation Grade Point Average (Cumulative) is an indication of a student's academic progress for all semester credit classes for all registered terms up to and including the selected term. Letter grades are assigned points (A=4, B=3, C=2, D=1, F=0) and the GPA is calculated by taking the number of grade points a student earned in a selected period of time divided by the total number of classes taken during that same period. GPA Graduation Cumulative = Sum of a student's total grade points earned in credit classes for all classes for all registered terms up to and including the selected term / Sum of student's total classes taken during that same period NOTES ON USING THIS TERM: GPA Graduation - Cumulative does not include grades from remedial classes. Related Terms: [GPA Graduation - Term]
- 41. Questions?
- 42. Resources Amazon Redshift Getting Started Guide: http://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html Scalar UDF Documentation: http://docs.aws.amazon.com/redshift/latest/dg/user-defined- functions.html Introduction to Python UDFs in Amazon Redshift: https://blogs.aws.amazon.com/bigdata/post/Tx1IHV1G67CY53T/Introduction-to-Python-UDFs-in- Amazon-Redshift Connecting R with Amazon Redshift: https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon- Redshift Databricks Apache Spark–Amazon Redshift Tutorial: https://github.com/databricks/spark- redshift/tree/master/tutorial Amazon ML Getting Started Guide: https://aws.amazon.com/machine-learning/getting-started/ Amazon QuickSight (Preview Registration): https://aws.amazon.com/quicksight/
- 43. Thank you!
Editor's Notes
- In this session, we will start by looking at current trends in data-driven development, then look at how Redshift is evolving from a traditional data warehouse and expanding its capabilities to handle more sophisticated data analytics use cases. We will then discuss how to build advanced data driven applications with Amazon Machine Learning and use Amazon QuickSight for visualization. Finally, Radhika will take over and do a brief demo and then we will open up the session for Q&A. Just to set expectations, this is not a deep dive in any of the services. There are SKO sessions on service deep dives and roadmaps.
- This is the AWS Big Data portfolio. We have tools like Direct Connect and Import Export that can bring in a lot of data. We can persist that data into a number of storage services from S3 to DynamoDB to EMR and RedShift for further analysis. Amazon Redshift provides a fast, fully managed, petabyte-scale data warehouse for less than $1000 per terabyte per year. Amazon Elastic MapReduce provides a managed, easy to use analytics platform built around the powerful Hadoop framework. Recently we announced Amazon Kinesis, a managed service for real-time processing of streaming big data. Amazon Glacier allows you to backup and archive an unlimited amount of data at just 1 cent per GB per month. Automate and schedule big data processing workloads with Data Pipeline. The tools to support big data collection, computation along with collaboration and sharing are all available in a couple of clicks, with AWS.
- These personas are not hard and fast rules but a good grouping of categories of our users. We can expect a single individual could wear multiple hats. We also want to emphasize the flexibility that AWS provides to mat h the use case to a broad, rich set of analytic tools. Contrast this to existing solutions that require a large capital expense that locks in future questions to previous platform decisions
- Thank you Greg. Let’s start by looking at the BI Analyst Again, just keep in mind these personas are general characterizations and may have significant overlap—depending on whatever project you are working on.
- What do BI Analysts do? Just to level set, they spend most of their time analyzing data to figure out market and business trends for companies with a goal to increase profits and efficiency. For example, they analyze past trends, and current conditions, and then communicate those trends using reports. A typical use case here is analyzing sales data for a retail company to optimize profits. For these types of use cases like historical analysis, Amazon Redshift is a great platform for such workloads. A BI Analysts can connect to Redshift using three options: They can install their existing BI Tools on EC2 and connect to Redshift We recently announced Amazon QuickSight, a low cost BI tool that connects to all your data in the cloud including Redshift We are working with our BI partners like Tableau, TIBCO Jaspersoft to integrate Amazon Quicksight and Greg will discuss QuickSight in more detail. BI Analysts primarily use SQL to manipulate data. With Redshift, they can extend SQL with new Redshift SQL functions and add custom functions via user defined functions or (UDFs). Lets look at how Redshift is evolving from a traditional data warehouse and expanding its capabilities to handle more sophisticated data analytics use cases.
- Redshift has a massively parallel processing (MPP) data warehouse architecture. Redshift automatically distributes data and query load across all the compute nodes to take advantage of all available resources. This means that all queries are executed in parallel across all the physical resources. You can extend Native SQL with Python UDFs or custom libraries. Python’s compiled byte code also runs in parallel across all your compute nodes. This horizontal scale out architecture makes it it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows. Redshift is a MPP—horizontally scalable Allows paralleization Native SQL or extension with python When the leader node receives a SQL query, it generates compiled C++ code. This compiled code is then broadcast to the compute nodes and executed in parallel. Python and UDFs also get executed at the compute nodes for parallelization. Python is an interpreted language. The generated python byte code is also distributed across the compute nodes and executed in parallel just as a normal SQL query. Keep in mind that compiled code executes faster than interpreted code and uses less compute capacity. When benchmarking your queries, you should always compare the times for the second execution of a query, because the first execution time includes the overhead of compiling the code. Python is a great language for data manipulation and analysis, but the programs are often a bottleneck when consuming data from large data warehouses. Python is an interpreted language, so for large data sets you may find yourself playing “tricks” with the language to scale across multiple processes in order to distribute the workload. In Amazon Redshift, the Python logic is pushed across the MPP system and all the scaling is handled by AWS. The Python execution in Amazon Redshift is done in parallel just as a normal SQL query, so Amazon Redshift will take advantage of all of the CPU cores in your cluster to execute your UDFs. They may not be as fast as native SQL, which is compiled to machine code, but the automated management of parallelism will alleviate many of the struggles associated with scaling the performance of your own Python code. Python and UDFs gets executed at the compute nodes for parallelization C++ code—does the same the same thing Interpreted and has the ability—scalability to overcome interpretation but otherwise the full python interpreter is available to you and it runs as compiled byte code in parallel on all the compute nodes. When benchmarking your queries, you should always compare the times for the second execution of a query, because the first execution time includes the overhead of compiling the code. Redshift is a columnar, multi-node, shared nothing cluster, database service There’s a leader node that you connect with SQL clients Data resides in the compute nodes and that’s where all the work happens All the work happens in parallel directly through the compute nodes Since this is advanced Redshift session, we will not focus on the details. There’s a Getting Started guide for Redshift and we will provide a link at the end of the webinar.
- We’ve been iterating quickly and we add SQL functions regularly This is because we want to expand the range of analytics you can do in SQL We have added over 25 since launch - new aggregate and window functions List Agg is quite popular with our customers - The idea here is that you can aggregate multiple values into a single string which is quite powerful We also added the Approximate count function for very large datasets REGEX capabilities and so on… We are going to continue iterating here but we also want to enable you to add your own functionality to Redshift, and so we’ve added … ----
- Scalar User Defined Functions. UDFs provide more flexible querying and analysis, enabling a broader range of analytic and data science use cases. You can write your custom functions in python 2.7 The syntax is very similar to what you can do with PostgreSQL UDF We do sandbox the UDFs so you can’t make filesystem or network calls, but otherwise the full python interpreter is available to you and it runs as compiled byte code in parallel on all the compute nodes. The python library in Redshift comes with Pandas, NumPy, Scipy for analytics, so those functions are available to you for use, but you can also bring in your own libraries. You also have the flexibility to import your own libraries into Redshift ---
- Time to first insight from months to minutes Goes from GBs to Terabytes Very fast response to queries Cost down to 1/10th Fast to get started—sign up and go
- The title Data Scientiest is realatively and has been around for only a few years. It was actually coined in 2008 by data and analytics leads at LinkedIn and Facebook. This title has come to replace data analyst.
- A Data Scientist can have formal training in a quantitavie field but they can also emerge from any field that has a strong data and computational focus. A data scientists’ most basic, universal skill is the ability to write code, but they also have a solid foundation in math, statistics, probability, and computer science. They work with complex datasets and they are good at converting masses of unstructured data into structured data, but most importantly, getting it into a form that can be analyzed. Often they are creative in displaying information visually and making the patterns they find clear and compelling. Data Scientists can use their existing toolsets to access Redshift. Let’s look at some toolsets that data scientists use to analyze data.
- Similarly, you can install packages in R to query Redshift. For example, you can install the RJDBC package to load Redshift’s JDBC driver and then send SQL queries to Amazon Redshift. The difference here between python UDFs and R in this scenario is that you have to take data out of the database to do your analysis. You can also write R code using the dplyr package to analyze your data in Redshift. First, you have to create a connection to Redshift via the RPostgreSQL package. Once connect R to Redshift, you have the flexibility to add predictive modeling using Rs packages. There’s a big data blog that shows you how to Connect R with Redshift in detail.
- Now that we better understand the role of a Data Engineer, lets switch gears and discuss the Application Developer persona.
- Yesterday, there was an article on The Wall Street Journal, and Hilary Mason, CEO and founder of Fast Forward Labs, which advises companies on data science and machine learning, observed that many many companies are now built on data and cannot exist without it. “Companies doing well … are building products using unique, proprietary data. Data and analytics has become a new competency for developers. All developers are doing something with data in addition to their web, mobile or backend development efforts. Developers now treat instrumenting and collecting events as one of the most important parts of the development process. And because of this, applications are logging tremendous amounts of data and so developers are picking up the skills to analyze this data to derive some value from it and to build useful applications on top of the data the application generates. Beyond historical analysis and knowing what is happening here, organizations are using the data they already have to make accurate, actionable predictions about what will happen in the future. They are building a new breed of smart applications using these predictions. As you can see, Machine learning is becoming an increasingly important tool to build advanced data-driven applications. The primary emphasis is that these application developers are not experts in machine learning but want to extend their application with ML constructs through an API.
- As airline passengers we always want to book flights when the likelihood of flight delays is minimum and with carriers that historically have the least amount of delays . In this demo we analyze those two use case by using historical flight and weather data to develop predictive model for flight delays using Amazon Analytic services . Before we take a look at a demo , let’s take a quick look at the architecture behind the solution. We begin the exercise by downloading publicly accessible flight and weather data and staging it in a s3 bucket . The datasets are then copied into a redshift cluster .We then use ML to create and develop models using redshift as a Datasource. We deploy a sample training model and store predictions in s3 bucket . Then we run batch predictions on the full data set based on that model we just created and finally use Quicksight for visualization. First step , We create a redshift cluster. This is the redshift cluster creation screen , once you create the cluster you would connect to it and load the raw data from a s3 bucket into it using the copy command. Here is the schema that I used. Here we have created a UDF in Redshift that calculates the numbers of days from a public holiday , This is important because we want to know the impact of holiday on the flight delays . We are using that in close_to_holiday column You will also see an example of copy command that can be used to copy data from s3 into redshift. Now let’s start with Machine learning . We start with creating a new Data source object and machine learning model using Redshift as a datasource. We need to fill in cluster detail ,as you see it identifies my demo cluster ,we then specify a IAM role , specify a staging location for ML model ,and issue a query against the redshift cluster and the table that contains my training data . After validation we are presented the schema , you can allow Amazon Machine learning to infer the data types or, you can modify these if needed ,the next step is to select a target which is the attribute that ML is learning to predict which is dep_delay attribute for us and you can optionally select a row identifier to include with your prediction.. we review the setting and create a DS object. Next we create a machine learning Model , here you see ML automatically detects this to be a binary classification model because the target attribute (flight_delay) is binary . It is attached to the right data source .We now review the settings and create a machine learning model. you can also control the training process by setting parameters as model size , number of passes you want over your training data . In this case we take the default values and review settings and create the model. When the model is ready , first thing you want to do is look at how it performed . For binary classification model , the prediction accuracy is defined using an industry standard area under the curve score . As you can see we received a score of 0.76 which is considered good . The actual output of a ML algorithm is a prediction score ,lets take a deeper look into model performance .so a model such as this is bound to have errors , false positives and false negatives . We want to reduce the number of false positives because that means people are going to miss their flights .We can do that by simply moving the slider to a threshold where it makes sense for our business case. for example if you look at target distribution report you will notice we have 13K rows corresponding to no delays and 8 k Reports corresponding to delay . Similarly you would also notice how the different attributes correlate t o your target, for example departure hour correlates to delay more than the day of the week. For numerical attributes , Machine learning provides information like mean median values , and neat histograms to understand how the data is distributed . Once you are done with that , we are ready to create the batch predictions.To create batch predictions we first need the model we want ti deploy. We will use the model we just created. In the second step we need test data , I have already specified a test data here , and finally you want a place where your predictions are stored in this case S3 bucket . You review your setting and you create your batch predictions . When batch predictions are ready ,their output is in s3 bucket that you can download , uncompress and take a peek at it . As you can see the results consists of a small number of columns including the prediction score given by ML and a best answer which is calculated by comparing the prediction score based on the cutoff value . Once we have the predictions ready to go , we can visualize them using Quicksight . I copied the predictions back to redshift cluster , connected the quick sight to redshift cluster and created couple of visualizations. steps are fairly straightforward .I am simply entering my cluster details here and I am on my way to create a connection were I have a graph where time Vs number of flights delayed and the We can see 6 pm is what you want to avoid . With that we encourage you to create your own analysis and applications with Machine learning , Redshift and Quicksight and I turn it over to Greg to wrap things up. Thank you !