The document discusses using Amazon Web Services (AWS) and Qlik to analyze and visualize raw data in the cloud. It describes how AWS provides various services for collecting, storing, processing, and analyzing data at massive scales. Qlik can then be used on AWS to build interactive business intelligence applications that allow users to explore and gain insights from large datasets. The demo shows how raw data can be automatically loaded into AWS services like S3, Redshift, and EMR and then used to trigger a reload of Qlik applications, providing up-to-date insights in near real-time.
3. 3#qonnections
Rahul Bhartia
• Ecosystem Solutions Architect, AWS
• Data and analytics applications for the Cloud
• Over 10 years in architecture of data processing
systems
Presenters
John Park
• Senior Solution Architect, Partner Engineering
• ETL, Data warehousing, Software design, *NIX
systems and architecture
• 2 years Qlik, 7 years DW Consultant
• Twitter: @jpark328
5. 5#qonnections
What can the cloud do for us ?
Collect
• EventsTransactions
• Files and Logs
Store
• Object Store
• Databases
• NoSQL
Capabilities of Qlik with Amazon Web Services (AWS)
Process
• Hadoop and data integration tools (ETL)
Think of the cloud as ‘The Matrix’ and you have infinite amount of cheap resources to create
your Business Intelligence environment
Analyze
• Query , Visualize and Collaborate
7. Why AWS?
Building and managing cloud since 2006
40+ services to support any cloud workload
History of rapid, customer-driven releases
11 regions, 29 availability zones, 53 edge locations
48 proactive price reductions to date
Thousands of SIs and ISVs; 2,100+ Marketplace listings
Experience
Service Breadth & Depth
Pace of Innovation
Global Footprint
Pricing Philosophy
Ecosystem
8. Administration
& Security
Access
Control
Identity
Management
Key Management
& Storage
Monitoring
& Logs
Resource &
Usage Auditing
Platform
Services
Analytics App Services Developer Tools & Operations Mobile Services
Data
Pipelines
Data
Warehouse
Hadoop
Real-time
Streaming Data
Application
Lifecycle
Management
Container
s
Deployment
DevOps
Event-driven
Computing
Resource
Templates
Identity
Mobile
Analytics
Push
Notifications
Sync
App
Streaming
Email
Queuing &
Notifications
Search
Transcoding
Workflow
Core
Services
CDN
Compute
(VMs, Auto-scaling, and
Load Balancing)
Databases
(Relational, NoSQL, and
Caching)
Networking
(VPC, DX, and DNS)
Storage
(Object, Block, EFS,
and Archival)
Infrastructure
Availability
Zones
Points of
PresenceRegions
Enterprise
Applications
Business
Email
Sharing &
Collaboration
Virtual
Desktop
Technical &
Business Support
Account
Management
Partner
Ecosystem
Professional
Services
Security &
Pricing Reports
Solutions
Architect
s
Support
Training &
Certification
Machine
Learning
AWS - cloud computing services
9. Culture of innovation
On-premises
Infrequent and long
experimentations
High risk
Lower innovation
Frequent and short
experimentations
Low risk
Higher innovation
Innovate faster by experimenting often with lower risks
10. “Unstructured data growth
explosive, with estimates
of compound annual
growth (CAGR) at 62%
from 2008 – 2012”
Source: IDC
GB
TB
PB
ZB
EB
Unconstrained data growth
11. Not only about growth…
Data
Velocity
Variety
Volume
Structured, Unstructured, Text, Binary
Gigabytes, Terabytes, Petabytes
Millisecond, Second, Minute, Hour, Day
23. 23#qonnections
Qlik®
with AWS - Andersen Corporation
Challenges
• With plans to transform the structure of its sales team,
Andersen Corporation needed a streamlined way to
realign, measure and track the success of its sales
team. Additionally, the team would have limited IT
bandwidth to support the project and just 90 days to
deploy.
Solution
• Andersen Corporation deployed QlikView in the cloud
with the help of Qlik Consulting to support the
restructure of its sales team. The tool is now at the
heart of all sales decisions.
Benefit
• Streamlined view of all sales data in one platform aligned
with new sales structure
• Increased efficiency with sales targeting based on
geography
• Created a self-service environment for sales team allowing
direct access to personal performance and sales pipeline.
• ROI in 90 Days
“With only three months to deploy and limited
bandwidth from our IT team, we went into this project
knowing we would need to rely heavily on the vendor.
Qlik Consulting was able to quickly understand the
idiosyncrasies of our business, digest the complexity
of the project and lay the proper foundation needed for
our team’s success”
Director of Business Analytics, Andersen
Corporation
• Andersen Corporations is the largest window
manufacturer in North America
• Employees 9,000 people
• 100+ Year company serving North America, South
America, Europe, Asia and Middle East.
24. 24#qonnections
QlikView - Business discovery on AWS
• Cloud-based associative business discovery
for Big Data
• Massive data scalability with Amazon EC2
• Rapid Big Data app development with
Amazon Redshift
• Seamlessly combines Big Data with other
data sources
• Easy to implement no upfront cost
Amazon EMR
Hadoop
Cluster
Amazon RDS
Operational
Systems
Amazon Redshift
Data
Warehouse
25. 25#qonnections
PresentationApplication
Windows Based
File Share
(Optional)
Third-Party Integration:
• Informatica
• Talend
DataAccess
Windows IIS
QlikView - Business Discovery Platform on AWS
Business Discovery Apps
Sales • Finance • Marketing • Operations
QlikView
Web Server
Business Users
IT Admins
QlikView
Management Console
QlikView Server
QlikView Publisher
Data Sources
Custom connectors; ODBC;
OLEDB; QVX; XML
QVW; QVD files
Data/Business
Analysts
Developers
QlikView
Developer
Security Integration:
• IAM
• AD Server Replication
Elastic IP
Instances
Amazon EC2 Amazon EC2
Amazon S3 MySQL DB
Instance
Oracle DB
Instance
MS SQL
Instance Amazon Redshift
AWS Management
Console
27. 27#qonnections
Take raw data and perform ETL and reload automatically in Qlik
• Use AWS CloudFormation to create our stack*
• Raw data into Amazon S3(manually)
• Use Amazon Data Pipeline as an orchestration engine
– Amazon EMR performs analysis with data on Amazon S3
– Load detail data into Amazon Redshift
– Amazon EMR summarizes data and place it in Amazon S3
– Trigger a QlikView re-load
• Automatically see the results on QlikView Access Point.
Our demo today
29. Architecture for Demo
Availability Zone us-east1a
Subnet 1 Subnet 2
Internet Gateway
User
Amazon VPC
Qlik
Server
Qlik
Server
Bastion
Amazon
Redshift
Amazon
S3
QlikView Server- Amazon
EC2*
31. 31#qonnections
Security
• AWS Identity and Access Management (IAM)
• Amazon VPC – subnets, security groups and network access
control list and
• Amazon S3 bucket policies and support for encryption
Performance
• Placement groups for low-latency, 10 Gbps network
• Enhanced networking for high packet-per-second performance
Cost Savings
• Use Amazon Elastic Block Store (Amazon EBS) and stop or
terminate Amazon EC2 instances
• Elasticity with Auto Scaling
• Amazon EC2 Spot Instances and Reserved Instances
Repeatability
• Script everything you can and automate
Considerations
32. 32#qonnections
Qlik
• All collateral from this session will be posted in
branch – http://branch.qlik.com/, GitHub, Slideshare
Qlik community - https://community.qlik.com/welcome
Slides will be posted on community and code will be posted on
branch.
AWS
• Test Drive - http://www.ipc-global.com/login-page-for-directit-test-
drive-labs/
• AWS Summits - http://aws.amazon.com/summits/
• AWS Documentations - http://aws.amazon.com/documentation/
• AWS Boot camp at Summits
Where can I learn more about Qlik and AWS
33. 33#qonnections
Combine AWS and Qlik for maximum benefits
• Provide your customers with production grade Qlik
environment(Use Stacks)
• Go from raw data to analysis in days not months with AWS
• An affordable flexible option that also scales with your customer
needs
– No hardware or software to buy to install Qlik
– Get started quickly
– Quick ROI for customers
Summary and key takeaways
35. 35#qonnections
Feedback survey
• Please complete the track session survey via the mobile app
• Access the track session survey through the mobile app
• Enter track session code T63
• Provide your feedback
TALKING POINTS
Customers have selected AWS for eight years because we have proven ourselves committed to customer success.
We believe we stand apart in the market because of six factors: Experience, Service Breadth and Depth, Pace of Innovation, Global Footprint, Pricing Philosophy, and Partner Ecosystem
AWS has developed the broadest collection of services available from any cloud provider.
Our approach to regions, availability zones, and POPs provides global coverage for high availability, low latency applications.
Foundation services across compute, storage, security, and networking offer customers flexibility in their architecture. We have a full spectrum of options to meet most price-to-performance scenarios.
We offer the capability for both managed and unmanaged database options.
The offerings for Analytics and Application Services enable advanced data processing and workloads.
AWS Redshift, our cloud-based data warehouse, is the fastest growing service in the history of AWS.
Our management tools offer a lot of insight and flexibility to let you manage your AWS resources through either our tools or the management tools you’re already familiar with.
Recent expansion into enterprise applications has been entirely driven by customer feedback on where they’d like us to deliver value.
We see our customers do amazing things when they reduce the cost of experimentation- it moves IT from being a roadblock, where each idea costs lots of money and takes lots of time, to being an enabler where you can launch a speculative project quickly and cheaply. It allows firms to take more chances on ideas, and gives them a shot at winning big, as opposed to being scared to even try.
Due to the convergence of many technologies of cloud, mobile, social, and advancements in many field such as genomics, life sciences, space, the size of the digital universe is growing at an ever increasing rate.
Customers have also found tremendous value in being able to mine this data to make better medicine, tailored purchasing recommendations, detect fraudulent financial transactions in real time, provide on-demand digital content such as movies and songs, predict weather forecasts, the list goes on and on.
However, we don’t believe that there is one tool that can do everything, but rather if you use the right tools, you can build a highly configurable big data architecture to meet your specific needs.
While I won’t be able to go over all of our big data services, I would like to spend some time introducing to you several key big data services that are designed for high availability and durability,
as a managed service where we provision the infrastructure on your behalf
where you can get significant big data storage and analytics with a few clicks or api calls.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Fundamental storage at internet scale, it can store any number of objects from 1 byte to 5 TB in size
It is engineered for 11 9’s of durability replicating your data at least three times in three distinct physical data centers we call availability zones
We have customers such as Dropbox, Spotify, Pinterest store billions of objects or files as photos, videos, songs, or any other type of file.
Elastic capability and on demand nature of Cloud with Qlikview.
High Level AWS EC2 Redshift
Redshift Being an WH Construct
Direct Connect
What is Big Data ?
What is Hadoop ?
Totally private web application with no public access to any AWS resources
Must be on the VPN to manage the resources or use the application