Journey to Cloud Analytics
•Download as PPTX, PDF•
0 likes•567 views
This presentation will discuss the stories of 3 companies that span different industries; what challenges they faced and how cloud analytics solved for them; what technologies were implemented to solve the challenges; and how they were able to benefit from their new cloud analytics environments. The objectives of this session include: • Detail and explain the key benefits and advantages of moving BI and analytics workloads to the cloud, and why companies shouldn’t wait any longer to make their move. • Compare the different analytics cloud options companies have, and the pros and cons of each. • Describe some of the challenges companies may face when moving their analytics to the cloud, and what they need to prepare for. • Provide the case studies of three companies, what issues they were solving for, what technologies they implemented and why, and how they benefited from their new solutions. • Learn what to look for one considering a partner and trusted advisor to assist with an analytics cloud migration.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Journey to Cloud Analytics
Similar to Journey to Cloud Analytics (20)
More from Datavail
More from Datavail (20)
Recently uploaded
Recently uploaded (20)
Journey to Cloud Analytics
- 1. Journey to Cloud Analytics How 3 Companies’ Analytics Challenges Were Solved by Moving to the Cloud Richmond Virtual CIO and IT Security Forum | May 25, 2021
- 2. T Tom Hoblitzell Datavail VP, Data Management Passion for solving complex global business challenges through advanced technology and leading-edge digital business intelligence A forward-thinking strategist to drive outstanding business results Built and successfully grew analytics practices at major systems integration firms leading growth from start-up to mature practice with over $60 MM in revenue Sold Practices as part of strategic acquisitions (Fujitsu, Capgemini) Acquired and integrated IT acquisitions into existing practices to increase growth and round out capabilities and competencies Acts as a strategic advisor to key clients enabling growth of analytics and digital transformation initiatives to leverage data as a strategic asset Over 30 years of experience www.datavail.com 2
- 3. Fill Out Our Cloud Analytics Survey Fill out the Cloud Analytics survey for a chance to WIN a Sonos Move, a battery-powered Smart Speaker! https://www.datavail.com/cloud-analytics-survey
- 4. A Our Agenda Simply migrating to the cloud with a “lift and shift”' approach does not result in innovation, but it does add another level of complexity to operations. - Gartner Why cloud for analytics? New capabilities in the cloud versus on prem Challenges moving analytics to the cloud Maturity matters Datavail’s approach Players in the Cloud Analytics Space Case Studies Two major players to watch • Snowflake • RedShift www.datavail.com 4
- 5. www.datavail.com 5 “IDC Forecasts Revenues for Big Data and Business Analytics Solutions Will Reach $189.1 Billion This Year(2019) with Double-Digit Annual Growth Through 2022 (to $274.3 billion)” Companies are investing in Cloud Analytics 0 50 100 150 200 250 300 350 2019 2020 2021 2022 Billions Year Cloud Analytics Growth: 44% of Market in 2022 Cloud Analytics=32% Annual Growth! On-Prem=13% Annual Growth
- 6. www.datavail.com 6 Why Cloud for Analytics? The Cloud as a Driver of Analytics Innovation Analytics Innovation Brainstorming Ideation Visibility Constant Refinement Design Thinking Incubation Focus Source: Gartner
- 7. A Cloud Analytics What Is It? “Analytics is the process of gathering, cleansing, transforming, and modeling data with the goal of discovering useful information to support decision making.” Source: Quantzig The goal of Analytics is to make data accessible, useful, and actionable, which leads to digital transformation. Cloud Analytics uses modern cloud technologies and approaches to achieve the goal with lower costs, faster scalability, and agile implementation. www.datavail.com 7
- 8. www.datavail.com 8 Analytics: New Capabilities in the Cloud vs on Prem – Help Generate and Manage innovation Source: Gartner On-premises Cloud Attracting users with emerging capabilities More prototypes with greater visibility Metadata powered collaboration Focus on high- quality analytics Desktop or web- based training Prototype with limited audience Discussion Tired of repetitive basic analytics Onboarding Prototype Pilot Production Ideation Design Incubation Focus Sandbox Constant refinement Elasticity/automation
- 9. www.datavail.com 9 Business needs self-service data exploration and discovery- oriented forms of advanced analytics Business needs data integrated into a single, trusted data store Want answers to any question across business processes Business wants both new and traditional data, thereby enabling analytics correlations across all data Low total cost of ownership (TCO) 360° view of any person or organization that touches the company Analytics systems should respond quickly and cheaply to changes in business conditions or acquisitions Scalable Fast response What Organizations want from their Analytics
- 10. www.datavail.com 10 But Moving to the Cloud Can Be Challenging… Top Internal Challenges Adopting Data & Analytics in the Cloud Challenges with technology infrastructure and/or architecture Solving risk and governance issues (security, ethics, privacy, data quality) Adding more agility and flexibility to our data and analytics initiatives Integrating multiple data sources Obtaining skills and capabilities needed Making data and analytics more usable for business consumers and front-line workers 0% 5% 10% 15% 20% 25% 30% 35% 33% 32% 29% 29% 27% 26% n= 270, total respondents, excluding “don’t know” Source: Gartner
- 11. M BI Maturity Stages Maturity is now critical to company competitiveness and success. www.datavail.com Creating Market Agility and Differentiation Fostering Innovation and People Productivity Integrating Performance Management & the Business Measuring and Monitoring the Business Running the Business 1 2 3 4 5 11
- 12. A Our Approach We provide a consultative and advisory service entrenched in technology, people, and process. A fined-tuned service and flexible solution with several successful engagements under our belt Solves a business problem (“pain points”) Includes Datavail developed accelerators Aims to be vendor software agnostic Delivered as an “outcomes” based solution with defined ‘Quick Wins’ www.datavail.com 12
- 13. Direct Efforts With A Focused Business Vision Goal is to enable Systems of Insight to drive business value and efficiencies. 3 2 1 Data Foundation Approach to delivering on a well architected journey for tools and framework to drive data integration and analytics through an execution roadmap and timeline in accordance with compliance. Transform Information Raise the bar on operational excellence and corporate success by converting data into actionable insights, along with ability to dynamically adjust to reporting requirements and compliance. Modern Analytics Guided, actionable analytics, providing self-service, distributed analytics, dashboards, and future-proof scaling of data to information integration. www.datavail.com 13
- 14. www.datavail.com 14 Players in the Cloud Analytics Space ETL MDM Cloud Providers Database Reporting/ Analytics Cloud & Big Data
- 15. www.datavail.com 15 Sample of our Analytics Clients
- 16. C Case Studies From small business to large enterprises, see how we’ve helped our clients gain value from their organizational data. Major Media Company Retail Company National Broadcasting Company www.datavail.com 16
- 17. C Case Studies From small business to large enterprises, see how we’ve helped our clients gain value from their organizational data. Major Media Company Retail Company National Broadcasting Company www.datavail.com 17
- 18. www.datavail.com 18 Challenge Client’s IT Staff was dedicated to providing custom reports based on client requirements that required two to three dedicated resources. Cost of Database Software license was becoming prohibitive Basic problem with the on-prem existing analytics solution: • Didn’t scale • Costly (licenses and VM Servers) • IT Bottleneck (required for each dataset developed) • Dependence on Affinity ERP email capability (performance and file-size limitations) • Dependency on internal staff for report customization Solution Proposed solution was to take advantage of the AWS Cloud Analytics services. Serverless solution reduced cost (pay as you go) Scaled easily Provide Self Service data visualization and data set delivery Automation of data movement and processing Case Study: Major Media Company
- 19. www.datavail.com 19 Media Company – New Architecture SQLServer DB OLTP OLTP 1. Existing Data Source bucket with incremental data Stage - S3 2. Stage Data 3. Data Marts RDS RDBMS Amazon RDS 4. Self Service Analysis Analysis Lambda function Amazon CloudWatch AWS Data Pipeline AWS Glue Amazon QuickSight Internal User External User 7. REST API for Data Integration Data Integration Services Business Intelligence bucket with data sets Data Set Delivery -S3 5. Build and Deliver Data Sets AWS Glue 6. Deliver Reports – signed URL in email
- 20. C Case Studies From small business to large enterprises, see how we’ve helped our clients gain value from their organizational data. Major Media Company Retail Company National Broadcasting Company www.datavail.com 20
- 21. www.datavail.com 21 Challenge Existing vendor solution was not providing the reporting and analytics environment required to manage the business. Technology was obsolete Support was minimal “keep the lights on” Needed to expand from B2B to include B2C Sales and Operational Data Expand to include additional data sources Solution Determined that an AWS “Data Lake” solution to bring both structured and unstructured data into the Data Lake for processing to drive analytics for the business. Utilized AWS Data Lab and POC to prove solution addressed business needs A support model was established so that Datavail was in a Build/Run opportunity to provide support for the new solution – from data loads, to reporting, to governance and managing the environment Case Study: International Retail Company
- 22. www.datavail.com 22 Existing Business Environment Agility for Today’s and Tomorrow’s Business Needs – Cloud Flexibility and Speed - Time to Deliver Updates and Data Availability Proactive Control of Data Quality
- 23. www.datavail.com 23 The Solution: Automated Data Profiling/Reporting Analysts CSV or Other Files On-Prem S3 Bucket AWS Athena Data Catalog Glue Crawler Glue Crawler Profiler Metrics Repository Data Profiler on EMR
- 24. C Case Studies From small business to large enterprises, see how we’ve helped our clients gain value from their organizational data. Major Media Company Retail Company National Broadcasting Company www.datavail.com 24
- 25. www.datavail.com 25 National Broadcasting Company Challenges Broadcasting Company has an existing data warehouse that is not meeting the user’s needs and they want to re-engineer this warehouse to meet the functional and analytical requirements of the user The existing DW has obscure field names which forces all reporting requests to go through a Data Scientist vs. enabling the user to create their own reports External data is not integrated into the warehouse for trend analysis or for other types of market analysis Improving the frequency of digital advertising data will improve and enhance fund raising campaigns and pledge drives The existing DW environment: • SQL Server • Tableau and Microsoft BI for reporting • Alteryx as the ETL tool
- 26. www.datavail.com 26 Solution Considerations Improve the flexibility, scalability and overall capabilities of the warehouse to support business reporting and analytics while providing data to the data science team to focus on analysis that is external to Broadcasting Company Improve and reduce the support structure to make the solution easily supportable by the existing support team including technical training, knowledge transfer, etc. Protect PCI and PII data in a secure manner Leverage the cloud to take advantage of potentially lower costs assuming security can be maintained Provide an approach to start with Broadcasting Company’s Digital business while extending the solution to other lines of business
- 27. www.datavail.com 27 A Modern Data Lake Architecture INGEST MODEL ANALYZE REPORTING STAGE & STORE DATA SOURCES Azure Data Factory Azure Data Lake Power BI Service Snowflake DB SaaS Other Data Sources Prayer Data Streaming Data Web Site Data Ad-hoc Reporting and Analysis Standard Reporting
- 29. www.datavail.com 29 Snowflake Snowflake’s Data Cloud is a Software-as-a-Service (SaaS) data platform that enables data storage, processing, and analytical solutions that are faster, easier to use, and more flexible than traditional analytics offerings Snowflake combines a new SQL query engine with an innovative architecture that is natively designed for the cloud Snowflake runs on the following cloud platforms: • Azure, AWS, Google Snowflake processes queries using MPP (Massively Parallel Processing) compute clusters storing a portion of the entire data set locally to offer data management simplicity of a shared-disk architecture but with the performance and scale-out benefits of a shared-noting architecture Snowflake stores data in a columnar format with the data only accessible through SQL query operations AWS Redshift Redshift can be described as a fully-managed, cloud-ready petabyte-scale data warehouse service that can be seamlessly integrated with business intelligence (BI) tools. An Amazon Redshift data warehouse is an enterprise-class relational database query and management system. Amazon Redshift integrates with various data loading and ETL (extract, transform, and load) tools and business intelligence (BI) reporting, data mining, and analytics tools. Amazon Redshift is based on industry-standard PostgreSQL. Amazon Redshift supports client connections with many types of applications, including business intelligence (BI), reporting, data, and analytics tools. When you execute analytic queries, you are retrieving, comparing, and evaluating large amounts of data in multiple-stage operations to produce a final result. Amazon Redshift achieves efficient storage and optimum query performance through a combination of massively parallel processing, columnar data storage, and very efficient, targeted data compression encoding schemes. Two Key Players to Watch and Learn From
- 30. www.datavail.com 30 AWS Data Hub – with Snowflake ERP On-Prem Data Sources Data Hub BI Tool(s) Data as a Service (data sets, 360 search, API, Web apps, predictive models) Information Delivery: Amazon Athena AWS Glue ETL Amazon Elasticsearch Service Crawler AWS Database Migration Service Data Lake (S3) Data Catalog Landing Tier Analytics 2 Tier Analytics 1 Tier Machine Learning Algorithms Data Source Model Train Data Test Data - Csv data files - Delta only (some full) - Granular level data - No transformations - Parquet/ORC files - Partitioned - Coelescing Partitions - Optimized for Analytics - Domain Level - Org by Use Cases - Optimized special analysis Views Accommodat e Updates and Deletes AWS Glue ETL AWS Lambda AWS Lambda Amazon SageMaker Amazon EMR Snowflake DB SaaS
- 31. www.datavail.com 31 AWS Data Hub – with Redshift ERP On-Prem Data Sources Data Hub BI Tool(s) Data as a Service (data sets, 360 search, API, Web apps, predictive models) Information Delivery: Amazon Athena AWS Glue ETL Amazon Elasticsearch Service Crawler AWS Database Migration Service Data Lake (S3) Data Catalog Landing Tier Analytics 2 Tier Analytics 1 Tier Machine Learning Algorithms Data Source Model Train Data Test Data - Csv data files - Delta only (some full) - Granular level data - No transformations - Parquet/ORC files - Partitioned - Coelescing Partitions - Optimized for Analytics - Domain Level - Org by Use Cases - Optimized special analysis Views Accommodat e Updates and Deletes AWS Glue ETL AWS Lambda AWS Lambda Amazon SageMaker Amazon EMR
- 32. What’s Next?
- 33. www.datavail.com 33 Get a clear view of your cloud strategy – and align • Expected Benefits of moving to the cloud • Cloud data strategy • XaaS strategy • Constraints • Roadmap Assess your current state Use the cloud for experimentation Set the right migration approach based on your priorities Put analytics wherever the data is Utilize the power of the cloud to scale Use multiple clouds depending on your purpose Enable self-service analytics Best Practices in moving to the Cloud
- 34. www.datavail.com 34 “As a Service” of cloud – pay as you go instead of capital outlay Increased scalability. Think about your on-site IT infrastructure Faster insights Easier maintenance and disaster recovery Stronger decision making Can start with a Small Project! Cost-Savings Agility Scalability Solves new analytics requirements (Use-Cases) Summary: Why Move Analytics to the Cloud?
- 35. Fill Out Our Cloud Analytics Survey Fill out the Cloud Analytics survey for a chance to WIN a Sonos Move, a battery-powered Smart Speaker! https://www.datavail.com/cloud-analytics-survey
- 36. Q&A
Editor's Notes
- This is not a “niche” solution – this is the future of how companies will achieve Strategic Agility and Differentiation. Why do we get deals or credits from AWS and Azure? Because they can’t keep up with the demand – they need companies like Datavail! Should Datavail focus on any particular area within that market?” Answer: Yes, the Cloud area of Big Data and Business Analytics Solutions. Cloud Analytics growing from 52 Billion this year to 121 Billion in 2022
- The lack of innovation is not the result of laziness. People and organizations are simply too busy providing descriptive analytics to engage in in-depth thinking. Simply migrating to the cloud with a “lift and shift”' approach does not result in innovation, but it does add another level of complexity to operations. The effort expended on maintaining the traditional analytics process turns into “analytics debt” for organizations, which impedes their ability to be creative and innovative. This lack of innovation ultimately costs organizations in terms of productivity in analytics, preventing them from adding value to the business. It is urgent, therefore, for organizations to explore the cloud, and inevitable that they will do so, as this is where new capabilities emerge owing to the effects of data gravity
- Cloud analytics offers new capabilities for users to generate business value through a trial-and-errorbased environment. Organizations can introduce cloud analytics as a use case for users to generate more visible analytics prototypes that form the basis for innovation (see Figure 3). Data and analytics leaders need to pitch onboarding with cloud analytics as an ideation process to start analytics with the following steps.
- Note: This is mostly business needs, not an IT needs! IT should know this, but often don’t. Basically they want answers to business questions – when they want to ask them – not all up front in a “requirements gathering” phase. Scalable – don’t want to wait for a procurement process…. Hopefully, Modern Analytics addresses many of these needs.
- Needed to rearchitect using new technologies and approaches.
- What’s different about this solution? Serverless. Less delivery of data sets and more interaction, ad-hoc analyses by both internal and external users. Delivers Business Insights more than data sets. Very elegant solution!
- Stan Add bullets about flexible data ecosystem that enables them to adjust and change to meet their business objectives. Add more content and bullet points about this. Cloud on demand Agility of the business Infrastructure flexibility – Cloud Future demands Real time reporting and inventory management Quality of the solution – flexibility and quality Time to delivery of changes to meet the reporting needs as they change The solution must provide the ability to re-develop the existing process while providing a capability for managing services for processing data from over 75,000 chains and 15,000 wholesalers and consolidating with master data (Store, SAP Customer, Product, Employee pro) to create cubes in SQL Server. These cubes will be consumed by over 30 analysts/data scientists and the resulting output (reports, dashboards) are viewed by over 300 users and must provide like functionality as to what the analysts/data scientists leverage today. Scope The overall scope of the solution is to process data from the existing data sources and to then create data cubes for analysis. Data for processing the RBH data received from the following sources: 75000 + stores 15000+ wholesalers The Master data that must be leveraged in the processing of the data received from these sources includes the following master data: Store SAP Customer Product Employee There are two File Specs expected for the Chains, however there are up to 5 different formats between 30 participating chains. Some chains will submit weekly aggregated data and the remaining chains will submit data at the daily aggregate level, but it will be submitted weekly. Some chains will be providing data in the Excel format and the remaining chains will be providing data in the text file format. The structure of the input data files from the store chains, wholesalers and master data will be provided by RBH or RBH will designate the service provider as the party that is authorized to communicate directly with the wholesalers and store chains to acquire the input data file structures. The overall solution must be responsible for keeping track of receiving the input files from the wholesalers, store chains and maser data from RBH. Vendor will also be responsible for following up with the input file and master data providers in case of any delay in receiving these files. The overall solution must be authorized by RBH to communicate with the wholesalers and store chains on behalf of RBH. The logic to perform the ETL process to transform the input files form the wholesalers and store chains into an output format that can be used by RBH analysts to execute existing reports, dashboards or ad hoc queries will be developed for use by the existing team of analysts/data scientists in a format similar to what they are using today. This format will be provided by RBH as well as the requirements for the overall process including any transformation business rules are not clear from the output format like in the case of calculated fields. The new process must also be designed and developed to cleanse and consolidate points of sales data for the products that are not present in the RBH master file.
- Stan
- A little like switching from film pictures to digital pictures. Answers the question, Who Needs Modern Analytics? Pretty much everyone. Cloud analytics lets companies leverage the power of analytics more quickly, more powerfully, and at lower cost. Cost Savings: Cost savings or financial benefits are one common reason for moving to the cloud. If you are only looking to just move your spending from a capital expense model to an operational expense model, then your achievement criterion is effortlessly met by simply moving to the cloud and subscribing to “as a service.” Agility: Another major reason for moving to the cloud is agility, but this is an option only if agility is important to you. For example, when on-premises capacity is made to handle the main jobs of the month, quarter, or year, then moving resources to the cloud can allow you to right-size the on-premises infrastructure for the workloads it needs to handle most of the time and only raise up to the peak demand when needed.
- A little like switching from film pictures to digital pictures. Answers the question, Who Needs Modern Analytics? Pretty much everyone. Cloud analytics lets companies leverage the power of analytics more quickly, more powerfully, and at lower cost. Cost Savings: Cost savings or financial benefits are one common reason for moving to the cloud. If you are only looking to just move your spending from a capital expense model to an operational expense model, then your achievement criterion is effortlessly met by simply moving to the cloud and subscribing to “as a service.” Agility: Another major reason for moving to the cloud is agility, but this is an option only if agility is important to you. For example, when on-premises capacity is made to handle the main jobs of the month, quarter, or year, then moving resources to the cloud can allow you to right-size the on-premises infrastructure for the workloads it needs to handle most of the time and only raise up to the peak demand when needed.