Sonny Khan, Senior Data Analytics Specialist, AWS
As everyone’s data is growing, there is a growing need to become data-driven and harness decision-making from data. The existing and legacy approaches created silos, dark data, and risk. AWS Modern Data Architecture is alleviating these issues and enabling new capabilities through cost effective and purpose-built services and architecture. Learn how it works and how customers have used it to realize their business goals and outcomes.
1/ Hi, I am Sonny Khan, Senior Data Analytics Specialist.
2/ Over the 40 minutes, I am going to share how organizations are using data to drive better business outcomes.
3/ I will also discuss how the various technology components come together to create a modern data strategy.
Transition --
Let’s get started by looking at your portfolio of AWS data and machine learning services.
1/ Data is getting more diverse.
2/ Customers are storing and analyzing data from all kinds of sources such as machine data from industrial equipment, digital media, data from social networks, online transactions, financial analysis, genomics research, and more.
Transition --
Businesses are also dealing with more volumes of data than ever before.
1/ According to Fortune magazine, the amount of data created over the next three years will be more than the data created over the past 30 years.
2/ Infact, 448 Zettabytes of data is expected to be created and replicated in the coming three years.
Transition –
It’s hard to even imagine how much data is in a Zettabyte, never mind 448 of them.
Forbes source: https://www.forbes.com/sites/gilpress/2021/12/30/54-predictions-about-the-state-of-data-in-2021/?sh=7034392397d3
1/ To put that into perspective, if you were to save all of that data to conventional 1TB hard disks, you could stack them as high as the Eiffel tower in Paris … over 20 million times.
2/ That’s the same distance as 527 round trips around Earth’s equator.
3/ We believe data growth is reaching a tipping point.
Transition –
And this data is hugely valuable!
1/ Data is the underlying force that fuels the insights and predictions that lead to better decision making and innovation. But harnessing this data to reinvent your business, while challenging, is imperative to staying relevant now and in the future. Data-driven organizations treat data like an asset, make it available and accessible to everyone, and put it the work to enable better, more informed decisions.
2/ In order to be ready for the future, organizations must be data-driven or be left behind. Richard Joyce, Senior Analyst at Forrester stated, “For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income.” Forrester also estimates data-driven businesses are growing at an average of more than 30% annually.
3/ That being said, most organizations are not putting their data to work. While companies are creating, collecting, and storing more data than ever before, much of it remains underutilized, or not used at all. Accenture found that 68% of companies are not able to realize tangible and measurable value from their data.
4/ In order to tap into that data, an organization must cultivate a data driven culture. Data alone can’t unlock trapped value. According to the same Accenture report, 55% of companies have a mostly manual approach to discovering data within their enterprise, and only 28% have a strategy in place to take advantage of analytics tools and infrastructure throughout the enterprise.
TRANSITION: While many organizations are struggling to manage their data, especially in the wake of the COVID-19 pandemic, others are thriving.
REFERENCE: https://www.accenture.com/_acnmedia/PDF-108/Accenture-closing-data-value-gap-fixed.pdf#zoom=50
According to a recent Forbes article, a typical Fortune 1000 company will see a $65 million increase in net income by making just 10% more data accessible. The same article states that Forrester estimates data-driven businesses are growing at an average of more than 30% annually. And while cultivating a data-driven culture hasn’t been easy for some, the benefits outweigh the challenges.
When it comes to leveraging AWS’ data lakes, analytics, and machine learning services, IDC calculated that organizations using those services would achieve average annual benefits of $6.15million per organization, resulting in a five-year return on investment of 415%. And all of these would be achieved by a 48% reduced total cost of operations over that time period.
Transition: These results can be seen by companies and organizations spanning a variety of industries. And event better, these results are already being realized.
Sources: https://www.forbes.com/sites/brentdykes/2019/03/28/the-four-key-pillars-to-fostering-a-data-driven-culture/?sh=184023137d90 | https://d1.awsstatic.com/analyst-reports/idc-bv-datalakes-analytics-ml-2020.pdf
1/ For a successful data strategy, customers modernized their data infrastructure by moving it to a scalable, trusted, and secure cloud, rather than self managing the infrastructure.
2/ They unified their data by breaking down silos, so data was put to work effectively across databases, data lakes, analytics and ML services.
3/ The third pillar of modern data strategy is innovate where they invented new experiences and reimagined existing processes.
4/ With the modern data strategy, customers can move and store any amount of data at scale, access that data seamlessly, and manage who has access to data with the proper security and data governance controls.
5/ These pillars do not require sequential implementation. A customer could be working on all three in parallel, depending on their data journey.
Transition --
But how do all the technology components come together in a modern data strategy?
Let’s start with modernize. Modernization could happen at various stages of the data journey. AWS offers scalable trusted and secure modernization for all types of data and workloads.
1/ Organizations running legacy, on-premises data stores or self-managing in the cloud still have to take care of management tasks such as database provisioning, patching, configuration, or backups. AWS takes care of this for you. Benefit from AWS’s unmatched experience, maturity, reliability, security, and performance that you can depend upon for your most important applications.
2/When organizations want to modernize their data infrastructure, AWS is the most scalable, trusted and secure cloud provider. AWS offers the broadest portfolio of purpose-built data stores to support the most demanding workloads at a fraction of the cost of old-guard databases. Modernizing with AWS means organizations can break free from legacy databases, move to fully-managed and purpose-built data services, and add ML into workloads to build modern applications.
3/When Samsung Electronics needed a more flexible, microservices-driven solution to replace its monolithic legacy internet data center, it looked to AWS to support it’s migration, significantly reducing database costs and increasing scalability. With AWS, Samsung enabled 60 ms latency or less 90% of the time, and reduced monthly database costs by 44%.
4/ Let’s talk about analytics modernization. At AWS, we acknowledge that one-size-fits-all approach to analytics eventually leads to compromises. With a modern data architecture on AWS, customers can rapidly build scalable data lakes, use a broad and deep collection of purpose-built data services, ensure compliance via a unified data access, security, and governance, scale their systems at a low cost without compromising performance.
5/ Intuit migrated to an Amazon Redshift-based solution that scales to more than 7X the data volume with zero effort and delivers 20X performance over the company's previous solution.
6/ And finally, machine learning modernization. AWS accelerates the pace of Machine learning led innovation by standardizing machine learning across the org - providing scalable infrastructure, integrated tooling, healthy practices for responsible use of ML and tools for users of all ML skill levels. It reduces ML model development time from months to weeks and improves data scientist productivity.
7/ Using Amazon SageMaker, NerdWallet reduced ML training costs by around 75 percent, even while increasing the number of models trained.
1/ Next is unify. Opportunities to transform the business with data exist all along the value chain. But making such a transformation requires that companies get a full picture and single source of truth of their customers and their business. This necessitates breaking down data silos and making data accessible and shared in a secure way to unlock the value of data for different constituencies and purposes
2/To make decisions quickly, you need new data stores that will scale and grow as business needs change. You also want to be able to connect everything together, including your data lake, data warehouse, and all of the purpose-built data stores into a coherent system that is secure and well governed. AWS helps you do this through the best of both data lakes and purpose-built data stores.
3/Data is most effectively unified when silos are broken down, allowing for unified security and governance across entire enterprise as well. Your organization can now move from insights to actions faster, with the help of purpose-built analytics and visualization services to extract the most value out of your data.
4/Swimming Australia, Australia’s national swing team, was looking to optimize their data management and analysis in preparation for the 2020 Tokyo Olympic Games. By utilizing AWS’ data lakes and ML services, Swimming Australia optimized the order of swimmers in the relay swim, resulting in the most successful relay team at the Games. Australia also received 20 medals overall in the pool – the country’s best ever medal haul for swimming.
5/ Nasdaq used Amazon S3 to build a data lake, allowing them to ingest 70 billion records per day. Nasdaq now loads financial market data five hours faster and runs Amazon Redshift queries 32 percent faster.
6/ Netflix uses Amazon Kinesis Data Streams to process multiple terabytes of log data each day, and yet surface events in analytics in seconds.
And Finally, innovate. Innovation too happens at various stages of the modern data strategy.
1/ Companies are innovating the way business applications are built. They are moving from building a single monolithic application—where every aspect of the application is tightly coupled—to applications built with modular independent components called microservices. AWS allows customers to access multiple, relational and non-relational database options that are purpose-built to solve specific use cases.
2/ We are also infusing ML into our other data-related service categories to allow for innovation in data management and analysis for users of all skill levels. Data analysts can use Amazon Redshift ML and Amazon Athena ML to run ML on their data in a data warehouse or data lake without having to select, build, or train an ML model. Business analysts can use QuickSight Q which uses ML to automatically generate a data model that understands the meaning of and relationships between business data, to ask questions of their data using plain language and receive answers in near-real time.
3/ On the machine learning services front, we’re innovating on behalf of our customers. We offer the broadest and deepest set of machine learning capabilities for builders of all levels of expertise, removing the undifferentiated heavy lifting so that our customers can move faster. We have purpose built AI services as well as an end-to-end machine learning suite to help our customers innovate. These services address common use cases such as personalized recommendations, contact center intelligence, document processing, fraud detection, intelligent search, business metrics analysis, and more.
4/Intuit was able to leverage AWS’ services to drive database innovation, which resulted in a 25% reduction in team costs, 60%-80% less time spent on maintenance, 20%-40% cost savings overall, and a 90% reduction in time to deploy models – all while freeing the teams to spend more time developing the next wave of innovations.
Innovate Data stores
Cathay Pacific Airways Modernizes Passenger Revenue Optimization System on AWS, Increases Performance by 20%
Experian turns to microservices driven architecture and achieves 100% Uptime with Amazon DynamoDB and Amazon Aurora
Innovate Analytics
Nielsen built an innovative cloud-native data reporting platform on AWS
Innovate ML
T-Mobile humanizes customer care with ML
Philips Uses AI and ML to Improve Medical Imaging Diagnostics for Philips HealthSuite Built on AWS
1/ A modern data strategy is enabled by a set of technology building blocks that help you manage, access, analyze, and act on data.
2/ It gives you multiple options to connect to data sources.
3/ It empowers your teams to run machine learning or analytics or using their preferred tools or techniques, and manage who has access to data with the proper security and data governance controls.
4/ It gives your end users timely access to insights through business intelligence tools.
5/ It lets you break down data silos, and gives you the best of both data lakes and purpose-built data stores.
6/ It enables you to store any amount of data, at low cost, and in open, standards-based data formats.
7/ AWS offers the most comprehensive set of tools for the end-to-end data journey.
Transition --
Let’s look at each of these building blocks, starting with data sources.
1/ There is tremendous growth in the variety of data sources from machine data from industrial equipment, digital media, data from social networks, online transactions, financial analysis, and more.
2/ And you need access to these data sources before you can run analysis and predictions.
3/ For this, we offer services like AWS Data Exchange that help you discover and use third party data.
4/ With Amazon Kinesis Data Streams, you can ingest and store streaming data.
5/ Amazon AppFlow lets you transfer data between SaaS applications like Salesforce, SAP, and Zendesk and AWS services.
6/ And, AWS IoT let's you connect and manage billions of devices.
7/ After you have ingested this data, you also need to think about how to move data between your data lake, data warehouse, and data stores. For this, we offer tools like AWS Glue Elastic Views that is currently in product preview. Glue Elastic Views enable you to effortlessly move and keep data in sync between the data lake, data warehouse, and purpose-built stores.
Transition –
When you have diversity of data, you also need purpose-built databases that can take advantage of diverse data models to give you the best price performance for your applications.
1/ We offer the industry’s broadest portfolio of databases that are purpose-built for your specific application needs.
2/ 15+ database engines support a variety of data models including relational, key-value, document, in-memory, graph, time-series, and ledger databases.
3/ Each of these purpose-built database engines is uniquely designed to provide optimal performance for the respective use cases so developers never have to compromise.
4/ For example, Amazon Aurora is a MySQL and PostgreSQL compatible relational database service designed for unparalleled performance including scalability, availability, and reliability - all at 1/10th of the cost of enterprise grade commercial databases.
5/ With Amazon DynamoDB, customers get a fast, flexible, and serverless NoSQL database for any scale, to support key-value and document workloads.
6/ If you are looking to store and analyze trillions of events per day, you can choose Amazon Timestream - a fast, scalable, and serverless time series database service.
7/ To-date we have more than 650,000 databases migrated to AWS using the AWS Database Migration Service (DBMS).
8/ Customers like Atlassian are using our database services to remove the undifferentiated heavy lifting associated with on-premises databases and to spend more time focusing on its customers. Further, Atlassian was able to build best practices for managing a large database fleet, and achieve its business objectives, including the introduction of a free pricing tier for workplace productivity products like Jira and Confluence.
Transition –
To make decisions quickly, organizations want to unify the data and make it available broadly across the organization to democratize analytics and ML.
Atlassian case study link: https://aws.amazon.com/solutions/case-studies/atlassian-case-study-rds/
1/ Data Lakes are a foundational element of your unification strategy.
2/ Amazon S3 is the best place to build a data lake because it has unmatched durability, availability, scalability, and more.
3/ Data lakes let you store all your data, your relational and non-relational data, your structured and unstructured data cost effectively.
4/ By storing your data in open formats, you are able to decouple storage from compute so when it’s time to analyze your data you can choose the best tool for the job from a variety of on-demand pay as you go analytics and ML engines.
5/ Our most advanced and largest customers are managing exabytes scale data lakes and often have multiple data lakes in multiple accounts across their organization. In fact, we have hundreds of thousands of data lakes running on AWS.
Transition –
Cataloging and governance are critical for making data lakes successful and for this we offer AWS Lake Formation.
1/ AWS Lake Formation helps build and secure data lakes in the cloud, in days instead of months.
2/ Lake Formation helps you collect and catalog data from databases and object storage, move the data into your new Amazon S3 data lake, and clean and classify your data using machine learning algorithms,
3/ It also helps you secure access to your sensitive data.
Transition –
But what good are data lakes and catalogs without insights?
1/ We offer the broadest and most cost-effective set of analytics services that help you create insights from data.
2/ Our analytics services support workloads like querying, big data processing, real-time analytics, data warehousing, and more.
3/ For interactive querying of unstructured data, we offer Amazon Athena. Athena makes it easy for you to query all your data, wherever it lives, in AWS, on premises, and in SaaS apps.
4/ Then there is Redshift, which offers up to 3x better price performance than other cloud data warehouses. It’s no wonder that tens of thousands of customers analyze exabytes of data every day in Amazon Redshift.
5/ We offer more serverless options for data analytics in the cloud than any other cloud provider.
6/ Taken together, our analytics services and data lakes help you innovate by helping you create insights faster from all your data.
Transition –
But innovation doesn’t stop with our analytics services. We also have a portfolio of ML services to help you innovate faster and invent new customer experiences.
1/ AWS helps you at every stage of your ML adoption journey with the most comprehensive set of artificial intelligence (AI) and ML services, and infrastructure.
2/ AWS ML services can help you make accurate predictions, get deeper insights from your data, reduce operational overhead, and improve customer experience.
3/ Our services support four approaches to machine learning.
4/ First, you can create ML predictions using our no-code solution, SageMaker Canvas. Using a simple drag and click user interface, business analysts can go through the entire ML workflow from preparing your data to picking the best ML model to doing prediction without writing a single line of code.
5/ Second, you can use a pre-trained model such as Amazon Rekognition, which has already been trained to recognize objects in images, or Amazon Lex, which has been trained to understand intentions in natural language. In fact, we offer 20+ of these pre-trained models that you can invoke through simple API calls.
6/ Third, you can use Amazon SageMaker to train and apply your own model based on any one of the common algorithms used for ML.
7/ Fourth, you can use your own algorithms and training approaches by working directly with AWS infrastructure that is optimized for machine learning.
8/ We offer the industry’s broadest and most complete set of ML capabilities. More than 100,000 customers are using our AI and ML services to make predictions from their data.
9/ Formula One is one such customer. They are using our ML services to make race predictions, and to create insights based on the split-second decisions made by teams and drivers. This enhances the fan viewing experience and increases fan engagement.
10/ Nerdwallet is another customer that modernized their data science and engineering workflows through automation with Amazon SageMaker. This enabled them to train ML models in days instead of months.
Transition –
We also need to make the insights available broadly across the organization.
1/ For this, we offer Amazon QuickSight.
2/ QuickSight is our cloud-native, serverless business intelligence service. It can help you enable BI for everyone in your organization.
3/ QuickSight makes it incredibly simple and intuitive to get to answers with QuickSight Q which allows users to query data in plain language.
4/ For example, you could have a simple question like “show me monthly sales for Americas versus Europe” but quickly want to know a more complex detail like “show me year to date sales for Americas versus Europe by year.”
5/ No longer do you have fiddle around with dashboards, controls and calculations – QuickSight Q can get you these answers in seconds without that or the back and forth with BI teams.
6/ With QuickSight, you can also embed interactive data visualizations, dashboards, or natural language querying into your applications.
Transition –
Our investments in ML, databases, and analytics are helping customers such as Intuit reimagine processes.
1/ AWS is the best place to extract value from your data and turn it into insight because we have the most experience among all cloud providers. We also have the the most reliable, scalable and secure cloud, and the most comprehensive set of services for your end-to-end data journey.
Transition –
And there are several ways for you to get started.
1/ We offer the broadest and most cost-effective set of analytics services from querying, to big data processing, to real-time analytics, data warehousing, and more.
2/ For ad-hoc universal querying of unstructured data, we offer Amazon Athena. Athena makes it easy for you to query all your data, wherever it lives, in AWS, on premises, and in SaaS apps.
3/ Then there is Redshift, which offers up to 3x better price performance than other cloud data warehouses.
4/ Taken together, our analytics services and data lakes help you gain insights faster from all your data.
1/ You can use our AI and ML services to create new customer experiences and reimagine existing processes.
2/ We’re innovating on behalf of our customers to deliver the broadest and deepest set of machine learning capabilities for builders of all levels of expertise.
3/ At the top layer are our AI Services, which allow developers to easily add intelligence to any application without needing ML skills. The pre-trained models provide ready-made intelligence for your applications and workflows to help you do things like personalize the customer experience, forecast business metrics, translate conversations, extract meaning from documents and more.
4/ At the middle layer is Amazon SageMaker, which provides every developer and data scientist with the ability to build, train, and deploy machine learning models at scale. It removes the complexity from each step of the machine learning workflow so you can more easily deploy your machine learning use cases, anything from predictive maintenance to computer vision to predicting customer behaviors.
5/ And at the bottom layer, expert practitioners can develop on the framework of their choice as a managed experience in Amazon SageMaker or use the AWS Deep Learning AMIs (Amazon machine images), which are fully configured with the latest versions of the most popular deep learning frameworks and tools.
6/ At each layer of the stack, we’re investing in removing the undifferentiated heavy lifting so your teams can move faster.
1/ We offer the most complete set of purpose-built database engines including relational, document, caching, and more.
2/ Each of these purpose-built database engines is uniquely designed to provide optimal performance for the respective use cases so developers never have to compromise.
3/ For example, Amazon Aurora is a MySQL and PostgreSQL compatible relational database service designed for unparalleled performance including scalability, availability, and reliability - all at 1/10th of the cost of enterprise grade commercial databases.
1/ Nasdaq is a multinational financial services company that owns and operates the Nasdaq Stock Exchange.
2/ Every night, Nasdaq receives billions of records that need to be processed for billing and reporting purposes before the markets open the following morning.
3/ With growth in data volumes from automated trading platforms, Nasdaq wanted to increase scale and performance, and lower operational costs of their data warehouse solution.
4/ Nasdaq selected AWS to modernize their data architecture using a range of technologies such as S3, Redshift, and EMR.
5/ They initially moved from a legacy on-premises data warehouse to an AWS data warehouse powered by an Amazon Redshift cluster.
6/ Later, they adopted a data lake architecture where they put all the data from exchanges into Amazon S3.
7/ They use Amazon Redshift Spectrum to query the data.
8/ With compute and storage scaling independently, Nasdaq can now flex its compute and storage layers to support high volumes of transactions. As a result, Nasdaq met it’s scaling requirements of processing 30 to 70 billion records daily. Even when the market volatility spiked in late February 2020, at the beginning of the COVID-19 pandemic, the solution scaled to ingest a peak volume of 113 billion records.
9/ The AWS solution also reduced the market data load times by up to 5 hours. This helped the Nasdaq economic research team provide more timely insights.
(Case study URL for reference: https://aws.amazon.com/solutions/case-studies/nasdaq-case-study/)
1/ Intuit is a global company that helps consumers and small businesses overcome their most important financial challenges.
2/ Intuit serves more than 100 million customers with products such as TurboTax, QuickBooks, Mint, and more.
3/ They were looking to improve their contact center customer experiences, and lower costs.
4/ Intuit deployed a range of AWS technologies including data lakes, data warehouse, databases, and ML.
5/ Intuit built a data lake using Amazon S3
6/ They deployed Amazon Redshift for complex analytical workloads and migrated MySQL to purpose-built databases such as Amazon RDS for MySQL and Aurora
7/ Intuit used Amazon SageMaker to train ML models quickly and at scale
8/ Intuit also deployed Contact Lens for Amazon Connect to identify customer sentiment and trends from contact center customer conversations
9/ This ML-powered speech analytics helped them train customer service agents, while identifying crucial company and product feedback.
10/ Further, Amazon SageMaker helped them reduce model deployment time by 90%.
11/ They also realized 25% lower costs from the migration to purpose-built databases.
Transition –
Just like Intuit, AWS can help you extract value from data.
Intuit case study link: https://aws.amazon.com/solutions/case-studies/innovators/intuit/
1/ You can build your data infrastructure with us in several ways.
2/ With AWS Data Labs, you can start joint engineering engagements with AWS to speed-up your data initiatives.
3/ ML Solutions Lab pairs your team with AWS ML experts to build high value ML solutions.
4/ AWS Professional Services offers advisory services and specialist practices for analytics, databases, ML, and all the major industries including financial services, automotive, telecom, public sector, and more.
5/ AWS Immersion Day workshops are day-long workshops that AWS Solutions Architects create to help customers walk through different areas of the AWS services and solutions.
6/ With the Data-Driven Everything (D2E) program, AWS partners with your company to accelerate the journey towards becoming data-driven
7/ The AWS Migration Acceleration Program is a comprehensive and proven cloud migration program based upon AWS’s experience migrating thousands of enterprise customers to the cloud.
8/ You can also build with our partners.
9/ You can choose an AWS partner from our global community of trusted cloud partners to help you design and implement a data solution.
10/ The AWS Marketplace features Independent Software Vendors that can help you extend and customize your AWS data stack for your industry or use case.
11/ You can also upskill your team by leveraging 100+ free and paid training courses for databases, analytics, and ML offered by the AWS Training and Certification team.
12/ For ML, we also offer the AWS ML Embark program that combines training, coaching, and implementation support needed to upskill teams on ML and accelerate business outcomes.
13/ These implementation offerings complement the industry’s broadest and deepest set of database, analytics and ML offerings to help you drive value from data.
Transition –
Thank you. I can answer any additional questions you may have.
D2E more information here: https://pages.awscloud.com/GLOBAL-field-GC-d2e-data-driven-Q421-NAMER-2021-learn.html
MAP more information here: https://aws.amazon.com/migration-acceleration-program/