The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Big Data in the Cloud: How the RISElab Enables Computers to Make Intelligent ...Amazon Web Services
Scientists, developers, and other technologists from many different industries are taking advantage of Amazon Web Services to perform big data workloads from analytics to using data lakes for better decision making to meet the challenges of the increasing volume, variety, and velocity of digital information. This session will feature UCB's RISELab (Real time Intelligent Secure Execution), a new lab recently created at UCB to enable computers to make intelligent, real-time decisions. You will hear how they are building on their earlier success with AMPLab to enable applications to interact intelligently and securely with their environment in real time, wherever computing decisions need to interact with the world. From cybersecurity to coordinating fleets of self-driving cars and drones to earthquake warning systems, you will come away with insight on how they are using AWS to develop and experiment with the systems for important research. Learn More: https://aws.amazon.com/government-education/
Configuration Management in the Cloud | AWS Public Sector Summit 2017Amazon Web Services
In order for your application to operate in a predictable manner in both your test and production environments, you must vigilantly maintain the configuration of your resources. By leveraging configuration management solutions, Dev and Ops engineers can define the state of their resources across their entire lifecycle. In this session, you will learn how to use AWS OpsWorks, AWS CodeDeploy, and AWS CodePipeline to build a reliable and consistent development pipeline that assures your production workloads behave in a predictable manner. Learn More: https://aws.amazon.com/government-education/
Incident Response in the Cloud | AWS Public Sector Summit 2017Amazon Web Services
We will walk you through a hypothetical incident response managed on AWS. Learn how to apply existing best practices as well as how to leverage the unique security visibility, control, and automation that AWS provides. We will cover how to setup your AWS environment to prevent a security event and how to build a cloud-specific incident response plan so that your organization is prepared before a security event occurs. This session also covers specific environment recovery steps available on AWS. Learn More: https://aws.amazon.com/government-education/
AWS Summit 2014 Melbourne - Breakout 5
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
Presenter: Craig Dickson, Solutions Architect, Amazon Web Services
(SEC320) Leveraging the Power of AWS to Automate Security & ComplianceAmazon Web Services
"You’ve made the move to AWS and are now reaping the benefits of decreased costs and increased business agility. How can you reap those same benefits for your cloud security and compliance operations? As building cloud-native applications requires different skill sets, architectures, integrations, and processes, implementing effective, scalable, and robust security for the cloud requires rethinking everything from your security tools to your team culture.
Attend this session to learn how to start down the path toward security and compliance automation and hear how DevSecOps leaders such as Intuit and Capital One are using AWS, DevOps, and automation to transform their security operations.
Session sponsored by evident.io"
What Organizational and Governance Changes Do I Need to Make Prior to Migrati...Amazon Web Services
The Center of Excellence (CoE) and Skills work stream is critical to establishing a customer’s migration readiness. To be prepared for an enterprise migration, the customer must have a critical mass of people with production AWS experience, established the foundational operational processes to support migrations, and a CoE dedicated to mobilizing the appropriate resources to lead the organization through the various organizational and business transformation challenges encountered over the course of a large-scale migration effort. Attend this session to learn about the organizational and governance changes an organization should make prior to migrating to AWS. Learn More: https://aws.amazon.com/government-education/
What is Innovation? How can cloud computing help you innovate? How can you make your applications smarter? Predictive? How can you interpret data and anticipate trends? With AWS Artificial Intelligence Solutions: Machine Learning, Rekognition, Polly; with serverless - Lambda, Step Functions.
Accelerating cloud adoption for your regulated workloads - AWS PS Summit Canb...Amazon Web Services
How can you architect your applications for regulatory and organisational compliance? How can you automate security, auditability, and governance controls using best practice? In this session, Accenture draws from real-world examples to showcase how the cloud can strengthen your security and compliance posture, while ensuring maximum agility – articulated through the lifecycle of an application moving to the cloud.
Speaker: Chris Fleischmann, Managing Director, Journey To Cloud, Accenture
Level: 200
Big Data in the Cloud: How the RISElab Enables Computers to Make Intelligent ...Amazon Web Services
Scientists, developers, and other technologists from many different industries are taking advantage of Amazon Web Services to perform big data workloads from analytics to using data lakes for better decision making to meet the challenges of the increasing volume, variety, and velocity of digital information. This session will feature UCB's RISELab (Real time Intelligent Secure Execution), a new lab recently created at UCB to enable computers to make intelligent, real-time decisions. You will hear how they are building on their earlier success with AMPLab to enable applications to interact intelligently and securely with their environment in real time, wherever computing decisions need to interact with the world. From cybersecurity to coordinating fleets of self-driving cars and drones to earthquake warning systems, you will come away with insight on how they are using AWS to develop and experiment with the systems for important research. Learn More: https://aws.amazon.com/government-education/
Configuration Management in the Cloud | AWS Public Sector Summit 2017Amazon Web Services
In order for your application to operate in a predictable manner in both your test and production environments, you must vigilantly maintain the configuration of your resources. By leveraging configuration management solutions, Dev and Ops engineers can define the state of their resources across their entire lifecycle. In this session, you will learn how to use AWS OpsWorks, AWS CodeDeploy, and AWS CodePipeline to build a reliable and consistent development pipeline that assures your production workloads behave in a predictable manner. Learn More: https://aws.amazon.com/government-education/
Incident Response in the Cloud | AWS Public Sector Summit 2017Amazon Web Services
We will walk you through a hypothetical incident response managed on AWS. Learn how to apply existing best practices as well as how to leverage the unique security visibility, control, and automation that AWS provides. We will cover how to setup your AWS environment to prevent a security event and how to build a cloud-specific incident response plan so that your organization is prepared before a security event occurs. This session also covers specific environment recovery steps available on AWS. Learn More: https://aws.amazon.com/government-education/
AWS Summit 2014 Melbourne - Breakout 5
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
Presenter: Craig Dickson, Solutions Architect, Amazon Web Services
(SEC320) Leveraging the Power of AWS to Automate Security & ComplianceAmazon Web Services
"You’ve made the move to AWS and are now reaping the benefits of decreased costs and increased business agility. How can you reap those same benefits for your cloud security and compliance operations? As building cloud-native applications requires different skill sets, architectures, integrations, and processes, implementing effective, scalable, and robust security for the cloud requires rethinking everything from your security tools to your team culture.
Attend this session to learn how to start down the path toward security and compliance automation and hear how DevSecOps leaders such as Intuit and Capital One are using AWS, DevOps, and automation to transform their security operations.
Session sponsored by evident.io"
What Organizational and Governance Changes Do I Need to Make Prior to Migrati...Amazon Web Services
The Center of Excellence (CoE) and Skills work stream is critical to establishing a customer’s migration readiness. To be prepared for an enterprise migration, the customer must have a critical mass of people with production AWS experience, established the foundational operational processes to support migrations, and a CoE dedicated to mobilizing the appropriate resources to lead the organization through the various organizational and business transformation challenges encountered over the course of a large-scale migration effort. Attend this session to learn about the organizational and governance changes an organization should make prior to migrating to AWS. Learn More: https://aws.amazon.com/government-education/
What is Innovation? How can cloud computing help you innovate? How can you make your applications smarter? Predictive? How can you interpret data and anticipate trends? With AWS Artificial Intelligence Solutions: Machine Learning, Rekognition, Polly; with serverless - Lambda, Step Functions.
Accelerating cloud adoption for your regulated workloads - AWS PS Summit Canb...Amazon Web Services
How can you architect your applications for regulatory and organisational compliance? How can you automate security, auditability, and governance controls using best practice? In this session, Accenture draws from real-world examples to showcase how the cloud can strengthen your security and compliance posture, while ensuring maximum agility – articulated through the lifecycle of an application moving to the cloud.
Speaker: Chris Fleischmann, Managing Director, Journey To Cloud, Accenture
Level: 200
An Evolving Security Landscape – Security Patterns in the CloudAmazon Web Services
Availability of cloud computing is helping Financial Services organizations realize accelerated go-to-market speeds, global scalability, and cost efficiencies. This new world forces considerations for security programs – what is different in the cloud and what do I do differently? AWS Security Architects will share protocols that need to be considered in the cloud, on premises, or in a hybrid model. They will also share best practices, lessons learned, efficiencies, and design patterns and architectures unique to cloud.
The New Normal: Benefits of Cloud Computing and Defining your IT StrategyAmazon Web Services
The standard business model is changing rapidly changing. Companies used to be built for the long haul. But now, success is powered by rapid-paced innovation and the ability to get disruptive products to market first.
You’re used to balancing resources between keeping things running and the development of new initiatives. But merely keeping the lights on doesn't differentiate you from your competitors.
Amazon CloudWatch Logs and AWS Lambda: A Match Made in Heaven | AWS Public Se...Amazon Web Services
In this session, we cover three common scenarios that include Amazon CloudWatch Logs and AWS Lambda. First, you learn to build an Elasticsearch cluster from historical data using Amazon S3, Lambda, and CloudWatch Logs. Next, you learn to add details to CloudWatch alarm notifications using Amazon SNS and Lambda. Finally, we show you how to bring Elastic Load Balancing logs to CloudWatch Logs using S3 bucket triggers from Lambda. Learn More: https://aws.amazon.com/government-education/
AWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API CallsAmazon Web Services
Monitoring and making sense of infrastructure data can be an arduous process. Managing a volume of API calls from more than one million active users every minute presents an even more complex and demanding challenge. Using Amazon Web Services (AWS) and Datadog, Grindr overcame a series of infrastructure challenges by both implementing and managing highly scalable, high availability, and top performing infrastructure, as well as aggregating, analyzing, and acting on key infrastructure data KPIs.
Getting Started with Managed Services | AWS Public Sector Summit 2016Amazon Web Services
The AWS cloud infrastructure is architected to be one of the most flexible and secure cloud computing environments available today. By leveraging services such as EC2, you are able to build highly scalable and performant architectures. AWS also provides a rich set of services which help to remove much of the potentially undifferentiated heavy lifting associated to managing your EC2 based infrastructure. This session will introduce some of these services in the areas of Application Management, Database, Analytics, Security and Enterprise Applications.
(SEC203) Journey to Securing Time Inc's Move to the CloudAmazon Web Services
"Learn how Time Inc. met security requirements as they transitioned from their data centers to the AWS cloud. Colin Bodell, CTO from Time Inc. will start off this session by presenting Time’s objective to move away from on-premise and co-location data centers to AWS and the cost savings that has been realized with this transition. Chris Nicodemo from Time Inc. and Derek Uzzle from Alert Logic will then share lessons learned in the journey to secure dozens of high volume media websites during the migration, and how it has enhanced overall security flexibility and scalability. They will also provide a deep dive on the solutions Time has leveraged for their enterprise security best practices, and show you how they were able to execute their security strategy.
Who should attend: InfoSec and IT management.
Session sponsored by Alert Logic."
SRV403 Deep Dive on Object Storage: Amazon S3 and Amazon GlacierAmazon Web Services
In this session, storage experts will walk you through Amazon S3 and Amazon Glacier, bulk data repositories that can deliver 99.999999999% durability and scale past trillions of objects worldwide – with cost points competitive against tape archives. Learn about the different ways you can accelerate data transfer into S3 and get a close look at new tools to secure and manage your data more efficiently. See how Amazon Athena runs serverless analytics on your data and hear about expedited and bulk retrievals from Amazon Glacier. Learn how AWS customers have built solutions that turn their data from a cost into a strategic asset, and bring your toughest questions straight to our experts.
(SEC310) Keeping Developers and Auditors Happy in the CloudAmazon Web Services
Often times, developers and auditors can be at odds. The agile, fast-moving environments that developers enjoy will typically give auditors heartburn. The more controlled and stable environments that auditors prefer to demonstrate and maintain compliance are traditionally not friendly to developers or innovation. We'll walk through how Netflix moved its PCI and SOX environments to the cloud and how we were able to leverage the benefits of the cloud and agile development to satisfy both auditors and developers. Topics covered will include shared responsibility, using compartmentalization and microservices for scope control, immutable infrastructure, and continuous security testing.
Want to learn more about Compliance in the Cloud? Attend the AWS Compliance Summit, where key verticals such as Financial Services, Government and Public Sector, and Healthcare and Life Sciences will be discussed, along with customer use cases and prescriptive guidance from AWS subject matter experts.
Operations: Security Crash Course — Best Practices for Securing your CompanyAmazon Web Services
All companies should build with security and protection of customer data as the number one priority. This talk will cover a wide range of best practices from MFA, root accounts, encrypting laptops, inventory management, MDM, and incident response. You'll learn key principles of how to build a secure organization to protect your data. Don't wait until your first security incident before putting these best practices in place.
Innovating IAM Protection for AWS with Dome9 - Session Sponsored by Dome9Amazon Web Services
Innovating IAM Protection for AWS. Protecting your IAM users and roles is a priority for security professionals and DevOps teams alike. The challenge becomes more complex when adding multiple AWS accounts, many users, and a growing list of local and cross account roles. By utilizing an innovative IAM protection solution, you can successfully defend your AWS cloud from new threats.
In this 30 min session you will learn:
How to identify and map out potential IAM risk factors and attack vectors.
How to prevent potentially dangerous activities over your AWS accounts directly from your mobile device.
How to defend your AWS investment from compromised credentials and malicious insiders that can impact your business.
Speaker: Patrick Pushor, Chief Technical Evangelist at Dome9
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...Amazon Web Services
Learning Objectives:
- Get an overview of streaming data and it's application in analytics and big data.
- Understand the factors driving the accelerating transformation of batch processing to real-time.
- Learn how you should plan for incorporating data streaming in your analytics and processing workloads.
Business can now easily perform real-time analytics on data that has been traditionally analyzed using batch processing in data warehouses or using Hadoop frameworks, and react to new information in minutes or seconds instead of hours or days. In this webinar, Forrester analyst Mike Gualtieri and Amazon Kinesis GM Roger Barga will discuss this prevalent trend, it's business significance, and how you should plan for it. You will also learn about the AWS services that can help you get started quickly with real-time, streaming applications fore your analytics and big data workloads.
Keeping Security In-Step with Your Application Demand CurveAmazon Web Services
Protecting dynamically scaled cloud compute resources can be challenging, especially for organizations that lack the time or money it takes to maintain dynamic security. Fortinet’s auto scaling security solution addresses this issue by providing the resources to help with deployment in order to optimize organizations’ AWS networks. Join the upcoming webinar hosted by Fortinet and AWS to learn how to leverage Fortinet for auto scaling complex security policies in your Amazon VPC. Fortinet has a broad set of capabilities that when combined with AWS services creates truly a complete security architecture.
AWS offers you the ability to add additional layers of security to your data at rest in the cloud, providing access control as well scalable and efficient encryption features. Flexible key management options allow you to choose whether to have AWS manage the encryption keys or to keep complete control over the keys yourself. In this session, you will learn how to secure data when using AWS services. We will discuss data encryption using Key Management Service, S3 access controls, edge and host access security, and database platform security features.
"Running enterprise workloads with sensitive data in AWS is hard and requires an in-depth understanding about software-defined security risks. At re:Invent 2014, Intuit and AWS presented ""Enterprise Cloud Security via DevSecOps"" to help the community understand how to embrace AWS features and a software-defined security model. Since then, we've learned quite a bit more about running sensitive workloads in AWS.
We've evaluated new security features, worked with vendors, and generally explored how to develop security-as-code skills. Come join Intuit and AWS to learn about second-year lessons and see how DevSecOps is evolving. We've built skills in security engineering, compliance operations, security science, and security operations to secure AWS-hosted applications. We will share stories and insights about DevSecOps experiments, and show you how to crawl, walk, and then run into the world of DevSecOps."
AWS re:Invent 2016: Deploying and Managing .NET Pipelines and Microsoft Workl...Amazon Web Services
In this session, we’ll look at the AWS services that customers are using to build and deploy Microsoft-based solutions that use technologies like Windows, .NET, SQL Server, and PowerShell. We’ll start by showing you how to build a Windows-based CI/CD pipeline on AWS using AWS CodeDeploy, AWS CodePipeline, AWS CloudFormation, and PowerShell using an AWS Quick Start. We’ll also cover best practices for how you can create templates that let you automatically deploy ready-to-use Windows products by leveraging services and tools like AWS CloudFormation, PowerShell, and Git. Woot, an online retailer for electronics, will share how it moved from using a complex mix of custom PowerShell code for its DevOps processes to using services like Amazon EC2 Simple Systems Manager (SSM), AWS CodeDeploy, and AWS Directory Service. This migration eliminated the need for complex PowerShell scripts and reduced the operational complexity of performing operational tasks like renaming servers, joining domains, and securely handling keys.
While there are many Cloud design patterns for infrastructure, there are also many Cloud design patterns for developers. Come and learn how you can take your software design patterns and apply them to the next generation of cloud applications, or simply modernise your existing software architectures.
Speaker: Arden Packeer, Solutions Architect, Amazon Web Services
Understanding AWS Managed Databases and Analytic Services - AWS Innovate Otta...Amazon Web Services
• Overview of database services to elevate your applications, analytic services to engage your data, and migration services to help you reach database freedom.
• Survey of how Canadian and other organizations are using the cloud to make data scalable, reliable, and secure.
Join us for a series of introductory and technical sessions on AWS Big Data solutions. Gain a thorough understanding of what Amazon Web Services offers across the big data lifecycle and learn architectural best practices for applying those solutions to your projects.
We will kick off this technical seminar in the morning with an introduction to the AWS Big Data platform, including a discussion of popular use cases and reference architectures. In the afternoon, we will deep dive into Machine Learning and Streaming Analytics. We will then walk everyone through building your first Big Data application with AWS.
An Evolving Security Landscape – Security Patterns in the CloudAmazon Web Services
Availability of cloud computing is helping Financial Services organizations realize accelerated go-to-market speeds, global scalability, and cost efficiencies. This new world forces considerations for security programs – what is different in the cloud and what do I do differently? AWS Security Architects will share protocols that need to be considered in the cloud, on premises, or in a hybrid model. They will also share best practices, lessons learned, efficiencies, and design patterns and architectures unique to cloud.
The New Normal: Benefits of Cloud Computing and Defining your IT StrategyAmazon Web Services
The standard business model is changing rapidly changing. Companies used to be built for the long haul. But now, success is powered by rapid-paced innovation and the ability to get disruptive products to market first.
You’re used to balancing resources between keeping things running and the development of new initiatives. But merely keeping the lights on doesn't differentiate you from your competitors.
Amazon CloudWatch Logs and AWS Lambda: A Match Made in Heaven | AWS Public Se...Amazon Web Services
In this session, we cover three common scenarios that include Amazon CloudWatch Logs and AWS Lambda. First, you learn to build an Elasticsearch cluster from historical data using Amazon S3, Lambda, and CloudWatch Logs. Next, you learn to add details to CloudWatch alarm notifications using Amazon SNS and Lambda. Finally, we show you how to bring Elastic Load Balancing logs to CloudWatch Logs using S3 bucket triggers from Lambda. Learn More: https://aws.amazon.com/government-education/
AWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API CallsAmazon Web Services
Monitoring and making sense of infrastructure data can be an arduous process. Managing a volume of API calls from more than one million active users every minute presents an even more complex and demanding challenge. Using Amazon Web Services (AWS) and Datadog, Grindr overcame a series of infrastructure challenges by both implementing and managing highly scalable, high availability, and top performing infrastructure, as well as aggregating, analyzing, and acting on key infrastructure data KPIs.
Getting Started with Managed Services | AWS Public Sector Summit 2016Amazon Web Services
The AWS cloud infrastructure is architected to be one of the most flexible and secure cloud computing environments available today. By leveraging services such as EC2, you are able to build highly scalable and performant architectures. AWS also provides a rich set of services which help to remove much of the potentially undifferentiated heavy lifting associated to managing your EC2 based infrastructure. This session will introduce some of these services in the areas of Application Management, Database, Analytics, Security and Enterprise Applications.
(SEC203) Journey to Securing Time Inc's Move to the CloudAmazon Web Services
"Learn how Time Inc. met security requirements as they transitioned from their data centers to the AWS cloud. Colin Bodell, CTO from Time Inc. will start off this session by presenting Time’s objective to move away from on-premise and co-location data centers to AWS and the cost savings that has been realized with this transition. Chris Nicodemo from Time Inc. and Derek Uzzle from Alert Logic will then share lessons learned in the journey to secure dozens of high volume media websites during the migration, and how it has enhanced overall security flexibility and scalability. They will also provide a deep dive on the solutions Time has leveraged for their enterprise security best practices, and show you how they were able to execute their security strategy.
Who should attend: InfoSec and IT management.
Session sponsored by Alert Logic."
SRV403 Deep Dive on Object Storage: Amazon S3 and Amazon GlacierAmazon Web Services
In this session, storage experts will walk you through Amazon S3 and Amazon Glacier, bulk data repositories that can deliver 99.999999999% durability and scale past trillions of objects worldwide – with cost points competitive against tape archives. Learn about the different ways you can accelerate data transfer into S3 and get a close look at new tools to secure and manage your data more efficiently. See how Amazon Athena runs serverless analytics on your data and hear about expedited and bulk retrievals from Amazon Glacier. Learn how AWS customers have built solutions that turn their data from a cost into a strategic asset, and bring your toughest questions straight to our experts.
(SEC310) Keeping Developers and Auditors Happy in the CloudAmazon Web Services
Often times, developers and auditors can be at odds. The agile, fast-moving environments that developers enjoy will typically give auditors heartburn. The more controlled and stable environments that auditors prefer to demonstrate and maintain compliance are traditionally not friendly to developers or innovation. We'll walk through how Netflix moved its PCI and SOX environments to the cloud and how we were able to leverage the benefits of the cloud and agile development to satisfy both auditors and developers. Topics covered will include shared responsibility, using compartmentalization and microservices for scope control, immutable infrastructure, and continuous security testing.
Want to learn more about Compliance in the Cloud? Attend the AWS Compliance Summit, where key verticals such as Financial Services, Government and Public Sector, and Healthcare and Life Sciences will be discussed, along with customer use cases and prescriptive guidance from AWS subject matter experts.
Operations: Security Crash Course — Best Practices for Securing your CompanyAmazon Web Services
All companies should build with security and protection of customer data as the number one priority. This talk will cover a wide range of best practices from MFA, root accounts, encrypting laptops, inventory management, MDM, and incident response. You'll learn key principles of how to build a secure organization to protect your data. Don't wait until your first security incident before putting these best practices in place.
Innovating IAM Protection for AWS with Dome9 - Session Sponsored by Dome9Amazon Web Services
Innovating IAM Protection for AWS. Protecting your IAM users and roles is a priority for security professionals and DevOps teams alike. The challenge becomes more complex when adding multiple AWS accounts, many users, and a growing list of local and cross account roles. By utilizing an innovative IAM protection solution, you can successfully defend your AWS cloud from new threats.
In this 30 min session you will learn:
How to identify and map out potential IAM risk factors and attack vectors.
How to prevent potentially dangerous activities over your AWS accounts directly from your mobile device.
How to defend your AWS investment from compromised credentials and malicious insiders that can impact your business.
Speaker: Patrick Pushor, Chief Technical Evangelist at Dome9
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...Amazon Web Services
Learning Objectives:
- Get an overview of streaming data and it's application in analytics and big data.
- Understand the factors driving the accelerating transformation of batch processing to real-time.
- Learn how you should plan for incorporating data streaming in your analytics and processing workloads.
Business can now easily perform real-time analytics on data that has been traditionally analyzed using batch processing in data warehouses or using Hadoop frameworks, and react to new information in minutes or seconds instead of hours or days. In this webinar, Forrester analyst Mike Gualtieri and Amazon Kinesis GM Roger Barga will discuss this prevalent trend, it's business significance, and how you should plan for it. You will also learn about the AWS services that can help you get started quickly with real-time, streaming applications fore your analytics and big data workloads.
Keeping Security In-Step with Your Application Demand CurveAmazon Web Services
Protecting dynamically scaled cloud compute resources can be challenging, especially for organizations that lack the time or money it takes to maintain dynamic security. Fortinet’s auto scaling security solution addresses this issue by providing the resources to help with deployment in order to optimize organizations’ AWS networks. Join the upcoming webinar hosted by Fortinet and AWS to learn how to leverage Fortinet for auto scaling complex security policies in your Amazon VPC. Fortinet has a broad set of capabilities that when combined with AWS services creates truly a complete security architecture.
AWS offers you the ability to add additional layers of security to your data at rest in the cloud, providing access control as well scalable and efficient encryption features. Flexible key management options allow you to choose whether to have AWS manage the encryption keys or to keep complete control over the keys yourself. In this session, you will learn how to secure data when using AWS services. We will discuss data encryption using Key Management Service, S3 access controls, edge and host access security, and database platform security features.
"Running enterprise workloads with sensitive data in AWS is hard and requires an in-depth understanding about software-defined security risks. At re:Invent 2014, Intuit and AWS presented ""Enterprise Cloud Security via DevSecOps"" to help the community understand how to embrace AWS features and a software-defined security model. Since then, we've learned quite a bit more about running sensitive workloads in AWS.
We've evaluated new security features, worked with vendors, and generally explored how to develop security-as-code skills. Come join Intuit and AWS to learn about second-year lessons and see how DevSecOps is evolving. We've built skills in security engineering, compliance operations, security science, and security operations to secure AWS-hosted applications. We will share stories and insights about DevSecOps experiments, and show you how to crawl, walk, and then run into the world of DevSecOps."
AWS re:Invent 2016: Deploying and Managing .NET Pipelines and Microsoft Workl...Amazon Web Services
In this session, we’ll look at the AWS services that customers are using to build and deploy Microsoft-based solutions that use technologies like Windows, .NET, SQL Server, and PowerShell. We’ll start by showing you how to build a Windows-based CI/CD pipeline on AWS using AWS CodeDeploy, AWS CodePipeline, AWS CloudFormation, and PowerShell using an AWS Quick Start. We’ll also cover best practices for how you can create templates that let you automatically deploy ready-to-use Windows products by leveraging services and tools like AWS CloudFormation, PowerShell, and Git. Woot, an online retailer for electronics, will share how it moved from using a complex mix of custom PowerShell code for its DevOps processes to using services like Amazon EC2 Simple Systems Manager (SSM), AWS CodeDeploy, and AWS Directory Service. This migration eliminated the need for complex PowerShell scripts and reduced the operational complexity of performing operational tasks like renaming servers, joining domains, and securely handling keys.
While there are many Cloud design patterns for infrastructure, there are also many Cloud design patterns for developers. Come and learn how you can take your software design patterns and apply them to the next generation of cloud applications, or simply modernise your existing software architectures.
Speaker: Arden Packeer, Solutions Architect, Amazon Web Services
Understanding AWS Managed Databases and Analytic Services - AWS Innovate Otta...Amazon Web Services
• Overview of database services to elevate your applications, analytic services to engage your data, and migration services to help you reach database freedom.
• Survey of how Canadian and other organizations are using the cloud to make data scalable, reliable, and secure.
Join us for a series of introductory and technical sessions on AWS Big Data solutions. Gain a thorough understanding of what Amazon Web Services offers across the big data lifecycle and learn architectural best practices for applying those solutions to your projects.
We will kick off this technical seminar in the morning with an introduction to the AWS Big Data platform, including a discussion of popular use cases and reference architectures. In the afternoon, we will deep dive into Machine Learning and Streaming Analytics. We will then walk everyone through building your first Big Data application with AWS.
Best Practices Using Big Data on AWS | AWS Public Sector Summit 2017Amazon Web Services
Join us for this general session where AWS big data experts present an in-depth look at the current state of big data. Learn about the latest big data trends and industry use cases. Hear how other organizations are using the AWS big data platform to innovate and remain competitive. Take a look at some of the most recent AWS big data developments. Learn More: https://aws.amazon.com/government-education/
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)Amazon Web Services
For discovery-phase research, life sciences companies have to support infrastructure that processes millions to billions of transactions. The advent of a data lake to accomplish such a task is showing itself to be a stable and productive data platform pattern to meet the goal. We discuss how to build a data lake on AWS, using services and techniques such as AWS CloudFormation, Amazon EC2, Amazon S3, IAM, and AWS Lambda. We also review a reference architecture from Amgen that uses a data lake to aid in their Life Science Research.
Join us for a series of introductory and technical sessions on AWS Big Data solutions. Gain a thorough understanding of what Amazon Web Services offers across the big data lifecycle and learn architectural best practices for applying those solutions to your projects.
We will kick off this technical seminar in the morning with an introduction to the AWS Big Data platform, including a discussion of popular use cases and reference architectures. In the afternoon, we will deep dive into Machine Learning and Streaming Analytics. We will then walk everyone through building your first Big Data application with AWS.
AWS Summit Singapore - Architecting a Serverless Data Lake on AWSAmazon Web Services
Unni Pillai, Specialist Solution Architect, ASEAN, AWS.
Daniel Muller, Head of Cloud Infrastructure, Spuul.
As the volume and types of data continues to grow, customers often have valuable data that is not easily discoverable and available for analytics. A common challenge for data engineering teams is architecting a data lake that can cater to the needs of diverse users - from developers to business analysts to data scientists.
In this session, we will dive deep into building a data lake using Amazon S3, Amazon Kinesis, Amazon Athena and AWS Glue. We will also see how AWS Glue crawlers can automatically discover your data, extracting and cataloguing relevant metadata to reduce operations in preparing your data for downstream consumers.
Furthermore, learn from our customer Spuul, on how they moved from a Data Warehouse based analytics to a serverless data lake. Why and how did Spuul undertake this journey? Hear about the benefits and challenges they encountered.
Learn how you can point Amazon QuickSight to AWS data stores, flat files, or other third-party data sources and begin visualizing your data in minutes.
Over 90% of today’s data was generated in the last 2 years, and the rate of data growth isn’t slowing down. In this session, we’ll step through the challenges and best practices on how to capture all the data that is being generated, understand what data you have, and start driving insights and even predict the future using purpose built AWS Services. We’ll frame the session and demonstrations around common pitfalls of building Data Lakes and how to successful drive analytics and insights from the data. This session will focus on the architecture patterns bringing together key AWS Services and rather than a deep dive on any single service. We’ll show how services such as Amazon S3, Amazon Glue, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, and Amazon Kinesis, and Amazon Machine Learning services are put together to build a successful data lake for various role including both data scientists and business users.
BDA303 Serverless big data architectures: Design patterns and best practicesAmazon Web Services
Serverless technologies let you build and scale applications and services rapidly without the need to provision or manage servers. But how can you incorporate serverless concepts into your big data architectures?
In this session, we explore the key concepts and benefits of serverless architectures for big data, diving into design patterns to ingest, store, process, and visualize your data. Along the way, we explain when and how you can use serverless technologies to streamline data processing, minimize infrastructure management, and improve agility and robustness. We will share reference architectures using a combination of services that include AWS Lambda, Amazon Kinesis, Amazon Athena, Amazon QuickSight, and AWS Glue.
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Speakers:
Neel Mitra - Solutions Architect, AWS
Roger Dahlstrom - Solutions Architect, AWS
Database and Analytics on the AWS Cloud - AWS Innovate TorontoAmazon Web Services
Antoine Genereux, AWS Solutions Architect, takes us on a tour of database solutions available for the AWS Cloud, and powerful analytics and business intelligence reporting tools.
AWS Summit 2013 | Singapore - Big Data Analytics, Presented by AWS, Intel and...Amazon Web Services
Learn more about the tools, techniques and technologies for working productively with data at any scale. This session will introduce the family of data analytics tools on AWS which you can use to collect, compute and collaborate around data, from gigabytes to petabytes. We'll discuss Amazon Elastic MapReduce, Hadoop, structured and unstructured data, and the EC2 instance types which enable high performance analytics.
In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.
Data Analytics Week at the San Francisco Loft
Using Data Lakes
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Speakers:
John Mallory - Principal Business Development Manager Storage (Object), AWS
Hemant Borole - Sr. Big Data Consultant, AWS
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Level: Intermediate
Speakers:
Tony Nguyen - Senior Consultant, ProServe, AWS
Hannah Marlowe - Consultant - Federal, AWS
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
2. AWS Data Services to Accelerate Your Move to the Cloud
RDS
Open
Source
RDS
Commercial
Aurora
Migration for DB Freedom
DynamoDB
& DAX
ElastiCache EMR Amazon
Redshift
Redshift
Spectrum
AthenaElasticsearch
Service
Amazon
QuickSight
Glue
Lex
Polly
Rekognition Machine
Learning
Databases to Elevate your Apps
Relational Non-Relational
& In-Memory
Analytics to Engage your Data
Inline Data Warehousing Reporting
Data Lake
Amazon AI to Drive the Future
Deep Learning, MXNet
Database Migration
Schema Conversion
4. AWS Data Services to Accelerate Your Move to the Cloud
RDS
Open
Source
RDS
Commercial
Aurora
Migration for DB Freedom
DynamoDB
& DAX
ElastiCache EMR Amazon
Redshift
Redshift
Spectrum
AthenaElasticsearch
Service
Amazon
QuickSight
Glue
Lex
Polly
Rekognition Machine
Learning
Databases to Elevate your Apps
Relational Non-Relational
& In-Memory
Analytics to Engage your Data
Inline Data Warehousing Reporting
Data Lake
Amazon AI to Drive the Future
Deep Learning, MXNet
Database Migration
Schema Conversion
5. Multi-engine support
Open Source
Commercial
Amazon Aurora
Automated provisioning, patching, scaling, backup/restore, failover
Use with General Purpose SSD or Provisioned IOPS SSD storage
High availability with RDS Multi-AZ
Amazon RDS: Cheaper, Easier, and Better
6. Enterprise-grade fault tolerant
solution for production
databases
Automatic failover
Synchronous replication
Inexpensive & enabled with one click
High Availability Multi-AZ Deployments
7. Up To 5x Performance
Of High-end MySQL
Highly Available
and Durable
MySQL
Compatible*
1/10th The Cost Of
Commercial Grade Databases
Fastest Growing
AWS Service, Ever
Amazon Aurora: Speed and Availability of Commercial
Databases, with Cost-Effectiveness of Open Source
*PostgreSQL compatibility in Open Preview
8. BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MySQL with Replica
Storage MirrorStorage Mirror
DC 1 DC 2
StorageStorage
Primary
Instance
Replica
Instance
AZ 1 AZ 3
Primary
Instance
Amazon S3
AZ 2
Replica
Instance
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
Replica
Instance
Amazon Aurora
780K transactions
7,388K I/Os per million txns (excludes mirroring, standby)
Average 7.4 I/Os per transaction
MySQL IO profile for 30 min. Sysbench run
27,378K transactions 35X MORE
0.95 I/Os per transaction 7.7X LESS
Aurora IO profile for 30 min. Sysbench run
Aurora, Faster Because it is Built for AWS
9. DynamoDB: Non-Relational
Managed NoSQL Database Service
Schemaless data model
Consistent low latency performance
Predictable provisioned throughput
Seamless scalability with no storage limits
High durability & availability (replication across 3 facilities)
Easy administration – we scale for you!
Low cost
DynamoDB
DAXApp
DynamoDB Accelerator (DAX) offers caching
without coding for sub-millisecond read
latency and up to 10x throughput
10. DynamoDB at Amtrak
Built and deployed an operational database and data mart
for near-real-time reporting of sales data
Developed and released the solution in six months
Used cloud native technologies: DynamoDB, Kinesis,
Lambda, and S3
Benefits
Improved accuracy and single source of truth for sales data
Allows decommissioning of four legacy systems
Low maintenance and operational costs. No servers to manage.
11. Make Almost Any Database Faster
and Less Expensive
In-Memory Cache
Memcached and Redis
Fully managed
High Speed In-Memory Data Store
Persistent high availability
Clusters up to 3.5TB
Average read and write time of
under 500µs (0.5ms)
Amazon ElastiCache Provides Sub-millisecond
Caching and In-Memory Data
12. AWS Data Services to Accelerate Your Move to the Cloud
RDS
Open
Source
RDS
Commercial
Aurora
Migration for DB Freedom
DynamoDB
& DAX
ElastiCache EMR Amazon
Redshift
Redshift
Spectrum
AthenaElasticsearch
Service
Amazon
QuickSight
Glue
Lex
Polly
Rekognition Machine
Learning
Databases to Elevate your Apps
Relational Non-Relational
& In-Memory
Analytics to Engage your Data
Inline Data Warehousing Reporting
Data Lake
Amazon AI to Drive the Future
Deep Learning, MXNet
Database Migration
Schema Conversion
13. Amazon EMR: the Hadoop and Spark Ecosystem,
Without the Chaos
Design Patterns
Amazon S3 as HDFS
Core Nodes and Task Nodes
Elastic Clusters
Transient + Always On Clusters
Leverage the Hadoop ecosystem
Use Cases
Recommendation Engines
Personalization Engines
Semi-structured/unstructured data
Combine disparate data sets
Next generation ETL
Sentiment analysis
Batch analytics
Taming Big Data in the Cloud
Hadoop, Spark, Presto, Hive and more
Easy to use, fully managed
Launch a cluster in minutes
Baked in security features
Pay by the hour and save with Spot
14. Amazon Elasticsearch Service
Log Analytics &
Operational Monitoring
Monitor the performance of your
apps, web servers, and
infrastructure
Easy to use, yet powerful data
visualization tools to detect issues in
near real-time
Ability to dig into your logs in an
intuitive, fine-grained way
Kibana provides fast, easy
visualization
Search
Application or website provides search
capabilities over diverse documents
Tasked with making this knowledge
base searchable and accessible
Key search features including text
matching, faceting, filtering, fuzzy
search, auto complete, and highlighting
Query API to support application search
15. Amazon Redshift: Cloud Data Warehousing
Leader Node
Simple SQL endpoint
Stores metadata
Optimizes query plan
Coordinates query execution
Compute Nodes
Local columnar storage
Parallel/distributed execution of all queries,
loads, backups, restores, resizes
Up to 2 petabytes of managed data
Automated ingestion from S3, Kinesis,
EMR and DynamoDB
Leader
Node
Compute Nodes
S3 EMR DynamoDB EC2
16. Large Data Lakes: PB and XB
Run SQL queries directly against data in S3
Fast @ exabyte scale Elastic & highly available
On-demand, pay-per-queryHigh concurrency: Multiple
clusters access same data
No ETL: Query data in-place
using open file formats
Full SQL support
S3
SQL
Amazon Redshift Spectrum
Run SQL queries directly against
data in S3 using thousands of nodes
Amazon Athena
Serverless interactive query service
Query an Exabyte of data in
under 3 minutes
17. Data Catalog
Hive metastore compatible metadata repository of data sources
Crawls data source to infer table, data type, partition format
Job Execution
Runs jobs in Spark containers – automatic scaling based on SLA
Glue is serverless – only pay for the resources you consume
Job Authoring
Generates Python code to move data from source to destination
Edit with your favorite IDE; share code snippets using Git
AWS Glue for Automated, Serverless ETL
18. Amazon QuickSight: Fast Business Analytics
Data from Many Sources
AWS Managed Databases
Amazon S3
Databases on Amazon EC2
On-premises databases
Excel and CSV Files
Salesforce and other SaaS
Mobile and Web Access
iPhone, Android and Tablet
Most popular web browsers
Powered by SPICE
Super-fast, Parallel, In-memory Calculation Engine
Run fast interactive queries on large datasets
Low monthly cost per user
19. Old-World Vendors and Old-World Policies…
You’ve Got
Mail!
AUDIT
Very Expensive Proprietary Lock-In Punitive
Licensing
Unshackle From
H stile Database Vendors
20. Freedom Begins with Choice; Migrating Data and Schema
AWS Schema Conversion Tool
Automatically convert & move tables,
views, stored procedures, metadata
Highlights and recommends custom
actions as needed
AWS Database Migration Service
Start a migration in literally a few minutes
Keep apps running during the migration
Replicate from, within, or to Amazon EC2 or
managed database services or on-premises
0
1
2
3
4
5
WorkloadQualification
Framework Assess workloads by
complexity, technology,
effort, and other factors
Recommends strategy
and plans for migration
AWS Workload Qualification Framework
21. Heterogeneous Migration
Oracle private DC to RDS PostgreSQL migration
Used the AWS Schema Conversion Tool (AWS SCT) to convert their
database schema
Used on-going Change Data Capture (CDC) replication to keep
databases in sync until they reached the cutover window
Benefits
Improved reliability of the cloud environment
Savings on Oracle licensing costs
SCT Assessment Report showed the scope of the migration
24. FINRA: Data Sharing Pre-Cloud
Built a data hub of to deal with growing problem of point-to-
point dependencies between databases in the data center.
FINRA data center
App 1 DB
App 2 DB
App 3 DB
App N DB
HUB DB
FINRA data center
App 1 DB
App 2 DB
App 3 DB
App N DB
31. FINRA: Analytics Impacts
• Removed obstacles
“Before data analysis of this magnitude required intervention from technology.”
“We are now able to see underlying data and visual representation of summaries together
with outliers and anomalies. This reduces our time to market on examinations.”
“We moved away from requesting raw reports to requesting dashboards that provide
meaningful information and tell a story…”
• Lowered the cost of curiosity
“Analysts are able to quickly obtain a full picture of what happens to an order over time,
helping to inform decision making as to whether a rule violation has occurred.”
“[W]ith a click we can now compare firms of our choice or defined peer groups. This helps
use by reducing a lot of noise…”
“Using machine learning algorithms validates our assumptions and makes us data driven”
• Optimize batch and interactive workloads without compromise
• Greater innovation and more engaged staff
21
I lead the AWS DB, Analytics, and Artificial Intelligence Customer Programs team.
Joining me is Scott Donaldson, a Sr. Dir. of Tech at Financial Industry Regulatory Authority, or FINRA.
FINRA is an independent, non-govt regulator for all securities comp’s that do biz with the investor public in US - 90 mill’n of us
FINRA reconstructs the market from tens of billions of events, detecting wrongdoing, & then the consequences from that.
Scott was a principal in moving FINRA’s DB and analytics onto the cloud over the last 3 years, a strong statement.
I appreciate his teamwork with me today.
gI will start with how AWS offers a broad portf. of Data Svcs across DB, Analytics, & AI to help cust. accelerate Your Move to the cloud.
At AWS > 90% of new features & Svcs are driven by cust f’back, and most requests are driven by a desire to make app experiences better. Better includes faster to deploy & run, more responsive & reliable, and easier & less expensive to operate.
For DB, we have a strong mix of rel. Open Source and Commer. Engine choices, and Amzn-created Aurora. For the highest scalability & perf. we have non-rel. Amazon-created DynamoDB and Open Source choices.
Moving right, our cust use a range of analytics Svcs to Engage your Data. I will talk about a some popular Open Source-based Svcs. And many cust. are migrating from DW appliances to on-demand Cloud services w/decoupled data, using Redshift and Redshift Spectrum. I’ll also cover integ. Svcs for Extract-Transform & Load, serverless query, & visualization.
You might know that AI is also a growing part of the AWS Data portf., with deep learning engines, an ML platforms, and Svcs to enable dev’s to easily add voice and pictorial capabilities to applications.
Completing this portfolio view, our cust use migration for a broad range of use cases across modernization of existing apps and moving between DB engines, and both for rel. and non-rel. DB and analytics Svcs.
Looking at momentum today, more than 2,300 gov’t agencies, 7,000 education institutions, and more than 22,000 nonprofit organizations are using our Svcs.
Here are the logos of some of the Public Sector agencies and orginaizations s using AWS DB, analytics, & AI services.
You might recognize severof them.
I will first going deeper in our DB portf., both Rel. and Non-Rel. and In-Memory
For rel DB, our foundation is Amzn’s Rel Data Svcs. RDS is a TCO win, and is Easier, and Better - providing cust. ever-inc. choice, autom. & innov.
5 DB eng. are avail w/RDS, the Open Source eng. of MySQL, PostgreSQL, and MariaDB; & the commercial eng. of Oracle RDBMS and Msft SQL Server. For even higher levels of reliability, scalability & perf, you can also use Amzn Aurora w/MySQL compat & PG compat in open preview.
For RDS commercial, you can bring your own licenses or rent them by-the-hour from AWS. The others are provided as part of our services.
For every DB engine, RDS automates the heavy lifting, including maint. high perf and HA. Inadequately managed DB are a leading cause of downtime in IT, and lost sleep.
For the typical DB team, perhaps 25% of the time goes to infra, 65% to mgmt., and only 10% to biz and app optimization. With DB from AWS, you can get more leverage from your teams and focus in areas that differentiate you, while Amazon takes care of the infras and DB mgmt.
On the final point, we have built-in HA and Cross Region replication across multiple availability zones and geographic regions. These are for all engines, including standard editions, not just for Enterprise editions, with over 99.95% availability.
Our experience running Amazon.com taught us what it takes to manage and operate relational DB with HA. We bring that learning to you.
On HA, to give sense of how simple it is, creating fault tol. w/RDS is just checking a box. AWS does to rest the create a highly reliable, auto failover system with replication.
If a primary fails, the application keeps running while the standby gets promoted to primary and new standby is created.
Now I want to talk about Amazon Aurora, which brings you (title).
Most rel. DB technologies were built to scale vert. and assume a fairly monolithic arch for each instance.
Making that type of arch perform at high Xactional volumes while providing HA and data durability is non-trivial.
It often takes a lot of engineering work and maintenance to sustain the target levels of perf, avail, and durability.
To solve that problem AWS built Aurora – a fully managed rel. DB taking full advantage of Amazon’s cloud infra.
Aurora is compat. with MySQL schema now, and compat with PostgreS in Open Preview is particularly interesting to Oracle B/C semantic compatibility of PostgreSQL with Oracle makes migration easier.
It has up to 5x times the perf of high-end MySQL, HA, and 1/10 the cost of commercial DB.
Since release in July 2015 Aurora is the fastest growing AWS service. Compelling to get ent class perf without having to purchase an expensive 3rd party solution. It is completely consumption based and has no license costs.
Here is an illustration of why Aurora performs so much better than native MYSQL. The key is in how Aurora uses AWS infra to its advantage.
With a typical DB you need to write everything to a disk which is mirrored to another disk for durability.
Then all of that data must be replicated to another instance where it is written to disk, twice, in order to achieve HA.
With Aurora the log data is the only thing that must be written to disk.
It is asynchronously written in a dist. manner 6 times, but 4 successful writes is a quorum for a successful write.
Data is then asynchronously written to S3 for durability/backup purposes.
Aurora keeps 3 instances in synch for failover purposes and for offloading read traffic.
The result is a dramatic increase in throughput and reduction in I/O while providing superior avail & durability.
Stepping over from Rel. BD to NoSQL DB, AWS built DynamoDB to provide very stable low latency perf. regardless of how many Xactional requests you might need to support. As with many of our services we provide durability by replicating across 3 physical facilities.
Provisioning DynamoDB is simple – you essentially define your table name, set up your key fields, and the R&W throughput you need. We scale the back end for you. If your needs change, you can change the R&W traffic you require and we will dynamically adjust the infra to meet your needs.
To provide further perf enhancements we recently announced the Dynamo DB Accel., or DAX. DAX adds a caching mechanism that dramatically increases perf.
You can enable a DAX cluster for a DynamoDB tables and offload read traffic from the DB. For workloads that repeatedly read the same key you can get up to 10X perf.
It is completely API compatible with Dynamo DB – you enable it by routing API calls to DAX instead of directly to Dynamo. No coding required. DAX is in Preview in US East 1 and US West 2, and eu-West-1 in Ireland.
Now lets look at a customer example on Dynamo…..
Amtrak teamed up w/Deloitte to overhaul their sales DW. It was built using typical legacy tech that was becoming increasingly expensive to maintain. And it was also not providing them the accuracy and speed they were looking for.
They went from concept to deployment in 6 mos. They leveraged AWS managed Svcs of DynamoDB, Kinesis, Lambda and S3 for the entire arch with DynamoDB as the core Svc to stores the data.
The result is a solution that will save Amtrak significant operational costs and that allows them to retire four legacy apps.
The new solution is much simpler to maintain because there are no servers to manage, and it also increases the accuracy of the data and provides it in near real time instead of daily batch loads.
If you want to learn more about this solution, they have a separate session at this summit this afternoon.
To complete our non-rel. DB portf, ElastiCache provides an in-memory cache for apps. It provides an AWS managed implementation of Memcached or Redis.
You can think of it a bit like a “shock absorber” for your DB. It functions similar to DAX, and will work with just about any DB, integrated into the persistence layer of an app.
It can speed up read heavy workloads to provide sub-millisecond response while reducing load on expensive DB resources.
And that concludes our tour through the DB portion of the AWS Data portfolio….
We will talk next about Analytics.
In addition to a comprehensive set of DB solutions, AWS provides a rich set of tools for big data and Analytics
These can be provisioned dynamically to allow you to quickly begin analyzing very large volumes of data.
To start with, Elastic Map Reduce has been in our analytics port. the longest of any tool. We launched it in 2009.
Hadoop, Spark, Presto, and Hive are natural fits for the cloud because you can apply vast resources to a data analytics problem quickly. So, we built a managed service to make it easy for customers to use the cloud for big data analytic purposes.
You might use EMR to analyze unstruct. or semi-struct. data sets, mult. data sets in various formats, for large scale batch anal. jobs, or ETL at scale.
EMR provisions a fully mang’d infras in a few mouse clicks. You can quickly launch clusters, then use tools of your choice from the Hadoop Ecosys.
You can have clusters that are persistent in nature that stay on 24X7, or you can have clusters that are transient in nature – just shut down the cluster when you are done and you stop paying for it.
For very large data volumes, EMR works seamlessly with S3. EMR can mount S3 as a FS to ingest data into the cluster and utilize tech such as Presto to query data directly in S3. This allows you to scale to access virtually limitless amounts of data.
EMR can take advantage of all of the EC2 pricing models such as reserved instances and spot instances to reduce cost.
Next, Elasticsearch Service is a relatively new addition to our analytics portfolio.
It is a managed version of the Open Source product of the same name and also includes Kibana, and integrates with Logstash.
It also integ. with many other AWS svcs incl. AWS IoT, AWS Kinesis Firehose, and also connectors for S3, CloudWatch, and CloudTrail.
With this service you can quickly stand up an ELK stack to ingest and analyze large volumes of data.
Typical use cases include analytics on log files with Kibana built in to quickly build dashboards to visualize data.
Another use area is for application search. Elasticsearch ingests your data and indexes it for easy searching.
Now we will talk about Amazon Redshift. Like Hadoop, DW is a natural fit for cloud computing. The ability to scale out across many compute nodes to query and analyze data is our sweet spot. Redshift is a Massively Parallel Processing, or MPP, DW that uses columnar storage which makes it optimal for analytics processing. There are many DW solutions on the market, many of them HW based, but Redshift is different
1st, Redshift is fully elastic – you can dynamically provision a warehouse from <1TB to 2PB in size, and can grow and shrink elastically with your data volumes. If you need more capacity, just go into the console and change the size of the cluster. We will build a new cluster, copy your data over and switch over to the new cluster for you.
2nd, Redshift is completely utility-based in its pricing – there are no licenses to buy, you pay for what you use. Similar to EMR, you can have persistent warehouses that are up 24X7 or have transient warehouses that you use for a short period of time and then bring down.
3rd it is fully managed, we do the backups, resizing, patching, etc. for you. You just provision it, load your data and start analyzing your data.
Redshift is available with several instance types, ranging from under $1000/TB/year using DS2 nodes with mag storage to 10x the perf of other DW options using DC1 nodes with SSD storage.
We just covered tools to analyze many TB or even PB of data. But we know sometimes you want to query and analyze at an even higher scale.
Ingesting many PB into a Hadoop cluster or DW can quickly become impractical, so AWS recently launched 2 tools to help with this problem – Amazon Athena and Amazon Redshift Spectrum. They do similar things in different ways.
Athena is a GP serverless query engine for S3. It uses Presto inside to do rel. queries against data in S3 in struct formats such as JSON, Parquet or Comma-separated values.
Using Athena is straightforward. You create table definitions for data stored in S3, then use your favorite BI or analytics tool to query & analyze the data. Beyond storing the data, you only pay for the volume of data you scan during the query.
Next, Spectrum is a new Svc that extends Redshift beyond the data stored in Redshift to query a data lake in S3.
Unlike Athena, you provision a cluster of servers that Redshift uses to query & join data in S3 w/data stored w/in the Redshift Cluster.
With Spectrum we were able to query an Exabyte of book sales data stored by Amazon.com to do analytics on projected sales of a given book title in under 3 minutes. So with these tech’s, you can scale to handle massive volumes of data without having to buy a DC full of HW.
Now I’ll talk about AWS Glue, a new service that is still in preview. It is a serverless ETL Svc.
It creates a Data Catalog across a metadata repository by crawling your data sources to understand formats & relationships
To Author Jobs, glue generates Python code (which you can then edit) to move or Xform the data from source to destination
Job Execution is done in Spark containers, and we ramp up the number of servers to meet your SLA.
Because it is serverless, you just pay for what you use.
And for reporting, Amazon Quicksight is a cloud-based biz analytics tool you can think of as the AWS UI into the data in the DB & analytic toolsets.
Quicksight understands a wide variety of DB formats, whether they are in the cloud or on prem. It also understands formats such as Excel and CSV, and it can query Athena or even SaaS vendors such as Salesforce.
It can query the data directly from the source or load data into SPICE – QuickSight’s in memory query engine.
Literally within minutes you can connect to a datasource and start visualizing and analyzing data.
QuickSight allows you to save queries and analysis into dashboards which can be shared and also has a mobile client.
And that wraps up our DB and Analytics portfolio.
Want to shift to a theme we hear a lot from DB cust: That is they like what they hear from us a lot more than what they are used to.
They want to be free from the “old-world” policies – about economics, choice & the ability to embrace leading edge innovation, & biz practices.
Old-world DB vendors are very expensive, proprietary, design for lock in, have punitive licensing terms.
And too often send untimely email that you're being audited
Many customers tell us they have just had enough of this.
In 2015 we announced our DB migration Svc. Since we opened it up the beginning of this year, over 28K DB have migrated.
DMS let you migrate to, from, or within the cloud safely & securely. And the source DB keeps running while the data is copied.
It takes only minutes to set up, and it runs at very low cost. There are dozens of DB supported as sources & targets, including most widely-used commercial and Open Source DB, like MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, Redshift, DynamoDB, MongoDB, and others.
When you want to change your DB from one engine to another, the Schema Conversion Tool, or SCT, creates an assessment report to guide you on your migration. Then SCT will read the metadata from your source and automatically convert to the right format for your target.
SCT is free and also non-disruptive, so it makes it easy to determine how easy it will be to move your DB.
As our customers examine their options for DB freedom, they can use our Workload Qualification Framework to look at each workload, assess the complexity and help determine which workloads are easiest to migrate, and which may need extra effort. Where WQF indicates higher complexity, AWS has programs that can help provide technology and expertise to make the migration easier and minimize risks.
Now lets look at a customer example of DMS and SCT…..
Located in the UK, Trimble is a global leader in geolocation svcs.
Their gov’t, health care, construction, & other clients depend upon Trimble to help know where critical items are located when needed.
Trimble’s Oracle solution was built over a decade ago. It was no longer flexible nor cost-effective for Trimble’s growing needs.
Using SCT and DMS, Trimble migrated their key apps from Oracle to RDS for PostgreSQL. In 6 weeks, for £40,000, or about US$55,000.
Immediate annual savings for Trimble of US$165,000, while allowing the app to grow to meet needs without requiring expensive licensing.
Completing our Portfolio view, there are other sessions at this summit where we will be diving deeper into AI. I just want to make a few points.
At Amazon, we’ve been using AI to better serve our cust for over 20 years. It’s a key part of our ops, from AWS to logistics, to new svcs like Alexa.
We anticipate a few years from now AI will be bigger than the rest of AWS combined.
Cloud computing, new low-cost high-speed processors, and new prog. frameworks combine to move AI from science fiction into everyday reality.
The AWS approach is to support a range of Svcs for our customers’ differing needs. Starting at the bottom.
Engines are for use by AI eng’rs, implementing new W/L. We’re contributing to the Apache MXNet project, and we also support a range of engines.
Platforms are for Data Scientists and Data Analysts, allowing the application of algorithms to data without requiring deep AI expertise.
On the top, Svcs let Devs to add facial recog, foto tagging, nat lang underst, text2voice, & other capab to apps w/o need’g any AI expertise.
And that covers our DB, analytics, and AI Portfolio. Now I want to ask Scott to talk about FINRA’s use of this portfolio, followed by a Q&A session.
The Financial Industry Regulatory Authority is an independent, non-governmental regulator for all securities firms during business with the public in United States.
FINRA’s 3-D mission is investor protection and market integrity. We protect 90 million investors every single day! Everyday we work to:
deter misconduct by enforcing the rules;
detect and prevent wrongdoing in the U.S. markets; and
discipline those who break the rules.
We get billions of trades, quotes and order data daily.
We conduct surveillance and monitor majority of the equities and options markets in the United States.
The point to points were proving to be problematic in terms of
Potential performance impacts on operational stores from heavy external query usage
Tight coupling of SLAs
If system A had an SLA of 8 – 8 and depended on System B which only had an SLA of 9 – 5 what do you do for planned maintenance
So we moved to a Hub to decouple the systems
Main Idea:
Challenge to support migration from on premise to the cloud. Need a data migration/replication services. Built this on top of AWS Data Migration Services (DMS) to expedite application migrations
Challenge:
Move our RDBMS-backed applications in our data center to the cloud on RDS PostgreSQL
Rely heavily on a data sharing hub in the center leveraging materialized views to share sets of data between applications
We want to support applications executing on their migration independent of their upstream / downstream partner’s cloud migration schedules while minimizing rework
Problems
We can’t expect 100+ databases and applications to move to the cloud all at once at the same time.
We need a way to try and provide a way for applications and databases to move without forcing their upstream/downstream dependencies to move with them or make coordinated changes.
Solution:
Automation / lights-out operations is first class consideration for FINRA
So our implementation is exposed through an API that wraps the DMS API
API covers DMS actions as well as non-DMS actions
Allows our partners to include as part of their deployment and go through normal SDLC
Business Challenges:
Exchanges are dynamically evolving
Regulatory landscape is changing
Market manipulators innovating
Normalizing data sets for common analytics and keep fidelity of original data
Main Idea:
AWS cloud has allowed us to separate infrastructure services from analytic services
Standardized infrastructure services
Focus greater investment on value add analytics rather than time consuming infrastructure items
Where We Started:
HBase – fast, interactive fetches based on keys
Hive & Tez - Batch analytics for surveillance patterns and creation of data marts for interactive exploration
Redshift – Data Marts & Interactive queries using power of MPP and Columnar access
RDS – workflow transactions, preferences, history, etc.
Where We Are Now & Where We are Going:
EMR:
Hive (decreasing)
Presto (increasing)
planning to move to Athena (service rather than own cluster)
Spark (increasing)
Batch analytics using Scala & Spark SQL
Interactive analytics for fast access to cached data setts
HBase (increasing)
Moved from static clusters to highly resilient Hbase on S3
Redshift (maintain)
Plan to move towards Spectrum as we are storage bound not compute
RDS (increasing)
Remainder of portfolio now moving to the cloud
Data Science/Machine Learning (increasing)
MetaStore (increasing)
Plan to look at Glue as complementary or alternative to our own Metastore
Main Idea:
AWS analytic services allows us to flexibility to customize interactive analytics
Use Cases:
Interactive Summaries
Market Reconstruction
Audit Trail
Data Science (e.g. regression analysis for purposeful sampling to hone firm examinations)
AWS Analytics Tech Stack:
EMR: Presto, Spark, Hive
Redshift: data warehouse and data marts created from EMR queries (Hive & Presto)
RDS: History, Preferences, etc.
Moving towards use of Spectrum, Athena, Glue, QuickSite/SPICE, Notebooks
R 3.2.5, Python (2.7.12 and 3.4.3)
Packages
R: 300+ Python: 100+
Tools for Building Packages
gcc, gfortran, make, java, maven, ant…
IDEs
Jupyter, RStudio Server
Deep Learning
CUDA, CuDNN (if GPU present)
Theano, Caffe, Torch
TensorFlow
Where we started:
HBase – fast, interactive fetches based on keys on static cluster
Hive – Batch analytics for surveillance patterns and creation of data marts for interactive exploration
Redshift – Data Marts & Interactive queries using power of MPP and Columnar access
RDS – workflow transactions, preferences, history, etc.
How We Evolved
EMR:
HBase on S3
Hive & Tez (decreasing)
Presto (increasing)
Spark (increasing)
Batch analytics using Scala & Spark SQL
Redshift (maintain)
Universal Data Catalog
MetaStore (increasing)
Data Science/Machine Learning (increasing)
Now and Future
Athena
Use service instead of own cluster
EMR:
Spark (increasing)
Batch analytics using Scala & Spark SQL
Interactive analytics for fast access to cached data setts
Spectrum
Plan to move towards Spectrum as we are storage bound not compute
RDS (increasing)
Remainder of portfolio now moving to the cloud
Data Science/Machine Learning (increasing)
Glue
Look as complement or alternative to Metastore
QuickSite / BI
More advanced interactive analytics with SPICE