The document provides an overview of key areas to review for production readiness including architecture design, monitoring, logging, documentation, alerting, service level agreements, expected throughput, testing, and deployment strategy. It summarizes best practices and considerations for each area such as using circuit breakers in monitoring, consistent logging formats, storing documentation near code, automating level 1 operations, and strategies for testing, deployments, and managing error budgets.
Yazid Boutejder: AWS San Francisco Startup Day, 9/7/17
Operations: Production Readiness Review – how to stop bad things from happening - There is more to deploying code than pushing the deploy button. A good practice that many companies follow is a Production Readiness Review (PRR) which is essentially a pre-flight check list before a service launches. This helps ensure new services are properly architected, monitored, secured, and more. We’ll walk through an example PRR and discuss the value of ensuring each of these is properly taken care of before your service launches.
Amazon Athena is a new serverless query service that makes it easy to analyze data in Amazon S3, using standard SQL. With Athena, there is no infrastructure to setup or manage, and you can start analyzing your data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3.
AWS-powered services for analytics can handle the scale, agility, and flexibility required to combine different types of data and analytics approaches that will allow you to transform your data into a valuable corporate asset. In this session, AWS will provide an overview of the different AWS services available for your data analytics needs. You can combine these blocks to build data flows that will extend your organization’s agility, ability to derive more insights and value from its data, and capability to adopt more sophisticated analytics tools and processes as your needs evolve. In the second part of the session, Paddy Power Betfair’s Data team will discuss the adoption and large scale operation of a broad range of AWS services that make up PPB’s scalable, mixed workload, multi-brand data platform. The data capabilities developed by PPB and powered by AWS were implemented to enable low-latency, high-volume and near real-time advanced analytics use cases, in the highly regulated and fast-paced betting industry. This was only possible through a focus on automation, innovation and continuous improvement.
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...Amazon Web Services
In this session, we discuss architectural principles that helps simplify big data analytics.
We'll apply principles to various stages of big data processing: collect, store, process, analyze, and visualize. We'll disucss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on.
Finally, we provide reference architectures, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
by Joyjeet Banerjee, Solutions Architect, AWS
Amazon Athena is a new serverless query service that makes it easy to analyze data in Amazon S3, using standard SQL. With Athena, there is no infrastructure to setup or manage, and you can start analyzing your data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3. Level 200
In this session, we will show you how easy it is to start querying your data stored in Amazon S3, with Amazon Athena. First we will use Athena to create the schema for data already in S3. Then, we will demonstrate how you can run interactive queries through the built-in query editor. We will provide best practices and use cases for Athena. Then, we will talk about supported queries, data formats, and strategies to save costs when querying data with Athena.
AWS delivers an integrated suite of services that provide everything needed to quickly and easily build and manage a data lake for analytics. AWS-powered data lakes can handle the scale, agility, and flexibility required to combine different types of data and analytics approaches to gain deeper insights, in ways that traditional data silos and data warehouses cannot. In this session, we will show you how you can quickly build a data lake on AWS that ingests, catalogs and processes incoming data and makes it ready for analysis. Using a live demo, we demonstrate the capabilities of AWS provided analytical services such as AWS Glue, Amazon Athena and Amazon EMR and how to build a Data Lake on AWS step-by-step.
Yazid Boutejder: AWS San Francisco Startup Day, 9/7/17
Operations: Production Readiness Review – how to stop bad things from happening - There is more to deploying code than pushing the deploy button. A good practice that many companies follow is a Production Readiness Review (PRR) which is essentially a pre-flight check list before a service launches. This helps ensure new services are properly architected, monitored, secured, and more. We’ll walk through an example PRR and discuss the value of ensuring each of these is properly taken care of before your service launches.
Amazon Athena is a new serverless query service that makes it easy to analyze data in Amazon S3, using standard SQL. With Athena, there is no infrastructure to setup or manage, and you can start analyzing your data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3.
AWS-powered services for analytics can handle the scale, agility, and flexibility required to combine different types of data and analytics approaches that will allow you to transform your data into a valuable corporate asset. In this session, AWS will provide an overview of the different AWS services available for your data analytics needs. You can combine these blocks to build data flows that will extend your organization’s agility, ability to derive more insights and value from its data, and capability to adopt more sophisticated analytics tools and processes as your needs evolve. In the second part of the session, Paddy Power Betfair’s Data team will discuss the adoption and large scale operation of a broad range of AWS services that make up PPB’s scalable, mixed workload, multi-brand data platform. The data capabilities developed by PPB and powered by AWS were implemented to enable low-latency, high-volume and near real-time advanced analytics use cases, in the highly regulated and fast-paced betting industry. This was only possible through a focus on automation, innovation and continuous improvement.
Big Data Analytics Architectural Patterns and Best Practices (ANT201-R1) - AW...Amazon Web Services
In this session, we discuss architectural principles that helps simplify big data analytics.
We'll apply principles to various stages of big data processing: collect, store, process, analyze, and visualize. We'll disucss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on.
Finally, we provide reference architectures, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
by Joyjeet Banerjee, Solutions Architect, AWS
Amazon Athena is a new serverless query service that makes it easy to analyze data in Amazon S3, using standard SQL. With Athena, there is no infrastructure to setup or manage, and you can start analyzing your data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3. Level 200
In this session, we will show you how easy it is to start querying your data stored in Amazon S3, with Amazon Athena. First we will use Athena to create the schema for data already in S3. Then, we will demonstrate how you can run interactive queries through the built-in query editor. We will provide best practices and use cases for Athena. Then, we will talk about supported queries, data formats, and strategies to save costs when querying data with Athena.
AWS delivers an integrated suite of services that provide everything needed to quickly and easily build and manage a data lake for analytics. AWS-powered data lakes can handle the scale, agility, and flexibility required to combine different types of data and analytics approaches to gain deeper insights, in ways that traditional data silos and data warehouses cannot. In this session, we will show you how you can quickly build a data lake on AWS that ingests, catalogs and processes incoming data and makes it ready for analysis. Using a live demo, we demonstrate the capabilities of AWS provided analytical services such as AWS Glue, Amazon Athena and Amazon EMR and how to build a Data Lake on AWS step-by-step.
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
Highlights of AWS ReInvent 2023 in Las Vegas. Contains new announcements, deep dive into existing services and best practices, recommended design patterns.
AWS App Mesh (Service Mesh Magic)- AWS Container Day 2019 BarcelonaAmazon Web Services
In this session, learn about how AWS App Mesh can help give you end-to-end visibility and manage traffic routing to ensure high availability for your microservice. We will cover what the need for a service mesh, capabilities of App Mesh, and show you a demo.
In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum and optimizing your overall capital expense can be challenging. This session presents AWS features and services along with disaster recovery architectures that you can leverage when building highly available and disaster-resilient strategies.
Amazon Kinesis Analytics is the easiest way to process streaming data in real time with standard SQL without having to learn new programming languages or processing frameworks. Amazon Kinesis analytics enables you to create and run SQL queries on streaming data so that you can gain actionable insights and respond to your business and customer needs promptly. In this session, we will provide an overview of the capabilities of the Amazon Kinesis Analytics. We will show you how you can build an entire stream processing pipeline to collect, ingest, process, and emit streaming data using Amazon Kinesis Analytics, Amazon Kinesis Firehose, and Amazon Kinesis Streams.
In this session we will introduce key ETL features of AWS Glue and cover common use cases ranging from scheduled nightly data warehouse loads to near real-time, event-driven ETL flows for your data lake. We will also discuss how to build scalable, efficient, and serverless ETL pipelines.
Choose Right Stream Storage: Amazon Kinesis Data Streams vs MSKSungmin Kim
This presentation compares Amazon Kinesis Data Streams to Managed Streaming for Kafka (MSK) in both architectural perspective and operational perspective. In addition, it shows common architectural patterns: (1) Data Hub: Event-Bus, (2) Log Aggregation, (3) IoT, (4) Event sourcing and CQRS.
Managing Container Images with Amazon ECR - AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- How Amazon ECR Lifecycle Policies work to lower costs and reduce image sprawl
- How to configure and test rules for automated image cleanup
- Best practices for getting started using Lifecycle Policies today
AWS CloudFormation: Infrastructure as Code | AWS Public Sector Summit 2016Amazon Web Services
This session provides the attendee with an overview of our AWS CloudFormation service and helps the customer to realize the benefits of "infrastructure as code." A demo is part of this session.
Need to start querying data instantly? Amazon Athena an interactive query service that makes it easy to interactive queries on data in Amazon S3, using standard SQL. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing your data immediately.
In this presentation, we will show you how Amazon Athena makes it easy it is to query your data stored in S3
Introduction to Amazon Kinesis Firehose - AWS August Webinar SeriesAmazon Web Services
Streaming data applications can deliver compelling, near real-time user experiences, but building the back-end infrastructure to collect and process streaming data is difficult. Amazon Kinesis Firehose makes it easy for you to load streaming data into AWS without having to build custom stream processing applications. In this webinar, we will introduce Amazon Kinesis Firehose and discuss how to ingest streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service using Amazon Kinesis Firehose. We will also highlight key use cases based on real-world examples from IoT, AdTech, E-Commerce, and Gaming. Join us to: - Get an introduction to streaming data and an overview of Amazon Kinesis Firehose - Learn about common streaming data use cases from IoT, Ad Tech, E-Commerce, and Gaming - Understand how to use Amazon Kinesis Firehose to load streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service Who should attend: Developers, data analysts, data engineers, architects
AWS Cost Management Workshop at the San Francisco Loft
AWS offers a number of products that allow you to access, organize, understand, optimize, and control your AWS costs and usage. This workshop will help you get started using AWS Cost Explorer to visualize your usage patterns and identify your underlying cost drivers. From there, you can take action on your insights by learning how to set custom cost and usage budgets and receive alerts via email or Amazon SNS topic using AWS Budgets.
Amazon API Gateway and AWS Lambda: Better TogetherDanilo Poccia
Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. Together they help you build a server-less event-driven backend that is easy to manage and scale.
Building Cloud-Native App Series - Part 3 of 11
Microservices Architecture Series
AWS Kinesis Data Streams
AWS Kinesis Firehose
AWS Kinesis Data Analytics
Apache Flink - Analytics
by Trevor Sullivan, Solutions Architect, AWS
Software release cycles are now measured in days instead of months. Cutting edge companies are continuously delivering high-quality software at a fast pace. In this session, we will cover how you can begin your DevOps journey by sharing best practices and tools used by the engineering teams at Amazon. We will showcase how you can accelerate developer productivity by implementing continuous Integration and delivery workflows. We will also cover an introduction to AWS CodeStar, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, AWS CodeDeploy, AWS Cloud9, and AWS X-Ray the services inspired by Amazon's internal developer tools and DevOps practice.
Introduction
Benefits
Concepts
Templates
CLI Tool
Cloud Formation Demo
Cloud Former (Intro)
Questions
The tutorial includes an introduction to Cloud formation, benefits to Cloud formation, concepts of Cloud formation, CLI tool, Cloud formation demo, introduction to Cloud former. The tutorial begins with an introduction to Cloud formation subsequent to which, there is another section talking about the benefits of Cloud formation. It also includes the services which are used by Cloud formation.
The next section is based on the concepts of Cloud formation. This section is important as it explains the concepts of Cloud formation which are template and stack. The Template section includes the description, objects, sample template, parameters, resources, types of resources and also the steps to create a template. Whereas, the Stack section includes the collection of resources, resources which are created or deleted. Afterward comes the CLI Tool. This section includes the CLI tool called CFN.
The CLI tool section is then followed by a Cloud formation demo. It not only gives a demo of Cloud formation and which templates would be useful. But, it also includes the issues which are present in the Cloud formation demo. The last section includes an introduction to Cloud former. It provides the description of Cloud former as to which tool and architecture it uses and also the things which are possible while using Cloud former.
Apache Spark Streaming + Kafka 0.10 with Joan ViladrosarieraSpark Summit
Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
Highlights of AWS ReInvent 2023 in Las Vegas. Contains new announcements, deep dive into existing services and best practices, recommended design patterns.
AWS App Mesh (Service Mesh Magic)- AWS Container Day 2019 BarcelonaAmazon Web Services
In this session, learn about how AWS App Mesh can help give you end-to-end visibility and manage traffic routing to ensure high availability for your microservice. We will cover what the need for a service mesh, capabilities of App Mesh, and show you a demo.
In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum and optimizing your overall capital expense can be challenging. This session presents AWS features and services along with disaster recovery architectures that you can leverage when building highly available and disaster-resilient strategies.
Amazon Kinesis Analytics is the easiest way to process streaming data in real time with standard SQL without having to learn new programming languages or processing frameworks. Amazon Kinesis analytics enables you to create and run SQL queries on streaming data so that you can gain actionable insights and respond to your business and customer needs promptly. In this session, we will provide an overview of the capabilities of the Amazon Kinesis Analytics. We will show you how you can build an entire stream processing pipeline to collect, ingest, process, and emit streaming data using Amazon Kinesis Analytics, Amazon Kinesis Firehose, and Amazon Kinesis Streams.
In this session we will introduce key ETL features of AWS Glue and cover common use cases ranging from scheduled nightly data warehouse loads to near real-time, event-driven ETL flows for your data lake. We will also discuss how to build scalable, efficient, and serverless ETL pipelines.
Choose Right Stream Storage: Amazon Kinesis Data Streams vs MSKSungmin Kim
This presentation compares Amazon Kinesis Data Streams to Managed Streaming for Kafka (MSK) in both architectural perspective and operational perspective. In addition, it shows common architectural patterns: (1) Data Hub: Event-Bus, (2) Log Aggregation, (3) IoT, (4) Event sourcing and CQRS.
Managing Container Images with Amazon ECR - AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- How Amazon ECR Lifecycle Policies work to lower costs and reduce image sprawl
- How to configure and test rules for automated image cleanup
- Best practices for getting started using Lifecycle Policies today
AWS CloudFormation: Infrastructure as Code | AWS Public Sector Summit 2016Amazon Web Services
This session provides the attendee with an overview of our AWS CloudFormation service and helps the customer to realize the benefits of "infrastructure as code." A demo is part of this session.
Need to start querying data instantly? Amazon Athena an interactive query service that makes it easy to interactive queries on data in Amazon S3, using standard SQL. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing your data immediately.
In this presentation, we will show you how Amazon Athena makes it easy it is to query your data stored in S3
Introduction to Amazon Kinesis Firehose - AWS August Webinar SeriesAmazon Web Services
Streaming data applications can deliver compelling, near real-time user experiences, but building the back-end infrastructure to collect and process streaming data is difficult. Amazon Kinesis Firehose makes it easy for you to load streaming data into AWS without having to build custom stream processing applications. In this webinar, we will introduce Amazon Kinesis Firehose and discuss how to ingest streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service using Amazon Kinesis Firehose. We will also highlight key use cases based on real-world examples from IoT, AdTech, E-Commerce, and Gaming. Join us to: - Get an introduction to streaming data and an overview of Amazon Kinesis Firehose - Learn about common streaming data use cases from IoT, Ad Tech, E-Commerce, and Gaming - Understand how to use Amazon Kinesis Firehose to load streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service Who should attend: Developers, data analysts, data engineers, architects
AWS Cost Management Workshop at the San Francisco Loft
AWS offers a number of products that allow you to access, organize, understand, optimize, and control your AWS costs and usage. This workshop will help you get started using AWS Cost Explorer to visualize your usage patterns and identify your underlying cost drivers. From there, you can take action on your insights by learning how to set custom cost and usage budgets and receive alerts via email or Amazon SNS topic using AWS Budgets.
Amazon API Gateway and AWS Lambda: Better TogetherDanilo Poccia
Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. Together they help you build a server-less event-driven backend that is easy to manage and scale.
Building Cloud-Native App Series - Part 3 of 11
Microservices Architecture Series
AWS Kinesis Data Streams
AWS Kinesis Firehose
AWS Kinesis Data Analytics
Apache Flink - Analytics
by Trevor Sullivan, Solutions Architect, AWS
Software release cycles are now measured in days instead of months. Cutting edge companies are continuously delivering high-quality software at a fast pace. In this session, we will cover how you can begin your DevOps journey by sharing best practices and tools used by the engineering teams at Amazon. We will showcase how you can accelerate developer productivity by implementing continuous Integration and delivery workflows. We will also cover an introduction to AWS CodeStar, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, AWS CodeDeploy, AWS Cloud9, and AWS X-Ray the services inspired by Amazon's internal developer tools and DevOps practice.
Introduction
Benefits
Concepts
Templates
CLI Tool
Cloud Formation Demo
Cloud Former (Intro)
Questions
The tutorial includes an introduction to Cloud formation, benefits to Cloud formation, concepts of Cloud formation, CLI tool, Cloud formation demo, introduction to Cloud former. The tutorial begins with an introduction to Cloud formation subsequent to which, there is another section talking about the benefits of Cloud formation. It also includes the services which are used by Cloud formation.
The next section is based on the concepts of Cloud formation. This section is important as it explains the concepts of Cloud formation which are template and stack. The Template section includes the description, objects, sample template, parameters, resources, types of resources and also the steps to create a template. Whereas, the Stack section includes the collection of resources, resources which are created or deleted. Afterward comes the CLI Tool. This section includes the CLI tool called CFN.
The CLI tool section is then followed by a Cloud formation demo. It not only gives a demo of Cloud formation and which templates would be useful. But, it also includes the issues which are present in the Cloud formation demo. The last section includes an introduction to Cloud former. It provides the description of Cloud former as to which tool and architecture it uses and also the things which are possible while using Cloud former.
Apache Spark Streaming + Kafka 0.10 with Joan ViladrosarieraSpark Summit
Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.
Streaming Data Analytics with Amazon Redshift and Kinesis FirehoseAmazon Web Services
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Evolving your analytics from batch processing to real-time processing can have a major business impact, but ingesting streaming data into your data warehouse requires building complex streaming data pipelines. Amazon Kinesis Firehose solves this problem by making it easy to transform and load streaming data into Amazon Redshift so that you can use existing analytics and business intelligence tools to extract information in near real-time and respond promptly. In this session, we will dive deep using Amazon Kinesis Firehose to load streaming data into Amazon Redshift reliably, scalably, and cost-effectively. Level: 200
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Amazon Aurora is a MySQL- and PostgreSQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. In this deep dive session, we’ll discuss best practices and explore new features in areas like high availability, security, performance management and database cloning. Level 300
The presentation at DevFest Tokyo 2017 / @__timakin__
An introduction of blockchain and why go is nice to implement blockchain.
Additionally described about the blockchain projects that are based on Go.
Andrew Betts Web Developer, The Financial Times at Fastly Altitude 2016
Running custom code at the Edge using a standard language is one of the biggest advantages of working with Fastly’s CDN. Andrew gives you a tour of all the problems the Financial Times and Nikkei solve in VCL and how their solutions work.
Go's simplicity and concurrency model make it an appealing choice for backend systems, but how does it fare for latency-sensitive applications? In this talk, we explore the other side of the coin by providing some tips on writing high-performance Go and lessons learned in the process. We do a deep dive on low-level performance optimizations in order to make Go a more compelling option in the world of systems programming, but we also consider the trade-offs involved.
Want to build a custom app for Google Home or Google Assistant? Learn the basic concepts and how you can create a custom app to reach your users on new platforms (Google Home, Android, iPhone, and more) and help them get things done.
We'll use serverless tools like Google Cloud Functions as well as API.AI to do intelligent routing of commands to entities and intents.
Video of this talk available at: https://www.youtube.com/watch?v=C492KgDMO0c&list=PLlCd2ljeqltbJQQ79eyxbresnaKkP0TgS&index=1
Today, it is critical that IT teams are able to easily, consistently deploy to production. Running Docker containers on Amazon Web Services makes it possible to engineer a compliant and DevOps-friendly environment from the ground up. Spring Venture Group successfully migrated to AWS with Docker containers and leveraged Logicworks to migrate to AWS and automate infrastructure build-out and deployment. Join our webinar to learn how Spring Venture Group, an innovative insurance brokerage, reduced risk and improved deployment velocity with Logicworks, AWS, and Docker.
Presentation from Cloud Expo Asia Hong Kong covering the rationale for "Compliance as Code" and how InSpec may be applied to servers, cloud platforms, and much more to keep track of your compliance everywhere.
Simplify and Scale Enterprise Spring Apps in the Cloud | March 23, 2023VMware Tanzu
Event Slides: Simplify and Scale Enterprise Spring Apps in the Cloud
Date: March 23, 2023
Speakers:
Adib Saikali, Principal Solutions Engineer, VMware Tanzu
Asir Selvasingh, Principal Architect, Java on Azure, Microsoft
Prometheus is a next-generation monitoring system. It lets you see you not just what your systems look like from the outside, but also gives visibility into the internals and business aspects of your systems. This allows everyone to benefit, including both operations and developers. This talk will look at the concepts behind monitoring with Prometheus, how it's designed, why it's suitable for Cloud Native environments and how you can get involved.
Best practice adoption (and lack there of)John Pape
This is a short presentation I created some time ago that details some of the developmental, procedural, and infrastructure best practices that I discovered while working with various customers.
Getting Started with Amazon Inspector - AWS June 2016 Webinar SeriesAmazon Web Services
The flexibility and scale of the AWS Cloud and the emergence of DevOps have combined to allow developers to build and deploy applications faster than ever before. Assessing these applications for security risks without slowing down the development process can be a challenge with traditional vulnerability assessment tools designed for on-premises infrastructure. Amazon Inspector, an automated security assessment service, addresses this by integrating security assessments directly into the development process of applications running on Amazon Elastic Compute Cloud (Amazon EC2).
In this session, we will review Amazon Inspector for performing host security assessments and how it can become a seamless part of your devops lifecycle. We will run through a demo of setting up assessment targets and templates, installing the AWS agent, and running assessments. We will explore the findings generated by an assessment and discuss how you can automate the running of assessments.
Learning Objectives:
An overview and the value of Security Assessment testing with Amazon Inspector
How customer sign up for, configure, and use the service
Understand AWS Agent and assessment data security
A presentation on the Netflix Cloud Architecture and NetflixOSS open source. For the All Things Open 2015 conference in Raleigh 2015/10/19. #ATO2015 #NetflixOSS
(SEC312) Taking a DevOps Approach to Security | AWS re:Invent 2014Amazon Web Services
More organizations are embracing DevOps to realize compelling business benefits, such as more frequent feature releases, increased application stability, and more productive resource utilization. However, security and compliance monitoring tools have not kept up. In fact, they often represent the largest single remaining barrier to continuous delivery. Learn how to integrate security controls in your DevOps program from experts at Alert Logic and George Miranda, engineer and evangelist at Chef. Sponsored by Alert Logic.
Modernizing Testing as Apps Re-ArchitectDevOps.com
Applications are moving to cloud and containers to boost reliability and speed delivery to production. However, if we use the same old approaches to testing, we'll fail to achieve the benefits of cloud. But what do we really need to change? We know we need to automate tests, but how do we keep our automation assets from becoming obsolete? Automatically provisioning test environments seems close, but some parts of our applications are hard to move to cloud.
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...Amazon Web Services
In this session, you will learn how to deploy complex Windows workloads and ways AWS CloudFormation, AWS OpsWorks, and AWS CodeDeploy enable you to automate your Windows application life-cycle management. We will also discuss the monitoring, logging, and automatically scaling of Windows applications. Learn More: https://aws.amazon.com/government-education/
Starting Your DevOps Journey – Practical Tips for OpsDynatrace
To watch, please see:
https://info.dynatrace.com/apm_wc_getting_started_with_devops_na_registration.html
Starting Your DevOps Journey: Practical Tips for Ops
In this webinar, Andreas Grabner, Chief DevOps Activist at Dynatrace, shares practical tips that all IT groups from Dev to Ops can use to start their DevOps journey quickly. With experience from hundreds of DevOps deployments, Andi provides insights it would take your team months or years to learn firsthand.
- Learn how everyone on your Ops team can use APM to better understand and monitor SLAs, Performance and End User Impact of their applications.
- Foster better collaboration between Ops and architects by extending basic system monitoring to monolith and microservices architectures.
- Shift-left your testing and QA by working with metrics that you and the architects agreed on up front, resulting in early relevant feedback and faster code deployments.
- Hear why changing the cultural mindset from “fear of change” to “Continuous Innovation and Optimization” is critical for success.
Andi is joined by guest speaker, Brian Chandler, Systems Engineer at Raymond James, who shares commonly used Ops dashboards that increase collaboration across IT teams and pro-actively break down silos!
AWS provides tools to improve your security posture, by providing ways of implementing detective and reactive controls that will detect and remediate security threats. We’ll look at the various services and the features that you can employee, such as AWS Inspector, AWS Trusted Advisor, AWS Config and Config Rules and CloudTrail. We’ll explore how they work and how they should be deployed as part of an overall security strategy.
Similar to Operations: Production Readiness Review – How to stop bad things from Happening (20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
2. About me:
Chris Munns - munns@amazon.com, @chrismunns
• Senior Developer Advocate - Serverless
• New Yorker
• Previously:
• AWS Business Development Manager – DevOps, July ’15 - Feb ‘17
• AWS Solutions Architect Nov, 2011- Dec 2014
• Formerly on operations teams @Etsy and @Meetup
• Little time at a hedge fund, Xerox and a few other startups
• Rochester Institute of Technology: Applied Networking and
Systems Administration ’05
• Internet infrastructure geek
4. Production Readiness Review
You don’t need all of these from day one, grow them as your teams grow.
Architecture Design Review
Monitoring
Logging
Documentation
Alerting
Service Level Agreement
Expected Throughput
Testing
Deploy Strategy
6. Architecture Design Review
Netflix Chaos Engineering
1. Define the system’s normal behavior — its “steady state” — based on
measurable output like overall throughput, error rates, latency, etc.
2. Hypothesize about the steady state behavior of an experimental group, as
compared to a stable control group.
3. Expose the experimental group to simulated real-world events such as server
crashes, malformed responses, or traffic spikes.
4. Test the hypothesis by comparing the steady state of the control group and
the experimental group. The smaller the differences, the more confidence we
have that the system is resilient.
TLDR; Intentionally break things, compare measured with expected impact, and correct any problems uncovered this way.
7. Architecture Design Review
Highly Available & Redundant
Problem Solution
Failure of a service in a specific
location
Run across multiple availability zones
or regions
Able to handle spikes of traffic Have auto-scaling in place with EC2,
Containers, or through leveraging
serverless architectures.
Avoid Single Points of Failure (SPOF) Be sure services are running in
clusters scaled across AZs.
Replication > Backups.
8. Architecture Design Review
Using Standard Libraries & Design Patterns
Standardizing on libraries, languages, styleguides makes onboarding new
developers and troubleshooting issues easier. Enforce these programmatically
where you can. (eslint, gofmt, etc)
Spot situations where code may be duplicated and able to be refactored.
Look for opportunities to implement good design patterns.
Know your licenses - OpenSource Permissive (MIT/Apache) vs Copy Left
(GNU/MPL)
9. Architecture Design Review
Review for Security Best Practices
Security should always be a top priority
Ensure no credentials are being stored in the application
Code defensively for SQL injections, XSS attacks, and more
Leverage Static Analysis tools
https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis
Consider using Pre-Commit by Yelp
http://pre-commit.com
10. Architecture Design Review
Leverage other startups or rotate teams to keep fresh eyes on your code
Partner with another startup to help each other with architecture, code review,
interviewing, and more.
Consider rotating developers off of projects every few months to gain fresh
eyes on projects.
13. Monitoring
Performance Metrics
Start by building a dashboard of “important” metrics. Continue iterating on this
as you learn more about your system under inspection. Each system has a
“heartbeat” that will appear off when things are unhealthy.
You always think you have enough metrics being gathered until you need the
one you’re missing. When applications fail, the more data you can observe the
easier it is to get to the root cause.
Averages hide issues. Be sure to leverage percentiles to expose where users
are experiencing issues.
Complicated systems build complicated dependency chains. Small fluctuations
in one part of your stack can manifest itself in other parts.
14. Monitoring
Application Level Visibility
Provides Insight To Application Performance
You need visibility into how your application itself is performing.
How long are certain calls to resources taking?
Is that trending up or down?
What part of the application is generating the most number of errors?
17. Monitoring
Real User Monitoring (RUM) & Synthetic Monitoring
Synthetic Monitoring
Automatic testing of your site and service to measure performance.
Real User Monitoring
Shows your exactly how users are interacting with your site or application.
Measures page load times, DNS resolution issues, traffic bottlenecks, and
more.
25. Logging
Consistent Log Format
Consider using JSON for logging
User Log Levels correctly [INFO/WARN/CRIT]
Add context for your logging statements
Log behaviors and errors
Consider how analytics will be used on this data
26. Logging
UTC Timestamps
Centrally aggregated logs make analysis easier
Helps prevent mismatch errors due to DST
Prepares you for multi-region
Log tool interfaces let you adjust time zones per user
[2017-07-13 14:49:24.436245]
27. Logging
Individual Transaction IDs
The session ID that generated the error
The user who encountered the error
The user’s location in the application
The ID of the transaction or product that caused the error
Be careful about what you log from a security perspective
Web App Database
ID 10948281 ID 10948281
28. Documentation
Store Your Documentation Close To Your Code: Read.me
What the code does
How to install and run it
How to interact with it (stop, start, restart)
How to configure it
How to troubleshoot it
What metrics and dashboards are available
30. Alerting
"Level 1" Operations Teams Should Be Automated
check process nginx with pidfile /var/run/nginx.pid
start program = "/etc/init.d/nginx start”
stop program = "/etc/init.d/nginx stop”
group www (for centos)
33. Alerting
Build Proper Escalation Paths For Alerts
Primary
Secondary
Team
Management
10 Minutes
10 Minutes
10 Minutes
Being paged when something fails is great, but you
always need a backup
These need to auto escalate when not acknowledged
As it escalates up it’s good to notify a wider range of
people to get more eyes on the issue
Review alerts that have been ack’d or silenced beyond
a tolerable threshold.
34. Alerting
Developers Code Should Only Burden Themselves
Operations Add Capacity
Developer Deploy Hotfix
Bad application code
causes 40% increase in
CPU usage across a
cluster.
Temporary Fix
Permanent Fix
36. Service Level Agreements/Objectives
Services Should Have An SLA/SLO
/Search
/Cart
/Avatars
99.99%
99.999%
99.9%
These are internal SLAs for the
company
Helps identify how much effort should
be put into the reliability of each
service
Important when using microservices
for teams to reliably build
dependencies on your service.
https://landing.google.com/sre/book/chapters/service-level-objectives.html
37. Service Level Agreements
Understand The Cost Of Adding Each 9
Level of
Availability
Percent of
Uptime
Downtime per
Year
Downtime per
Day
1 Nine 90% 36.5 days 2.4 hours
2 Nines 99% 3.65 days 14 minutes
3 Nines 99.9% 8.76 hours 86 seconds
4 Nines 99.99% 52.6 minutes 8.6 seconds
5 Nines 99.999% 5.25 minutes .86 seconds
6 Nines 99.9999% 31.5 seconds 8.6 milliseconds
38. Expected Throughput
Run Load Tests & Understand Your Limits
Before a service goes live, know where your breaking points are.
Know the bare minimum number of instances needed to run your average
throughput
Know the maximum throughput you can handle with your current architecture
Calculate the throughput per instance ratio so you can accurately setup
proper auto-scaling in a cost optimized way.
40. Expected Throughput
Provides Performance Baseline For Future Release
0
500
1000
1500
2000
2500
3000
3500
Max RPS
V1
V14
As code evolves, so does your
performance.
Understand the impact of additional
libraries, added lines of code, and new
external calls.
Here we see a 63.58% increase in
performance from V1 to V14. This
directly correlates to your infrastructure
cost.
42. Testing
Adopt Automated Testing Early
Builds confidence in the code being
released
Allows you to test more of your
application in less time
Manual testing can become error
prone
45. Deployment Strategy
Database Migrations
Understand what changes to the database need to happen to support new
code releases.
Avoid removing columns, only make additions to reduce risk.
Be sure to test migrations against test copies of the database
Keep a revision history of database migrations for reference
Snapshot databases before doing migrations
47. Deployment Strategy
Dark Deploys & Feature Flags
Opt In
Test new features with selected
users
Kill Switch
Disable poorly performing features
Scalable Roll Outs
Do % roll outs of new features
Block Users
Prevent selected users from features
Run A/B Tests
Test and compare new features
Sunset Old Features
Safely decommission old features
48. Error Budget
Spend it! It’s there for you to use.
Error budget is there for you to take calculated risks in your environment.
Allows you to save up a high budget to spend it on major architectural
changes.
Some companies force the spending of this budget when it’s not utilized to
encourage services built on it to gracefully fail. If the SLA is 99.99% and it’s
running at 100%, they will manually force downtime to stay at 99.99%.
49. Production Readiness Review
Summary of key areas for a PRR
Architecture Design Review
Monitoring
Logging
Documentation
Alerting
Service Level Agreement
Expected Throughput
Testing
Deploy Strategy
50. Resources
Useful resources related to the topics covered
Production Readiness Review:
https://arxiv.org/pdf/1305.2402.pdf
Netflix Hystrix Circuit Breaker:
https://github.com/Netflix/Hystrix/wiki/How-it-Works
Feature Flags:
https://en.wikipedia.org/wiki/Feature_toggle
Error Budgets:
https://landing.google.com/sre/interview/ben-treynor.html
Monitoring Philosophies:
https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit