Scalable serverless architectures using event-driven design - MAD308 - New Yo...Amazon Web Services
There are a number of design options to apply when you build serverless architecture, such as which AWS services to use with AWS Lambda and the Lambda invocation pattern. Lambda can be invoked by many different AWS services or events, such as in response to infrastructure activities, developer tools actions, lifecycle events, and more. Lambda can also be invoked as part of a workflow within an application. In this session, we discuss the various design patterns for Lambda and when you should use them. We also cover some best practices for building apps using the full serverless stack, not just Lambda.
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...Amazon Web Services
Come learn what's new with Amazon CloudWatch, and watch as we leverage new capabilities to better monitor our systems and resources. We also walk you through the journey that BBC took in monitoring its custom off-cloud infrastructure alongside its AWS cloud resources.
In this session, AWS will present an overview of the AWS Landing Zone – an automated solution for setting up a robust and flexible AWS environment. Customers can expect to learn how AWS works with customers to accelerate their journey to AWS confidently and securely and how the AWS Landing Zone can be customized to meet each organization’s specific needs.
Presenter: Sadegh Nadimi, Senior Consultant, Global Migrations, AWS
대용량 트래픽을 처리하는 최적의 서버리스 애플리케이션 - 안효빈, 구성완 AWS 솔루션즈 아키텍트 :: AWS Summit Seoul 2021Amazon Web Services Korea
서버리스는 오늘날 클라우드에서 가장 인기있는 디자인 패턴 중 하나입니다. 서버 및 운영 체제 운영의 부담은 덜고 서비스의 개발 및 구축을 통해 빠르게 혁신에 집중할 수 있기 때문입니다. 본 강연에서는 클라우드 네이티브한 방식으로 서버리스 애플리케이션을 고도화 하는 방법에 관해 알아봅니다. 아키텍처를 디커플링하고 비동기적으로 구성함으로써 보다 유연한 확장이 가능하게 됩니다. Amazon API Gateway, AWS Lambda 를 활용한 기본적인 서버리스 애플리케이션을 Lambda의 신규 기능과 Amazon SQS, Amazon RDS Proxy, AWS Secrets Manager 등을 활용해 보다 안정적으로 확장하는 방법에 관해 알아봅니다.
Security and governance with AWS Control Tower and AWS Organizations - SEC204...Amazon Web Services
Whether it is per business unit or per application, many AWS customers use multiple accounts to meet their infrastructure isolation, separation of duties, and billing requirements. In this session, learn about the considerations, limitations, and security patterns of building a multi-account strategy. Get insight into topics such as thought pattern, identity federation, cross-account roles, consolidated logging, and account governance. Finally, see an enterprise-ready landing zone framework and the background needed to implement an AWS Landing Zone using AWS Control Tower and AWS Organizations.
by Fritz Kunstler, Sr. Security Consultant, AWS
AWS Organizations offers policy-based management for multiple AWS Accounts. Learn how Organizations helps you more easily manage policies for groups of accounts and automate account creation.
Scalable serverless architectures using event-driven design - MAD308 - New Yo...Amazon Web Services
There are a number of design options to apply when you build serverless architecture, such as which AWS services to use with AWS Lambda and the Lambda invocation pattern. Lambda can be invoked by many different AWS services or events, such as in response to infrastructure activities, developer tools actions, lifecycle events, and more. Lambda can also be invoked as part of a workflow within an application. In this session, we discuss the various design patterns for Lambda and when you should use them. We also cover some best practices for building apps using the full serverless stack, not just Lambda.
Monitor All Your Things: Amazon CloudWatch in Action with BBC (DEV302) - AWS ...Amazon Web Services
Come learn what's new with Amazon CloudWatch, and watch as we leverage new capabilities to better monitor our systems and resources. We also walk you through the journey that BBC took in monitoring its custom off-cloud infrastructure alongside its AWS cloud resources.
In this session, AWS will present an overview of the AWS Landing Zone – an automated solution for setting up a robust and flexible AWS environment. Customers can expect to learn how AWS works with customers to accelerate their journey to AWS confidently and securely and how the AWS Landing Zone can be customized to meet each organization’s specific needs.
Presenter: Sadegh Nadimi, Senior Consultant, Global Migrations, AWS
대용량 트래픽을 처리하는 최적의 서버리스 애플리케이션 - 안효빈, 구성완 AWS 솔루션즈 아키텍트 :: AWS Summit Seoul 2021Amazon Web Services Korea
서버리스는 오늘날 클라우드에서 가장 인기있는 디자인 패턴 중 하나입니다. 서버 및 운영 체제 운영의 부담은 덜고 서비스의 개발 및 구축을 통해 빠르게 혁신에 집중할 수 있기 때문입니다. 본 강연에서는 클라우드 네이티브한 방식으로 서버리스 애플리케이션을 고도화 하는 방법에 관해 알아봅니다. 아키텍처를 디커플링하고 비동기적으로 구성함으로써 보다 유연한 확장이 가능하게 됩니다. Amazon API Gateway, AWS Lambda 를 활용한 기본적인 서버리스 애플리케이션을 Lambda의 신규 기능과 Amazon SQS, Amazon RDS Proxy, AWS Secrets Manager 등을 활용해 보다 안정적으로 확장하는 방법에 관해 알아봅니다.
Security and governance with AWS Control Tower and AWS Organizations - SEC204...Amazon Web Services
Whether it is per business unit or per application, many AWS customers use multiple accounts to meet their infrastructure isolation, separation of duties, and billing requirements. In this session, learn about the considerations, limitations, and security patterns of building a multi-account strategy. Get insight into topics such as thought pattern, identity federation, cross-account roles, consolidated logging, and account governance. Finally, see an enterprise-ready landing zone framework and the background needed to implement an AWS Landing Zone using AWS Control Tower and AWS Organizations.
by Fritz Kunstler, Sr. Security Consultant, AWS
AWS Organizations offers policy-based management for multiple AWS Accounts. Learn how Organizations helps you more easily manage policies for groups of accounts and automate account creation.
Amazon WorkSpaces is a secure, managed, virtual desktop service running on the AWS cloud. The service helps organizations support a modern mobile workforce, improve information security, and save money with a pay-as-you-go model. In this session, we'll cover how cloud desktops can benefit your organization, what's new with Amazon WorkSpaces, and some of the top technical considerations like user identity and access management, VPC design, network traffic flow, and application delivery. This session is for IT professionals and business decision makers interested in learning how to simplify desktop management and productivity for their organizations.
Managing and governing multi-account AWS environments using AWS Organizations...Amazon Web Services
As you continue to grow your footprint on AWS, centralized tools and features are required to help govern multiple AWS accounts for account management, security and access control, and resource sharing. This session discusses how you can use AWS Organizations to manage and govern multi-account environments on AWS with security and compliance in mind. This session covers AWS Organizations, IAM, AWS Config, AWS Firewall Manager, CloudTrail, CloudWatch Events, Directory Service, License Manager, Resource Access Manager, and Single Sign-On.
Landing Zones - Creating a Foundation for Your AWS MigrationsAmazon Web Services
Dean Samuels, Head of Solutions Architecture, Hong Kong & Taiwan, AWS
When migrating lots of applications to the cloud, it's important to architect cloud environments that are efficient, secure and compliant. AWS Landing Zones are a prescriptive set of instructions for deploying an AWS-recommended foundation of interrelated AWS accounts, networks, and core services for your initial AWS application environments. This session will review the benefits and best practices.
Accelerating Your Portfolio Migration to AWS Using AWS Migration Hub - ENT321...Amazon Web Services
When migrating a large number of workloads to AWS, tracking progress across the various applications and services involved can distract your team from core migration activities. In this session, learn how AWS Migration Hub provides a single place to discover your existing servers and track the status of each application migration. It provides you with better visibility into your application portfolio and streamlines migration tracking, at no additional cost beyond the services you use.
AWS 기반 지속 가능한 데이터 분석 플랫폼 구축하기
소성운, 지그재그
한명의 데이터 분석가로서 AWS를 통해 어떻게 손쉽게 분석 플랫폼을 구축할 수 있는지 사례를 통해 알아봅니다. 1위 초등 스마트 홈러닝 서비스인 아이스크림 홈런에서 사용자의 학습 데이터를 분석하기 위해 Amazon Kinesis, Glue, Redshift 등의 서비스를 사용하면서 겪었던 문제와 해결 노하우를 공유합니다. 또한, 매일 신상품이 1만건 이상 등록되고, 월 활성화유저(MAU) 230만에 육박하는 쇼핑몰 모음 서비스인 지그재그에서 데이터 전처리 부터 시각화까지 Amazon S3, EMR, Glue, Tableau(BI) 도구를 통해 통해 어떻게 클라우드를 통해 단발성이 아닌 지속 가능한 관리 가능한 데이터 분석 프로세스를 만들 수 있는지 알아봅니다.
Looking for a way to build beautiful, cloud-native web and mobile apps? Seeking to eliminate the undifferentiated heavy lifting of a traditional, three-tier web application? AWS Amplify makes it a snap with an easy-to-use library, a powerful toolchain, a user-interface component library, and built-in capabilities such as authentication and analytics. Uncover how to bring your idea to life, scale it to the max, and delight your users in less time than before.
Providing more control around how to manage your AWS accounts with our newly launched service - AWS Organizations. In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Speaker: Pierre Liddle, Solutions Architect, Amazon Web Services
API Gateways can simplify the work that a developer needs to do to build API based services by helping to standardize authentication and authorization, consumer interfaces, and management needs. With Amazon API Gateway you get all of this and more, including a completely serverless management of your APIs and the ability to host them at almost any scale. You also can get the benefits of the numerous types of APIs that are supported, from pubic to private, REST to Websockets, backed by almost any backend you can think of. In this session we’ll review the powerful capabilities of Amazon API Gateway and how you can get started building awesome APIs.
Speaker: Chris Munns - Principal Developer Advocate, AWS Serverless Applications, AWS
[NEW LAUNCH!] Deep Dive on Amazon FSx for Windows File Server (STG322-R) - AW...Amazon Web Services
If you have compute-intensive workloads like high performance computing, machine learning, and media processing then this is the workshop for you! Our new file storage service, Amazon FSx for Lustre, provides compute-optimized storage with fully managed Lustre file systems that can deliver hundreds of gigabytes of throughput and sub-millisecond latencies. You will learn how to spin up an FSx for Lustre file system in minutes, feed data to it from an S3 data lake, run analyses while writing results back to S3, and then spin down the file system once the workload is finished.
Join us to learn about the state of serverless computing from Dr. Tim Wagner, General Manager of AWS Lambda. Dr. Wagner discusses the latest developments from AWS Lambda and the serverless computing ecosystem. He talks about how serverless computing is becoming a core component in how companies build and run their applications and services, and he also discusses how serverless computing will continue to evolve.
Amazon WorkSpaces is a secure, managed, virtual desktop service running on the AWS cloud. The service helps organizations support a modern mobile workforce, improve information security, and save money with a pay-as-you-go model. In this session, we'll cover how cloud desktops can benefit your organization, what's new with Amazon WorkSpaces, and some of the top technical considerations like user identity and access management, VPC design, network traffic flow, and application delivery. This session is for IT professionals and business decision makers interested in learning how to simplify desktop management and productivity for their organizations.
Managing and governing multi-account AWS environments using AWS Organizations...Amazon Web Services
As you continue to grow your footprint on AWS, centralized tools and features are required to help govern multiple AWS accounts for account management, security and access control, and resource sharing. This session discusses how you can use AWS Organizations to manage and govern multi-account environments on AWS with security and compliance in mind. This session covers AWS Organizations, IAM, AWS Config, AWS Firewall Manager, CloudTrail, CloudWatch Events, Directory Service, License Manager, Resource Access Manager, and Single Sign-On.
Landing Zones - Creating a Foundation for Your AWS MigrationsAmazon Web Services
Dean Samuels, Head of Solutions Architecture, Hong Kong & Taiwan, AWS
When migrating lots of applications to the cloud, it's important to architect cloud environments that are efficient, secure and compliant. AWS Landing Zones are a prescriptive set of instructions for deploying an AWS-recommended foundation of interrelated AWS accounts, networks, and core services for your initial AWS application environments. This session will review the benefits and best practices.
Accelerating Your Portfolio Migration to AWS Using AWS Migration Hub - ENT321...Amazon Web Services
When migrating a large number of workloads to AWS, tracking progress across the various applications and services involved can distract your team from core migration activities. In this session, learn how AWS Migration Hub provides a single place to discover your existing servers and track the status of each application migration. It provides you with better visibility into your application portfolio and streamlines migration tracking, at no additional cost beyond the services you use.
AWS 기반 지속 가능한 데이터 분석 플랫폼 구축하기
소성운, 지그재그
한명의 데이터 분석가로서 AWS를 통해 어떻게 손쉽게 분석 플랫폼을 구축할 수 있는지 사례를 통해 알아봅니다. 1위 초등 스마트 홈러닝 서비스인 아이스크림 홈런에서 사용자의 학습 데이터를 분석하기 위해 Amazon Kinesis, Glue, Redshift 등의 서비스를 사용하면서 겪었던 문제와 해결 노하우를 공유합니다. 또한, 매일 신상품이 1만건 이상 등록되고, 월 활성화유저(MAU) 230만에 육박하는 쇼핑몰 모음 서비스인 지그재그에서 데이터 전처리 부터 시각화까지 Amazon S3, EMR, Glue, Tableau(BI) 도구를 통해 통해 어떻게 클라우드를 통해 단발성이 아닌 지속 가능한 관리 가능한 데이터 분석 프로세스를 만들 수 있는지 알아봅니다.
Looking for a way to build beautiful, cloud-native web and mobile apps? Seeking to eliminate the undifferentiated heavy lifting of a traditional, three-tier web application? AWS Amplify makes it a snap with an easy-to-use library, a powerful toolchain, a user-interface component library, and built-in capabilities such as authentication and analytics. Uncover how to bring your idea to life, scale it to the max, and delight your users in less time than before.
Providing more control around how to manage your AWS accounts with our newly launched service - AWS Organizations. In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Speaker: Pierre Liddle, Solutions Architect, Amazon Web Services
API Gateways can simplify the work that a developer needs to do to build API based services by helping to standardize authentication and authorization, consumer interfaces, and management needs. With Amazon API Gateway you get all of this and more, including a completely serverless management of your APIs and the ability to host them at almost any scale. You also can get the benefits of the numerous types of APIs that are supported, from pubic to private, REST to Websockets, backed by almost any backend you can think of. In this session we’ll review the powerful capabilities of Amazon API Gateway and how you can get started building awesome APIs.
Speaker: Chris Munns - Principal Developer Advocate, AWS Serverless Applications, AWS
[NEW LAUNCH!] Deep Dive on Amazon FSx for Windows File Server (STG322-R) - AW...Amazon Web Services
If you have compute-intensive workloads like high performance computing, machine learning, and media processing then this is the workshop for you! Our new file storage service, Amazon FSx for Lustre, provides compute-optimized storage with fully managed Lustre file systems that can deliver hundreds of gigabytes of throughput and sub-millisecond latencies. You will learn how to spin up an FSx for Lustre file system in minutes, feed data to it from an S3 data lake, run analyses while writing results back to S3, and then spin down the file system once the workload is finished.
Join us to learn about the state of serverless computing from Dr. Tim Wagner, General Manager of AWS Lambda. Dr. Wagner discusses the latest developments from AWS Lambda and the serverless computing ecosystem. He talks about how serverless computing is becoming a core component in how companies build and run their applications and services, and he also discusses how serverless computing will continue to evolve.
AWS re:Invent serverless recap day: Controls for-managing-events-in-serverles...⛷️ Ben Smith
Event-driven architectures can help decouple services and simplify dependencies as your application grows. There are now new options for developers looking to benefit from this approach with the release of AWS Lambda controls for asynchronous and stream processing events, Lambda event destinations, and Amazon EventBridge, a serverless event bus service. In this tech talk, we'll demonstrate strategies for managing events to help you build more resilient event-driven applications. You will learn how to customize responses to Lambda function results with new service features to handle retries, failures, and prevent “poison pill” events. We will also demonstrate how to configure Lambda event destinations to route execution results to other AWS services, and how to set up routing rules in Amazon EventBridge that react to all of your data sources.
The State of Serverless Computing | AWS Public Sector Summit 2017Amazon Web Services
oin us to learn about the state of serverless computing from Dougal Ballantyne, Principal Product Manager, Serverless. Dougal Ballantyne discusses the latest developments from AWS Lambda and the serverless computing ecosystem. He talks about how serverless computing is becoming a core component in how companies build and run their applications and services, and he also discusses how serverless computing will continue to evolve. Learn More: https://aws.amazon.com/government-education/
The technical advantages of a microservices architecture pattern are understood by many AWS customers. In this session, the innovation advantages of microservices are explored from a business perspective together with business agility lessons learned during an evolution from a single monolithic application to cloud based microservices.
Speaker: Craig Dickson, Solutions Architect, Amazon Web Services
Getting Started with AWS Lambda and the Serverless Cloud - AWS Summit Cape T...Amazon Web Services
Serverless computing allows you to build and run applications without the need for provisioning or managing servers. With serverless computing, you can build web, mobile, and IoT backends; run stream processing or big data workloads; run chatbots, and more. In this session, you’ll learn how to get started with serverless computing with AWS Lambda, which lets you run code without provisioning or managing servers. We’ll introduce you to the basics of building with Lambda and how you can benefit from features such as continuous scaling, built-in high availability, integrations with AWS and third-party apps, and subsecond metering pricing. We’ll also introduce you to the broader portfolio of AWS services that help you build serverless applications with Lambda, including Amazon API Gateway, Amazon DynamoDB, AWS Step Functions, and more.
AWS Speaker : Danilo Poccia, Technical Evangelist - Amazon Web Services
Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. Microservices architectures make applications easier to scale and faster to develop, enabling innovation and accelerating time-to-market for new features. We’ll explore best practices for microservice design and the data design needed to support microservices, using Aurora, RDS, DynamoDB, DAX, ElastiCache, and Lambda and we’ll do a design exercise in converting a monolithic solution to a microservices design.
Speaker: Rajanikanth Bhargava Chilakapati - Solutions Architect, AWS
2016-06 - Design your api management strategy - AWS - Microservices on AWSSmartWave
Morning session started with a presentation on working with a micro-services API gateway in hybrid architectures, by Jean-Pierre LeGoaller, Architect at AWS. We learned how to greatly reduce coding efforts, make applications far more efficient, and decrease errors all at the same time, using small and flexible Micro-services with an API Gateway. Jean-Pierre then illustrated the benefits of AWS lambda function to run seamlessly codes as a service in AWS high-availability compute infrastructure.
AWS re:Invent 2016: Deploying Amazon WorkSpaces at Enterprise Scale to Delive...Amazon Web Services
Your enterprise has decided it is exiting the traditional desktop business and migrating to Amazon WorkSpaces. Your challenge: how do you provide end users a high quality experience using DaaS while integrating key enterprise services? Focusing on user adoption and simplified operational management DaaS offers significant benefits over traditional physical desktops and VDI solutions. These benefits include hourly consumption pricing, sizing flexibility, linear scalability, and simplified management.
We will dive into the methodology and design decisions that Informa, a global leader in business intelligence and part of the Information Economy, made to migrate to Amazon WorkSpaces. The goal of this strategy for Informa was to provide their end users with a Windows 10 desktop that could scale across three AWS regions.
Following this session you will be equipped with a methodology for migrating to Amazon WorkSpaces and key design decision points that will assist your organization in rolling out this service for your end users. Best practices will be presented and there will be discussion on the capabilities required to build an enterprise-scale solution around WorkSpaces. All of this will include the fundamental value proposition AWS has brought to market with their industry-leading DaaS offering. Session sponsored by AHEAD.
Cloud computing gives you a number of advantages, such as the ability to scale your web application or website on demand. If you have a new web application and want to use cloud computing, you might be asking yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We show you how to best combine different AWS services, how to make smarter decisions for architecting your application, and how to scale your infrastructure in the cloud.
How to Build a Big Data Application: Serverless Editionecobold
Come learn how to build, launch, and scale a Big Data application in a serverless context. This is going to be an information packed meetup around Big Data processing, Lambda functions, Lambda Step functions, and everything that ties them together.
Big Data is something we're very passionate about. As the cost of servers have come down and the cost of software has become free, using data to drive your business has become much more obtainable to a larger group of companies. The serverless methodology has recently come in the scene, and it's proving to be just as transformational as cloud has been to the Big Data analytics space. We will be sharing some of our learnings and experiences over the last two years of working with Big Data in a serverless context. We will cover one or two examples of eventful Big Data processing, and the impact it can have on your business in terms of speed of analytics and cost savings to the bottom line.
AWS re:Invent 2016: The State of Serverless Computing (SVR311)Amazon Web Services
Join us to learn about the state of serverless computing from Dr. Tim Wagner, General Manager of AWS Lambda. Dr. Wagner discusses the latest developments from AWS Lambda and the serverless computing ecosystem. He talks about how serverless computing is becoming a core component in how companies build and run their applications and services, and he also discusses how serverless computing will continue to evolve.
Application architecture is increasingly migrating to an event driven model - which are more efficient and have lower operational overhead. In this talk we will cover how to build a fully event driven architecture using AWS Lambda, chaining together multiple functions to build a complex application, which can be easily scaled to millions of users.
Serverless architectures let you build and deploy applications and services with infrastructure resources that require zero administration. In the past, you had to provision and scale servers to run your application code, install and operate distributed databases, and build and run custom software to handle API requests. Now, AWS provides a stack of scalable, fully-managed services that eliminates these operational complexities.
In this session, you will learn about the benefits of serverless architectures and the basics of the serverless stack AWS provides. We will also walk through how you can use serverless architectures for everything from data processing to mobile and web backends.
Migrating Microsoft Applications to AWS like an Expert - AWS Summit Sydney 2018Amazon Web Services
Migrating Microsoft Applications to AWS like an Expert
With Microsoft applications making up 60% of most enterprises, this session will illustrate mechanisms to migrate to AWS ensuring your migration strategy encompasses a strong, security focused foundation with a focus on removing undifferentiated heavy lifting. Join us on a journey of migrating the Unicornshop.lol core infrastructure to AWS. This includes business productivity systems, Microsoft SQL server, and a number of .NET applications that require seamless migration with minimal downtime. We will walk through the journey of creating our landing zone and fully automated compliance controls before embarking on our migration journey together!
Danny Jenkins, Solutions Architect, Amazon Web Services
SaaS architectures can be deployed onto AWS in a number of ways, and each optimizes for different factors from security to cost optimization. Come learn more about common deployment models used on AWS for SaaS architectures and how each of those models are tuned for customer specific needs. We will also review options and tradeoffs for common SaaS architectures, including cost optimization, resource optimization, performance optimization, and security and data isolation.
Getting started building your first serverless web application on AWSIoannis Polyzos
Getting started with AWS serverless computing. A cloud computing model where developers can build and run applications without having to manage or provision servers. With AWS serverless services like AWS Lambda, developers can focus solely on writing code for their application's business logic, and AWS handles the infrastructure management, scaling, and availability aspects. This allows for highly scalable, event-driven architectures and pay-per-use pricing, where you only pay for the actual usage of your applications or functions.
Similar to Track 4 Session 3_ 利用 AWS Step Functions 建構穩健的業務處理流程 (20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
Durante i laboratori pratici, gli esperti AWS ti mostrano quali strumenti aiutano a sviluppare le applicazioni Serverless in locale e nel cloud AWS e ti aiuteranno a programmare i prossimi passi per iniziare ad utilizzare questa tecnologia nella tua azienda.
6. Is this you?
“I want
try/catch/finally”“I want to retry failed tasks”
A
“I want to sequence tasks”
BA
“I want to select tasks
based on data”
B C
A
?
“I want to run tasks in parallel”
BA
7. Coordination must-haves
• Easy to build & operate
• Scales out
• Doesn’t lose state
• Deals with errors/exceptions
• Auditable
9. AWS Step Functions Features
Visualize in the
Console
Define in JSON Monitor
Executions
10. Workflow Type
Standard Workflows
• Long-running, durable and auditable
workflows
• 2000 per second (stared rate)
• 4000 per second (state transition)
• Log events on console & CloudWatch Log
• Max execution time of 1 year
• $0.025 per 1,000 state transitions
Express Workflows
• Simplified capabilities, geared
for speed and scale
• Supporting event rates up to
100,000 per second
• No logging events on console
• Max execution time of 5 mins
• Pay per use at $1.00 per 1M
requests and $0.000001 per
request
Flexibility to select the right workflow type for your needs, or integrate them together where needed
11. Orchestrating Your Business Logic with Eight States
The Amazon States Language is a JSON-based, structured language used to define your state machine
12. Tasks: Activity
Hybrid workflows with servers and instances
Long poll
On premises server
Worker requests
tasks from Step
Functions
13. Tasks: Callback
Call an external resource.
Pause workflow for a
callback event from the
resource. Wait for as long
as you need – minutes,
days, weeks, or months:
• Human activity
• Third party API
• Legacy application
Start
End
Callback Task
Next Task
14. Callback tasks : Human Activities
Call an external resource.
Pause workflow for a
callback event from the
resource. Wait for as long
as you need – minutes,
days, weeks, or months:
• Human activity
• Third party API
• Legacy application
23. Build Workflows Using Service Integrations
AWS Lambda
AWS Lambda
Mountains
People
Snow
Amazon
DynamoDB
Amazon SNS
Amazon
DynamoDB
24. Payment and Order Processing Flow
Storing payment information
Authentication and Payment
AWS Step Functions workflow
Processing shipmentUpdating Payment Status
Financial Organization
Fail
30. Use Cases for AWS Step Functions
Process
data
Automate
tasks
Modernize
monoliths
Orchestrate
applications
31. Challenge
• Transcode ~350 news video clips per day
into 14 formats each – as quickly as possible
Results
• Able to process video segments in parallel
• Reduced processing time from ~20 min to
~2 minutes
• The bigger the source video, the more
segments, the bigger the savings
Thomson Reuters
Serverless split video transcoding Industry: Media & Entertainment
Solution: Big Data / Machine Learning
Architecture: Serverless
32. coinbase
Buy and sell digital currency
Challenge
• Secure, reliable, immutable deployments with visibility by
their engineers
• Scale to multiple AWS accounts easily and quickly
Results
• Reduced new account deployment from days to seconds
• Increased the time a deployment can reliably run from 2
hours to 48 hours
• Mission critical services deployment rate rose 7%
33. Challenge
• Make a very old, very critical monolithic billing application
easier to modify & maintain
• Improve account billing process
• High refactoring risk
Results
• Able to refactor gradually and safely
• Able to process multiple accounts at once (concurrency)
• Stable and improved performance (parallelism)
• Enhanced observability
• Simple evolution to Lambda
Yelp
Transformed a decade-old monolith to microservices
Industry: Internet
Solution: Application modernization
Architecture: Serverless
34. Challenge
• Workflow management platform for video
teams has to be able to process high volumes of
media, transcode to different formats, cost-
effectively
Results
• Improved performance and lower costs
• Code is easily managed and debugged
• Increased code releases 20x using CI/CD
• Better performance, expect to scale to running
1,000 state machines a minute
Frame.io
Custom solution to optimize real-time transcoding
Industry: Media & Entertainment
Solution: Digital Supply Chain
Architecture: Microservices
36. Business Processes on AWS Step Functions
AWS Step Functions lets you
build visual workflows that
enable fast translation of
business requirements into
technical requirements. You
can build applications in a
matter of minutes, and
when needs change, you can
swap or reorganize
components without
customizing any code.
Build and update
apps quickly
AWS Step Functions
manages the logic of your
application for you, and
implements basic primitives
such as branching, parallel
execution, and timeouts.
This removes extra code that
may be repeated in your
microservices and functions.
Write less code
AWS Step Functions
manages state, checkpoints
and restarts for you to make
sure that your application
executes in order and as
expected. Built-in try/catch,
retry and rollback
capabilities deal with errors
and exceptions
automatically.
Improve resiliency
37. Best Practice
• Use Timeouts to Avoid Stuck Executions
• Use ARNs Instead of Passing Large Payloads
• Avoid Reaching the History Quota
• Handle Lambda Service Exceptions
• Avoid Latency When Polling for Activity Tasks
• Choosing Standard or Express Workflows
https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
Standard:
2000 per second started rate
4000 state transition
Charged by
Express:
100000 per second started rate
Nearly unlimited state transition per seconds
- Pass state can inject some fixed data into the state machine,
- The Map state ("Type": "Map") can be used to run a set of steps for each element of an input array. While the Parallel state executes multiple branches of steps using the same input, a Map state will execute the same steps for multiple entries of an array in the state input.
GetActivityTask
taskTokenA token that identifies the scheduled task. This token must be copied and included in subsequent calls to SendTaskHeartbeat, SendTaskSuccess or SendTaskFailure in order to report the progress or completion of the task.
TimeoutSeconds
Heartbeat
Wait up to 1-year
在 callback pattern 中,Step Functions 可以暫停工作流程,直到目標服務(例如 Amazon SNS, SQS, AWS Lambda 等等)透過 Step Functions API回傳 token 後再繼續執行工作流程,這樣的 pattern 讓您可以加入一些使用場景,例如需要加入人類互動場景如 review 和 approval 之類的管理流程
In a callback pattern, Step Functions pauses execution of the workflow until your application returns a token through the Step Functions application programming interface. This helps you automate workflows for applications that need human activities such as review and approval of documents or creating and managing work items in a customer support application. You can also use the callback pattern to build custom integrations with third-party services. Step Functions supports callback patterns with Amazon ECS, Amazon SNS, Amazon SQS, AWS Fargate, and AWS Lambda.
https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-human-approval.html
In the email account you used for the Amazon SNS topic earlier, open the message with the subject Required approval from AWS Step Functions.
The message includes separate URLs for Approve and Reject.
Use "Parameters" field to create a collection of key-value pairs that are passed as input. The values of each can either be static values that you include in your state machine definition, or selected from either the input or the context object with a path. For key-value pairs where the value is selected using a path, the key name must end in .$.
ResultPath can completely replace the input with the result of the task.
{“ResultPath”:”$”}
Discard result: {“ResultPath”:null}
Collect result : {“ResultPath”:$.”result”}
Use "Parameters" field to create a collection of key-value pairs that are passed as input. The values of each can either be static values that you include in your state machine definition, or selected from either the input or the context object with a path. For key-value pairs where the value is selected using a path, the key name must end in .$.
ResultPath can completely replace the input with the result of the task.
{“ResultPath”:”$”}
Discard result: {“ResultPath”:null}
Collect result : {“ResultPath”:$.”result”}
Use "Parameters" field to create a collection of key-value pairs that are passed as input. The values of each can either be static values that you include in your state machine definition, or selected from either the input or the context object with a path. For key-value pairs where the value is selected using a path, the key name must end in .$.
Use "Parameters" field to create a collection of key-value pairs that are passed as input. The values of each can either be static values that you include in your state machine definition, or selected from either the input or the context object with a path. For key-value pairs where the value is selected using a path, the key name must end in .$.
ResultPath can completely replace the input with the result of the task.
{“ResultPath”:”$”}
Discard result: {“ResultPath”:null}
Collect result : {“ResultPath”:$.”result”}
https://github.com/aws-samples/aws-codepipeline-stepfunctions
This action invokes a Lambda function (1) called the State Machine Trigger Lambda, which, in turn, triggers a Step Functions State Machine to process the request (2). The Lambda function sends a continuation token back to the pipeline (3) to continue its execution later and terminates. Seconds later, the pipeline invokes the Lambda function again (4), passing the continuation token received. The Lambda function checks the execution state of the state machine (5,6) and communicates the status to the pipeline. The process is repeated until the state machine execution is complete. Then the Lambda function notifies the pipeline that the corresponding pipeline action is complete (7). If the state machine has failed, the Lambda function will then fail the pipeline action and stop its execution (7). While running, the state machine triggers various Lambda functions to perform different tasks. The state machine and the pipeline are fully decoupled. Their interaction is handled by the Lambda function.
Developers have been able to use CodePipeline to build sophisticated automation pipelines that often require a single CodePipeline action to perform multiple tasks, fork into different execution paths, and deal with asynchronous behavior. For example, to deploy a Lambda function, a CodePipeline action might first inspect the changes pushed to the code repository. If only the Lambda code has changed, the action can simply update the Lambda code package, create a new version, and point the Lambda alias to the new version. If the changes also affect infrastructure resources managed by AWS CloudFormation, the pipeline action might have to create a stack or update an existing one through the use of a change set. In addition, if an update is required, the pipeline action might enforce a safety policy to infrastructure resources that prevents the deletion and replacement of resources. You can do this by creating a change set and having the pipeline action inspect its changes before updating the stack. Change sets that do not conform to the policy are deleted.
Developers have been able to use CodePipeline to build sophisticated automation pipelines that often require a single CodePipeline action to perform multiple tasks, fork into different execution paths, and deal with asynchronous behavior. For example, to deploy a Lambda function, a CodePipeline action might first inspect the changes pushed to the code repository. If only the Lambda code has changed, the action can simply update the Lambda code package, create a new version, and point the Lambda alias to the new version. If the changes also affect infrastructure resources managed by AWS CloudFormation, the pipeline action might have to create a stack or update an existing one through the use of a change set. In addition, if an update is required, the pipeline action might enforce a safety policy to infrastructure resources that prevents the deletion and replacement of resources. You can do this by creating a change set and having the pipeline action inspect its changes before updating the stack. Change sets that do not conform to the policy are deleted.
Developers have been able to use CodePipeline to build sophisticated automation pipelines that often require a single CodePipeline action to perform multiple tasks, fork into different execution paths, and deal with asynchronous behavior. For example, to deploy a Lambda function, a CodePipeline action might first inspect the changes pushed to the code repository. If only the Lambda code has changed, the action can simply update the Lambda code package, create a new version, and point the Lambda alias to the new version. If the changes also affect infrastructure resources managed by AWS CloudFormation, the pipeline action might have to create a stack or update an existing one through the use of a change set. In addition, if an update is required, the pipeline action might enforce a safety policy to infrastructure resources that prevents the deletion and replacement of resources. You can do this by creating a change set and having the pipeline action inspect its changes before updating the stack. Change sets that do not conform to the policy are deleted.
Thosoms Reuters 是個媒體與資訊供應商 ,每天提供大量的新聞, 訪談等影片給用戶。原本使用 Ffmpeg (singled thread) 30秒長就需要30秒,一小時訪談影片就會需要一小時,那樣慢慢的處理影片轉檔工作,之後他們時做了新的影片轉檔機制,搭配 AWS Step Functions 將原本只能慢慢從頭做到尾的轉檔工作,變成可以從影片中的關鍵影格並行處理,大大提升影片製作效率!
Most people have at least heard of Thomson. They have a lot of publications and a lot of TV channels and if you consume any news at all, it’s probably hard to get through the course of a day without reading some of their stories or watching some of their video clips.
There’s the problem. They have lots of video incoming, which is as usual. The incoming clips are, fortunately for them, all in the same format, so unlike many media companies, they don’t have too much of an input-conversion problem. However, they vary wildly in length, from a 3-second clip of a bomb going off to a half-hour-long interview with a newsmaker. They need to convert for delivery and this is news, so they need to do it fast. Historically they’ve used Ffmppeg, which you know about if you’re in this business, it’s what everybody uses. The problem is that it’s single-threaded, and so if you want to transcode a half-hour video, well, you can easily end up waiting a half-hour for Ffmpeg, which is not what you want.
It turns out that if you look at a stream of video frames, some of them are special and are called “Keyframes”. Results show that the actual video processing time of each segment is pretty much static at around 3s for a 0.5s segment. The initial identification of keyframes within the source video does depend on the size of the source video as you have to tick through the whole video to identify the keyframes. We’re currently doing further experimentation to make this stage more efficient to save even more time. This could feasibly take the processing of ANY video down to a few seconds.
APPROACH (for each video to each format):
Derive keyframe locations within source video
Split source video on keyframe boundaries
Process segments in parallel (typically 0.5s of video each segment)
Concat all resultant segments together
RESULTS:
Processing time for 15min video down from ~20min to ~2min
The bigger the source video, the more segments, the bigger the saving
LESSONS LEARNED:
Used S3 events to trigger processing of segments due to Step not supporting dynamic number of tasks
Used S3 URLs to stream source video in Lambda rather than having to copy full file (avoids exhausting disk space)
Scaling to 1000 Lambda easier than scaling 1000 EC2’s!
Coinbase – digital currency exchange
MAD principles: release features and update services quickly, reliably, securely.
Need for changes in their deployments:
Reduced deployment time investment from days to seconds
Increased reliability of deployments
Were able to deploy more frequently: iterating, innovating
原本單一且巨大的 billing 系統只能批次的一筆筆處理每個帳戶billing事務,透過 AWS Step functions 拆解成幾個子系統,變可以在同一時間處理好幾個帳戶的帳務問題,大大提升了執行效率
Yelp is focused on local businesses - they’re around to help you find what you're looking for, whatever city you happen to be in
When you buy something on Yelp, such as food delivery, restaurant reservations, advertising contracts and so on, all of those different products go through a set of teams in the revenue and commerce side of the company that are responsible for processing payments, making sure they get the right money at the right time from people who owe it to Yelp, and then paying out money that they owe to other people.
Their subscription billing process handles recurring purchases, and it's really important.
Monthly invoicing for advertising accounts
Business critical, happens at scale (>100k accts)
Older code, evolved over ~10 years
They have 3 steps to subscription billing: bill all accounts, create invoices and collect all accounts, in the middle of a much larger scheduled job. They were chaining one after another, operating over all accounts at Yelp.
Yelp wanted to make this system a lot better and improve it, wanted to take advantage of new technologies like serverless, but needed to dig out from 10 years of maintenance costs and burdens and had a couple of challenges
Why haven’t we refactored this before? It was too big, too messy, and frankly, it worked and they were scared to touch it
What made refactoring this up hard? Refactoring this whole system at once was very complicated. It spanned teams, on-call rotations, conceptual subsystems and quite a bit more. Doing any of these was really hard, expensive and frankly dangerous.
There also wasn't any API for this business practice, it was essentially a job that ran nightly, and if you never touched it, it kept doing the same thing and the company makes money. So API is a really generous word for what it was. That makes it really hard to turn it into something like lambdas that require a little bite-sized functionality and it makes it really hard to turn it into services if you just don't know what's in the middle of this process - and that's where Step Functions came in
Solution
To improve their old subscription billing process and to be able to use these nicer technologies in the future, Yelp turned to Step Functions to help with the gradual refactoring.
Their business-critical process lived in the sequence of three jobs and they decided to take the first of those steps - billing of accounts - and move that onto step function state machine
Unfortunately scoping it down to just billing of accounts didn't actually solve our problem that didn't magically make this easy to do. That big box of functionality was written a long time ago, had gotten modified and had grown organically over those ten years and so it was still complicated, and they didn't really know what was going on inside of it. If they changed it and broke it they might not have any feedback to tell us they were in trouble you need to go reverse whatever you just did.
It also had a lot of old kind of complex logic. Python's greatest skill is not concurrency and over the years they had had problems with scaling this out. They were processing a hundred thousand accounts a night and didn’t want it to take more than 24 hours - ideally a lot less than that. They built out a custom parallelism framework to make that possible, so very roughly we took all the accounts we needed to bill, broke them into chunks and farmed out the work to other individual processes. They started here in the refactoring.
They took that high level goal - bill all the accounts - and instead of breaking it into chunks said we can have each individual accounts job billing one account in a step functions workflow, then do a bunch of these in parallel
Turns out it has only one state which had all of the code and they literally lifted it out and put it in the middle of a workflow as a single step
But they got to act like we're a service now and now have an API interface they can manage - really basic stuff we get from just moving into this single step workflow
When they started they were able to use these activity tasks API driven workflows and essentially just talked back and forth with Amazon over their API is to get work and tell that that we had succeeded
Using SFN as the engine for handling concurrency meant they could easily scale up or down the degree of concurrency by just submitting more workflows at once instead of needing to provision more machines to handle the additional capacity – right off the bat Yelp got a boost in efficiency of the parallelism
They also had improved observability. One problem we had is our old process could get slow it could get slow for every single account and then 24 hours later we would be 10% through our day and we'd say oh what happened that was unfortunate right we want fast feedback when stuff is slowing down and we want to get yelled at really quickly so we can put in these timeouts and we can say this activity needs to take no more than 20 minutes we can put in little heartbeats and we can say if I don't check in with you every minute something is going wrong please don't wait 20 minutes to tell me that the whole thing is screwed up and there's really good flexible support for this and we also got better observability.
Old world: Broken accounts have to block all other accounts; recovery is incredibly urgent and time-sensitive
New world: “Normal” errors are handled explicitly, Decouple error handling behavior from actual billing behavior, very obvious metrics (did we hit the “handle failure” state too much today?) when things go wrong
I talked about that first workflow having just one step right build a whole account do all this stuff but one of the first changes we made was to actually break out two use cases away from that and essentially decouple the error handling so if things were going really well that's awesome we're going to have this handle success state that basically just tells us at Yelp hey it worked and if things go poorly we're going to have a handle failure state and in the beginning we didn't put any complexity into this we're essentially just saying let's represent the fact that it didn't go well in the future maybe we want to page somebody maybe we want to wake Scott up in the middle of the night and say hey you need to go look at something it's broken but for now we're just going to literally have a state that says a thing is going wrong in in one particular workflow so we can keep track of it and one really benefit here is we can say no matter what happens here whether a thing went well or a thing went poorly the state machines shouldn't die in the middle of it right I'm explicitly representing failure here in a way that should theoretically be cleanly handled and if we can automate what to do in the case of failure the account itself can still get billed in a consistent way if not a finally successful way and for us when we're handling money being able to be consistent is worth a tremendous amount
And, if when they decide the time is right, we can flexibly pick individual tasks and move them over to lambdas and SFN stays with us the whole way from monolith to serverless, helping us quickly build more efficient and resilient systems.
Benefits
Really safe refactoring now so when we take that middle bill account step and we say I wish that this were more parallel Amazon step functions has an ability to run things in parallel we can break apart one big task into three parallel tasks I'm not handling any parallelism any joining any waiting for completion which I love doing because it's hard and instead I get to focus on writing really bite-sized functionality in here and so that kind of brings us to what this has to do with building towards serverless.
One of the things that we really found was wonderful is step functions isn't very opinionated on how my architecture looks it's happy to support me if I want to write a workflow with one activity and step functions isn't going to judge me or not run if I'm just running a ton of code inside of one activity for 20 minutes at a time but as I sort of refactor my own system as I get ready to move towards a more bite-sized more decoupled future it's there to keep up with us and in the meantime it's providing some really valuable stuff even before I'm in this decoupled serverless microservices future so it's got a lot of ability to add parallelism. It's really segmenting my work into different activities that can't talk to each other except through the step function itself and we get this observability of any errors or of how the system is working and that strong foundation for the future experimentation is what we've gotten really excited about
so again just running through the history of what we just described:
started out with one big function doing all the stuff and that was where we were when we began
put it into a workflow right and the workflow did all the stuff in one big function no big changes there
started to decouple it so we took out that error handling we were made ourselves aware of the success and failure and the difference between the two cases so that we could write code that handled them explicitly
started to do this refactoring more aggressively over time so we were able to say hey let's break out a parallel state because we think we can we can think it'll make individual functionality more targeted and to the point so I can write something that just knows how to do bookkeeping for the upgrades product versus ad campaign product versus something
and all my code gets Dumber in the process
longer-term we get to do this sort of directionally: if we think serverless is the right answer if we think lambda is something that's relevant to us, step functions is happy for us to make individual activities lambda functions
this is nice from my perspective because I don't want to rush into these kinds of things I want to be ready for it and decide it's the right choice step functions will support us whether we're all on these API driven sort of old school batch jobs or if we want to go ahead and try one lambda function or a few lambda functions where we think they're appropriate - either way the framework still works
使用 Step Functions 將功能拆解成一個一個小部分,程式碼更好維護和除錯,code release 效率提高了 20倍,並得到更好的整體效能
Frame.io is a leading workflow management platform for video teams. Frame.io built their own custom solution.
Frame.io is a general-purpose toolkit for doing lots of useful things with videos. And, quite sensibly, they decided to offload the actual video processing into the cloud. From small production agencies to major broadcast media companies, video teams of all sizes rely on Frame.io to streamline their media review and collaboration process. Frame.io uses Step Functions to process media, transcode to different formats, create thumbnails, and much more.
No fleet code required
No deployment code required
Frame.io broke everything down into tiny segments, so they could always do the transcoding in a Lambda function. These guys sometimes need to process way bigger chunks, so for those they think an ECS task is a better choice, both because they might run into Lambda limits, and for economic reasons – if you’re going to keep a Docker container running hot for an appreciable number of minutes, you might save some money, compared to Lambda. Anyhow, they were trying to stich this together in a pub/sub architecture and experiencing some pain.
Use a collection of Step Functions to process user media files. Job poller pattern is used to run ECS batch tasks. Choice states decide whether to use AWS Lambda or AWS ECS to run transcodes, depending on the duration and file size of the source. Began with SNS for internal service to service messaging, but realized that CloudWatch Events could do everything with more flexibility than SNS, and trigger our Step Functions without a Lambda proxy.
Frame.io now has a reliable process for manipulating videos, with a code that is easily managed and debugged, which allows them to increase their code releases 20x. They are also experiencing better overall performance and expect to scale to running 1,000 state machines a minute.
No timeout is set for the activity worker by default, so Step Functions will keep waiting. Explicitly configure timeout in order to prevent from stuck executions
ARNs: Storing parameters on S3 and only pass the configuration to the step functions (Executions that pass large payloads of data between states can be terminated. If the data you are passing between states might grow to over 32 KB)
AWS Step Functions has a hard quota of 25,000 entries
GetActivityTask is designed to provide taskToken exactly once. If a taskToken is dropped while communicating with an activity worker, a number of GetActivityTask requests can be blocked for 60 seconds waiting for a response until GetActivityTask times out. For production systems, we recommend at least 100 open polls per activity ARN's at each point in time. If one poll gets blocked, and a portion of those polls queue up behind it, there are still many more requests that will receive a taskToken to process work while the GetActivityTask request is blocked.