Do you have the next best idea? How will you quickly migrate a legacy feature to new world for almost free? This talk will give you how to architect and implement your scenario for a cloud-oriented solution. We will share the best practices for storing your state in database; ways to decouple by events and suggested patterns for serverless. You will be equipped with taking advantage of low-cost serverless computing in a secure way and how to minimize operational costs. It will mostly focus AWS offerings like Serverless Aurora, API Gateway and Lambda functions for solutions blueprint
With AWS you can choose the right database for the right job. Given the myriad of choices, from relational databases to non-relational stores, this session will profile details and examples of some of the choices available to you (MySQL, RDS, Elasticache, Redis, Cassandra, MongoDB and DynamoDB), with details on real world deployments from customers using Amazon RDS, ElastiCache and DynamoDB.
(BDT310) Big Data Architectural Patterns and Best Practices on AWSAmazon Web Services
The world is producing an ever increasing volume, velocity, and variety of big data. Consumers and businesses are demanding up-to-the-second (or even millisecond) analytics on their fast-moving data, in addition to classic batch processing. AWS delivers many technologies for solving big data problems. But what services should you use, why, when, and how? In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
JustGiving – Serverless Data Pipelines, API, Messaging and Stream ProcessingLuis Gonzalez
What to Expect from the Session
• Recap of some AWS services
• Event-driven data platform at JustGiving
• Serverless computing
• Six serverless patterns
• Serverless recommendations and best practices
(SOV202) Choosing Among AWS Managed Database Services | AWS re:Invent 2014Amazon Web Services
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We'll cover how each service might help support your application, how much each service costs, and how to get started.
2016 Utah Cloud Summit: Big Data Architectural Patterns and Best Practices on...1Strategy
In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
AWS offers customers a range of different database options. These include Amazon DynamoDB, a fully-managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data as well as Amazon Relational Database Service (RDS), a service that makes it easy to set up, operate, and scale a relational database in the cloud with support for MySQL, Microsoft SQL Server, PostgreSQL, and Oracle Database. In this session you’ll get an overview of AWS database options and how they might help support your application and see how to get started.
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We’ll cover how each service might help support your application, how much each service costs, and how to get started.
With AWS you can choose the right database for the right job. Given the myriad of choices, from relational databases to non-relational stores, this session will profile details and examples of some of the choices available to you (MySQL, RDS, Elasticache, Redis, Cassandra, MongoDB and DynamoDB), with details on real world deployments from customers using Amazon RDS, ElastiCache and DynamoDB.
(BDT310) Big Data Architectural Patterns and Best Practices on AWSAmazon Web Services
The world is producing an ever increasing volume, velocity, and variety of big data. Consumers and businesses are demanding up-to-the-second (or even millisecond) analytics on their fast-moving data, in addition to classic batch processing. AWS delivers many technologies for solving big data problems. But what services should you use, why, when, and how? In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
JustGiving – Serverless Data Pipelines, API, Messaging and Stream ProcessingLuis Gonzalez
What to Expect from the Session
• Recap of some AWS services
• Event-driven data platform at JustGiving
• Serverless computing
• Six serverless patterns
• Serverless recommendations and best practices
(SOV202) Choosing Among AWS Managed Database Services | AWS re:Invent 2014Amazon Web Services
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We'll cover how each service might help support your application, how much each service costs, and how to get started.
2016 Utah Cloud Summit: Big Data Architectural Patterns and Best Practices on...1Strategy
In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
AWS offers customers a range of different database options. These include Amazon DynamoDB, a fully-managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data as well as Amazon Relational Database Service (RDS), a service that makes it easy to set up, operate, and scale a relational database in the cloud with support for MySQL, Microsoft SQL Server, PostgreSQL, and Oracle Database. In this session you’ll get an overview of AWS database options and how they might help support your application and see how to get started.
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We’ll cover how each service might help support your application, how much each service costs, and how to get started.
MongoDB is one of the fastest growing NoSQL workloads on AWS due to its simplicity and scalability, and recent product additions by the AWS team have only improved those traits. In this session, we’ll talk about various AWS offerings and how they fit together with MongoDB -- including CloudFormation, Elastic MapReduce, Route53, Elastic Beanstalk, Elastic Load Balancing, and more -- and how they can be leveraged to enhance your MongoDB experience.
Getting to 1.5M Ads/sec: How DataXu manages Big DataQubole
DataXu sits at the heart of the all-digital world, providing a data platform that manages tens of millions of dollars of digital advertising investments from Global 500 brands. The DataXu data platform evaluates 1.5 million online ad opportunities every second for our customers, allowing them to manage and optimize their marketing investments across all digital channels. DataXu employs a wide range of AWS services: Cloud Front, Cloud Trail, CloudWatch, Data Pipeline, Direct Connect, Dynamo DB, EC2, EMR, Glacier, IAM, Kinesis, RDS, Redshift, Route53, S3, SNS, SQS, and VPC to run various workloads at scale for DataXu data platform.
In addition, DataXu also uses Qubole Data Service, QDS, to offer a Unified Analytics Interface tool to DataXu customers. Qubole, a member of APN provides self-managing Big data infrastructure in the Cloud which leverages spot pricing for cost-efficiencies, provides fast performance, and most importantly a streamlined user-interface for ease of use.
Attendees will learn how Qubole provided self-managing Hadoop clusters in the AWS Cloud accelerated DataXu’s batch-oriented analysis jobs; and how Qubole integration with Amazon Redshift enabled DataXu to preform low latency and interactive analysis. Further, in the session we'll take a look at how DataXu opened up QDS access to their customers using QDS user interface thereby providing them with a single tool for both batch-oriented and interactive analysis. By using the QDS user interface buyers of the DataXu data service could perform all manner of analysis against the data stored in their AWS S3 bucket.
Speakers:
Scott Ward
Solutions Architect at Amazon Web Services
Ashish Dubey
Solutions Architect at Qubole
Yekesa Kosuru
VP Engineering at DataXu
(BDT322) How Redfin & Twitter Leverage Amazon S3 For Big DataAmazon Web Services
Analyzing large data sets requires significant compute and storage capacity that can vary in size based on the amount of input data and the analysis required. This characteristic of big data workloads is ideally suited to the pay-as-you-go cloud model, where applications can easily scale up and down based on demand. Learn how Amazon S3 can help scale your big data platform. Hear from Redfin and Twitter about how they build their big data platforms on AWS and how they use S3 as an integral piece of their big data platforms.
Learning Objectives: - Get an overview of Amazon DynamoDB improvements in 2017
- Learn about the new features of Amazon DynamoDB, including Time-to-live (TTL), Tagging, VPC-Endpoints, DynamoDB Accelerator (DAX), Database Migration Service (DMS) support, and more.
- Learn about the benefits these new features deliver to you
Day 2 - Amazon RDS - Letting AWS run your Low Admin, High Performance DatabaseAmazon Web Services
Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and re-sizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. In this webinar we review the different types of Amazon RDS available and how to move your existing databases to Amazon RDS with minimum disruption.
Reasons to attend:
- Learn how Amazon RDS can reduce the overhead of running high performance mission critical databases.
- Learn how to migrate your existing database workloads into Amazon RDS running on the AWS Cloud.
- Learn how to scale up and scale down your Amazon RDS instance and save money with reserved instances.
For our eReader development project, we had to find a persistent storage for our JSON documents. After initial scanning we zeroed into two products DynamoDB and MongoDB. These slides take a deeper dive in the selection of our JSON data store.
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...Rustem Feyzkhanov
One of the main issues with ML and DL deployment is finding the right way to train and operationalize the model within the company. Serverless approach for deep learning provides simple, scalable, affordable yet reliable architecture. The challenge of this approach is to keep in mind certain limitations in CPU, GPU and RAM, and organize training and inference of your model.
My presentation will show how to utilize services like Amazon SageMaker, AWS Batch, AWS Fargate, AWS Lambda and AWS Step Functions to organize deep learning workflows.
Database Migration – Simple, Cross-Engine and Cross-Platform MigrationAmazon Web Services
Learn about the new AWS Database Migration Service, which helps you migrate databases with minimal downtime from on-premises and Amazon EC2 environments to Amazon RDS, Amazon Redshift, Amazon Aurora and EC2 databases.
Cassandra Day SV 2014: Spark, Shark, and Apache CassandraDataStax Academy
This session covers our experience with using the Spark and Shark frameworks for running real-time queries on top of Cassandra data.We will start by surveying the current Cassandra analytics landscape, including Hadoop and HIVE, and touch on the use of custom input formats to extract data from Cassandra. We will then dive into Spark and Shark, two memory-based cluster computing frameworks, and how they enable often dramatic improvements in query speed and productivity, over the standard solutions today.
We have seen tremendous growth in near real-time ("nearline") processing at LinkedIn in recent years. LinkedIn now uses Apache Samza to process well over a Trillion messages every day across thousands of applications. Apache Samza serves as the foundation for several application platforms at LinkedIn, spanning a wide variety of use cases like security, notifications, machine learning, monitoring, search, and more. In this talk we will explore various features of Apache Samza that provide the flexibility and scalability to we need to power stream processing at massive scale.
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & DataductAmazon Web Services
"As data volumes grow, managing and scaling data pipelines for ETL and batch processing can be daunting. With more than 13.5 million learners worldwide, hundreds of courses, and thousands of instructors, Coursera manages over a hundred data pipelines for ETL, batch processing, and new product development.
In this session, we dive deep into AWS Data Pipeline and Dataduct, an open source framework built at Coursera to manage pipelines and create reusable patterns to expedite developer productivity. We share the lessons learned during our journey: from basic ETL processes, such as loading data from Amazon RDS to Amazon Redshift, to more sophisticated pipelines to power recommendation engines and search services.
Attendees learn:
Do's and don’ts of Data Pipeline
Using Dataduct to streamline your data pipelines
How to use Data Pipeline to power other data products, such as recommendation systems
What’s next for Dataduct"
(GAM301) Real-Time Game Analytics with Amazon Kinesis, Amazon Redshift, and A...Amazon Web Services
Success in free-to-play gaming requires knowing what your players love most. The faster you can respond to players' behavior, the better your chances of success. Learn how mobile game company GREE, with over 150 million users worldwide, built a real-time analytics pipeline for their games using Amazon Kinesis, Amazon Redshift, and Amazon DynamoDB. They walk through their analytics architecture, the choices they made, the challenges they overcame, and the benefits they gained. Also hear how GREE migrated to the new system while keeping their games running and collecting metrics.
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We’ll cover how each service might help support your application, how much each service costs, and how to get started.
Speaker:
Shaun Pearce, AWS Solutions Architect
NASA LandSat data can be stored, transformed, navigated, and visualized. In this session we will explore how the LandSat dataset is stored in Amazon Simple Storage Service (S3), one of the recommended cloud storage services in AWS for storage of petabytes of data, and how data stored in S3 can be processed on the server with the Lambda service, visualized for users, and made available to search engines.
Create by: Ben Snively, Senior Solutions Architect
This presentation covers best practices for running MongoDB on AWS. We also discuss how to utilize the automation features of MMS to spin up new clusters in minutes on AWS.
For people who start to create a cloud service, it’s really important to know how to create a scalable cloud service to fit the growth of the future workloads. In this session, we will introduce how to design a scalable cloud service including AWS services introduction and best practices.
Amazon Aurora New Features - September 2016 Webinar SeriesAmazon Web Services
Amazon Aurora is a fully managed MySQL-compatible database with high-end commercial database features and performance at one-tenth the cost. Since launching Aurora a year ago we have added many new capabilities and features. Some of these features include encryption, database snapshot sharing, enhanced monitoring, cross-region replication, S3 binary snapshot ingestion and customized failover priority. In this session we'll demonstrate how these features work and discuss how you can make the best use of them.
Learning Objectives:
• Learn about the newly added features of Aurora
• Learn how to use those features
• Learn when and why to use those features
Who Should Attend:
• IT Managers, DBAs, Enterprise and Solution Architects, Devops Engineers and Developers
AWS Webcast - Managing Big Data in the AWS Cloud_20140924Amazon Web Services
This presentation deck will cover specific services such as Amazon S3, Kinesis, Redshift, Elastic MapReduce, and DynamoDB, including their features and performance characteristics. It will also cover architectural designs for the optimal use of these services based on dimensions of your data source (structured or unstructured data, volume, item size and transfer rates) and application considerations - for latency, cost and durability. It will also share customer success stories and resources to help you get started.
MongoDB is one of the fastest growing NoSQL workloads on AWS due to its simplicity and scalability, and recent product additions by the AWS team have only improved those traits. In this session, we’ll talk about various AWS offerings and how they fit together with MongoDB -- including CloudFormation, Elastic MapReduce, Route53, Elastic Beanstalk, Elastic Load Balancing, and more -- and how they can be leveraged to enhance your MongoDB experience.
Getting to 1.5M Ads/sec: How DataXu manages Big DataQubole
DataXu sits at the heart of the all-digital world, providing a data platform that manages tens of millions of dollars of digital advertising investments from Global 500 brands. The DataXu data platform evaluates 1.5 million online ad opportunities every second for our customers, allowing them to manage and optimize their marketing investments across all digital channels. DataXu employs a wide range of AWS services: Cloud Front, Cloud Trail, CloudWatch, Data Pipeline, Direct Connect, Dynamo DB, EC2, EMR, Glacier, IAM, Kinesis, RDS, Redshift, Route53, S3, SNS, SQS, and VPC to run various workloads at scale for DataXu data platform.
In addition, DataXu also uses Qubole Data Service, QDS, to offer a Unified Analytics Interface tool to DataXu customers. Qubole, a member of APN provides self-managing Big data infrastructure in the Cloud which leverages spot pricing for cost-efficiencies, provides fast performance, and most importantly a streamlined user-interface for ease of use.
Attendees will learn how Qubole provided self-managing Hadoop clusters in the AWS Cloud accelerated DataXu’s batch-oriented analysis jobs; and how Qubole integration with Amazon Redshift enabled DataXu to preform low latency and interactive analysis. Further, in the session we'll take a look at how DataXu opened up QDS access to their customers using QDS user interface thereby providing them with a single tool for both batch-oriented and interactive analysis. By using the QDS user interface buyers of the DataXu data service could perform all manner of analysis against the data stored in their AWS S3 bucket.
Speakers:
Scott Ward
Solutions Architect at Amazon Web Services
Ashish Dubey
Solutions Architect at Qubole
Yekesa Kosuru
VP Engineering at DataXu
(BDT322) How Redfin & Twitter Leverage Amazon S3 For Big DataAmazon Web Services
Analyzing large data sets requires significant compute and storage capacity that can vary in size based on the amount of input data and the analysis required. This characteristic of big data workloads is ideally suited to the pay-as-you-go cloud model, where applications can easily scale up and down based on demand. Learn how Amazon S3 can help scale your big data platform. Hear from Redfin and Twitter about how they build their big data platforms on AWS and how they use S3 as an integral piece of their big data platforms.
Learning Objectives: - Get an overview of Amazon DynamoDB improvements in 2017
- Learn about the new features of Amazon DynamoDB, including Time-to-live (TTL), Tagging, VPC-Endpoints, DynamoDB Accelerator (DAX), Database Migration Service (DMS) support, and more.
- Learn about the benefits these new features deliver to you
Day 2 - Amazon RDS - Letting AWS run your Low Admin, High Performance DatabaseAmazon Web Services
Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and re-sizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. In this webinar we review the different types of Amazon RDS available and how to move your existing databases to Amazon RDS with minimum disruption.
Reasons to attend:
- Learn how Amazon RDS can reduce the overhead of running high performance mission critical databases.
- Learn how to migrate your existing database workloads into Amazon RDS running on the AWS Cloud.
- Learn how to scale up and scale down your Amazon RDS instance and save money with reserved instances.
For our eReader development project, we had to find a persistent storage for our JSON documents. After initial scanning we zeroed into two products DynamoDB and MongoDB. These slides take a deeper dive in the selection of our JSON data store.
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...Rustem Feyzkhanov
One of the main issues with ML and DL deployment is finding the right way to train and operationalize the model within the company. Serverless approach for deep learning provides simple, scalable, affordable yet reliable architecture. The challenge of this approach is to keep in mind certain limitations in CPU, GPU and RAM, and organize training and inference of your model.
My presentation will show how to utilize services like Amazon SageMaker, AWS Batch, AWS Fargate, AWS Lambda and AWS Step Functions to organize deep learning workflows.
Database Migration – Simple, Cross-Engine and Cross-Platform MigrationAmazon Web Services
Learn about the new AWS Database Migration Service, which helps you migrate databases with minimal downtime from on-premises and Amazon EC2 environments to Amazon RDS, Amazon Redshift, Amazon Aurora and EC2 databases.
Cassandra Day SV 2014: Spark, Shark, and Apache CassandraDataStax Academy
This session covers our experience with using the Spark and Shark frameworks for running real-time queries on top of Cassandra data.We will start by surveying the current Cassandra analytics landscape, including Hadoop and HIVE, and touch on the use of custom input formats to extract data from Cassandra. We will then dive into Spark and Shark, two memory-based cluster computing frameworks, and how they enable often dramatic improvements in query speed and productivity, over the standard solutions today.
We have seen tremendous growth in near real-time ("nearline") processing at LinkedIn in recent years. LinkedIn now uses Apache Samza to process well over a Trillion messages every day across thousands of applications. Apache Samza serves as the foundation for several application platforms at LinkedIn, spanning a wide variety of use cases like security, notifications, machine learning, monitoring, search, and more. In this talk we will explore various features of Apache Samza that provide the flexibility and scalability to we need to power stream processing at massive scale.
(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & DataductAmazon Web Services
"As data volumes grow, managing and scaling data pipelines for ETL and batch processing can be daunting. With more than 13.5 million learners worldwide, hundreds of courses, and thousands of instructors, Coursera manages over a hundred data pipelines for ETL, batch processing, and new product development.
In this session, we dive deep into AWS Data Pipeline and Dataduct, an open source framework built at Coursera to manage pipelines and create reusable patterns to expedite developer productivity. We share the lessons learned during our journey: from basic ETL processes, such as loading data from Amazon RDS to Amazon Redshift, to more sophisticated pipelines to power recommendation engines and search services.
Attendees learn:
Do's and don’ts of Data Pipeline
Using Dataduct to streamline your data pipelines
How to use Data Pipeline to power other data products, such as recommendation systems
What’s next for Dataduct"
(GAM301) Real-Time Game Analytics with Amazon Kinesis, Amazon Redshift, and A...Amazon Web Services
Success in free-to-play gaming requires knowing what your players love most. The faster you can respond to players' behavior, the better your chances of success. Learn how mobile game company GREE, with over 150 million users worldwide, built a real-time analytics pipeline for their games using Amazon Kinesis, Amazon Redshift, and Amazon DynamoDB. They walk through their analytics architecture, the choices they made, the challenges they overcame, and the benefits they gained. Also hear how GREE migrated to the new system while keeping their games running and collecting metrics.
In addition to running databases in Amazon EC2, AWS customers can choose among a variety of managed database services. These services save effort, save time, and unlock new capabilities and economies. In this session, we make it easy to understand how they differ, what they have in common, and how to choose one or more. We explain the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service; Amazon RDS, a relational database service in the cloud; Amazon ElastiCache, a fast, in-memory caching service in the cloud; and Amazon Redshift, a fully managed, petabyte-scale data-warehouse solution that can be surprisingly economical. We’ll cover how each service might help support your application, how much each service costs, and how to get started.
Speaker:
Shaun Pearce, AWS Solutions Architect
NASA LandSat data can be stored, transformed, navigated, and visualized. In this session we will explore how the LandSat dataset is stored in Amazon Simple Storage Service (S3), one of the recommended cloud storage services in AWS for storage of petabytes of data, and how data stored in S3 can be processed on the server with the Lambda service, visualized for users, and made available to search engines.
Create by: Ben Snively, Senior Solutions Architect
This presentation covers best practices for running MongoDB on AWS. We also discuss how to utilize the automation features of MMS to spin up new clusters in minutes on AWS.
For people who start to create a cloud service, it’s really important to know how to create a scalable cloud service to fit the growth of the future workloads. In this session, we will introduce how to design a scalable cloud service including AWS services introduction and best practices.
Amazon Aurora New Features - September 2016 Webinar SeriesAmazon Web Services
Amazon Aurora is a fully managed MySQL-compatible database with high-end commercial database features and performance at one-tenth the cost. Since launching Aurora a year ago we have added many new capabilities and features. Some of these features include encryption, database snapshot sharing, enhanced monitoring, cross-region replication, S3 binary snapshot ingestion and customized failover priority. In this session we'll demonstrate how these features work and discuss how you can make the best use of them.
Learning Objectives:
• Learn about the newly added features of Aurora
• Learn how to use those features
• Learn when and why to use those features
Who Should Attend:
• IT Managers, DBAs, Enterprise and Solution Architects, Devops Engineers and Developers
AWS Webcast - Managing Big Data in the AWS Cloud_20140924Amazon Web Services
This presentation deck will cover specific services such as Amazon S3, Kinesis, Redshift, Elastic MapReduce, and DynamoDB, including their features and performance characteristics. It will also cover architectural designs for the optimal use of these services based on dimensions of your data source (structured or unstructured data, volume, item size and transfer rates) and application considerations - for latency, cost and durability. It will also share customer success stories and resources to help you get started.
Antoine Genereux takes us on a detailed overview of the Database solutions available on the AWS Cloud, addressing the needs and requirements of customers at all levels. He also discusses Business Intelligence and Analytics solutions.
JustGiving | Serverless Data Pipelines, API, Messaging and Stream ProcessingBEEVA_es
PPT de la presentación de Richard T. Freeman en el Meetup de BEEVA. Marzo 2017.
https://www.meetup.com/es-ES/Innovative-technology-BEEVA/events/238027581/
Data warehousing in the era of Big Data: Deep Dive into Amazon RedshiftAmazon Web Services
Analyzing big data quickly and efficiently requires a data warehouse optimized to handle and scale for large datasets. Amazon Redshift is a fast, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all of your data for a fraction of the cost of traditional data warehouses. In this session, we take an in-depth look at data warehousing with Amazon Redshift for big data analytics. We cover best practices to take advantage of Amazon Redshift's columnar technology and parallel processing capabilities to deliver high throughput and query performance. We also discuss how to design optimal schemas, load data efficiently, and use work load management.
Highly available and scalable web hosting can be complex and expensive. Learn how Amazon Web Services provides the reliable, scalable, secure, and high performance infrastructure required for web applications while enabling an elastic, scale out and scale down infrastructure to match IT costs in real time as customer traffic fluctuates.
Serverless design considerations for Cloud Native workloadsTensult
We have built a news website with more than a billion views per month and we are sharing the learnings from that experience covering Serverless architectures, Design considerations, and Gotchas.
Scaling on AWS for the First 10 Million Users at Websummit DublinAmazon Web Services
In this talk from the Dublin Websummit 2014 AWS Technical Evangelist Ian Massingham discusses the techniques that AWS customers can use to create highly scalable infrastructure to support the operation of large scale applications on the AWS cloud.
Includes a walk-through of how you can evolve your architecture as your application becomes more popular and you need to scale up your infrastructure to support increased demand.
Scaling on AWS for the First 10 Million Users at Websummit DublinIan Massingham
In this talk from the Dublin Websummit 2014 AWS Technical Evangelist Ian Massingham discusses the techniques that AWS customers can use to create highly scalable infrastructure to support the operation of large scale applications on the AWS cloud.
Includes a walk-through of how you can evolve your architecture as your application becomes more popular and you need to scale up your infrastructure to support increased demand.
Amazon Aurora is a MySQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. The service is now in preview. Come to our session for an overview of the service and learn how Aurora delivers up to five times the performance of MySQL yet is priced at a fraction of what you'd pay for a commercial database with similar performance and availability.
Speakers:
Ronan Guilfoyle, AWS Solutions Architect
Brian Scanlan, Engineer, Intercom.io
Storage options for Analytics are not one size fits all. To deliver the best solution, you need to understand the use case, performance requirements, and users of the system. This session will break down the options you have in Azure to build a data analytics ecosystem, and explain why everyone's talking about data lakes and where's best to build your data warehouse.
AWS Summit 2014 Melbourne - Breakout 5
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
Presenter: Craig Dickson, Solutions Architect, Amazon Web Services
As serverless architectures become more popular, AWS customers need a framework of patterns to help them deploy their workloads without managing servers or operating systems.
As serverless architectures become more popular, AWS customers need a framework of patterns to help them deploy their workloads without managing servers or operating systems.
AWS re:Invent 2016: Amazon Aurora Best Practices: Getting the Best Out of You...Amazon Web Services
Amazon Aurora is a fully managed relational database engine that provides higher performance, availability and durability than previously possible using conventional monolithic database architectures. After launching a year ago, we continued adding many new features and capabilities to Aurora. In this session AWS Aurora experts will discuss the best practices that will help you put these capabilities to the best use. You will also hear from Amazon Aurora customer Intercom on the best practices they adopted for moving live databases with over two billion rows to a new datastore in Amazon Aurora with almost no downtime or lost records.
Intercom was founded to provide a fundamentally new way for Internet businesses to communicate with customers at scale. For growing startups like Intercom, it’s natural for the load on datastores to grow on a weekly basis. The usual solution to this problem is to get a bigger box from AWS. But very soon you reach a point where bigger boat is not an option anymore. You will learn about the benefits of moving to such a datastore, the problems it introduced, and all about the new ability for scaling that was not there before.
Cloud computing gives you a number of advantages, such as the ability to scale your web application or website on demand. If you have a new web application and want to use cloud computing, you might be asking yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We show you how to best combine different AWS services, how to make smarter decisions for architecting your application, and how to scale your infrastructure in the cloud.
Cloud computing gives you a number of advantages, such as the ability to scale your web application or website on demand. If you have a new web application and want to use cloud computing, you might be asking yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We show you how to best combine different AWS services, how to make smarter decisions for architecting your application, and how to scale your infrastructure in the cloud.
Similar to Serverless architectures: APIs, Serverless Functions, Microservices - How to get your product quickly to market? (20)
How do you improve the Config Model? Where to use Windows Server AppFabric? How to provide a RoutingService in the Framework? How to enable dynamic apps with Discovery?
How to find out production issues? Where to look for errors when application crashes in live environment? How to Visual Studio 2010 for replicating post mortem scenarios in difficult to reproduce errors? Using Source server, PDB symbols in old fashioned way for new age WCF services.
Basics & Intro to SQL Server Reporting Services: Sql Server Ssrs 2008 R2Bala Subra
How to use SQL Server 2008 R2 reporting services instead of ASP.NET for every data presentation problems? Where SSRS is superior to raw SQL? How it helps QA to automate their test cases?
Denny Lee\'s Data Camp v1.0 talk on SSRS Best Practices for ITBala Subra
Building and Deploying Large Scale SQL Server Reporting Services Environments Technical Note:
* Report Catalog sizing
* The benefits of File System snapshots for SSRS 2005
* Why File System snapshots may not help for SSRS 2008
* Using Cache Execution
* Load Balancing your Network
* Isolate your workloads
* Report Data Performance Considerations
BizTalk 2010 with Appfabric Hosting in the Cloud: WCF Services vs BT2010Bala Subra
How do you decide which Appfabric offering to use? When to prefer WCF services vs BizTalk solution? How to get the best performance with horizontal scaling in SOA?
How to ace your .NET technical interview :: .Net Technical Check TuneupBala Subra
This session is just not a brain dump of a technical interview on Microsoft technologies. It will be refresher on various pieces of the .NET, Database, OO, Process world. It will serve as a caution for interviewers to red flag their questions which are circulated on the webdom. For all the inquisitive and MCP certified brains, this will serve as a ‘whodunnit’ challenge. It will be a useful reference for the rest of us. The talk is divided into four sections. We will cover the typical BrainBench type questions to start with. Then we will attack the common problems encountered on the field and ideal solution paths to each of them. Third part will be about architectural trade-offs and ‘it depends’ scenarios. Finally, there will be discussion on best practices, books needed for interview preparation and open Q&A among the participants.
This session is for you if you want to learn tips and techniques that are used to optimize database development with special emphasis on SQL Server 2005. If you write lot of stored procedures and want to learn the tools of a DBA, this is the session for you. If you are new to SQL Server development environment, you will learn how the various constructs compare to each other and better performance can be produced every time with a brief introduction to understanding Execution Plans.
Generate reports with SSRS - SQL Server Reporting Services: This session will be a cornucopia of three sub-sessions. The first part will be to convince the skeptics. Why does every organization should consider SQL Server Reporting as part of its front-end solution? What will SSRS do better than a typical web application/site or a client-server application? The second portion will be a quick demo of the possibility and will be the shortest. The final part will talk about the best practices, tips from the field and will cover the implementation techniques.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
10. @bsubra
WHY?
• Smallest Unit of Compute – Pure Code
• Super, Hyper-Scalable
• Rapid Iteration, CI, CD, Quickest Delivery
• Extreme multi-tenancy
• Very Polyglot friendly (Java, JS, C#, Python,
Powershell, F#, Typescript, PHP, Go and so on)
• Easier to Collaborate – Microservices, APIs
• Deploy Independently – Go live to production
in seconds
16. @bsubra
WHAT?
• Abstraction of backend infrastructure
Completely
• Execution environment for single purpose
functions
• Hosted in public cloud or on-premise
• Serverless functions have a runtime in a
stateless container
• Node.JS, JavaScript, C#, Python, Java
• Event-driven Startup Triggers / Instant Scale-
out or Scale-in
• Micro-billing instead of per-hour / per-month
billing
20. @bsubra
MICROSERVICES ARCHITECTURE
• Services can communicate via
• through synchronous API requests (for example API
Gateway)
• through asynchronous messaging between services (for
example SQS and SNS)
• through a state machine orchestrator
• State: Backend-as-a-Service (Baas): infrastructure
components managed by a Cloud Provider
• Google Cloud Firestore
• Azure Storage / AWS S3 / Google Cloud Storage
• CosmosDB / DynamoDB
• Azure Kubernetes Service (AKS)
• Google BigQuery
22. @bsubra
EVENT DRIVEN ARCHITECTURE (EDA)
• Design Pattern around the Production
and Reaction to Events
• Serverless Functions are triggered by
Events
• Examples of Events
• A File uploaded to an S3 bucket
• Inserts on a DynamoDB Table
• A Message published to an SNS topic –
PubSub
• A CloudWatch Alert
• A message is available to read from a
Kinesis stream
24. @bsubra
EXAMPLES OF SERVERLESS USE CASES
• Multimedia processing
• Database changes or Change Data Capture
• IoT sensor input messages
• Stream processing at scale
• Chat bots
• Batch jobs scheduled tasks
• HTTP REST APIs and web apps
• Mobile back ends
• Business logic
• Continuous integration pipeline
Source: CNCF Serverless Whitepaper
v1.0
25. @bsubra
TIPS AND TRICKS
• Limit your Function Size (JVM Startup time especially)
• Remember Execution is Async
• Do not assume function container reuse but take advantage of it
• Audit dependencies to keep bundle sizes small.
• CloudWatch's default log retention period is forever.
• Structure your logs to ease alert creation and debugging
• Understand AWS account limits
• Think twice before dynamically provisioning AWS resources.
• Price - Is Serverless really as cheap as everyone claims? (dev.to) Unless you are
operating at massive scale, serverless is not just cheap, it's a steal.
26. @bsubra
BEST PRACTICES
• Scrutinize and research before attaching new input/event sources
• Keep the functions’ permissions least privileged and maintain a least privileged build system
• Amount of code that can access sensitive data be reduced, exceptions are handled and input is
validated
• Avoid embedding secrets and access keys in code
• Do not store access keys or credentials in source code repositories
• Throttle and define quotas on the amount of requests that go through
• Keep data encrypted that is stored at rest
• Scrutinize and keep tab on third party API services for vulnerabilities
• Scan third party libraries for potential vulnerabilities and try to keep them up-to-date
• Carefully examine granting permissions that allow creation, modification or removal of the resources
28. @bsubra
SERVERLESS FAILURE STORIES
• Serverless out of control
• Mo’ developers, mo’ problems: How serverless has trouble with teams
• Beware “RunOnStartup” in Azure Functions – a serverless horror story
• Serverless: A lesson learned. The hard way.
• Serverless: 15% slower and 8x more expensive
29. @bsubra
DYNAMODB - BEST PRACTICES
• Where it could be Used?
• Wherever we see relationships…
• Document management
• Process control
• Social network
• Data trees
• IT monitoring
• Takeaways
• NoSQL does not mean non-relational
• The ERD still matters
• RDBMS is not deprecated by NoSQL
• Use NoSQL for OLTP or DSS at Scale.
• Use RDBMS for OLAP or OLTP when Scale is not important.
30. @bsubra
DYNAMODB ANTI-PATTERNS
• Prewritten application tied to a traditional relational database – If you are attempting to port an
existing application to the AWS cloud and need to continue using a relational database, you may elect
to use either Amazon RDS (Amazon Aurora, MySQL, PostgreSQL, Oracle, or SQL Server), or one of the
many pre-configured Amazon EC2 database AMIs. You can also install your choice of database
software on an EC2 instance that you manage.
• Joins or complex transactions – While many solutions are able to leverage DynamoDB to support
their users, it’s possible that your application may require joins, complex transactions, and other
relational infrastructure provided by traditional database platforms. If this is the case, you may want to
explore Amazon Redshift, Amazon RDS, or Amazon EC2 with a self-managed database.
• Binary large objects (BLOB) data – If you plan on storing large (greater than 400 KB) BLOB data,
such as digital video, images, or music, you’ll want to consider Amazon S3. However, DynamoDB still
has a role to play in this scenario, for keeping track of metadata (e.g., item name, size, date created,
owner, location, etc.) about your binary objects.
• Large data with low I/O rate –DynamoDB uses SSD drives and is optimized for workloads with a
high I/O rate per GB stored. If you plan to store very large amounts of data that are infrequently
accessed, other storage options may be a better choice, such as Amazon S3.
31. @bsubra
LAMBDA - BEST PRACTICES
• Minimize package size to necessities
• Separate the Lambda handler from core logic
• Read only what you need
• Properly Indexed Databases
• Query Filters in Aurora
• Use S3 Select
• Dynamic Logic via Configuration
• Per Function - Use Environment Variables to modify operational behavior
• Cross Function – Amazon Parameter Store / Secrets Manager
• Self-contain dependencies in your function package
• Leverage “Max Memory Used” to right-size your functions
• Use Functions to TRANSORM (not for TRANSPORT)
• Delete large unused functions (used to be 75GB limit)
32. @bsubra
LAMBDA – ANTI-PATTERNS
• Long-running applications – Each Lambda function must complete within 300
seconds. For long-running applications that may require jobs to run longer than five
minutes, Amazon EC2 is recommended or you create a chain of Lambda functions
where function 1 calls function 2, which calls function 3, and so on until the process is
completed.
• Dynamic websites – While it is possible to run a static website with Lambda, running
a highly dynamic and large volume website can be performance prohibitive. Using
Amazon EC2 and Amazon CloudFront would be a recommended use case.
• Stateful applications –Lambda code must be written in a “stateless” style; that is, it
should assume there is no affinity to the underlying compute infrastructure. Local file
system access, child processes, and similar artifacts may not extend beyond the
lifetime of the request, and any persistent state should be stored in Amazon S3,
DynamoDB, or another Internet-available storage service.
40. @bsubra
Features AWS Lambda Azure Functions Google Cloud Functions
Language
Support Node.js, Java, C#, Python Node.js, C#, F#, Python, PHP Node.js
Security
AWS IAM, VPC
OAuth providers such as Azure Active Directory, Facebook, Google,
Twitter, and Microsoft Account Cloud IAM
Monitoring Cloudwatch Application Insights Stackdriver Monitoring
Logging Cloudwatch Application Insights Analytics Stackdriver Logging
Auditing CloudTrail Azure Audit Logs Cloud Audit Logging
Alerts Cloudwatch Alarms Application Insights, Log Analytics, and Azure Monitor Stackdriver Monitoring
Tooling
Support CodePipeline, Code Build Azure Portal, Azure Powershell, Azure CLI, Azure SDK gcloud CLI for functions
Debugging
Support AWS X-Ray
Azure CLI - local debugging,
Azure App Service - remote debugging Stackdriver Debugger
Pricing
it depends
Execution Time: $0.000016/GBs, 400,000 GBs/month are free
Total Executions: $0.20/million executions, with 1 million
executions/month free
Invocations: $0.40/million invocations with 2 million invocations free
Compute Time: $0.0000025/GBsec with 400,00 GB-sec/month free &
$0.0000100/GHzsec with 200,000 GHz-sec/month free
Limits
Memory allocation range: Min. 128 MB / Max. 1536 MB (with 64 MB
increments)
Ephemeral disk capacity ("/tmp" space): 512 MB
Number of file descriptors: 1,024
Number of processes and threads (combined total): 1,024
Maximum execution duration per request: 300 seconds
Invoke request body payload size (RequestResponse): 6 MB
Invoke request body payload size (Event): 128 K
Invoke response body payload size (RequestResponse): 6 MB
Allow only 10 concurrent executions per function
No limitations on max. execution time limit
Resource, Time and Rate Limits are defined under Google Cloud Functions
Quota limits
41. @bsubra
DISADVANTAGES
• Portability
• Automation / DevOps
• Cross-Cloud Communication
• The Cold Start Problem
• Applications that haven't been used recently
take longer to startup and to handle the first
request.
• Because serverless happens on use, there
aren’t dedicated instances ready to handle
requests
• Solution: Run a function in a dedicated
container/VM, not serverless
46. @bsubra
CONSIDERATIONS FOR THE ENTERPRISE
• Cloud Silos
• Enterprise Eventing
• Serverless skills gap
• Monitoring vs. Observability
• Stateful vs. Stateless
• Forecasting costs
• Where serverless falls short
Source: The New Stack Serverless Survey 2018
Q: What are the top three areas in which serverless falls short of expectations?
47. @bsubra
WORKS CITED
• Serverless Architectures - Martin Fowler
• Serverless on Google Cloud Platform: an
Introduction with Serverless Store Demo
• Introduction to Serverless: RISE Conference 2018 -
Hong Kong Developer Workshop: Introduction to
Serverless - Shaun Ray
• Serverless is a Doctrine, not a Technology - Paul
Johnston – Medium
• The Serverless Sea Change
• How AWS built a production service using
serverless technologies | AWS Open Source Blog
• Serverless Architectural Patterns and Best Practices
| AWS Architecture Blog
• Contemporary Views on Serverless and
Implications - Subbu’s Blog
@bsubra
48. @bsubra
SERVERLESS ON GOOGLE CLOUD PLATFORM
• App Engine | Google Cloud
• 1. Auth, Storage, Event Streaming, and Third-Party APIs (Firebase Authentication,
Cloud Storage, Cloud Firestore, Cloud Pub/Sub and Third-Party APIs)
• 11. Machine Learning, AI, Data Analytics, and Data Visualization (Cloud Vision API,
Cloud AutoML, DialogFlow +Google Assistant integration, BigQuery and Data Studio)
• 111. Serverless Computing, Management Tools, and Cron Jobs (App Engine, Cloud
Functions, Cloud Scheduler, Stackdriver Logging, Stackdriver Monitoring, and
Stackdriver Tracing)
• 1111. Hacking Google Cloud Run
49. @bsubra
CALL TO ACTION
1. Write an AWS Lambda from Template
• How do I get started with creating serverless applications?
2. Make your monolithic app use Step Functions
3. Try how Azure Kubernetes Service (AKS) can deploy your legacy apps
4. Make your next project be event driven with Azure Event Grid, or GCP Cloud Pub/Sub
• Building a Reliable Serverless Application in a Weekend
5. Become familiar with application observability with comprehensive logging, metrics, and
tracing
6. Follow folks pushing Serverless in twitter
7. Read a book
• Serverless apps: Architecture, patterns, and Azure implementation | Microsoft Docs
8. Watch Videos like
• AWS re:Invent: Accelerate Innovation & Maximize Business Value w/ Serverless Apps (SRV212-R1)
• Keynote: Serverless, Not So FaaS - Kelsey Hightower, Kubernetes Community Member, Google
APIs, Serverless Functions, Microservices - How to get your product quickly to market?
Do you have the next best idea? How will you quickly migrate a legacy feature to new world for almost free? This talk will give you how to architect and implement your scenario for a cloud-oriented solution. We will share the best practices for storing your state in database; ways to decouple by events and suggested patterns for serverless. You will be equipped with taking advantage of low-cost serverless computing in a secure way and how to minimize operational costs. It will mostly focus AWS offerings like Serverless Aurora, API Gateway and Lambda functions for solutions blueprint.
Balaji is a Boston-based developer, writer, speaker and debugger. He has expertise in infrastructure, internet technologies, and systems integration obtained through 20 plus years of working as a manager, architect and developer on projects for mission critical back end systems that have included various cloud platforms such as OpenStack, AWS and Azure. As a degreed Industrial engineer, he also has experience in distributive controls systems as a controls engineer at Honeywell. He has also worked extensively with global hedge fund industry providing SaaS solutions on the public cloud.
An elderly couple had dinner at another couple's house, and after eating, the wives left the table and went into the kitchen. The two gentlemen were talking, and one said, 'Last night we went out to a new restaurant and it was really great. . . I would recommend it very highly.' The other man said, 'What is the name of the restaurant?' The first man thought and thought and finally said, 'What is the name of that flower you give to someone you love? You know... The one that's red and has thorns.' 'Do you mean a rose?' 'Yes, that's the one,' replied the man. He then turned towards the kitchen and yelled, 'Rose, what's the name of that restaurant we went to last night?'
Development, debugging, deployment, delivery, and decommissioning of application services are the major portion of any application lifecycle management (ALM) process.
This serverless deployment pattern recommends a kind of deployment infrastructure that hides the concept of servers (whether physical or virtual). The infrastructure takes the application service's code and runs it. The user has to pay for each of his requests based on the resources consumed. The performance, scalability, and availability requirements are being automatically met. Almost all the established cloud service providers are providing this new deployment pattern. This is a new cloud service ensuring every function is being delivered as a service.
Serverless Can Help You To Focus - By Simona Cotin: Time is crucial in startups. We are experiencing a constant race against time. Your market might be time sensitive, and you need to grow fast. Most startups fail because they run out of cash and time. Serverless helps you spend time only on things that matter instead of stuff that seems shiny but isn’t essential.
Two Uni Students From Queensland Built a Better Census Site in One Weekend - VICE: Their website is faster, cheaper, and more secure. And they didn't even fill out the Census this year.
It cost the Australian Bureau of Statistics (ABS) many years and $9 million to build its somewhat inadequate Census website, which was taken down by a DDoS attack after being online for mere hours. Over the weekend, two uni students from Queensland were able to build a better Census site in two days for only $500.
The two young computer geniuses behind the aptly-named Make Census Great Again website are students Austin Wilshire (18) and Bernd Hartzer (24). Both hail from the Queensland University of Technology. Their system can handle nearly 40 times as many census submissions per second as the ABS one can, and is specifically designed to prevent the DDoS attack that forced the Census offline on August 9.
Intro to Serverless: A little bit of history, evolution, misconceptions around serverless
Benefits: Get into why serverless, its benefits and characteristics
Adoption: Insight into the adoption by enterprises and the tech community in general
Architecture: Discussions around serverless architecture, FaaS, evolving patterns and solutions
Security: Auth. services, access controls, surface areas for attack, data isolation...
Development, Deployment & Testing: Rethinking around developing, deploying and testing serverless
applications and services
Toolsets: Evolving toolsets, frameworks and methodologies
Changing DevOps: A look into the changing roles of DevOps teams and the Mindshift
Challenges: Concerns around debugging, logging, and monitoring, of serverless applications
Providers: A comparative look at the serverless providers out there
Case Studies: Examples of real-life implementations of serverless technologies
No server management
Scales automatically
Pay only while your code runs
Runs code in response to events
Open and familiar
Connects and extends cloud serviceshttps://twitter.com/PaulDJohnston/status/1186908841100886017
AMG Earnings Q3 19: The Future of the Tech Industry – Bernard Golden
Last week all three of the most important cloud providers (I refer to them collectively as AMG — Amazon, Microsoft, and Google) announced their financial results. All continue to see impressive growth and, as a group, have achieved very significant results. The chart below indicates that the three are generated something like $16B or $17B for the quarter, although it’s impossible to be precise because Microsoft and Google do not break out their pure cloud computing revenues.
The Leading Open Source Serverless Solutions for Kubernetes: The universe of serverless-wielding software architects and Kubernetes cluster operators has started to collide and, yet again, Google is in the driver's seat. In this article we'll wander down the CNCF's Serverless Landscape in chronological order, quickly discovering that Knative is the sweet mamba jamba of open source lambda competitors.
6 Strategies for Migrating Applications to the Cloud
Rehosting — Otherwise known as “lift-and-shift.”
Replatforming — I sometimes call this “lift-tinker-and-shift.”
Repurchasing — Moving to a different product.
Refactoring / Re-architecting — Re-imagining how the application is architected and developed, typically using cloud-native features.
Retire — Get rid of.
Retain — Usually this means “revisit” or do nothing (for now).
Serverless computing promises a pay-as-you-go future with (almost) no server management at all. Serverless platforms take the code from developers and perform all the deployment tasks (networking, dependencies, maintenance, etc.) automatically behind the scenes
Serverless application
Event Source
Function
StoreCloud Functions - Event-driven Serverless Computing | Cloud Functions | Google Cloud
1. Zero Administration: This is the most exciting thing about serverless. Whereas previous abstractions like VMs and containers still shared a lot of the same configuration and administration properties of servers, serverless is a completely different experience. When you're ready to deploy code, you don't have to provision anything beforehand, or manage anything afterward. There is no concept of a fleet, an instance, or even an operating system. Everything runs in the cloud and the provider manages scaling for you.
2. Pay-per-execution: This is what typically incentivizes developers to try serverless for the first time. It’s alluring to have complete resource utilization without paying a cent for idle time. This tenet alone results in over 90% cost-savings over a cloud VM and immeasurable developer satisfaction in knowing that you never have to pay for resources that you don’t use.
3. Function as unit of deployment: Serverless architectures are composed of very small, independent bits of code (functions) that are loosely coupled and collaborative—also known as a microservice architecture. The main advantage? Pieces of the system are contained. They can be developed and deployed independently. The result is fewer blockers and far greater developer autonomy and productivity.
4. Event-Driven: This aspect of serverless is the most under-the-radar right now, but is shaping up to be the most important in the long-term. Serverless functions are stateless, and essentially dormant, until they have an event to react to. The event is what brings them to life and provides them with the data/context to do their job. Event-driven architectures are nothing new, but the rise of serverless compute has renewed interest in them because serverless architectures are by definition event-driven.
https://twitter.com/ben11kehoe/status/1097804380777205760
Which serverless compute platform is right for you?
https://cloud.google.com/serverless-options/
AWS Step Functions enables us to construct a state machine graph with custom logic, where each processing node can be either AWS Lambda, AWS Batch or AWS Fargate. The Step Function service tracks the completion of the task as well as if an exception occurred. It enables us to branch out logic in case of error (with the ability to customize the handling of an error), execute jobs in parallel and implement retry logic.
“clean architecture”, “hexagonal architecture” (ports and adapters), and “onion architecture”. Some argue it’s all the same.
Serverless Event Sourcing in AWS (Lambda, DynamoDB, SQS): In this post, I have presented the project structured using Event Sourcing and CQRS patterns, written in TypeScript. The project I was building was a “Twitter-like” service
Source: Ajay Nair, Principal Product Manager (AWS Lambda) at Amazon Web Services (AWS)
Event-driven architecture (EDA) is conceptually different than the client/server architecture most commonly used todayAll logic in an event-driven architecture is embodied in functions. Events trigger these functions and then the functions, in turn, trigger something downstream, which themselves may be functions, according to Ajay Nair, principal product manager for AWS Lambda at Amazon Web Services
the infrastructure provisioning, setting up and administering time, and treasure and talent get reduced significantly.
Software engineers can coolly focus on their core strengths without any botheration of the readying infrastructure to run their applications.
Cost
Elasticity versus scalability
Productivity
https://twitter.com/ShortJared/status/1100887501047132160
The job of a person in a business is not to provide technology, but to provide business value.
Follow AWS Best Practices. The Lambda and Using Lambda with SQS Best Practices helps avoid Lambda throttles, understand SQS message batches (they succeed or fail together) and configure redrive policies high enough to prevent prematurely sending messages to dead-letter queues.
A Common response to a recurring problem that is usually ineffective and risks being highly counterproductive
https://github.com/cristim/serverless-failure-stories
https://twitter.com/ajaynairthinks/status/1122197970743468033
Serverless and Microservices: a match made in heaven?: When the first tutorials started to come out using AWS Lambda and API Gateway, back in 2015, it was unsurprising to find that they focused largely on replicating the microservice. However, it…
Tenets of NoSQL Data Modeling
Understand the use case
Nature of the application
OLTP / OLAP / DSS
Define the Entity-Relationship Model
Identify Data Life Cycle
TTL, Backup/Archival, etc
Identify the access patterns
Identify data sources
Define query aggregations
Document all workflows
Read/Write workloads
Query dimensions and aggregations
Data-modeling
Avoid relational design patterns, use one table
1 application service = 1 table
Reduce round trips
Simplify access patterns
Identify Primary Keys
How will items be inserted and read?
Overload items into partitions
Define indexes for secondary access patterns
Review -> Repeat -> Review
Amazon DynamoDB helps you build serverless applications by providing a managed NoSQL database for persistent storage.
Combined with DynamoDB Streams you can respond in near real-time to changes in your DynamoDB table by invoking Lambda functions.
DynamoDB Accelerator (DAX) adds a highly available in-memory cache for DynamoDB that delivers up to 10x performance improvement from milliseconds to microseconds.
Exposed AWS Lambda Functions: Identify any publicly accessible AWS Lambda functions and update their access policy in order to protect against unauthorized users that are sending requests to invoke these functions
Allowing anonymous users to invoke your Amazon Lambda functions is considered bad practice and can lead to data exposure, data loss and unexpected charges on your AWS bill. To prevent any unauthorized invocation requests to your Lambda functions, restrict access only to trusted entities
Lambda Functions with Admin Privileges
AWS Lambda Unknown Cross Account Access
Lambda Runtime Environment Version
Using An IAM Role For More Than One Lambda Function
Tracing Enabled
Orchestrate your application with state machines, not functions: Chaining Lambda executions within the code to orchestrate the workflow of your application results in a monolithic and tightly coupled application. Instead, use a state machine to orchestrate transactions and communication flows.
AWS Serverless Application Model (AWS SAM) is an extension of AWS CloudFormation that is used to package, test, and deploy serverless applications. SAM Local can also enable faster debugging cycles when developing Lambda functions locally.
AWS Step Functions orchestrates serverless workflows including coordination, state, and function chaining as well as combining long-running executions not supported within Lambda execution limits by breaking into multiple steps or by calling workers running on Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises: Applying the Saga pattern with AWS Lambda and Step Functions | theburningmonk.comWith Amazon API Gateway, you can run a fully managed REST API that integrates with Lambda to execute your business logic and includes traffic management, authorization and access control, monitoring, and API versioning: Error Handling Patterns in Amazon API Gateway and AWS Lambda | AWS Compute BlogAccessing Amazon CloudWatch Logs for AWS Lambda - AWS LambdaA Simple Serverless Test Harness using AWS Lambda | AWS Compute BlogAWS Lambda Function Versioning and Aliases - AWS LambdaEnvironment Variables - AWS LambdaAWS Lambda and AWS X-Ray - AWS X-RayDead Letter Queues (DLQ) - AWS LambdaUnderstanding Container Reuse in AWS Lambda | AWS Compute Blog
Serverless web application - Azure Reference Architectures | Microsoft Docs: Recommended architecture for a serverless web application and web API.
Azure Serverless | Microsoft Azure
1 User accesses the web app in browser and signs in.
2 Browser pulls static resources such as images from Azure Content Delivery Network.
3 User searches for products and queries SQL database.
4 Web site pulls product catalog from database.
5 Web app pulls product images from Blob Storage.
6 Page output is cached in Azure Cache for Redis for better performance.
7 User submits order and order is placed in the queue.
8 Azure Functions processes order payment.
9 Azure Functions makes payment to third party and records payment in SQL database.
AWS - Multiple points to optimize - Invocations, Functions, Interactions
AWS Batch and AWS Fargate: why they are beneficial and what their differences are.
AWS Step Functions: how it is different from other ways of connecting services and what the advantages are.
What are some specific cases where hybrid infrastructure could be beneficial
Build a real-life serverless app with AWS Amplify
last year’s Re:Invent deep-dive video on DynamoDB
open source project designed to help customers learn from Amazon’s approach to building serverless applications. The project captures key architectural components, code structure, deployment techniques, testing approaches, and operational practices of the AWS Serverless Application Repository, a production-grade AWS service written mainly in Java and built using serverless technologies.
available in the project wiki.
Further Discussions for some tips and notes about Serverless Store.part of the Serverless on Google Cloud Platform: an Introduction with Serverless Store Demo How-to Guide. It discusses the authentication, storage, event streaming solutions, and third-party APIs Serverless Store uses.
It is only suitable for deploying stateless applications that run quickly and respond to requests.
Running long-running stateful applications such as a database or message broker in the serverless model is not possible.
If an application takes a long time to start, then the application is not a good fit for serverless deployment.
Similarly, legacy monolithic and massive applications are not suitable for serverless computing.
Serverless deployment is typically reactive, not proactive, and hence the issue of high latency can arise
https://twitter.com/jessfraz/status/942031487809085440
On the face of it, it would appear that choosing the lowest memory allocation for compute would be the cheapest. But that’s not always the case. Execution time may lengthen with smaller memory size allocations, thus raising the overall cost,
the company also used Amazon’s CloudWatch tools, which turned out to be surprisingly costly.
But perhaps the biggest hidden cost comes from API requests.
when a bug in its CloudWatch function prevented the service from scaling well. They discovered that 5,000 functions were repeatedly getting timeouts that amounted to a cost of $12,000 per month — a fortune for a small startup.
Cloud providers do, thankfully, offer the ability to set rate limits and spending thresholds on service accounts and will issue automatic billing alerts if you reach your limits. But closely monitoring risks associated with spiking and ensuring that estimated costs will not jump from poor performance of serverless workflows and systems will be a focus in the coming year
LambdaGuard - tool which allows you to visualize and audit the security of your serverless assets, an open-source AWS Lambda Serverless Security Scanner. https://darknet.org.uk/2019/10/lambdaguard-aws-lambda-serverless-security-scanner/… via @THEdarknet
statistical analysis
AWS service dependencies
configuration checks
AWS Lambda automatically monitors Lambda functions on your behalf. Through Amazon CloudWatch, it reports metrics such as the number of requests, the execution duration per request, and the number of requests resulting in an error. These metrics are exposed at the function level, which you can then leverage to set CloudWatch alarms.
Using Amazon CloudTrail, you can implement governance, compliance, operational auditing, and risk auditing of your entire AWS account, including Lambda. CloudTrail enables you to log, continuously monitor, and retain account activity related to actions across your AWS infrastructure, providing a complete event history of actions taken through the AWS Management Console, AWS SDKs, command line tools, and other AWS services. Using CloudTrail, you can optionally encrypt the log files using Amazon Key Management Service (KMS) and also leverage the CloudTrail log file integrity validation for positive assertion.
Using AWS X-Ray, you can analyze and debug production and distributed Lambda-based applications, which enables you to understand the performance of your application and its underlying services, so you can eventually identify and troubleshoot the root cause of performance issues and errors. X-Ray’s end-to-end view of requests as they travel through your application shows a map of the application’s underlying components, so you can analyze applications during development and in production.
With AWS Config, you can track configuration changes to the Lambda functions (including deleted functions), runtime environments, tags, handler name, code size, memory allocation, timeout settings, and concurrency settings, along with Lambda IAM execution role, subnet, and security group associations. This gives you a holistic view of the Lambda function’s lifecycle and enables you to surface that data for potential audit and compliance requirements
Whole-Event Serverless Observability - IOpipe Blog
The Serverless Vector Map Stack Lives! - Alexander Rolek - Medium: TL;DR: With the release of AWS Aurora Serverless Postgres (with PostGIS support), building an entirely serverless map stack is now possible, including the database.
How to FaaS like a pro: 12 less common ways to invoke your serverless functions on AWS
Watch Google Global Digital Conference 2019.Watch more #io19 here:
GCP at Google I/O 2019 Playlist → https://goo.gle/2ZPLejw
Google I/O 2019 All Sessions Playlist → https://goo.gle/io19allsessions
Learn more on the I/O Website → https://google.com/io
Google Cloud Platform Channel → https://goo.gle/GCP
Get started at → https://cloud.google.com/gcp
When you’re building serverless applications on AWS, you can use AWS CloudFormation directly, or choose the AWS Serverless Application Model, also known as AWS SAM.
Introduction to Serverless
Building large scale applications with AWS Lambda
Building Enterprise-Grade Serverless Applications
Best Practices for Optimizing your Data Lake with AWS Serverless
Real-Time Streaming Analytics and Machine Learning in Serverless Era
Serverless Development, CI/CD
Serverless database for your Serverless App
Serverless intelligence - the fastest path to smart applications