In recent years, Docker containers have become a key component of modern application design. Increasingly, developers are breaking their applications apart into smaller components and distributing them across a pool of compute resources. Using Docker on your local development machine is simple, but running Docker applications at scale in production can be difficult. In this session, we will discuss the difficulties of running Docker in production and how Amazon EC2 Container Service (ECS) can be used to reduce the operational burdens. We will give an overview of the core architectural principles underlying Amazon ECS., and we will walk through a number of patterns used by our customers to run their microservices platforms, to run batch jobs, and for deployments and continuous integration. We will also demonstrate how to define multi-container applications, deploy and scale them seamlessly on a cluster with Amazon ECS.
AWS Fundamentals @Back2School by CloudZoneIdan Tohami
This class is all about the basics. Here you will learn about the services AWS has to offer in compute, storage, databases and various application services.
Cost is often the conversation starter when customers think about moving to the cloud. AWS helps lower costs for customers through its “pay only for what you use” pricing model, frequent price drops, and pricing model choice to support variable & stable workloads. In this session, you will learn about the financial considerations of owning and operating a traditional data center or managed hosting provider versus utilizing AWS. We will detail our TCO methodology and showcase cost comparisons for some common customer use-cases. We’ll also cover a few AWS cost optimization areas, including Spot and Reserved Instances, EC2 Auto Scaling, and consolidated billing.
Presenter:
Amit Sharma, Solution Architect, Amazon Internet Services
Krishnenjit Roy, Director IT Operations, Freshdesk
In recent years, Docker containers have become a key component of modern application design. Increasingly, developers are breaking their applications apart into smaller components and distributing them across a pool of compute resources. Using Docker on your local development machine is simple, but running Docker applications at scale in production can be difficult. In this session, we will discuss the difficulties of running Docker in production and how Amazon EC2 Container Service (ECS) can be used to reduce the operational burdens. We will give an overview of the core architectural principles underlying Amazon ECS., and we will walk through a number of patterns used by our customers to run their microservices platforms, to run batch jobs, and for deployments and continuous integration. We will also demonstrate how to define multi-container applications, deploy and scale them seamlessly on a cluster with Amazon ECS.
AWS Fundamentals @Back2School by CloudZoneIdan Tohami
This class is all about the basics. Here you will learn about the services AWS has to offer in compute, storage, databases and various application services.
Cost is often the conversation starter when customers think about moving to the cloud. AWS helps lower costs for customers through its “pay only for what you use” pricing model, frequent price drops, and pricing model choice to support variable & stable workloads. In this session, you will learn about the financial considerations of owning and operating a traditional data center or managed hosting provider versus utilizing AWS. We will detail our TCO methodology and showcase cost comparisons for some common customer use-cases. We’ll also cover a few AWS cost optimization areas, including Spot and Reserved Instances, EC2 Auto Scaling, and consolidated billing.
Presenter:
Amit Sharma, Solution Architect, Amazon Internet Services
Krishnenjit Roy, Director IT Operations, Freshdesk
"Fast Start to Building on AWS", Igor IvaniukFwdays
We will look into different stages of startup lifecycle from a technology point of view, and talk about how does AWS could support each of it. We’ll cover multiple scenarios and also discuss initial decisions that will help to deliver the MVPs quickly and not break the bank along the way. The session will be suitable both, for business- and tech- founders – so bring your co-founders with you. After the session we will have a time for free style Q&A.
Running solutions in Amazon Web Services can help you get your applications up and running faster. By harnessing the power of the Amazon Web Services platform services you no longer have to focus on the underlying infrastructure, but instead you can focus on your application and business logic. This session will enable you to learn about the rich array of Amazon Web Services platform services available, find out how other customers are leveraging platform services and identify which services are relevant to your business. This session also includes a demonstration of Amazon Web Services for compute, database and deployment which you can use to accelerate your cloud adoption journey.
Mark Statham, Senior Cloud Architect - Professional Services, Amazon Web Services, ASEAN
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...Amazon Web Services
Fortune 500 companies are increasingly using cloud services to run enterprise workloads to improve security, increase agility, and enable scale. Learn how OpenEye is running their AWS-native platform and workflow engine to support collaboration and data sharing at large pharmaceutical companies like Pfizer. In this session, OpenEye will share cloud best practiced around security controls, cross-departmental collaboration across the enterprise, and agility at scale. Attendees will gain practical tips for using AWS in the enterprise and healthcare industries.
If you could not be one of the 60,000+ in attendance at Amazon AWS re:Invent, the yearly Amazon Cloud Conference, get the 411 on what major announcements that were made in Las Vegas. This presentation covers new AWS services & products, exciting announcements, and updated features.
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
Integrating Deep Learning into your Enterprise
In this workshop we return to one of the popular Machine Learning Framework - scikit-learn. We scikit-learn's decision tree classifier to train the model. Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. We follow the whole machine learning pipeline from algorithm selection, training and finally deployment of an endpoint. We would be working with the widely available Iris dataset and the endpoint would be predicting what species the sample belongs to from the Sepal width and length, Petal width and length. Through this workshop we would know all the internal details of how we use containers to train and deploy our machine learning workloads.
Level: 300-400
AWS Machine Learning Week SF: Integrating Deep Learning into Your EnterpriseAmazon Web Services
AWS Machine Learning Week SF: Integrating Deep Learning into your Enterprise
Hands on Workshop based on BYOD Scikit learn and use of Docker containers in the workflow. More detailed description forthcoming.
Cloud Computing with Amazon Web Services.
AWS Cloud Solutions - Websites, Archiving, Data Lakes and Analytics, Serverless Computing, Internet of Things and more.
Containers in AWS - Amazon Elastic Container Service, Fargate, and EKS
Big Data and the Data lake implementation in AWS
Machine Learning with Amazon SageMaker - Build, train, and deploy machine learning models at scale.
AWS Identity and Access Management (IAM) - Securely manage access to AWS services and resources.
AWS Pricing - How does AWS pricing work?
AWS 101 Webinar: Journey to the AWS Cloud - Introduction to Cloud Computing w...Amazon Web Services
Whether you are running applications that share photos or support critical operations of your business, you need rapid access to flexible and low cost IT resources. The term "cloud computing" refers to the on-demand delivery of IT resources via the Internet with pay-as-you-go pricing. Whether you are a start-up who wants to accelerate growth without a big upfront investment in cash or time for technology or an Enterprise looking for IT innovation, agility and resiliency while reducing costs, the AWS Cloud provides a complete set of web services at zero upfront costs which are available with a few clicks and within minutes. In this session learn more about the benefits of Cloud Computing with AWS.
Los AWSome day son eventos gratuitos enfocados en educación, propocionandole una introducción a los servicios básicos de AWS desde computación, almacenamiento, bases de datos y redes. Los instructores de AWS explicarán las características claves de cada producto y sus casos de uso, compartirán las mejores prácticas, harán demostraciones técnicas y responderán sus preguntas. Este entrenamiento virtual es un extracto del curso de AWS Technical Essentials diseñado para líderes de TI, responsables de articular las ventajas técnicas de los servicios de AWS a sus clientes. Administradores de Sistemas, arquitectos de soluciones y desarrolladores que quieran usar los servicios de AWS y personas interesadas en aprender a usar AWS.
Agenda:
- Introducción a los conceptos de AWS y Cloud Computing
- Servicios básicos de AWS: EC2, VPC, S3, EBS
- Seguridad, identidad y administración del acceso en AWS: IAM
Connect and Interconnect – The Mesh of Event-Driven Compute and Marvelous Vir...Amazon Web Services
Let’s enter the new world of serverless, voice and event-driven compute to build a broad mesh of interconnected smart devices. Services like Amazon API Gateway, AWS Lambda, Amazon S3, AWS IoT, Amazon Mobile Hub and Alexa Skills Kit all help to build completely serverless, smart, voice-enabled architectures within minutes without managing any servers. We will demonstrate interesting Webhook integrations with Facebook and Slack, build mobile apps on the fly, send containers into the cloud and give Amazon Echo new skills. In addition, experience Amazon Lumberyard, a free, cross-platform, 3D game engine to create the highest-quality games, connect your virtual worlds to the vast compute and storage of the AWS Cloud, and engage fans on Twitch. It has never been a better time to build!
Wild Rydes (www.wildrydes.com) needs your help! With fresh funding from its seed investors, Wild Rydes is seeking to build the world’s greatest mobile/VR/AR unicorn transportation system. The scrappy startup needs a first-class webpage to begin marketing to new users and to begin its plans for global domination. Join us to help Wild Rydes build a website using a serverless architecture. You’ll build a scalable website using services like AWS Lambda, Amazon API Gateway, Amazon DynamoDB, and Amazon S3. Join this workshop to hop on the rocket ship!
To complete this workshop, you'll need:
Your laptop
AWS Account
AWS Command Line Interface
Google Chrome
git
Text Editor
Your Basic Building Blocks - AWS Compute - AWS Summit Tel Aviv 2017Amazon Web Services
AWS offers multiple compute products allowing you to deploy, run, and scale your applications as virtual servers, containers, or code. This session will cover the main service starting with Amazon EC2 - Resizable cloud-based compute capacity, Amazon Lightsail - The easiest way to launch and manage a virtual private server, Amazon EC2 Container Service (ECS) - highly scalable, high performance container management service that supports Docker containers and AWS Lambda which allows you to run code without provisioning or managing servers.
"Fast Start to Building on AWS", Igor IvaniukFwdays
We will look into different stages of startup lifecycle from a technology point of view, and talk about how does AWS could support each of it. We’ll cover multiple scenarios and also discuss initial decisions that will help to deliver the MVPs quickly and not break the bank along the way. The session will be suitable both, for business- and tech- founders – so bring your co-founders with you. After the session we will have a time for free style Q&A.
Running solutions in Amazon Web Services can help you get your applications up and running faster. By harnessing the power of the Amazon Web Services platform services you no longer have to focus on the underlying infrastructure, but instead you can focus on your application and business logic. This session will enable you to learn about the rich array of Amazon Web Services platform services available, find out how other customers are leveraging platform services and identify which services are relevant to your business. This session also includes a demonstration of Amazon Web Services for compute, database and deployment which you can use to accelerate your cloud adoption journey.
Mark Statham, Senior Cloud Architect - Professional Services, Amazon Web Services, ASEAN
AWS re:Invent 2016: Best practices for running enterprise workloads on AWS (E...Amazon Web Services
Fortune 500 companies are increasingly using cloud services to run enterprise workloads to improve security, increase agility, and enable scale. Learn how OpenEye is running their AWS-native platform and workflow engine to support collaboration and data sharing at large pharmaceutical companies like Pfizer. In this session, OpenEye will share cloud best practiced around security controls, cross-departmental collaboration across the enterprise, and agility at scale. Attendees will gain practical tips for using AWS in the enterprise and healthcare industries.
If you could not be one of the 60,000+ in attendance at Amazon AWS re:Invent, the yearly Amazon Cloud Conference, get the 411 on what major announcements that were made in Las Vegas. This presentation covers new AWS services & products, exciting announcements, and updated features.
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
Integrating Deep Learning into your Enterprise
In this workshop we return to one of the popular Machine Learning Framework - scikit-learn. We scikit-learn's decision tree classifier to train the model. Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. We follow the whole machine learning pipeline from algorithm selection, training and finally deployment of an endpoint. We would be working with the widely available Iris dataset and the endpoint would be predicting what species the sample belongs to from the Sepal width and length, Petal width and length. Through this workshop we would know all the internal details of how we use containers to train and deploy our machine learning workloads.
Level: 300-400
AWS Machine Learning Week SF: Integrating Deep Learning into Your EnterpriseAmazon Web Services
AWS Machine Learning Week SF: Integrating Deep Learning into your Enterprise
Hands on Workshop based on BYOD Scikit learn and use of Docker containers in the workflow. More detailed description forthcoming.
Cloud Computing with Amazon Web Services.
AWS Cloud Solutions - Websites, Archiving, Data Lakes and Analytics, Serverless Computing, Internet of Things and more.
Containers in AWS - Amazon Elastic Container Service, Fargate, and EKS
Big Data and the Data lake implementation in AWS
Machine Learning with Amazon SageMaker - Build, train, and deploy machine learning models at scale.
AWS Identity and Access Management (IAM) - Securely manage access to AWS services and resources.
AWS Pricing - How does AWS pricing work?
AWS 101 Webinar: Journey to the AWS Cloud - Introduction to Cloud Computing w...Amazon Web Services
Whether you are running applications that share photos or support critical operations of your business, you need rapid access to flexible and low cost IT resources. The term "cloud computing" refers to the on-demand delivery of IT resources via the Internet with pay-as-you-go pricing. Whether you are a start-up who wants to accelerate growth without a big upfront investment in cash or time for technology or an Enterprise looking for IT innovation, agility and resiliency while reducing costs, the AWS Cloud provides a complete set of web services at zero upfront costs which are available with a few clicks and within minutes. In this session learn more about the benefits of Cloud Computing with AWS.
Los AWSome day son eventos gratuitos enfocados en educación, propocionandole una introducción a los servicios básicos de AWS desde computación, almacenamiento, bases de datos y redes. Los instructores de AWS explicarán las características claves de cada producto y sus casos de uso, compartirán las mejores prácticas, harán demostraciones técnicas y responderán sus preguntas. Este entrenamiento virtual es un extracto del curso de AWS Technical Essentials diseñado para líderes de TI, responsables de articular las ventajas técnicas de los servicios de AWS a sus clientes. Administradores de Sistemas, arquitectos de soluciones y desarrolladores que quieran usar los servicios de AWS y personas interesadas en aprender a usar AWS.
Agenda:
- Introducción a los conceptos de AWS y Cloud Computing
- Servicios básicos de AWS: EC2, VPC, S3, EBS
- Seguridad, identidad y administración del acceso en AWS: IAM
Connect and Interconnect – The Mesh of Event-Driven Compute and Marvelous Vir...Amazon Web Services
Let’s enter the new world of serverless, voice and event-driven compute to build a broad mesh of interconnected smart devices. Services like Amazon API Gateway, AWS Lambda, Amazon S3, AWS IoT, Amazon Mobile Hub and Alexa Skills Kit all help to build completely serverless, smart, voice-enabled architectures within minutes without managing any servers. We will demonstrate interesting Webhook integrations with Facebook and Slack, build mobile apps on the fly, send containers into the cloud and give Amazon Echo new skills. In addition, experience Amazon Lumberyard, a free, cross-platform, 3D game engine to create the highest-quality games, connect your virtual worlds to the vast compute and storage of the AWS Cloud, and engage fans on Twitch. It has never been a better time to build!
Wild Rydes (www.wildrydes.com) needs your help! With fresh funding from its seed investors, Wild Rydes is seeking to build the world’s greatest mobile/VR/AR unicorn transportation system. The scrappy startup needs a first-class webpage to begin marketing to new users and to begin its plans for global domination. Join us to help Wild Rydes build a website using a serverless architecture. You’ll build a scalable website using services like AWS Lambda, Amazon API Gateway, Amazon DynamoDB, and Amazon S3. Join this workshop to hop on the rocket ship!
To complete this workshop, you'll need:
Your laptop
AWS Account
AWS Command Line Interface
Google Chrome
git
Text Editor
Your Basic Building Blocks - AWS Compute - AWS Summit Tel Aviv 2017Amazon Web Services
AWS offers multiple compute products allowing you to deploy, run, and scale your applications as virtual servers, containers, or code. This session will cover the main service starting with Amazon EC2 - Resizable cloud-based compute capacity, Amazon Lightsail - The easiest way to launch and manage a virtual private server, Amazon EC2 Container Service (ECS) - highly scalable, high performance container management service that supports Docker containers and AWS Lambda which allows you to run code without provisioning or managing servers.
An introduction to the Transformers architecture and BERTSuman Debnath
The transformer is one of the most popular state-of-the-art deep (SOTA) learning architectures that is mostly used for natural language processing (NLP) tasks. Ever since the advent of the transformer, it has replaced RNN and LSTM for various tasks. The transformer also created a major breakthrough in the field of NLP and also paved the way for new revolutionary architectures such as BERT.
These days we see a lot of buzz about Machine Learning(ML)/Artificial Intelligence(AI), and why not, we all are consumers of ML directly or indirectly, irrespective of our profession. AI and ML is a fantastic field, everyone is excited about it, and rightly so. In this tutorial series, we will try to explore and demystify the complicated world of {maths, equations, and theory} that functions in tandem to bring out the "magic" which we experience on many application(s)/software(s). In this talk we will learn about Supervised Learning, Decision Tree, and shall solve some problem with SageMaker.
Blog: https://dev.to/aws/an-introduction-to-decision-tree-and-ensemble-methods-part-1-24p0
Code: https://github.com/debnsuma/AI-ML-Algo2020/tree/master/01.Decision_Tree
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
StarCompliance is a leading firm specializing in the recovery of stolen cryptocurrency. Our comprehensive services are designed to assist individuals and organizations in navigating the complex process of fraud reporting, investigation, and fund recovery. We combine cutting-edge technology with expert legal support to provide a robust solution for victims of crypto theft.
Our Services Include:
Reporting to Tracking Authorities:
We immediately notify all relevant centralized exchanges (CEX), decentralized exchanges (DEX), and wallet providers about the stolen cryptocurrency. This ensures that the stolen assets are flagged as scam transactions, making it impossible for the thief to use them.
Assistance with Filing Police Reports:
We guide you through the process of filing a valid police report. Our support team provides detailed instructions on which police department to contact and helps you complete the necessary paperwork within the critical 72-hour window.
Launching the Refund Process:
Our team of experienced lawyers can initiate lawsuits on your behalf and represent you in various jurisdictions around the world. They work diligently to recover your stolen funds and ensure that justice is served.
At StarCompliance, we understand the urgency and stress involved in dealing with cryptocurrency theft. Our dedicated team works quickly and efficiently to provide you with the support and expertise needed to recover your assets. Trust us to be your partner in navigating the complexities of the crypto world and safeguarding your investments.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
12. Leaders start with the customer and work
backwards. They work vigorously to earn
and keep customer trust. Although leaders
pay attention to competitors,
they obsess over customers.
13. 90% of the AWS roadmap is driven by
direct customer feedback, and this is
also true of open source.
14. Open source your way. Customers tell
us that they love the choice available
to them when it comes to how they
want to run their open source
workloads.
15. Customer obsession also means that
we want to understand the pain points
customers have when it comes to
using open source software and then
helping them tackle these pain points.
23. Leaders have relentlessly high
standards—many people may think
these standards are unreasonably high.
Leaders are continually raising the bar
and driving their teams to deliver high-
quality products, services, and
processes.
Leaders ensure that defects do not get
sent down the line and that problems are
fixed so they stay fixed.
24. Amazon
CloudSearch
(Apache Solr)
Amazon
Elasticsearch
Service
Amazon EMR
(Apache
Hadoop,
Apache Hudi)
Amazon Managed
Streaming for
Apache Kafka
Analytics
Amazon Kinesis
Data Analytics for
Java (powered by
Apache Flink)
Database
Amazon
ElastiCache
for Redis,
Memcached
Amazon RDS for
MySQL,
MariaDB,
PostgreSQL
Amazon Aurora
for MySQL,
PostgreSQL
Amazon
Keyspaces
Apache
MXNet on
AWS
PyTorch
on AWS
TensorFlow
on AWS
Machine learning
Amazon
SageMaker
Neo
Amazon
SageMaker
Clarify
Amazon Neptune
TorchServe
25. Leaders expect and require innovation
and invention from their teams and
always find ways to simplify.
They are externally aware, look for new
ideas from everywhere, and are not
limited by "not invented here."
As we do new things, we accept that we
may be misunderstood for long periods
of time.
26. Self Manage
• Open Source technology skill and expertise
• Installation and configuration of OSS
• Scaling and performance tuning
• Security configuration
• Patching – updates and security fixes
• Upstream contributions and fixes
• Support of open source technology
• Infrastructure patching – security and functional
• Compute orchestration, provisioning
• Cluster scaling
• Physical hardware, host OS/kernel, networking, and facilities
• Physical data centre capabilities
27. Managed Open Source
Self Manage
• Open Source technology skill and expertise
• Installation and configuration of OSS
• Scaling and performance tuning
• Security configuration
• Patching – updates and security fixes
• Upstream contributions and fixes
• Support of open source technology
• Infrastructure patching – security and functional
• Compute orchestration, provisioning
• Cluster scaling
• Physical hardware, host OS/kernel, networking, and facilities
• Physical data centre capabilities
• Use open source technology
29. David Woodhouse
Madyln Olson Robert Zhu
Michael McCandless Clare Ligouri James Gosling
Iliana Etaoin
This is just a small
selection of open
source builders at
Amazon…
31. Database & analytics Compute Machine learning Developer tools
• OpenSearch
• PartiQL
• Deequ
• Amazon Athena
Federated Query
• Babelfish for
PostgreSQL
• AWS SDK for pandas
• Amazon Linux
• Bottlerocket
• Firecracker
• AWS Nitro
Enclaves
• AWS
ParallelCluster
• Amazon EKS
Distro
• Amazon SageMaker Neo
• Apache MXNet
• Gluon, AutoGluon
• Sockeye
• TorchServe
• AWS Neuron SDK
• Deep Java Library
• Deep Graph Library
• Amazon SageMaker Clarify
• Amazon SageMaker
Jumpstart
• Amazon Corretto
• AWS Cloud
Development Kit
• Boto 3
• Cdk8s
• .NET Porting Assistant
Serverless Web development Security Other
• Chalice
• AWS SAM and SAM
CLI
• AWS Amplify
• AWS SDKs
• s2n
• Amazon Corretto Crypto
Provider
• Open Distro for Telemtry
• AWS IoT Greengrass v2
• FreeRTOS LTS
• AWS SaaS Boost
32. Database & analytics Compute Machine learning Developer tools
• OpenSearch
• PartiQL
• Deequ
• Amazon Athena
Federated Query
• Babelfish for
PostgreSQL
• AWS SDK for pandas
• Amazon Linux
• Bottlerocket
• Firecracker
• AWS Nitro
Enclaves
• AWS
ParallelCluster
• Amazon EKS
Distro
• Amazon SageMaker Neo
• Apache MXNet
• Gluon, AutoGluon
• Sockeye
• TorchServe
• AWS Neuron SDK
• Deep Java Library
• Deep Graph Library
• Amazon SageMaker Clarify
• Amazon SageMaker
Jumpstart
• Amazon Corretto
• AWS Cloud
Development Kit
• Boto 3
• Cdk8s
• .NET Porting Assistant
Serverless Web development Security Other
• Chalice
• AWS SAM and SAM
CLI
• AWS Amplify
• AWS SDKs
• s2n
• Amazon Corretto Crypto
Provider
• Open Distro for Telemtry
• AWS IoT Greengrass v2
• FreeRTOS LTS
• AWS SaaS Boost
33. Linux-based open-source operating system that is
purpose-built by Amazon Web Services for running
containers on virtual machines or bare metal hosts.
Security from the ground up
Only essential software need to run containers, SELinus enabled by default, dm-
verity protects read only root file system, no shell and more
Flexible
Ability to easily create different builds (variants)
Transactional
TUF – The Update Framework used to publish and secure repositories. Full disk
images are used to toggle updates for robust fall back on failure recovery
34. Code contributions
Amazon Neptune is compatible with Apache TinkerPop and AWS has contributed bug fixes and
enhancements to the Apache TinkerPop project
Part of the community
AWS developer currently serving as the PMC Chair
Members of the Amazon Neptune team include the author of “Practical Gremlin”,
An open source graph computing framework and
provides a standard way to interface with a number of
different databases.
35. Thinking small is a self-fulfilling prophecy.
Leaders create and communicate a bold
direction that inspires results.
They think differently and look around
corners for ways to serve customers.
For over 20 years open source has been helping Builders to innovate and develop solutions. Whether it is the tools they use to develop, the libraries they combine within their applications, or the runtimes on which those applications are deployed. Hello, my name is Suman Debnath and over the next 30 minutes I want to share how we think and work with open source at AWS, how we work backwards from our customers to provide Builders with greater choice in how they develop and run their preferred open source technologies, and share how Cloud open source is the future of IT.
But before we get started I want to thank each of you for the significant commitment you're making to come and spend an entire day with us at this wonderful Open Source India Conference.
We are so excited to have you here.
It’s been 2 years since we’ve been together in person, and its great to be back in an physical event like this
At Amazon, everything starts with our Customers, and when we speak to customers and ask them why they choose AWS, we get a lot of different answers because customers, have lots of different types of workloads.
Still, beyond the primary benefits like agility, cost and elasticity, there is one benefit that customers often mention and that is the pace of innovation.
We are constantly thinking of ways to invent on behalf of customers.
In 2021 alone, we launched 3,084 meaningful services and features.
That’s 3,084 more capabilities that our customers are equipped with to address their needs.
And the challenges for a customer today are often different from the ones of tomorrow, and that makes it all the more important to choose the cloud provider that will be that best technology partner now, and for the future.
Picking the right platform is incredibly important because decisions you make can be with you for a very long time.
It reminds me of a story I heard about the space shuttle.
If you look at a Space Shuttle, you will notice two big booster rockets attached to the sides.
The rockets need to generate maximum thrust, and to make this happen they need to be as wide as possible.
But in trying to increase the width, they ran into a surprising limitation.
You see, to get to the launch site, the booster rocket has to travel by train through tunnels, and in the US, these tunnels restrict the width of railroad cargo to less than 13 feet.
And tunnel width is limited by the width of the railroad tracks, which is 4 feet, 8 and a half inches.
The US inherited this standard from the first railroads built in England.
And these first railways were built to the same width used for building wagons.
And these wagons were built to roll on old roads that were first made by the wheels from Roman War Chariots.
The chariots were standardized to accommodate the width of two war-horses which just happened to be (pause) 4 feet, 8 and a half inches.
And so, as you can see, a significant design feature of a Space Shuttle was initially influenced by the width of two horses 2000 years ago.
When you look at business processes and technology, there are often many constraints and the reasons for them can date back to decisions made many years before.
It’s why it’s so important then when you make decisions about the cloud today, you make the right ones. As those decision will have a long-term impact on what you are able to achieve.
You should pick the platform with the most capabilities and the most innovation because you don’t know what the future holds, but you want to be ready.
AWS is innovative, in part, because we have a different approach and culture to many companies.
One of our 16 leadership principles is "Invent and simplify."
We hire builders, expecting employees of all types to always look at ways to simplify, looking for innovative ideas from everywhere, and avoid falling into the trap of doing things just because it's the way things have always been done.
Amazon Culture - LPs To understand how we think about open source at Amazon, we first need to look at our culture. At Amazon we have 16 LPs, and these form the basis for how everything is done. From finding and hiring builders, our day to day interactions with each other, how we get our work done, our leadership principles enables us to better understand how we can work together to serve our customers. These also influence us how we interact with open source. Lets take a look at some of these.
The first, and most important LP is Customer Obsession. What does this mean?
As a company, we are customer obsessed. We don’t lead with what the coolest tech is, what competitors are doing, or what delivers the best short-term results. We aim to build relationships with customers that outlast any of us.
It means we start with the customer and work backwards to understand their needs, their pain points and frustrations. It also means that we look between the lines and we discover opportunities to delight customers. In the context of open source this means a few things.
Many of our customers are enthusiastic users and creators of open source software.
We listen to our customers when it comes to what products and services we create. 90% of the AWS roadmap is driven by direct customer feedback, and this is true of open source. When customers tell us that they love open source technologies but they would like to benefit from integration to the broad range of AWS products and services, we listen and we work to create managed services for those open source technologies.
Customers tell us that they love the choice available to them when it comes to how they want to run their open source workloads. For some workloads, they do want to take on and self manage those open source workloads, other times it might be they want to use one of the many hundreds of open source products available via the AWS Marketplace.
Customer obsession also means that we want to understand the pain points customers have when it comes to using open source software and then helping them tackle these pain points. One of the common pain points that customers have when running open source workloads is how to remove all the undifferentiated heavy lifting involved. Installation, configuration, patching and security updates, scaling up and down, upgrades and more. They love the open source project, but want to focus on using it to help their customers.
Let’s take a closer look at some examples of this.
In 2014, the engineering team at Airbnb open sourced a tool that they had built to help them scale how they built their data pipelines.
Apache Airflow is a workflow orchestration technology that helps you create and manage complex workflows, and has been adopted by many business and data engineering teams.
It became a top level Apache project in 2019, and has been growing in popularity.
We have customers that prefer to manage their own Apache Airflow environments. Deploying Apache Airflow on AWS provides customers with lots of choice and flexibility, from deploying on VMs via EC2 to deploying on containers via ECS, EKS or your own Kubernetes environments. Customers can benefit from the integration of other AWS services such as S3 for storage, or perhaps data services such as Amazon Redshift and Athena.
But there are a number of challenges around self-managing Apache Airflow.
The first is really around setup. It's typically a very manual process—so much so that some customers end up developing their own environment creation tools. And there are lots of choices where it's not always easy to understand which is the best choice to make.
Scaling can also be a challenge. This can be done with EC2 autoscaling or Kubernetes, but either of those options bring their own complexity.
For security, supporting role-based Authentication and Authorization typically involves a process where you have to authenticate in one place and then go into the Airflow user interface and authorize that particular person to have a certain role, like administrator or viewer. Sometimes customers find it's easier just to make everyone administrator not even worry about it. . It's also pretty easy to make mistakes like accidentally opening up the Web server to the world.
Upgrades and patches can be challenging as there are hundreds of Python libraries and other dependencies to keep track of and knowing which ones are stable, which ones are required, and which versions offer a security risk can be tough. Upgrading Airflow can challenge as when you perform an upgrade, sometimes things don’t go the way you expect it can also be painful to roll back.
In December 2020, we launched a managed service for Apache Airflow called Managed Workflows for Apache Airflow that provides an upstream version of Apache Airflow, integrated with AWS services in your own AWS account.
Customers also have the choice of using Apache Airflow via a number AWS Partners. We can see here, there are many good choices for running Apache Airflow.
https://aws.amazon.com/marketplace/search?searchTerms=airflow&FULFILLMENT_OPTION_TYPE=SAAS&filters=FULFILLMENT_OPTION_TYPE
We also contributed a new project that helps builders easily develop and test their apache airflow workflows locally, before commiting them to your live environments.
Over the past few years we have worked with a number of leading open source ISVs, including Confluent, Databricks, HashiCorp, and Redis Labs to help them build and optimize their cloud services. Our partnership with Confluent is representative of how we obsess over partner and customer success.
Confluent is an Advanced tier Technology Partner, and works with AWS across a range of programs, including AWS Marketplace.
AWS and Confluent has also built tight service integrations, including integration with AWS Outposts to offer a hybrid data streaming solution.
According to Confluent, working with AWS gives their customers innovations unmatched by other clouds. These include:
Using Amazon S3 for tiered storage to enable infinite Kafka storage,
Multiple private networking options including PrivateLink,
Self-managed encryption keys for storage volume encryption using AWS Key Management Service (KMS).
AWS works with customers to help them effectively build and operate their own open source projects, while also contributing code to improve these projects.
For example, AWS has partnered with Intuit and Weaveworks to develop Argo Flux, which provides a single tool chain for continuous deployment and fleet-scale automated workflows using GitOps.
Another LP is Insist on the highest standard.
AWS services are built to meet the needs of very high scale multitenanted operations and when we build a service based on open source, we provide customers with a fully managed, highly scalable, multi-tenant service.
Here is a look at some of our AWS services for open source, across different categories such as data analytics, databases, compute, machine learning and many more.
These managed services allow customers to quickly get started using these open source technologies and not have to worry about having to run them.
When AWS launches a service based on an open source project, we make a long-term commitment to support our customers. We contribute bug fixes, security, scalability, performance, and feature enhancements back to the community. For example the Amazon EMR team has been making contributions to the Hadoop ecosystem for many years, and the Amazon Elastic Container Service for Kubernetes (EKS) team has been contributing to Kubernetes both code and broader contributions.
AWS has always aimed to take technology that was traditionally cost prohibitive and difficult for many organizations to adopt, and make it accessible to a much broader audience. Open source is just one more way that AWS seeks to make technology more accessible to everyone. Core to Amazon’s mission is helping make open source easier to use.
Thanks to the four freedoms of free software, customers are able to access many thousands of open source projects.
But just because you can access the source code, does that mean you can use it? Well not really – for many open source projects, there is “work” required to get it into a state where it can be useful to you.
When you think about what you typically need to do to have source code that is useful to you, there are a number of things you might look for.
Installation instructions and examples, you might need to learn the project and how it works if there are lots of moving parts, you will need to deploy this onto some sort of infrastructure, you will need to configure security and make sure that it can scale to meet your needs, and many more things.
When we dive into the details, we can see that operating open source technologies is a lot of work.
In order to be able to benefit from open source technologies, you will typically need to think about all of these activities.
This is one of the things that customers love about open source – the customer decides what and how they want to run their open source based on criteria they choose.
When you think about what you typically need to do to have source code that is useful to you, there are a number of things you might look for.
Installation instructions and examples, you might need to learn the project and how it works if there are lots of moving parts, you will need to deploy this onto some sort of infrastructure, you will need to configure security and make sure that it can scale to meet your needs, and many more things.
When we dive into the details, we can see that operating open source technologies is a lot of work.
In order to be able to benefit from open source technologies, you will typically need to think about all of these activities.
This is one of the things that customers love about open source – the customer decides what and how they want to run their open source based on criteria they choose.
At Amazon we hire builders.
We hire pioneers. People who challenge the status quo. People who like to invent, who want to build the future, people who like to look at different customer experiences and assess what's wrong with them and iterate them or reinvent them entirely. People who get that launch is the starting line, not the finish line
Many of these builders are also users and contributors to open source software.
This influences how we think about open source and we aim to build relationships with open source projects that outlast any of us.
We draw upon the expertise and experience from folk who have worked at the likes of Red Hat, the Apache Software Foundation, and other open source communities, as well as Amazonians contributing to diverse open source projects as you can see from this small selection.
We have a distinctly Amazonian term for the way we organize people to optimize innovation and execution. This is what we call our “two-pizza teams.” Meaning that no team should be big enough that it would take more than two pizzas to feed them.
This concept is fundamentally around creating a little startup of 10 or less people. Smaller teams minimize the need for matrixed communication and unnecessary meetings and bureaucracy. This helps accelerate decision making. With less technological dependencies and freer access to the tools they need to innovate, two-pizza teams have more time to apply to rapidly innovate on behalf of customers.
Many of these teams develop new open source projects.
This is a sample, some of these projects may not be familiar to you but are super interesting. Lets take a look at some of them.
Partiql is an open source project that allows you to provide a single query language across all your data sources, t easy to efficiently query data, regardless of where or in what format it is stored
S2n, short for signal to noise, is a new open source transport layer security implementation,
Amplify is a development platform for building secure, scalable modern mobile and web applications that makes it really easy to add capabilities such as authentication, machine learning and more.
Here are just some of those projects and lets take a closer look at one of these projects, Firecracker.
In addition to the more than 2,500 Amazon GitHub repositories we build, we contribute to thousands of third-party projects.
Sometimes this is on pre-existing projects that are meaningful to our customers such as Linux or Kubernetes, other times it might be new projects we release such as Firecracker or more recently Bottlerocket.
Many of these teams develop new open source projects.
This is a sample, some of these projects may not be familiar to you but are super interesting. Lets take a look at some of them.
Partiql is an open source project that allows you to provide a single query language across all your data sources, t easy to efficiently query data, regardless of where or in what format it is stored
Here are just some of those projects and lets take a closer look at one of these projects, Firecracker.
In addition to the more than 2,500 Amazon GitHub repositories we build, we contribute to thousands of third-party projects.
Sometimes this is on pre-existing projects that are meaningful to our customers such as Linux or Kubernetes, other times it might be new projects we release such as Firecracker or more recently Bottlerocket.
Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers on virtual machines or bare metal hosts.
Most customers today run containerized applications on general-purpose operating systems that are updated package-by-package, which makes OS updates difficult to automate. Updates to Bottlerocket are applied in a single step rather than package-by-package. This single-step update process helps reduce management overhead by making OS updates easy to automate using container orchestration services such as Amazon EKS and Amazon ECS. The single-step updates also improve uptime for container applications by minimizing update failures and enabling easy update rollbacks. Additionally, Bottlerocket includes only the essential software to run containers, which improves resource usage and reduces the attack surface.
Stephen Mallette is a member of the Amazon Neptune team at AWS. He has developed graph database and graph processing technology for many years. He is a decade long contributor to the Apache TinkerPop project, the home of the Gremlin graph query language, and is currently serving as its PMC Chair.
http://stephen.genoprime.com/
Amazon Neptine – TinkerPop compatibility -> https://docs.aws.amazon.com/neptune/latest/userguide/engine-releases-1.0.4.0.html
We have two other Neptune team members who are official Apache committers on the project, so that might be of interest as well. One of these folks is Kelvin Lawrence who authored "Practical Gremlin" a free online book which has been teaching people the Gremlin query language for many years now.
While some of this work was done at his time with IBM I believe that he has continually enhanced that work for the community in his time here at Amazon. You might check with him as to how much credit should go to Amazon for that work if you think you'd like to see that bullet in your slide...it is a highly recognized element of the TinkerPop Community.
AWS has been one of the top contributors to open source for years. No other company has done more to foster the rise and success of open source than AWS.
Our participation in and support of open source encompasses our own projects, code contributions to other projects, and financial support of open source foundations and projects, as well as the force multiplier effects of customers being able to run open source in the cloud, either on their own or via managed services.
We’ve made significant contributions to a myriad of open source projects, including the Xen, Linux, KVM, Java, Kubernetes, Chromium, Robot Operating System, and many more.
These are just a subset of projects and our contributions accelerate.
Make sure you are keeping up to date with everything that is happening on open source on AWS. Check these valuable resources.
I am always looking to hear from builders who are creating interesting open source projects to feature on the AWS open source blog or my newsletter, so if you want to share something you are working on then please let me know.
The cloud has changed the way we build apps, because now we have flexibility.
You can scale up without buying more hardware. You can actually scale down easily as well with the click of a button.
Today you can build things that would have been impossible to build before the cloud.
Everything is a programmable resource, getting access to capacity is a click of a button.
Networks became VPCs
Elastic load balancers replaced physical load balancers.
AWS provides you with the broadest and deepest set of services, and we are constantly innovating on your behalf. Unlocking new capabilities and possibilities.
It now means that AWS can be used for any type of application. Mission critical, Low latency.
And once you move to AWS you will be in the perfect position to build a modern data strategy so you can make faster, higher quality decisions.
At AWS we say that Leaders are never done learning and always seek to improve themselves.
They are curious about new possibilities and act to explore them.
There is a lot to learn at this Summit, I encourage you to take this opportunity to Learn and be Curious.
Thank you!
