AWS re:Invent 2016: Open Source at AWS—Contributions, Support, and Engagement(ARC213)

409 views

Published on

Over the last few years, we have seen a dramatic increase in the use of open source projects as the mainstay of architectures in both startups and enterprises. Many of our customers and partners also run their own open source programs and contribute key technologies to the industry as a whole (see DCS201). At AWS we engage with open source projects in a number of ways. We contribute bug fixes and enhancements to popular projects including our work with the Hadoop ecosystem (see BDM401), Chromium (see BAP305) and (obviously) Boto. We have our own standalone projects including the security library s2n (see NET405) and machine learning project MXnet (see MAC401). We also have services that make open source easier to use like ECS for Docker (see CON316), and RDS for MySQL and PostgreSQL (see DAT305). In this session you will learn about our existing open source work across AWS, and our next steps.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
409
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
53
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

AWS re:Invent 2016: Open Source at AWS—Contributions, Support, and Engagement(ARC213)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adrian Cockcroft - AWS VP Cloud Architecture Strategy December 1, 2016 Open Source at AWS Contributions, Support and Engagement ARC213
  2. 2. What to Expect from the Session • Examine how leading AWS customers use open source • See the value of an open source program • Understand the trends in open source adoption • Baseline existing contributions from AWS • Begin the discussion of next steps in engagement
  3. 3. What do customers want?
  4. 4. Capital One use of open source
  5. 5. Capital One https://developer.capitalone.com/open-source/ “We’re an open source first organization — actively using, contributing to and managing open source software projects.”
  6. 6. Capital One Open Source Projects https://capitalone.github.io Cloud Custodian – AWS Lambda based policy engine – ARC312 Hygieia – DevOps delivery pipeline dashboard OAS-nodegen – Open API Specification support Ops-Pipeline – scripts for creating environments Hydrograph – User interface for ETL management Giraffez – Client library for Teradata React Native PathJS Charts SWHttpTrafficRecorder – Swift mobile library CQRS Manager for Distributed Reactive Services
  7. 7. Netflix open source program
  8. 8. From 2012 NetflixOSS Presentation by @adrianco
  9. 9. Current NetflixOSS Program http://www.slideshare.net/aspyker/netflix-open-source-building-a-distributed-and-automated-open-source-program
  10. 10. NetflixOSS in 2016 http://netflix.github.io Around 200 projects currently on Github Deep integration into SpringCloud Java framework Hystrix circuit-breaker ported and adopted widely Curator migrated from NetflixOSS to Apache Curator New: Titus team working with AWS ECS – See CON313 Startups inspired by NetflixOSS: • DynomiteDB supporting and extending Dynomite Redis clustering • Armory providing Spinnaker deployment pipeline as a service • Gremlin implementing Chaos Monkey testing as a product
  11. 11. Many Benefits of Open Source Programs Give back to the communities that you depend on Attract and curate better developers Improved code quality, docs, modularity and re-use Pooled development and testing resources Validate architectural decisions in public Enable fast-followers and remain centered in an ecosystem Build trust and basis for collaboration
  12. 12. Open Source is the Default Leading customers have their own open source programs Many AWS products have an open source component Blackduck Annual Future Of Open Source survey “The survey reports that 78 percent of its respondents are now running their businesses with open source software, and two-thirds are building software for their customers that’s based on open source software. More significant, the percentage of respondents actually participating in open source projects has increased from 50 percent to 64 percent, and 88 percent say they expect to contribute to projects within the next three years.”
  13. 13. Baseline current status
  14. 14. Memberships Code contributions Projects hosted on GitHub Open source based services
  15. 15. Open Source Memberships Linux Foundation Silver Member • Founding advisory member of Xen @ Linux Foundation • Founding member of the Core Infrastructure Initiative • Founding member of the Open Container Initiative ToDo Group - Companies committed to open source Alliance for Open Media – Video format Open-MPI - consortium for high performance computing
  16. 16. Amazon & AWS contributions
  17. 17. Bug Fix and Feature Contributions Linux & Drivers Xen Apache Tomcat PostgreSQL Docker Boto Apache Hadoop Apache Hive Apache Bigtop Apache Oozie Apache Drill Apache Zeppelin Apache Pig Cloudera HUE Apache Lucene Apache Solr Kuromoji ElasticSearch CBMC MXNet Moses Apache Joshua
  18. 18. Highlights for Bug/Feature Contributions Linux Contributions • ENA – network driver • Annapurna – ARM64 chipset support • AWS recently hired Linux committer David Woodhouse Boto – Python SDK – AWS hired creator Mitch Garnaat Hadoop ecosystem projects – over 100 contributions
  19. 19. MXNet – Deep Learning Framework http://www.allthingsdistributed.com/2016/11/mxnet-default-framework-deep-learning-aws.html Created at CMU Major investments by AWS Scalable, high performance Supported alongside many other deep learning frameworks Contributions to developer experience, documentation Tools for visualization, development, migration Supporting academic contributions with AWS credits See MAC401 - Scalable Deep Learning Using MXNet
  20. 20. Repositories Owned by AWS & Amazon https://github.com/aws https://github.com/awslabs https://github.com/amznlabs https://blox.github.io/ Blox – Container orchestration for ECS – CON316 s2n – Secure replacement for openssl – NET405 Chalice – Python serverless microframework for AWS – DEV308 AWS-CLI – Built by Mitch Garnaat – DEV402 AWS-shell – Autocomplete based user interface for AWS-CLI AWS SDKs AWS Mobile SDKs Cloud9 Ace – Cloud based interactive development editor Cfncluster – Build and manage HPC clusters ION – Amazon data serialization libraries
  21. 21. Blox – Container Orchestration Framework https://blox.github.io/ New governance model for Blox – built in the open All development and roadmap in public Provides a venue for collaboration with AWS customers Intent to build an ecosystem Contribute/port schedulers See more details in CON316
  22. 22. Contributing to Blox https://blox.github.io/ • Blox is licensed under Apache 2.0 • Open an issue or pull request • Watch our roadmap on GitHub • Check out our Gitter channel
  23. 23. Open Source Made Easy by AWS Services Amazon EMR – Elastic Hadoop Amazon ElastiCache - Memcached Amazon RDS – MySQL Amazon RDS – PostgreSQL Amazon RDS – Scalable Aurora back-end for MySQL & PostgreSQL AWS OpsWorks – Chef Automate Server Amazon ECS – Docker Orchestration Amazon CloudSearch Amazon Elasticsearch
  24. 24. Let AWS Do The Undifferentiated Heavy Lifting Amazon EMR updates & patches are frequent and automatic
  25. 25. Engagement and next steps
  26. 26. Program Team Represents all of Amazon Manage consumption of projects Manage engineering contributions Represent AWS (foundations etc.) Work with legal on licensing Assist with standalone projects Operate GitHub accounts Did everything to this point AWS Teams Managing Open Source New – Outbound Programs Team Engage with AWS customers Support academic credit program Create content: blogs, talks etc. Represent AWS (foundations etc.) Present at industry events Manage example code projects Measure sentiment/engagement Starting now!
  27. 27. AWS Cloud Credits for Research
  28. 28. AWS has been supporting global research projects through the Cloud Credits for Research program since 2009.  AWS believes that researchers are key drivers of technological innovation  AWS Cloud Credits for Research is aimed at helping researchers move their research endeavors to the cloud so they can innovate more quickly and at lower cost
  29. 29. AWS has been supporting global research projects through the Cloud Credits for Research program since 2009. We support researchers who seek to:  Build reusable tools to facilitate their future research and the research of their community  Perform pilot, proof of concept, or benchmark tests evaluating the efficacy of moving research workloads or open data sets to the cloud  Train a broader community on the usage of cloud for research workloads via workshops or tutorials
  30. 30. AWS Cloud Credits for Research Recipients https://aws.amazon.com/research-credits/2016-recipients/ Categories Proof of Concept Development of Open Source Software Development of a Cloud Service Open Dataset Development Benchmark for Comparison Workshop or Tutorial
  31. 31. AWS Credits for Open Source Projects Awarded on a case-by-case basis: Berkeley AMP Lab – Founding Sponsor Apache Mesos, Spark and other open source tools Open-MPI Consortium support PostgreSQL project was given credits to support testing
  32. 32. How can we help?
  33. 33. Adrian Cockcroft @adrianco Experience: Startup Open Source Ecosystems NetflixOSS New - Outbound Team Contact us via AWS account management, or in person at #reinvent Zaheda Bhorat @zahedab Experience: UK Gov Open Document Format Salesforce Open Source Program Google Summer of Code Sun OpenOffice & Netbeans
  34. 34. First steps, we’re listening…
  35. 35. Thank you!
  36. 36. Remember to complete your evaluations!
  37. 37. Related Sessions DCS201 - Open-Source Resources and Certifications BDM401 - Deep Dive: Amazon EMR Best Practices & Design Patterns BAP305 - Zero to Google Chrome in 60 Minutes: Lightweight and Inexpensive Client Devices for Amazon WorkSpaces NET405 - Amazon s2n: Cryptography and Open Source at AWS MAC401 - Scalable Deep Learning Using MXNet CON316 - State of the Union: Containers CON313 - Netflix: Container Scheduling, Execution, and Integration with AWS DAT305 - Deep Dive on Amazon Relational Database Service DEV308 - Chalice: a Serverless Microframework for Python DEV402 - The Effective AWS-CLI User

×