Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Source at AWS: Code, Contributions, Collaboration, and Communication

297 views

Published on

At OSCON 2018, Adrian Cockcroft detailed the many ways AWS participates in open source: contributing to open source projects, reporting bugs, contributing fixes and enhancements to a wide spectrum of projects ranging from the Linux kernel to PostgreSQL and Kubernetes, and managing the hundreds of projects of its own.

  • Be the first to comment

  • Be the first to like this

Open Source at AWS: Code, Contributions, Collaboration, and Communication

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark The @AWSOpen Team OSCON July 18, 2018 Open Source at AWS Community, Code, Contributions, and Communication
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Happy 20th Birthday Open Source!
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Welcome! Community Contributions CommunicationCode
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark OSS is Everywhere Today
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Courtesy of The Linux Foundation Sources: Sourceclear, Sonatype, Github 64M+ Repositories on GitHub 23M+ Open source developers 41B+ Lines of code 1,100 New projects a day
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Growing Community Support open source foundations and organizations Cloud Native Computing Foundation OSI Apache Software Foundation Linux FoundationOpen Container Initiative Open Network User Group
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Mission To be Earth’s most customer-centric company
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Growing Community Engage customers to collaborate and promote their projects
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Motivation for Open Source We are motivated to collaborate by many of our partners and customers Open source projects help us to innovate Scaling open source projects as AWS services meets customer needs Open data helps customers build innovative services
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Open Source Code as a Service Upstream code and expertise Make open source code easy to use Release new open software
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Make Open Source Code Easy to Use Open Source Machine Learning Frameworks Suneel Marthi @suneelmarthi @AWSOpen AI/ML/Analytics Specialist Principal Open Source Technologist
  12. 12. Put machine learning in the hands of every developer and data scientist M L @ A W S O U R M I S S I O N
  13. 13. F R A M E W O R K S KERAS
  14. 14. F R A M E W O R K S KERAS P L A T F O R M S A M A Z O N S A G E M A K E R
  15. 15. F R A M E W O R K S KERAS P L A T F O R M S A M A Z O N S A G E M A K E R A P P L I C A T I O N S E R V I C E S R E K O G N I T I O N R E K O G N I T I O N V I D E O P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X
  16. 16. F R A M E W O R K S KERAS P L A T F O R M S A M A Z O N S A G E M A K E R A P P L I C A T I O N S E R V I C E S R E K O G N I T I O N R E K O G N I T I O N V I D E O P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X
  17. 17. KERAS O P E N S O U R C E F R A M E W O R K S A N D I N T E R F A C E S F r a m e w o r k s I n t e r f a c e s Complete control over the entire stack
  18. 18. KERAS O P E N S O U R C E F R A M E W O R K S A N D I N T E R F A C E S Complete control over the entire stack F r a m e w o r k s I n t e r f a c e s NVIDIA Tesla V100 GPUs (14x faster than P2) P3 Open Source Machine Learning AMIs 5,120 Tensor cores 128GB of memory 1 Petaflop of compute NVLink 2.0 I N F R A S T R U C T U R E
  19. 19. Where do we spend our time? B UI L D
  20. 20. Where do we spend our time? B UI L D TRAI N
  21. 21. Where do we spend our time? B UI L D TRAI N D EP L O Y
  22. 22. A P P L I C A T I O N S E R V I C E S R E K O G N I T I O N R E K O G N I T I O N V I D E O P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X F R A M E W O R K S KERAS P L A T F O R M S A M A Z O N S A G E M A K E R
  23. 23. Amazon SageMaker Pre-built notebooks for common problems Built-in, high performance algorithms TRAI N D EP L O Y B UI L D
  24. 24. Amazon SageMaker Pre-built notebooks for common problems Built-in, high performance algorithms One-click training Hyperparameter optimization D EP L O Y B UI L D TRAI N
  25. 25. Amazon SageMaker Fully managed hosting with auto-scaling One-click deployment Pre-built notebooks for common problems Built-in, high performance algorithms One-click training Hyperparameter optimization B UI L D TRAI N D EP L O Y
  26. 26. Model NEW I TEM PREDICTED SALES ACTIVITY Inventory K-means clustering
  27. 27. Model NEW P URC HAS E FRAUD RISK Purchases Logistic regression
  28. 28. Model NEW D ATA Algorithms Amazon SageMaker PREDICTIONS
  29. 29. Collect and prepare training data Choose and optimize your ML algorithm Set up and manage environments for training Train and tune model (trial and error) Deploy model in production Scale and manage the production environment Amazon SageMaker
  30. 30. Put open source machine learning in the hands of every developer and data scientist M L @ A W S O U R M I S S I O N
  31. 31. B R I N G I N G C L O U D S C A L E T O D a t a b a s e s
  32. 32. M i g ra t e b e t w e e n o n -p re m a n d A W S M i g ra t e b e t w e e n d a t a b a s e s A u t o m a t e d s c h e m a c o n v e rs i o n D a t a re p l i c a t i o n fo r z e ro d o w n t i m e AWS Database Migration Service Helps convert from proprietary enterprise databases to open source 76,000+ unique databases migrated using DMS
  33. 33. Amazon Aurora The fastest-growing service in AWS history MySQL and PostgreSQL compatible Several times faster than standard MySQL and PostgreSQL Highly available and durable 1/10th the cost of commercial grade database
  34. 34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Make Open Source Code Easy to Use Open Source Container Technologies Arun Gupta @arungupta @AWSOpen Container Specialist, CNCF Board Member Principal Open Source Technologist
  35. 35. “Run Kubernetes for me”
  36. 36. EKS Platform for enterprises to run production-grade Kubernetes M a n a g e d a n d c o n s i s t e n t e x p e r i e n c e S e a m l e s s , n a t i v e i n t e g r a t i o n w i t h A W S s e r v i c e s B u i l t w i t h t h e O S S c o m m u n i t y U p s t r e a m a n d C e r t i f i e d
  37. 37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Open Source Code Contributions
  38. 38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark gvsbuild cbmc golang apache joshua apache mxnet chromium wing airflow mqtt lantram tuplopenfire moses s3_file elasticsearch xen netty beanstalker reactdaterangepicker pigredis linuxkernel apache hive hue docker apache bigtop libgxps apachespark apache oozie libsoup joshua decoder presto apache hbase exoplayer gpyopt glib go-dockerclient apache phoenix botoflow cloud-init rdoc glib-networking grpc-go chromium vim redhat rails scikit-fuzzy fast align californium logstash-input-udp pygresql eclipse paho apache zeppelin qemu containerd cnn catwatch ios-webkit-debug-proxy celery dynamodb backend rosette yarn boto 3 awsiotdevkit hana mariadb-connector-j appiumadb googleguava elastalert jacksondatabind esp-open-rtos kenlm dovecot haproxy goa tsung mshadow uml-utilities appium-chromedriver github-plugin pyzmq statsd backend lombok capybara apachetomcat gerrit-check mysql workbench dynamic-object angulartooltips apache commons lang appium-gulp-plugin ruby-simple-queue go-statsd-client ldns sentry facebookwebdriveragent littleproxy bixie troposphere git-bigstore fop django-debug-toolbar bandit elephas akka grpc parquet-mr rspec-retry glisten wordpress filequeue oci-fetch zipper pkg errors jcommander aws-scala cmis_5 java dpdk rebouncer parquet-python libfuse appiumremotedebugger leveldb mvel webdriveragent yield-java immutables.org cocoapods assertj-core appium-dot-exe safe-eval xcpretty ews java api streetaddress wycheproof mqtt js appium fabric scapy react-hot-loader sys-proctable httpcomponentshttpclient apachepig sync word segmentation treeextra ultipart-parser tez wired tiger cmock go-winio postgresql apache commons dbutils gradle dtcoretext libarchive p5-protocol-http2 react-native apache ant open fire appium-android-bootstrap iputils gulp-typescript dynamodb-milkhatx kinesalite yubico apache commons io presto hadoop apache 2 ansible nginx 2016
  39. 39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark gvsbuild cbmc golang apache joshua apache mxnet wing airflow mqtt lantram tupl moses s3_file elasticsearch xen netty beanstalker reactdaterangepicker pigredis linuxkernel apache hive hue docker apache bigtop libgxps apachespark apache oozie libsoup joshua decoder presto apache hbase exoplayer gpyopt glib go-dockerclient apache phoenix botoflow cloud-init rdoc glib-networking grpc-go chromium vim redhat rails scikit-fuzzy fast align logstash-input-udp pygresql eclipse paho apache zeppelin qemu containerd cnn catwatch ios-webkit-debug-proxy celery dynamodb backend rosette yarn boto 3 awsiotdevkit hana mariadb-connector-j appiumadb googleguava elastalert jacksondatabind esp-open-rtos kenlm dovecot haproxy goa tsung mshadow uml-utilities appium-chromedriver github-plugin pyzmq statsd backend lombok capybara apachetomcat gerrit-check mysql workbench dynamic-object angulartooltips apache commons lang appium-gulp-plugin ruby-simple-queue go-statsd-client ldns sentry facebookwebdriveragent littleproxy bixie troposphere git-bigstore fop django-debug-toolbar bandit elephas akka grpc parquet-mr rspec-retry glisten wordpress filequeue oci-fetch zipper pkgerrors jcommander aws-scala cmis_5 java dpdk rebouncer parquet-python libfuse appiumremotedebugger leveldb mvel webdriveragent yield-java immutables.org cocoapods assertj-core appium-dot-exe safe-eval xcpretty ews java api streetaddress wycheproof mqtt js appium fabric scapy react-hot-loader sys-proctable httpcomponentshttpclient apachepig sync word segmentation treeextra ultipart-parser tez wired tiger cmock go-winio apache commons dbutils gradle dtcoretext libarchive p5-protocol-http2 react-native apacheant open fire appium-android-bootstrap iputils gulp-typescript dynamodb-milkhatx kinesalite yubico apache commons io presto hadoop apache 2 ansible nginx chromium openfire nodejs linux (arch) janusgraph wing wingtk open mpi repolinter deep visualization toolbox kubernetes flink tacacsplus glib-openssl sparklingml openjdk tercom react-draft-wysiwyg incubator mxnet mxnet notebooks libmoon gpy moby oss-attribution-generator licensee glib seabios ompi linux(xendrivers) linux (kvm) postgresql tslint-eslint-rules cni dmlc containernetworking gbm jackson-dataformats-binary tinyxml12 peewee netlink definitelytyped phoenix pynamodb jgi t rgp gpyopt tabular smack sqoop firefox postcss-extract-animations irate irate astrum gatsby cerebro mithriljs keras geckodriver wireshark cmake gtk+ tslint behave bro appium-uiautomator2-server linux xfs nginx paramz pydeepgp rust emrun hypothesis-python linux (pm) intellij-community xenbits apache (http server) typescript-json-schema mysql 389ds conda-build spring framework yarn gitlfs gsettings-test git-lfs ex_aws web socket sharp conda liblouis eslint bencode-go emscripten react-native-swipeout kappa apachehadoopyarn macports parsimonious coreos ublockorigin vowpal wabbit airlift wycheproof s2n linux (blkfront) kotlin homebrew linux (fbdev) pyaad libkv curator libc allure ts-test slight.alexa lombok linux-pci immutableassign apache lucene react-int dynamic-object jscolor chart.js rigor linux(nvme) s3fs apache beam aalto-xml deniswernert/udev koji ntpsec lmdbjava apache hadoop react-spinkit redux bin webhooks corenlp papaparse counterfeiter go-btrfs jruby cairo sequel pidgin sockeye rollbar typescript-fsa alks-cli gucumber libjson autovpn json11 moto r509-ocsp-responder bambamboogieman pkcs7pad flask-sqlalchemy californium multi-module-mave tensorboard fortuneserver linux-nvme-cli securitymonkey functional-streams-for-scala spark-redshift plantuml-syntax multi-module-maven-release-plugin 2017
  40. 40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark gvsbuild cbmc golang apache joshua apache mxnet wing airflow mqtt lantram tupl moses s3_file elasticsearch xen netty beanstalker reactdaterangepicker pigredis linuxkernel apache hive hue docker apache bigtop libgxps apachespark apache oozie libsoup joshua decoder presto apache hbase exoplayer gpyopt glib go-dockerclient apache phoenix botoflow cloud-init rdoc glib-networking grpc-go chromium vim redhat rails scikit-fuzzy fast align logstash-input-udp pygresql eclipse paho apache zeppelin qemu containerd cnn catwatch ios-webkit-debug-proxy celery dynamodb backend rosette yarn boto 3 awsiotdevkit hana mariadb-connector-j appiumadb googleguava elastalert jacksondatabind esp-open-rtos kenlm dovecot haproxy goa tsung mshadow uml-utilities appium-chromedriver github-plugin pyzmq statsd backend lombok capybara apachetomcat gerrit-check mysql workbench dynamic-object angulartooltips apache commons lang appium-gulp-plugin ruby-simple-queue go-statsd-client ldns sentry facebookwebdriveragent littleproxy bixie troposphere git-bigstore fop django-debug-toolbar bandit elephas akka grpc parquet-mr rspec-retry glisten wordpress filequeue oci-fetch zipper pkgerrors jcommander aws-scala cmis_5 java dpdk rebouncer parquet-python libfuse appiumremotedebugger leveldb mvel webdriveragent yield-java immutables.org cocoapods assertj-core appium-dot-exe safe-eval xcpretty ews java api streetaddress wycheproof mqtt js appium fabric scapy react-hot-loader sys-proctable httpcomponentshttpclient apachepig sync word segmentation treeextra ultipart-parser tez wired tiger cmock go-winio apache commons dbutils gradle dtcoretext libarchive p5-protocol-http2 react-native apacheant open fire appium-android-bootstrap iputils gulp-typescript dynamodb-milkhatx kinesalite yubico apache commons io presto hadoop apache 2 ansible nginx chromium openfire nodejs linux (arch) janusgraph wing wingtk open mpi repolinter deep visualization toolbox kubernetes flink tacacsplus glib-openssl sparklingml openjdk tercom react-draft-wysiwyg incubator mxnet mxnet notebooks libmoon gpy moby oss-attribution-generator licensee glib seabios ompi linux(xendrivers) linux (kvm) postgresql tslint-eslint-rules cni dmlc containernetworking gbm jackson-dataformats-binary tinyxml12 peewee netlink definitelytyped phoenix pynamodb jgi t rgp gpyopt tabular smack sqoop firefox postcss-extract-animations irate irate astrum gatsby cerebro mithriljs keras geckodriver wireshark cmake gtk+ tslint behave bro appium-uiautomator2-server linux xfs nginx paramz pydeepgp rust emrun hypothesis-python linux (pm) intellij-community xenbits apache (http server) typescript-json-schema mysql 389ds conda-build spring framework yarn gitlfs gsettings-test git-lfs ex_aws web socket sharp conda liblouis eslint bencode-go emscripten react-native-swipeout kappa apachehadoopyarn macports parsimonious coreos ublockorigin vowpal wabbit airlift wycheproof s2n linux (blkfront) kotlin homebrew linux (fbdev) pyaad libkv curator libc allure ts-test slight.alexa lombok linux-pci immutableassign apache lucene react-int dynamic-object jscolor chart.js rigor linux(nvme) s3fs apache beam aalto-xml deniswernert/udev koji ntpsec lmdbjava apache hadoop react-spinkit redux bin webhooks corenlp papaparse counterfeiter go-btrfs jruby cairo sequel pidgin sockeye rollbar typescript-fsa alks-cli gucumber libjson autovpn json11 moto r509-ocsp-responder bambamboogieman pkcs7pad flask-sqlalchemy californium multi-module-mave tensorboard fortuneserver linux-nvme-cli securitymonkey functional-streams-for-scalaspark-redshift plantuml-syntax multi-module-maven-release-plugin 2018 ndbench openjml tinycorelinux xen (devel) nnvm tvm llvm libuavcan gdk-pixbufgraphql-go cynical authenticator protobuf scipy apache activemq tacacs_plus graphql-java clearlydefined cryptography sparkmagic packer ubuntu xen devellibfabric jupyter notebook go-git arrow jest teleport convolutional-pose-machines-tensorflow irqbalance focus-android miniredis jdk spack apacheincubatorairflownimbus-jose-jwt packr mosquito apache jena rusqlite webkit kcov jawk ffmpeg cargo-kcov quicktype swaggercodegen bundler linux-fsdevel apache httpcomponents core recrud netty reactive streams txthings goleveldb seunjeon jose pysnmp retool openquantumsafe amphtml liboqs oauth2 kotlinpoet kuromoji diaporama badssl.com xcode-install unicodecldr kpatch graal foolbox apache commons-compress goics linux kexec glide jedis apache solr istioselenium notebook pyinvestigate hypothesis lsp-intellij-spacemacs boringssl cyrus-sasl openssl jepsen jetty.project
  41. 41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Open Source Communications Events
  42. 42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 2018 Major Events funded and operated by @AWSopen team FOSDEM RailsConf KubeCon Europe PyCon FOSS Backstage OSCON and CLS OS Summit North America ApacheCon OS Summit Europe All Things Open ONUG Fall KubeCon North America Belgium Minneapolis Copenhagen Cleveland Berlin Portland OR Vancouver Montreal Edinburgh Raleigh New York City Seattle
  43. 43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Open Source Communications Telling the Story
  44. 44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Some developers think (or hope) great code will be enough
  45. 45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Open Source Blog aws.amazon.com/blogs/opensource
  46. 46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark • Open source projects that we “own” (mostly on GitHub) • Projects to which we contribute significantly • Projects that we and others make easier for customers to use on AWS (which may also involve contribution) • Projects our customers have created that run on/improve/extend AWS • Other open source projects/activities/ events that employees are involved in and/or we’re sponsoring • Excluded: AI/ML—that material goes on the Machine Learning blog. Blog Scope
  47. 47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
  48. 48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Open Social Media Twitter: @AWSOpen, @AWSCloud | Facebook | LinkedIn
  49. 49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Promote Your AWS-Related Open Source Project Contact Deirdré Straughan dbs@amazon.com
  50. 50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Open Source Communications Sharing Experiences
  51. 51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Collect your copy at AWS Booth 301 Book signing at Thursday 10:30am Download, share and support opensource.amazon.com/enterprise-oss-book Content will be available for translations under a CC license
  52. 52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Closing Summary Community Contributions CommunicationCode
  53. 53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Thank you from the @AWSOpen team attending OSCON & CLS Zaheda Bhorat Head of Open Source Strategy @zahedab Alolita Sharma Principal Open Source Technologist @alolita Shirley Bailes Open Source Events Lead @sbailes Deidré Straughan Open Source Social & Blog Lead @deirdres Adrian Cockcroft VP Architecture Strategy @adrianco @AWSOpen | opensource.amazon.com | aws.github.io

×