Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Bridging the Gap Between Real Time/Offline and AI/ML Capabilities in Modern Serverless Apps (MOB310) - AWS re:Invent 2018

491 views

Published on

Building real-time collaboration applications can be difficult, and adding intelligence to an app to make it stand out remains a challenge. In this session, learn how to build real-time chat serverless apps infused with AWS machine learning (ML) services. We dive into enhancing a real-time chat application with search capabilities, chatroom bots providing automated responses , and on-demand message translation using Amazon AI/ML services.

Bridging the Gap Between Real Time/Offline and AI/ML Capabilities in Modern Serverless Apps (MOB310) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bridging the Gap Between Real Time/Offline and AI/ML Capabilities in Modern Serverless Apps Brice Pellé Enterprise Support Lead Amazon Web Services M O B 3 1 0 Ed Lima Solutions Architect Amazon Web Services
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • The mobile/web app challenges in 2018 • Addressing the difficult basics • Solving the challenges • Boosting your application with intelligence
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://www.flickr.com/photos/118006733@N05/12619573163/ The mobile and web app challenges in 2018
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://www.flickr.com/photos/atbaker/302838820/ Anybody (AuthN/Z)
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Anywhere (mobility) https://www.flickr.com/photos/mrtiking/8615241850/
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. On anything (platform) https://www.flickr.com/photos/mikecogh/7348035690/
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. In an instant (Real-time) https://www.flickr.com/photos/claustom/15288837503/
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. At any time (offline) https://www.flickr.com/photos/labdog2010/32955584723/
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The mobile/web app challenge in 2018 • In 2018, these problems are still difficult to solve • But they should be the basic requirements • Making an app stand out is becoming even harder • Millions of apps available across various application stores
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. To solve the challenge, developers need to • Iterate quickly • Focus on development, not servers • Provide the basics at a low cost of development • Offer innovative ways for users to interact with their apps • Make applications smarter, without a PhD in data sciences
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The modern mobile/web app must Solve the security challenge • AuthN and AuthZ by default Solve the data challenge • Network friendly (less calls / more data / just right) • Instantaneous, with real-time feedback Solve the screen and mobility challenge • Responsive & progressive • Always available, even offline Solve the differentiation challenge • Innovative & new ways to interact • How can your app stand out? Oh and … no servers!
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Messaging application architecture overview Machine learning services AppSync data sources Raw data
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://github.com/aws-samples/aws-appsync-chat-starter-react Machine learning services AppSync data sources Raw data
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Iterating quickly with AWS Amplify • CLI & toolchain for the client • Client library & Javascript framework support • Supported services • Database • API • Lambda functions • Authentication • Analytics • Hosting • Storage > npm install aws-amplify > npm install –g @aws-amplify/cli
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solve the security challenge Identity is mission critical for applications Three key areas to address • Authentication • Authorization • User management Implementing auth infrastructure is hard!
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secure, always! • Leverage Amazon Cognito • Built with best security practices • Iterate quickly with AWS Amplify • With toolchain • Provision serverless backend resources • Use Auth module to easily interact • Wrap app in authenticator to require sign-in • Uses customizable pre-built UI components amplify import awsmobile from './aws-exports' import Amplify, { Auth } from 'aws-amplify' import { withAuthenticator } from 'aws- amplify-react’ ... Amplify.configure(awsmobile) ... class App extends Component { async componentDidMount() { session = await Auth.currentSession() this.setState({ session }) } } export default withAuthenticator(App) © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solve the data challenge: GraphQL /posts /comments /authors REST API posts comments authors GraphQL API
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS AppSync Managed serverless GraphQL service Connect to data sources in your account Add data sync, real-time, and offline capabilities for any data source or API GraphQL facade for any AWS service Conflict detection and resolution in the cloud Enterprise security features: IAM, Amazon Cognito, OIDC, API keys
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building a serverless messaging app with NoSQL 1..1 1..n 1..1 1..n 1..n Relations between NoSQL Tables User MessageConversation User Conversation
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Schemas with AWS Amplify > amplify add api type Conversation @model { id: ID! name: String! createdAt: String messages: [Message] @connection(name: "ConvoMsgs") } type Message @searchable @model { id: ID! content: String file: S3Object messageConversationId: ID conversation: Conversation @connection(name: "ConvoMsgs") } • Define and deploy DynamoDB tables based on schema • Creates AWS AppSync API & resolvers • Supports custom transformers > amplify api gql-compile; amplify api push type Mutation @custom { createMessage(input: CustomCreateMessageInput!): Message @custom(subscription: true, watchedFields: ["messageConversationId"]) } © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GraphQL transformer – mix and match data sources searchMessages mutations queries listMessages createMessage type Message @searchable @model { id: ID! content: String file: S3Object messageConversationId: ID conversation: Conversation @connection( name: "ConvoMsgs") }
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Application overview • Data presentation split between components • Design independent of data source • Components wrapped in graphql HOC InputBar ConversationBar SearchBar const SearchResultListWithData = graphql(search, { skip: props => !props.term, options: props => ({ variables: { userTerm: `.*${props.term}.*`, msgFilter: buildMsgFilter(props.term, props.conversations) }, fetchPolicy: 'cache-and-network' }) })(SearchResultList)
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS AppSync with AWS Amplify Four easy steps to configure AWS AppSync 1. Import modules 2. Configure client 3. Advanced settings a. Complex data handling b. Union and interface handling 4. Wrap app in HOCs a. Make client available to provider b. Load app data from storage c. Launch app when data ready d. Require authentication import AWSAppSyncClient from 'aws-appsync' import { Rehydrated } from 'aws-appsync-react’ import awsmobile from './aws-exports' const client = new AWSAppSyncClient({ url: awsmobile.aws_appsync_graphqlEndpoint, region: awsmobile.aws_appsync_region, auth: { type: awsmobile.aws_appsync_authenticationType, jwtToken: async () =>(await Auth.currentSession()) .getIdToken().getJwtToken() }, complexObjectsCredentials: () => Auth.currentCredentials(), cacheOptions: { fragmentMatcher } }) const WithProvider = () => ( <ApolloProvider client={client}> <Rehydrated> <App /> </Rehydrated> </ApolloProvider> ) export default withAuthenticator(WithProvider) a b c d a b © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Network friendly— Batch requests • Wrap component with graphql • Data fetch when component is mounted but skipped if necessary • Cache lookup for quick user feedback • Use fetchMore to fetch additional data Messenger const MessengerWithData = graphql(getConvoMessages, { skip: props => !props.conversation, options: props => ({ variables: {convoId: props.conversation}, fetchPolicy: 'cache-and-network' }) })(Messenger) export default Messenger export { MessengerWithData } © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instantaneous—Using subscriptions Subscriptions are • Granular • Real-time • In line notifications Initiated on component mount for each opened conversation Canceled on unmount or conversation change componentDidMount() { const { subscribeToMore, conversation: { id: convoId }, userId } = this.props this.unsubscribe = subscribeToMore({ document: onCreateMessage, variables: { messageConvoId: convoId }, updateQuery: (previous, data) => {...} }) } componentWillUnmount() { this.unsubscribe() } MessagePane © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. subscription onCreateMessage { createMessage {…} } Websocket URL and connection payload Secure Websocket connection (wss://) Subscriptions—Handshake process AppSync Data Sources
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solve the screen challenge: Responsive Using a responsive design allows for “write once, run anywhere” Several well-known frameworks available
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secure ShareableNetwork agnostic Always up-to-date Solve the mobility challenge: Progressive Write once, run anywhere
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solve the mobility challenge: Always available With AWS AppSync SDK • Configured by default • Use Rehydrated component to rehydrate cache from storage • Use: fetchPolicy to control cache and network usage • AWS AppSync SDK maintains an outbox of pending actions • If no connection, then store action • On reconnection, send action • Development: Use redux tools
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Offline data rendering • Provide optimistic response • Useful when offline or slow network • Required when using offline capabilities! • Provide all fields present in the returned selection set mutation({ variables, optimisticResponse: { createMessage: {...} }, update: (proxy, data) => { const QUERY = { query: getConvoMessages, variables: { convoId } } const prev = proxy.readQuery(QUERY) const data = {...} proxy.writeQuery({ ...QUERY, data }) } }) MessagePane © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Platform services Application services Frameworks & interfaces Caffe2 CNTK Apache MXNet PyTorch TensorFlow Torch Keras Gluon AWS Deep Learning AMIs Amazon SageMaker Amazon Rekognition Amazon Transcribe Amazon Translate Amazon Polly Amazon Comprehend Amazon Lex Amazon Mechanical Turk The AWS Machine Learning stack
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PLATFORM SERVICES Application services FRAMEWORKS & INTERFACES Caffe2 CNTK Apache MXNet PyTorch TensorFlow Torch Keras Gluon AWS Deep Learning AMIs Amazon SageMaker Amazon Rekognition Amazon Transcribe Amazon Translate Amazon Polly Amazon Comprehend Amazon Lex Amazon Mechanical Turk The AWS Machine Learning stack
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The AWS Machine Learning stack—App services Vision Speech Language Chatbots
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vision Speech Language Chatbots The AWS Machine Learning stack—ChatQL app
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Common language use cases Information bots Education Accessibility Knowledge management Voice of customer applications Customer service/ call centers Enterprise digital assistant Semantic search Captioning workflows LocalizationPersonalization
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Translate Natural and fluent language translation Real-time translation Batch analysis Automatic language recognition Low cost
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Polly Turn text into lifelike speech using deep learning Wide selection of voices and languages Synchronize speech Fine-grained control Unlimited replay
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Comprehend Storm World Series Australia Stock market Washington Health Crisis Machine learning Library of news articles Amazon Comprehend Discover insights and relationships in text using deep learning
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Comprehend Discover insights and relationships in text using deep learning Entities Key phrases Language Sentiment Amazon Comprehend
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A m a z o n . c o m , I n c . i s l o c a t e d i n S e a t t l e , W A a n d w a s f o u n d e d J u l y 5 t h , 1 9 9 4 b y J e f f B e z o s . O u r c u s t o m e r s l o v e b u y i n g e v e r y t h i n g f r o m b o o k s t o b l e n d e r s a t g r e a t p r i c e s N a m e d e n t i t i e s A m a z o n . c o m : O r g a n i z a t i o n S e a t t l e , W A : L o c a t i o n J u l y 5 t h , 1 9 9 4 : D a t e J e f f B e z o s : P e r s o n K e y p h r a s e s O u r c u s t o m e r s b o o k s b l e n d e r s g r e a t p r i c e s S e n t i m e n t P o s i t i v e L a n g u a g e E n g l i s h Text analysis
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Lex Building conversational interfaces into any application using voice and text Integrated development in the AWS console Trigger Lambda functions Multi-step conversations One-click deployment Enterprise connectors Fully managed
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Rekognition Image Detect objects, scenes, and faces, extract text and identify unsafe content in images Object and scene detection Facial analysis Face recognition Text in imageUnsafe image detection Celebrity recognition Face comparison
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AppSync Data Sources Raw Data Machine Learning Services
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Request template Response template Data sources 1 2 3 4 5 GraphQL data flow in AWS AppSync
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. An AI component import React from "react"; import { graphql } from "react-apollo"; import rekognition from "../../images/rekognition.png"; import detectLabels from "../../graphql/queries/detectLabels"; class Labels extends React.Component { render() { const { data: { loading, error, detectLabels } } = this.props; if (loading) { return (<JSX…>); } else if (error) { const err = JSON.stringify(error.message); return ( return (<JSX…>); } else { const response = JSON.parse(detectLabels.response); return (<JSX…>); } } export default graphql(detectLabels, { skip: props => !props.path, options: props => ({ variables: { bucket: props.bucket, key: props.path }, fetchPolicy: "cache-and-network" }) })(Labels);
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS AppSync as GraphQL facade for AI services import gql from "graphql-tag"; export default gql` query invokeBot($bot: String, $text: String) { invokeBot(bot: $bot, text: $text) { response } } `; { "version" : "2017-02-28", "operation": "Invoke", "payload": { "field": "lex", "arguments": { "bot": "$context.arguments.bot", "sender": "$context.identity.sub", "text": "$context.arguments.text" } } } exports.handler = (event, context, callback) => { switch (event.field) { case "lex": const lexparams = { botAlias: "$LATEST", botName: event.arguments.bot, inputText: event.arguments.text, userId: event.arguments.sender }; lex.postText(lexparams, function(err, data) { if (err) { console.error("Error JSON: ", JSON.stringify(err, null, 2)); callback(err); } else { let result = { bot: event.arguments.bot, text: event.arguments.text, response: data.message }; callback(null, result); } }); break;
  51. 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversation Real-time intelligence
  53. 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversation Real-time intelligence
  54. 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversation Real-time intelligence
  55. 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversation Real-Time Intelligence
  56. 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversation Real-Time Intelligence
  57. 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Real-time intelligence
  58. 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automatic language translation with speech
  59. 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sentiment and entity analysis
  60. 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Offline intelligence
  61. 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Offline intelligence
  62. 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Offline intelligence
  63. 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence
  64. 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence
  65. 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence
  66. 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence
  67. 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence
  68. 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  69. 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  70. 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  71. 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  72. 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  73. 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Image and audio intelligence bucket object
  74. 74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  75. 75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The modern mobile/web app must Solve the security challenge • AWS Amplify and Amazon Cognito Solve the data challenge • AWS AppSync and GraphQL Solve the screen and mobility challenge • AWS AppSync + PWA built-in offline capabilities Solve the differentiation challenge • Great user (+ developer) experience (real-time, pagination, toolchain) • Add Intelligence to your app with AWS AI/ML Oh and … no servers!
  76. 76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://github.com/aws-samples/aws-appsync-chat-starter-react
  77. 77. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Brice Pellé Enterprise Support Lead Amazon Web Services Ed Lima Solutions Architect Amazon Web Services @ednergizer@BicePelle
  78. 78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×