Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit

383 views

Published on

GraphQL is a query language for APIs and a runtime to fulfill these queries, allowing applications to easily connect and access data stored on any type of database technology or API. AWS AppSync provides a powerful and flexible serverless GraphQL API that securely accesses, manipulates, and combines data from multiple sources at any scale, enabling you to build any kind of application on a range of data sources independently of the underlying database technology. In this session, we discuss different use cases where AWS AppSync and GraphQL power next-generation applications. Special guest, Candid Partners, shares how it uses AWS AppSync in its Data Fabric solution to simplify large-scale data management using a GraphQL API to interact with data lakes.

  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Next generation intelligent data lakes, powered by GraphQL & AWS AppSync Ed Lima Sr. Solutions Architect AWS AppSync & AWS Amplify Amazon Web Services M A D 3 0 9 Aaron Bawcom Chief Architect Candid Partners
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Agenda • Data and APIs • GraphQL: A query language for APIs • AWS AppSync: Use cases and capabilities • Data lakes, GraphQL, and AWS AppSync
  3. 3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  4. 4. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. “You can have data without information, but you cannot have information without data.” Daniel Keys Moran “Data really powers everything that we do.” Jeff Weiner
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T “A one-size-fits-all database doesnʼt fit anyone.” Werner Vogels
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS databases and analytics Broad and deep portfolio, built for builders AWS Marketplace Amazon Redshift Data warehousing Amazon EMR Hadoop + Spark Amazon Athena Interactive analytics Amazon Kinesis Data Analytics Real time Amazon ES Operational analytics Amazon RDS MySQL, PostgreSQL, MariaDB, Oracle, SQL Server Amazon Aurora MySQL, PostgreSQL Amazon QuickSight Amazon SageMaker Amazon DynamoDB Key value, document Amazon ElastiCache Redis, Memcached Amazon Neptune Graph Amazon Timestream Time series Amazon QLDB Ledger database Amazon S3/ Amazon S3 Glacier AWS Glue ETL and Data Catalog AWS Lake Formation Data lakes AWS DMS | AWS Snowball | AWS Snowmobile | Amazon Kinesis Data Firehose | Amazon Kinesis Data Streams | AWS Data Pipeline | AWS Direct Connect Data movement AnalyticsDatabases Business intelligence and machine learning Data lake Amazon Managed Blockchain AWS Blockchain Templates Blockchain Amazon Comprehend Amazon Rekognition Amazon Lex Amazon Transcribe AWS DeepLens 250+ solutions 730+ Database solutions 600+ Analytics solutions 25+ Blockchain solutions 20+ Data lake solutions 30+ solutions Amazon RDS on VMWare
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  8. 8. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Databases/ data stores Basic API technology stack Failover Load balancers Web/application servers Message buses Workers ?API backend ?API “server”
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Basic API technology stack Databases/ data stores API backendAPI “server”
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Databases/ data stores Basic API technology stack ?API backend ?API “server” AWS Fail over Load balancers Web/application servers Message buses Workers
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T No servers to provision or manage Scales with usage Never pay for idle Availability and fault tolerance built in Serverless means…
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Basic serverless API technology stack AWS Serverless business logic Serverless APIs Databases/ data stores Fail over Load balancers Web/application servers Message buses Workers
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T /posts /comments /authors REST API posts comments authors GraphQL API What is GraphQL?
  16. 16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://2018.stateofjs.com/data-layer
  17. 17. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://2018.stateofjs.com/data-layer
  18. 18. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://2018.stateofjs.com/data-layer
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Queries MutationsTypes Subscriptions GraphQL schema and operations A query language for APIs—and a runtime! type User { id: ID! username: String! firstName: String lastName: String daysActive: Int }
  20. 20. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. A query language for APIs Queries query GetPost { getPost(id: ”1”) { id title } } mutation CreatePost { createPost(title: “Summit”) { id title } } subscription OnCreatePost { onCreatePost { id title } } Mutations Subscriptions
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS AppSync Managed serverless GraphQL service Connect to data sources in your account Add data sync, real-time, and offline capabilities for any data source or API GraphQL facade for any AWS service Conflict detection and resolution in the cloud Enterprise security features: IAM, Amazon Cognito, OIDC, API keys
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Who’s using AWS AppSync in production?
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does AWS AppSync work? , ,
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does AWS AppSync work? , ,
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does AWS AppSync work? , ,
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does AWS AppSync work? , ,
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How does AWS AppSync work? , ,
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T GraphQL data flow in AWS AppSync: Resolvers Request template Response template Data sources 1 2 3 4 5
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Pipeline resolvers
  31. 31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Multi-auth Type, operation, and field-level authorization Default authorization mode Additional providers: • OpenID Connect • Amazon Cognito user pools • AWS IAM • API key
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Multi-auth Type, operation, and field-level authorization Default authorization mode Additional providers: • OpenID Connect • Amazon Cognito user pools • AWS IAM • API key
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Log insights
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Log insights nanoseconds
  36. 36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Real-time data broadcasting
  38. 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Real-time subscriptions subscription onCreateMessage { createMessage {…} } WebSocket URL and connection payload Secure WebSocket connection (wss://) AppSync Data Sources AWS AppSync Amazon DynamoDB Amazon Elasticsearch Service AWS Lambda Amazon Aurora HTTP Local (Pub/Sub)
  39. 39. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Offline
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Chat applications
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Internet of Things (IoT)
  43. 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Microservices data access
  44. 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Polyglot backend data access
  45. 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  46. 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T July 2019 Data Fabric
  47. 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Introduction Chief Architect, Candid Partners Aaron Bawcom
  48. 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Overview - Terminology - Data is everywhere - Data management challenges - The ideal solution - Data fabric solution
  49. 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Terminology • Data lake • Data warehouse • Data mart • Universal semantic layer • Operational data lake
  50. 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Terminology • Data provisioning • Data promotion • Storage zones • Data governance
  51. 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data is everywhere Acquisitions Business units Geographies External partners Supply chains Products Customer engagement IoT Websites Mobile
  52. 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data management challenges Most companies only analyze 12%of the data they have By 2020, every person will generate 1.7 megabytes in just a second Gartner estimates that poor-quality data is costing organizations on average $14.2 million annually “Nearly one third of analysts spend more than 40% of their time vetting and validating their analytics data before it can be used for strategic decision-making” Organizations make (often erroneous) assumptions about the state of their data and continue to experience inefficiencies, excessive costs, compliance risks, and customer satisfaction issues as a result Corporate data grows 40% per year (Hollis Tibbetts) Business cost of bad data may be as high as 10% to 25% of an organizations revenue 40% of businesses lose sales due to inaccessible data
  53. 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 54 Asinglesystemtoacquire,store, progress,refine,andmakeprovisioned dataaccessibletosystemsofinsight, dataanalysts,datascientists,and decisionmakers. view Operations •Helpdesk •Workflows •Stewardship Catalog •Glossary •Dictionary •Lineage •Search Universal semantic layer •API •Data warehouse Acquisition •Database push/pull •Streaming •File Promotion •Landing, raw, consolidated, processed, USL, and data science Governance •Policy •Process •Security •Entitlement The ideal solution
  54. 54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data acquisition Refinement Storage Lineage Data fabric solution AccessibilityandsystemIntegration usingcloud-nativetechnologies
  55. 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Conceptual pattern Storage Configuration Metadata Acquisition Router Processors Delivery Userinterface Efficiently acquire data Large-scale data transformation governance Bad ETL/ELT codeAPI access
  56. 56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  57. 57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 58 Problem: Efficiently acquire data Acquire data Cost optimization Operations Scale Flexibility
  58. 58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 59 Configuration YAML based with UI RDBMS push/pull API pull AWS Lambda, Amazon SQS Files AWS Transfer for SFTP, AWS Lambda, Amazon SQS Streaming Amazon API Gateway, Amazon Kinesis, AWS Lambda, AWS Service Endpoint Solution: Continuous data acquisition
  59. 59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 60 Problem: Large-scale data transformation governance Bad source data Source data systems Source data systems ETL dependency challenges
  60. 60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 61 Solution: Create simplified orchestration of data transformation Create data router Amazon DynamoDB, AWS Lambda, Amazon SQS Create data processors Amazon ECS task, AWS Lambda, AWS Glue, Amazon DynamoDB, Amazon Aurora, Amazon S3, Amazon Redshift
  61. 61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 62 Problem: Bad ETL/ELT code Data transformation challenges Difficult to conduct root-cause analysis Difficult ETL code engineering and operations
  62. 62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 63 Solution: Create modernized ETL logic using AWS AppSync Automated creation of GraphQL schema Automated creation of resolvers to cloud- native storage Amazon DynamoDB, Amazon S3, Amazon Redshift, Amazon Aurora Customized creation of resolvers to external data Simplified ETL business logic Simplified debugging Amazon CloudWatch Higher-quality ETL code
  63. 63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T GraphQL standardized ETL rules: - name: eidrProductUpdate src: s3:/eidr/product extract: query: “query { … }” variables: id: $src.productID transform: | ( { data, src } ) => return { …data, …src } load: mutate: “mutate (…) { … }” variables: input: $transform
  64. 64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 65 Problem: No unified API provided by the organization Different suppliers had different forms of EDI interfaces into the organization Different portals were provided for different types of users to order products Different integrations had different “views” of product master data Lack of consistent order execution lead to penalty fines paid to major customers No external enterprise authorization and authentication structure was available from a consumable API
  65. 65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 66 Solution: Create a single unified external API for e- commerce transactions Continuous data acquisition Amazon DynamoDB Single GraphQL schema AWS AppSync
  66. 66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T E-commerce API type Mutation { orderProduct(…) @aws_auth( cognito_groups: [”corpECommerce"] ) }
  67. 67. Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ed Lima @ednergizer Aaron Bawcom aaron.bawcom@candidpartners.com

×