Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ian Robinson, Specialist SA, Data and Analytics,...
AWS Glue automates
the undifferentiated heavy lifting of ETL
Automatically discover and categorize your data making it imm...
AWS Glue: Components
Data Catalog
 Hive Metastore compatible with enhanced functionality
 Crawlers automatically extract...
Common Use Cases for AWS Glue
Understand your data assets
Instantly query your data lake on Amazon S3
ETL data into your data warehouse
Build event-driven ETL pipelines
Demo
AWS Glue Data Catalog
Glue data catalog
Manage table metadata through a Hive metastore API or Hive SQL.
Supported by tools like Hive, Presto, Sp...
Data Catalog: Crawlers
Automatically discover new data, extracts schema definitions
• Detect schema changes and version ta...
AWS Glue Data Catalog
Bring in metadata from a variety of data sources (Amazon S3, Amazon Redshift, etc.) into a single
ca...
Data Catalog: Table details
Table schema
Table properties
Data statistics
Nested fields
Data Catalog: Version control
List of table versionsCompare schema versions
Data Catalog: Detecting partitions
file 1 file N… file 1 file N…
date=10 date=15…
month=Nov
S3 bucket hierarchy Table defi...
Data Catalog: Automatic partition detection
Automatically register available partitions
Table
partitions
Job Authoring in AWS Glue
 Python code generated by AWS Glue
 Connect a notebook or IDE to AWS Glue
 Existing code brought into AWS Glue
Job Auth...
1. Customize the mappings
2. Glue generates transformation graph and Python code
3. Connect your notebook to development e...
 Human-readable, editable, and portable PySpark code
 Flexible: Glue’s ETL library simplifies manipulating complex, semi...
Job Authoring: Glue Dynamic Frames
Dynamic frame schema
A C D [ ]
X Y
B1 B2
Like Spark’s Data Frames, but better for:
• Cl...
Job Authoring: Glue transforms
ResolveChoice() B B B
project
B
cast
B
separate into cols
B B
Apply Mapping() A
X Y
A X Y
A...
Job authoring: Relationalize() transform
Semi-structured schema Relational schema
FKA B B C.X C.Y
PK ValueOffset
A C D [ ]...
Job authoring: Glue transformations
 Prebuilt transformation: Click and
add to your job with simple
configuration
 Spigo...
Job authoring: Write your own scripts
Import custom libraries required by your code
Convert to a Spark Data Frame
for comp...
Job authoring: Developer endpoints
 Environment to iteratively develop and test ETL code.
 Connect your IDE or notebook ...
Job Authoring: Leveraging the community
No need to start from scratch.
Use Glue samples stored in Github to share, reuse,
...
Orchestration and Job
Scheduling
Job execution: Scheduling and monitoring
Compose jobs globally with event-
based dependencies
 Easy to reuse and leverage...
Job execution: Job bookmarks
For example, you get new files everyday
in your S3 bucket. By default, AWS Glue
keeps track o...
Job execution: Serverless
 Auto-configure VPC and role-based access
 Customers can specify the capacity that
gets alloca...
Introduction to AWS Glue
Introduction to AWS Glue
Upcoming SlideShare
Loading in …5
×

Introduction to AWS Glue

3,602 views

Published on

In this session, we introduce AWS Glue, provide an overview of its components, and share how you can use AWS Glue to automate discovering your data, cataloging it, and preparing it for analysis.

  • Have you ever heard of taking paid surveys on the internet before? We have one right now that pays $50, and takes less than 10 minutes! If you want to take it, here is your personal link ■■■ https://tinyurl.com/make2793amonth
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I made $2,600 with this. I already have 7 days with this... ●●● https://tinyurl.com/make2793amonth
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Introduction to AWS Glue

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ian Robinson, Specialist SA, Data and Analytics, EMEA 20 September 2017 Introduction to AWS Glue
  2. 2. AWS Glue automates the undifferentiated heavy lifting of ETL Automatically discover and categorize your data making it immediately searchable and queryable across data sources Generate code to clean, enrich, and reliably move data between various data sources; you can also use their favorite tools to build ETL jobs Run your jobs on a serverless, fully managed, scale-out environment. No compute resources to provision or manage. Discover Develop Deploy
  3. 3. AWS Glue: Components Data Catalog  Hive Metastore compatible with enhanced functionality  Crawlers automatically extracts metadata and creates tables  Integrated with Amazon Athena, Amazon Redshift Spectrum Job Execution  Run jobs on a serverless Spark platform  Provides flexible scheduling  Handles dependency resolution, monitoring and alerting Job Authoring  Auto-generates ETL code  Build on open frameworks – Python and Spark  Developer-centric – editing, debugging, sharing
  4. 4. Common Use Cases for AWS Glue
  5. 5. Understand your data assets
  6. 6. Instantly query your data lake on Amazon S3
  7. 7. ETL data into your data warehouse
  8. 8. Build event-driven ETL pipelines
  9. 9. Demo
  10. 10. AWS Glue Data Catalog
  11. 11. Glue data catalog Manage table metadata through a Hive metastore API or Hive SQL. Supported by tools like Hive, Presto, Spark etc. We added a few extensions:  Search over metadata for data discovery  Connection info – JDBC URLs, credentials  Classification for identifying and parsing files  Versioning of table metadata as schemas evolve and other metadata are updated Populate using Hive DDL, bulk import, or automatically through Crawlers.
  12. 12. Data Catalog: Crawlers Automatically discover new data, extracts schema definitions • Detect schema changes and version tables • Detect Hive style partitions on Amazon S3 Built-in classifiers for popular types; custom classifiers using Grok expressions Run ad hoc or on a schedule; serverless – only pay when crawler runs Crawlers automatically build your Data Catalog and keep it in sync
  13. 13. AWS Glue Data Catalog Bring in metadata from a variety of data sources (Amazon S3, Amazon Redshift, etc.) into a single categorized list that is searchable
  14. 14. Data Catalog: Table details Table schema Table properties Data statistics Nested fields
  15. 15. Data Catalog: Version control List of table versionsCompare schema versions
  16. 16. Data Catalog: Detecting partitions file 1 file N… file 1 file N… date=10 date=15… month=Nov S3 bucket hierarchy Table definition Estimate schema similarity among files at each level to handle semi-structured logs, schema evolution… sim=.99 sim=.95 sim=.93 month date col 1 col 2 str str int float Column Type
  17. 17. Data Catalog: Automatic partition detection Automatically register available partitions Table partitions
  18. 18. Job Authoring in AWS Glue
  19. 19.  Python code generated by AWS Glue  Connect a notebook or IDE to AWS Glue  Existing code brought into AWS Glue Job Authoring Choices
  20. 20. 1. Customize the mappings 2. Glue generates transformation graph and Python code 3. Connect your notebook to development endpoints to customize your code Job authoring: Automatic code generation
  21. 21.  Human-readable, editable, and portable PySpark code  Flexible: Glue’s ETL library simplifies manipulating complex, semi-structured data  Customizable: Use native PySpark, import custom libraries, and/or leverage Glue’s libraries  Collaborative: share code snippets via GitHub, reuse code across jobs Job authoring: ETL code
  22. 22. Job Authoring: Glue Dynamic Frames Dynamic frame schema A C D [ ] X Y B1 B2 Like Spark’s Data Frames, but better for: • Cleaning and (re)-structuring semi-structured data sets, e.g. JSON, Avro, Apache logs ... No upfront schema needed: • Infers schema on-the-fly, enabling transformations in a single pass Easy to handle the unexpected: • Tracks new fields, and inconsistent changing data types with choices, e.g. integer or string • Automatically mark and separate error records
  23. 23. Job Authoring: Glue transforms ResolveChoice() B B B project B cast B separate into cols B B Apply Mapping() A X Y A X Y Adaptive and flexible C
  24. 24. Job authoring: Relationalize() transform Semi-structured schema Relational schema FKA B B C.X C.Y PK ValueOffset A C D [ ] X Y B B • Transforms and adds new columns, types, and tables on-the-fly • Tracks keys and foreign keys across runs • SQL on the relational schema is orders of magnitude faster than JSON processing
  25. 25. Job authoring: Glue transformations  Prebuilt transformation: Click and add to your job with simple configuration  Spigot writes sample data from DynamicFrame to S3 in JSON format  Expanding… more transformations to come
  26. 26. Job authoring: Write your own scripts Import custom libraries required by your code Convert to a Spark Data Frame for complex SQL-based ETL Convert back to Glue Dynamic Frame for semi-structured processing and AWS Glue connectors
  27. 27. Job authoring: Developer endpoints  Environment to iteratively develop and test ETL code.  Connect your IDE or notebook (e.g. Zeppelin) to a Glue development endpoint.  When you are satisfied with the results you can create an ETL job that runs your code. Glue Spark environment Remote interpreter Interpreter server
  28. 28. Job Authoring: Leveraging the community No need to start from scratch. Use Glue samples stored in Github to share, reuse, contribute: https://github.com/awslabs/aws-glue-samples • Migration scripts to import existing Hive Metastore data into AWS Glue Data Catalog • Examples of how to use Dynamic Frames and Relationalize() transform • Examples of how to use arbitrary PySpark code with Glue’s Python ETL library Download Glue’s Python ETL library to start developing code in your IDE: https://github.com/awslabs/aws-glue-libs
  29. 29. Orchestration and Job Scheduling
  30. 30. Job execution: Scheduling and monitoring Compose jobs globally with event- based dependencies  Easy to reuse and leverage work across organization boundaries Multiple triggering mechanisms  Schedule-based: e.g., time of day  Event-based: e.g., job completion  On-demand: e.g., AWS Lambda  More coming soon: Data Catalog based events, S3 notifications and Amazon CloudWatch events Logs and alerts are available in Amazon CloudWatch Marketing: Ad-spend by customer segment Event Based Lambda Trigger Sales: Revenue by customer segment Schedule Data based Central: ROI by customer segment Weekly sales Data based
  31. 31. Job execution: Job bookmarks For example, you get new files everyday in your S3 bucket. By default, AWS Glue keeps track of which files have been successfully processed by the job to prevent data duplication. Option Behavior Enable Pick up from where you left off Disable Ignore and process the entire dataset every time Pause Temporarily disable advancing the bookmark Marketing: Ad-spend by customer segment Data objects Glue keeps track of data that has already been processed by a previous run of an ETL job. This persisted state information is called a bookmark.
  32. 32. Job execution: Serverless  Auto-configure VPC and role-based access  Customers can specify the capacity that gets allocated to each job  Automatically scale resources (on post-GA roadmap)  You pay only for the resources you consume while consuming them There is no need to provision, configure, or manage servers Customer VPC Customer VPC Compute instances

×