Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data integration with embulk

586 views

Published on

Data Integration with Embulk
Data Science Weekend 2016, Yogyakarta
December 3rd, 2016

Published in: Data & Analytics
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Data integration with embulk

  1. 1. Data Integration with Embulk DATA SCIENCE WEEKEND 2016, YOGYAKARTA TEGUH NUGRAHA
  2. 2. Multi Data Formats and Storages MySQL PostgreSQL MongoDB CSV files BigQuery Redshift HDFS Google Analytics Mixpanel Zendesk Elasticsearch
  3. 3. Multi Data Sources Users data in MySQL Offline data in CSV Traffics data in Google Analytics Log data Bulk Data Loading: Load data from A to B
  4. 4. Problems Parsing files Error handling Idempotent Retrying Performance Scalability Format compatibility
  5. 5. Solution Reliable framework with parallel execution, data validation, error recovery, auto guessing, resuming and extensive plugins github.com/embulk/embulk
  6. 6. Embulk: Bulk Data Loader
  7. 7. Plugins by Category •Input plugins •Output plugins •Filter plugins •File parser plugins •File decoder plugins •File formatter plugins •File encoder plugins •Executor plugins
  8. 8. Getting Started 1. Embulk requires Java 2. Download embulk: http://dl.embulk.org/embulk-latest.jar 3. Make it executable $ embulk --version 4. Run an example: $ embulk example
  9. 9. Installing Embulk Plugin $ embulk gem install embulk-input-mysql $ embulk gem install embulk-output-postgresql List of plugins: https://embulk.org/plugins
  10. 10. Embulk Configuration File
  11. 11. Embulk Configuration File (YAML) in: Input plugin options. ◦ parser: If the input is file-based, parser plugin parses a file format (built-in csv, json, etc). ◦ decoder: If the input is file-based, decoder plugin decodes compression or encryption (built-in gzip, bzip2, zip, tar.gz, etc). out: Output plugin options. ◦ formatter: If the output is file-based, formatter plugin formats a file format (such as built-in csv, JSON) ◦ encoder: If the output is file-based, encoder plugin encodes compression or encryption (such as built-in gzip or bzip2) filters: Filter plugins options (optional). exec: Executor plugin options. An executor plugin control parallel processing (such as built-in thread executor, Hadoop MapReduce executor)
  12. 12. Using Guess Command Guess command guesses parser and decoder options $ embulk guess seed.yml –o config.yml
  13. 13. Using guess command
  14. 14. Previewing and Running $ embulk preview config.yml $ embulk run config.yml Setup cron schedule
  15. 15. embulk-input-mysql https://github.com/embulk/embulk-input-jdbc/tree/master/embulk- input-mysql $ embulk gem install embulk-input-mysql
  16. 16. embulk-output-postgresql https://github.com/embulk/embulk-output-jdbc/tree/master/embulk- output-postgresql $ embulk gem install embulk-output-postgresql
  17. 17. Using Variables configuration file name must end with .yml.liquid Environment variables are set to env variable
  18. 18. Include file File will be searched from the relative path of the input configuration file and file name will be _<name>.yml.liquid
  19. 19. Thank You TEGUH NUGRAHA DATA SCIENCE LEAD, BUKALAPAK TEGUH@BUKALAPAK.COM HTTP://WWW.SLIDESHARE.NET/TEGUHN
  20. 20. References https://embulk.org https://github.com/embulk/embulk http://www.slideshare.net/frsyuki/fighting-against-chaotically- separated-values-with-embulk

×