Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OWF12/Java Michael hirt

1,900 views

Published on

  • Be the first to comment

OWF12/Java Michael hirt

  1. 1. Tackling Big Data with Hadoop andGraphical Open Source Integration Michaël Hirt Data Integration Product Manager
  2. 2. Agenda1. What is Big Data ?2. Talend’s Goal3. What’s next ? Big Data Quality and Big Data management4. Talend Open Studio for Big Data in action© Talend 2011 2
  3. 3. What is Big Data?
  4. 4. What Is BIG Data? 2,300 tweets per second "Big data" (June 2011) is information of extreme size, 50 gigabytes of data diversity, complexity per person on Earth 50,000,000,000 and need for rapid 300 exabytes total processing. 200 billion Ted Friedman - Information intelligent devices Infrastructure and Big Data Projects Key Initiative Overview - July 2011 200,000,000,000 2015 275 exabytes of data flowing over the Internet each day 275,000,000,000,000,000,000© Talend 2011 2020 4
  5. 5. How to define Big data is…. Hans Rosling – uses big data to analyze world health trends Key Takeaway #1 volume, variety, velocity© Talend 2011 – Stri2y Private & Confidential© Talend 2011 5
  6. 6. The 6 Dimensions of BIG Data Primary challenges  Volume  Velocity  Variety  Complexity And also  Validation  Lineage © Talend 2011 6
  7. 7. Key Takeaway #2 Forces us to think© Talend 2011 differently© Talend 2011 – Stri2y Private & Confidential 7
  8. 8. Traditional Data Flows CRM ETL Normalized Traditional Data ERP Data Data Warehouse Quality Finance • Scheduled–daily or weekly, sometimes more frequently. Business Business Analyst User • Volumes rarely exceed terabytes Warehouse Administrator Executives© Talend 2011 – Stri2y Private & Confidential© Talend 2011 8
  9. 9. The new world of big data Social Networking CRM ERP Big Data Finance© Talend 2011 – Stri2y Private & Confidential© Talend 2011 9
  10. 10. The new world of big data Social Networking CRM Mobile Devices ERP Big Data Finance© Talend 2011 – Stri2y Private & Confidential© Talend 2011 10
  11. 11. The new world of big data Social Networking CRM Mobile Devices ERP Transactions Finance Network Devices Big Data Sensors© Talend 2011 – Stri2y Private & Confidential© Talend 2011 11
  12. 12. Data driven business enables data governance supports information decisions drives Information provides value to the business If you cant rely on your information then Your the result can be missed opportunities, or business higher costs. Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE).© Talend 2011 – Stri2y Private & Confidential© Talend 2011 12
  13. 13. BIG Data Management Big Data Big Data Management Production Big Data Big Data Big Data RDBMS Integration Quality Consumption Analytical DB NoSQL DB ERP/CRM Mining SaaS Social Media Analytics Web Analytics Log Files Storage Processing Search RFID Filtering Call Data Records Sensors Enrichment Machine-Generated Turn Big Data into actionable information© Talend 2011 13
  14. 14. BIG data driven business enables BIG data governance supports BIG BIG information decisions drives Information provides value to the business If you cant rely on your information then the result can be missed opportunities, or BIG higher costs. business Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE).© Talend 2011 – Stri2y Private & Confidential© Talend 2011 14
  15. 15. Our goal
  16. 16. Talend – The Market Leading Unified Integration Platform Talend Enterprise Data Data MDM ESB BPM Quality Integration  Commercial license  Subscription model Studio Repository Deployment Execution Monitoring  Open source license Talend Open Studio for  Free of charge  Optional support Data Data Quality Integration MDM ESBRecognized as the open source leader in each of its market category by all industry analysts© Talend 2011 16
  17. 17. Trying to get from this… © Talend 2011 – Stri2y Private & Confidential © Talend 2011 17
  18. 18. to this… Why Talend… ONLY Talend generates code that is executed within map reduce. This open approach removes the limitation of a proprietary “engine” to provide a truly unique and powerful set of tools for big data. © Talend 2011 – Stri2y Private & Confidential © Talend 2011 18
  19. 19. “Big Data for the Masses”
  20. 20. Goal: Democratize Big Data Talend Open Studio for Big Data  “Big Data for the Masses”  Improves efficiency of big data job design with graphic interface  Abstracts and generates code  Run transforms inside Hadoop  Native support for HDFS, Pig, HBase, Pig Sqoop and Hive  Apache License 2.0  Embedded in Hortonworks Data Platform …an open source  Certifed with Cloudera, MapR and ecosystem Grenplum© Talend 2011 – Stri2y Private & Confidential© Talend 2011 20
  21. 21. Big Data – How about Data Quality?© Talend 2012
  22. 22. Poor Data Quality + Big Data = Big ProblemsPoor Data Quality * Big Data = Big Problems^2 Key Takeaway #3 In big data… poor data quality can be magnified at huge scale© Talend 2011 23
  23. 23. Two methods for inserting data quality into a big data job 1. Pipelining: as part of the load process 2. Load the cluster then implement and execute a data quality map reduce job© Talend 2011 24
  24. 24. E-T-L Extract – Transform - Load© Talend 2011 – Stri2y Private & Confidential© Talend 2011 25
  25. 25. E- DQ -L Extract – Improve/Cleanse - Load© Talend 2011 – Stri2y Private & Confidential© Talend 2011 26
  26. 26. Pipelining: data quality with big data CRM DQ ERP DQ Finance Big Data Social Networking • Use traditional data quality tools • No new programming, no PHDs • Once and done Mobile Devices© Talend 2011 – Stri2y Private & Confidential© Talend 2011 27
  27. 27. Big data alternative: Load and improve within the cluster CRM DQ ERP DQ Finance Big Data Social Networking • Load first, improve later • Really complex to build, limited tools • Constant on, increments Mobile Devices • Insane performance© Talend 2011 – Stri2y Private & Confidential© Talend 2011 28
  28. 28. Let us show you…© Talend 2012
  29. 29. What’s next for Talend Big Data?© Talend 2012
  30. 30. Talend Open Studio for Big Data 4.1: Hive & 5.1:HCatalog 4.0: HDFS 4.2: Pig 5.0: Hbase Sqoop & Oozie© Talend 2011 31
  31. 31. big2012 data now Q4 2013Talend Open Studio for Big DataPackaged within Hortonworks Data Platform …Eclipse tools for HIVE, HDFS, PIG, SCOOP …supports Oozie, Hcatalog, KerberosFree to download and use under the Apache license …democratizing big data through intuitive tools© Talend 2011 – Stri2y Private & Confidential© Talend 2011 32
  32. 32. Questions / Thanks for attending mhirt_at_talend.com

×