How to Automate Offloading ETL Processes to Hadoop
•
0 likes•1,222 views
Offloading ETL processes to Hadoop is often one of the first Big Data efforts because of the obvious ROI benefits. However, you have hundreds, maybe thousands, of legacy ETL processes to migrate which makes achieving the benefits of Hadoop and ROI a distant goal. What if you could automatically convert up to 70% of your existing ETL processes to run on Hadoop with no code changes? In this presentation you will see: - A detailed walk-through of migrating existing ETL processes to Hadoop without changing anything - How you can cut development time of new ETL process on Hadoop by up to 50% - How you can leverage your existing developers’ Java skills to turn them into Hadoop developers - Best practices for monitoring the performance of your ETL processes to ensure you meet your service level agreements
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How to Automate Offloading ETL Processes to Hadoop
Similar to How to Automate Offloading ETL Processes to Hadoop (20)
Recently uploaded
Recently uploaded (20)
How to Automate Offloading ETL Processes to Hadoop
- 1. Confidential OPERATIONAL EXCELLENCE FORBIG DATAAPPS
- 2. Confidential 2 TRUSTED by over 10,000 companies as their big data app platform BACKED by top Silicon Valley investors True Ventures, Rembrandt VP, Bain Capital FOUNDED in 2008, with headquarters in San Francisco
- 3. Confidential PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS your big data apps MONITOR to resolve issues faster big data apps more effectively MANAGECOLLABORATE
- 4. Confidential 4 Java, Scala (Scalding), SQL SIMPLE Ensure best practices at any scale thanks to easy-to-learn design principles FLEXIBLE Leverage existing Java, Scala, and SQL skills and easily adapt to new systems WE ARE THEDEVELOPERS BEHIND CASCADING RELIABLE Always get optimal performance and reliability for big data applications
- 5. Confidential • Use Hadoop for ETL / ELT • Ensure quality and manageability of our ELT / ELT applications • Translate existing ETL work to Hadoop • GUI ETL tool for developers that don’t know Java, Scala, SQL 5 MIGRATING TO HADOOPFOR ETLATENTERPRISE SCALE. Cascading Driven ? ?
- 6. Confidential6 TODAY’S SPEAKERS Shahab Kamal Vice President at BitWise Inc. Shahab is responsible for strategy, growth and client relations. Shahab works with client executives on IT Strategy for Business Intelligence, Big Data, Data Warehousing and Enterprise Applications. Shahab has worked at Ford Motors, Aon Hewitt and Tribune Company on their PeopleSoft ERP implementation and support. His expertise has been around retrofitting data from legacy applications without loss of data integrity. Mark Castillo Driven, Inc. Mark is a Solutions Architect with 15+ years of software engineering background. He has worked in the finance, security, healthcare, streaming music, marketing, and social networking industries. His technical knowledge and skills are focused on distributed systems, data processing, networking, Linux appliances and Big Data.
- 10. Bitwise Migration Solution Approach ~70% Effort Saving~60% Effort Saving Inventory Deep Dive Migration Design Migration Validation ~30% Effort Saving Migration Automation Assessment Automation Test Automation 1 2 3
- 11. Big Data Processing Platform OTHER CUSTOM Local In-Memory MapReduce& Tez COMPUTATION FABRIC CASCADINGEnterprise Data Application BitWise Big Data Processing Platform ETL Migration QualiDI Data Quality Framework ELT Development Development Migration Engine Testing Checks & Balances
- 16. Axes UI
- 17. Axes UI
- 18. Axes UI
- 19. Axes UI
- 20. Axes UI
- 22. Concurrent – Cascading and Driven OTHER CUSTOM Local In-Memory MapReduce & Tez COMPUTATION FABRIC CASCADING Enterprise Data Application
- 23. Bitwise helped a large Fortune 500 company save millions of dollars and an estimated 30-50% time in ETL development through utilization of the Bitwise proprietary ETL migration accelerator, offloading from a costly legacy platform to Hadoop. It began when the client expressed their interest in moving to Hadoop/Big Data by migrating their existing Recovery Ab Initio ETLs. Bitwise came up with a phased approach to Proof, Validate and Convert the existing ETLs. Taking the partnership further, Bitwise proposed a GUI to the ELT tool to act as a developer IDE based on Eclipse as a Next Step. Proofing the Technology Stack Validation of the Bitwise Hadoop ELT Stack ETL Migration using Accelero Conversion Engine Partnership in Accelero Development Engine Partnership in Accelero GUI Development Stage 1 Stage 2 Stage 3 Stage 4 Stage 5
- 24. Bitwise has been working with fortune 500 company to move data from Data lake to Hadoop and identify risks that need to be addressed. The primary focus on developing templates and framework for Data Ingestion and Testing after the data is transferred and build reports on the offloaded data. Prior Bitwise has helped the client with Data Integration migration through utilization of the Bitwise proprietary Data Integration migration excelerator Accelero, offloading from a costly legacy platform to Hadoop, saving 30-50% time in ETL migration. Stage 1 Stage 2 Stage 3 Data Ingestion into Hadoop – Proof of Concept Taking the entire Proof of Concept ahead – data lake moving to Hive Build optimized reports running of the offloaded data on Hadoop Conversion of proprietary ETL to Accelero ELT using Cascading and Driven Stage 4
- 25. Thank You
- 26. Confidential • Bitwise website: http://www.bitwiseglobal.com/ • Driven website: http://www.driven.io/ • Speakers contact information: - Bob Taylor: bobt@driven.io - Shahab Kamel: Shahab.Kamal@bitwiseglobal.com - Mark Catillo: mark@drive.io ADDITIONALRESOURCES