Lessons learned from over 25 Data Virtualization implementations
•
0 likes•86 views
Watch full webinar here: https://bit.ly/327fzQ0 If you have been part of the Denodo community for a while, you likely have heard of SimplicityBI (now part of the BDO family). This expert firm located in Canada has done over 25 implementations of Denodo across North America in over 10 different industries.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lessons learned from over 25 Data Virtualization implementations
Similar to Lessons learned from over 25 Data Virtualization implementations (20)
More from Denodo
More from Denodo (20)
Recently uploaded
Recently uploaded (20)
Lessons learned from over 25 Data Virtualization implementations
- 1. Logical Data Fabric: The Future of Data Management and Analytics
- 2. #DenodoDataFest Lessons Learned From Over 25 Data Virtualization Implementations Partner, Technology Consulting Vincent L'Archeveque Manohar Vajpey Partner, Technology Consulting
- 4. WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US CANADA Calgary Vancouver Ottawa Montreal Quebec UNITED STATES San Francisco San Diego Denver Hartford Princeton & Newark Fort Lauderdale Based in Calgary, Alberta Focus on Data Management Data Virtualization Experts End-to-end Data Solutions CLIENT LOCATIONS
- 5. + TOGETHER, we meet the evolving needs of clients and create opportunity Leader in data management Providing end-to-end solutions WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 6. KEY TECHNOLOGY PARTNERS WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 7. INDUSTRIES WE SERVE Oil and Gas Energy and Utilities Financial Services Staffing and Recruiting Data Center Management Government Insurance Social Media Telco Transportation E-Commerce and Food Healthcare WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 8. System or Cloud Migration Future Proof (Abstraction ) System Integration Virtual MDM Security Layer Data Lake Access Real Time Access Data Service Provisioning Curated Enterprise Views Self-Serve DS/BI/Analy tics Logical Semantic OUR CLIENTS USE CASES WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US General Data Provisioning Virtual DM
- 10. STRATEGIC LESSON This is not the end of Data Warehouses & Data Lakes 1 STRATEGIC LESSON No, it is not your last Data Warehouse or Data Lake 2 Complement to other data integration styles
- 11. WHERE ARE YOU? WHAT IS THE NEXT EVOLUTION? This is a Second Major Cycle of Analytical Consolidation 1990s EDW Operational Application Operational Application Operational Application Data Warehouse 2010s Operational Application Operational Application Operational Application IoT Data Other NewData LDW Data Warehouse ODS Marts Data Lake Staging/ Ingest 1980s Pre EDW Operational Application Operational Application Operational Application Cube Cube ? 2000s Post EDW Operational Application Operational Application Operational Application IoT Data Other NewData Data Warehouse Data Lake ? Fragmented/Non-existent Analysis • Multiple sources • Multiple structured sources Unified Analysis • Consolidated data • “Collect the data” • Single server, multiple nodes • More analysis than any one server can provide Fragmented/Non-existent Analysis • “Collect the data” (into different repositories) • New data types, processing, requirements • Uncoordinated views Unified Analysis • Logically consolidated view of all data • “Connect and collect” • Multiple servers, of multiple nodes • More analysis than any one system can provide WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 12. WHERE ARE YOU? WHAT IS THE NEXT EVOLUTION? This is a Second Major Cycle of Analytical Consolidation 1980s Pre EDW 1990s 2000s 2010s EDW Post EDW Operational Application Operational Application Operational Application Cube Cube Operational Application Operational Application Operational Application Data Warehouse Operational Application Operational Application Operational Application IoT Data Other NewData Data Warehouse Data Lake Operational Application Operational Application Operational Application IoT Data Other NewData ? ? Fragmented/Non-existent Analysis • Multiple sources • Multiple structured sources Unified Analysis • Consolidated data • “Collect the data” • Single server, multiple nodes • More analysis than any one server can provide Fragmented/Non-existent Analysis • “Collect the data” (into different repositories) • New data types, processing, requirements • Uncoordinated views Unified Analysis • Logically consolidated view of all data • “Connect and collect” • Multiple servers, of multiple nodes • More analysis than any one system can provide Where are you? WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Data Warehouse ODS Marts Staging/ Ingest Data Lake Data Virtualization LDW
- 13. STRATEGIC LESSON Get consensus on the value of abstraction for your organization 3
- 14. TRADITIONAL POSITIONING OF DENODO Structured Data Unstructured Data Application Data Ingestion Storage Compute Data Access Data Insight • RDBMS • Excel • Flat files • JSON • XML • Email • RFID • Sensors (IIIoT) • Wearables • Social media • LOG • Batch • CDC • Real Time/Data Streaming • Data Warehouse • RDBMS • noSQL • Big Data Lakes Metadata Enrichment Search and Index Data Virtualization Data Services Dashboards Reporting Data Discovery and Self-Service Data Mining Governance and Metadata Management Security DATA VIRTUALIZATION WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 15. STRATEGIC ABSTRACTION In our opinion, the single best benefit of adding Denodo to your data landscape is the power of abstraction that it brings. Abstraction by Denodo fully abstracts your data consumption from your data storage. Abstraction is an architecture decision Abstraction will • Future proof your architecture • Make you agile and faster • Always be a relevant feature of your architecture The need for abstraction can be forgotten or less obvious when your users design a single solution WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 16. STRATEGIC LESSON Evaluate the right use case for Denodo 4
- 17. CHOOSE THE RIGHT USE CASE Denodo is more than just a Data Hub, possibilities are above and beyond data integration. Possibilities need to be shared with the internal dev community. 1 2 3 Carefully validate use cases to put on Denodo, the enthusiasm for the platform can get out of control Denodo is a Swiss Knife, it can play a role in most solutions but it shouldn’t play every role Learning: Educating a Denodo developer requires some time to “Think Virtual” For best performance: new Denodo developers need to go through an onboarding process to understand the platform WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 18. STRATEGIC LESSON Pay attention to Governance 5
- 19. PAY ATTENTION TO GOVERNANCE Denodo can become the place to understand and monitor enterprise data flows 02 • Who is using this data? • How often is this data being used? Denodo can gradually become the entry point into all your data assets. 01 Denodo can become the best place to centralize your security policy implementation 04 • Where else will you maintain security across multiple sources that need to be integrated? Data pipelines seem to be getting longer and longer. Denodo can become the best place to maintain an abstraction while the data pipelines are re-engineered and shortened. 0 3 WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US
- 20. STRATEGIC LESSON Not all Virtualization is Equal 6
- 21. STRATEGIC LESSON Involve Both IT and Business In The Evaluation Of Each Case 7
- 22. STRATEGIC LESSON Build An Internal Data Virtualization Community 6
- 23. STRATEGIC LESSON Consider Data Security Concerns Upfront 7
- 25. IMPLEMENTATION LESSON Denodo as your scalable query orchestrator and not as a compute enginestorage 1
- 26. IMPLEMENTATION LESSON Understand & respect the optimizer…and turn it on! 2
- 27. UNDERSTAND AND RESPECT THE OPTIMIZER WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Query push down is your primary goal, understanding it Activate all key features of the optimizer including Data Movement Minimize “black boxes” at the source (views, stored procedures, DB Links) Structure your solution with your sources in mind Remember Temporary Data Movement Option 1? Option 2? group by join create temp table group by join Sales Custome r 300M 2M Naïve Strategy Sales Custome r Temp_Customer 2M Temporary Data Movement 50 Statistics, statistics, statistics!
- 28. UNDERSTAND AND RESPECT THE OPTIMIZER WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Query push down is your primary goal, understanding it Activate all key features of the optimizer including Data Movement Minimize “black boxes” at the source (views, stored procedures, DB Links) Structure your solution with your sources in mind
- 29. IMPLEMENTATION LESSON Work closely with the source owners and admins 3
- 30. COLLABORATE WITH THE SOURCE ADMINS WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Get the buy in to use their compute Get the buy in to use their storage Work with them on alternative strategy for compute and storage
- 31. IMPLEMENTATION LESSON Have a Gate Keeper 4
- 32. HAVE A GATE KEEPER WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US We recommend every client to have a "Release Manager" or "Librarian". That person or team is responsible for the health of the platform. They monitor developer's work, supervise promotions of and block unhealthy code Why is this required? Ensures the quality of metadata transitioning between environments is high A “Librarian” will be the gatekeeper of the environment and will prevent any code that does not respect your best practices from being promoted to another environment
- 34. OUR NEW WORLD WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US US WEST Azure Data Lake CANADA CENTRAL ON-PREM
- 35. LESSONS LEARNED LATENCY MATTERS WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Place the Denodo servers close to where the data is gravitating towards 01 02 One Denodo server in each cloud One Denodo server on-prem Multiple Denodo servers can often be used to integrate data closer to the stores
- 36. SELF-SERVING LESSON Start on the right foot with expert support
- 37. HOW TO BE SUCCESSFUL WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Start with a single project That has: • Disparate data • Value in speed • Value in abstraction • Unclear roadmap Learn 🡪 Adjust 🡪 Repeat Get expert support upfront
- 38. FINAL LESSON Remember why you are doing this
- 39. REMEMBER WHY YOU ARE DOING THIS WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US 1 SPEED OF DELIVERY Virtual first, physical later 2 ABSTRACTION CONNECTIVITY
- 40. THE TEAM WHO WE ARE STRATEGIC LESSONS IMPLEMENTATION LESSONS ADDITIONAL LESSONS CONTACT US Vincent L’Archeveque Partner, Technology Consulting Mano Vajpey Partner, Technology Consulting Arif Rajwani Partner, Technology Consulting
- 41. +
- 42. © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies. Thank You!