Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Accelerating Data Warehouse Modernization

864 views

Published on

Accelerating Data Warehouse Modernization

Published in: Technology
  • Be the first to comment

Accelerating Data Warehouse Modernization

  1. 1. Accelerating Data Warehouse Modernization Ajay Anand, VP Products, Kyvos Insights Vineet Tyagi, CTO, Impetus
  2. 2. Our 40 Minutes Today • Drivers for Data Warehouse Modernization • What is a Modern Data Warehouse • Challenges for implementing a Modern Data Warehouse • Driving adoption and usage within the enterprise • Measuring success factors and ROI
  3. 3. Data Warehouse Modernization – Drivers Optimize Existing DW/BI Infrastructure of Create New Capabilities Handle Big Data and the 3 V’s • Volume, Variety, Velocity Integrate Multiple Data Silos • ERP, CRM, HRM and others Reduce Cost • ETL process • Analytical process • Mainframe process • Cloud feasibility for data analytics Applying Science • Unstructured data for enhancing analytics • Data Science for advanced analytics Reduce Time to Market by Faster Processing Analytics
  4. 4. Blueprint of a Modern Data Warehouse with Hadoop The enterprise data warehouse (EDW) and Hadoop based warehouse would co-exist to allow the enterprise to leverage the strengths of each architecture. Landing and ingestionStructured Unstructured External Social Machine Geospatial Time Series Streaming Provisioning,Workflow, Monitoring and Security Enterprise Data Lake Real-Time applications Predictive applications Exploration & discovery Enterprise applications Traditional data repositories RDBMS MPP
  5. 5. Key Challenges for Modernization “Through 2018, 90% of modernized warehouses will be useless as they are overwhelmed with information assets captured for uncertain use cases”
  6. 6. Key Challenges for Modernization “Visual data-discovery, an important enabler of end user self-service, will grow 2.5 x faster than the rest of the market, becoming by 2018 a requirement for all enterprises.” Making insights and data in the warehouse readily discoverable, accessible and usable
  7. 7. Key Challenges for Modernization Is the opposite of “Dumb” data • hard to find • hard to understand • hard to combine Data in the Lake has to be Smart Rethink the information plumbing • Supplement first , transform later • Maximize ROI by protecting investments Rethink ETL – Light weight data blending tools that can allow for data wrangling when business cannot wait
  8. 8. Key Challenges for Modernization “By 2017, most business users and analysts in organizations will have access to self-service tools to prepare data for analysis” “Managed BI Self-Service Will Continue to Close the Business and Technology Gap.” Self Service BI over Hadoop
  9. 9. Using big data capabilities as a “landing zone” before determining what data should be moved to the data warehouse PRE-PROCESSING Moving infrequently accessed data from data warehouses into enterprise- grade Hadoop Moving associated workloads to be serviced from Hadoop OFFLOADING Using big data capabilities to explore and discover new high value data from massive amounts of raw data EXPLORATION Top 3 Tactics for Modernization
  10. 10. • Barriers to adoption: Complex, slow, needs expertise Kyvos Solution: Build a BI Consumption Layer on your Data Lake • Enable business users to explore data visually and interactively • No waiting for reports • Self service – no learning curve • No need to move data out of Hadoop • Eliminate scalability restrictions for BI • Drill down to lowest levels of granularity Bridging the Gap for Business User
  11. 11. BI Consumption Layer with OLAP on Hadoop
  12. 12. BI Consumption Layer – Secure, Scalable Access for All Users • Fine-grained access control • Row and column level security • Integration with kerberos, LDAP, Active Directory • Integration with security frameworks • Role based access control • Support for third party encryption tools • Support for single sign-on
  13. 13. Excel Spotfire ICE JAVA APP MDX Clients Other Transformations (Java / Scala) Hive HDFS Jacobian Transformation (Scala) Impala SQL Server/ SSAS Spark Business Need • Evaluate risk across all asset classes • Deliver interactive access at massive scale • Interface with Spotfire and in-house apps • Reduce time to market Challenges • DATA SILOS – Teradata, SQL Server, and HDFS • BIG DATA • Data too large to look at all asset classes across desired time period • 700 M transactions per day • WEEKS – time to get results • SLOW - response time to queries Use Case Investment Bank Risk Analysis
  14. 14. Excel Spotfire ICE JAVA APP MDX Clients Other Transformations (Java / Scala) HDFS Jacobian Transformation (Scala) KYVOS Spark Solution Highlights • One OLAP / caching layer for all three UI’s: Excel, Spotfire, In-house • Consolidated view of all asset classes • Drill down to trade level – never possible before Results Obtained • 20-day trend of risk – not achievable with previous Hive or Impala solutions • Daily updates of cubes • Reduced time to market: eliminated need to move data to SSAS • Interactive response times for users, even at massive scale • No learning curve: support for all business UI’s Use Case Investment Bank Risk Analysis
  15. 15. • Can it deal with the scalability and granularity needed? • How does it perform with “cold” queries for ad-hoc analysis? • How efficiently does it deal with “warm” or repeated queries? • Can business users access data seamlessly with their BI tools? • Can diverse data sets be transformed and combined with no coding? • Can it deal with incremental data updates efficiently? • Can it deal with concurrent access without significant degradation? • Is it enterprise ready to support availability and security requirements? Evaluating Criteria
  16. 16. • Reduction in time to market • Reduction in development time • Increased business user productivity • Reduced latency – reduced number of “hops” or diverse systems supported • Reduced operational costs • Top-line benefits of insights that were not possible before Measuring ROI
  17. 17. Visit us at Booth 1105 ajay.anand@kyvosinsights.com vineet.tyagi@impetus.com Q&A

×