Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling Real-Time Analytics with Attunity

2,207 views

Published on

Today’s reporting and analytics expectations are more demanding than ever. However, IT resources and budgets are not growing nearly as rapidly. In spite of those challenges, IT and BI pros are faced with the task of accelerating data warehouse development, data connectivity, and data transfers across on-premises and cloud platforms to support real-time and just-in-time BI/analytics cost-effectively. In this session, we will explore current data warehouse trends in integration/CDC, automation, design patterns and technologies for hybrid BI infrastructures to/from SQL Server, Azure and beyond.

Attendees will learn about:

• Hybrid data warehousing architecture
• Data warehouse development automation
• Just-in-time and real-time BI/analytics technologies
• Real-world user success stories
• Data integration tips and best practices to save time, $ and sanity

Published in: Data & Analytics
  • Be the first to comment

Enabling Real-Time Analytics with Attunity

  1. 1. Enabling Real-time Analytics Jen Underwood, Founder, Impact Analytics Kevin Petrie, Senior Director, & Reza Khan, Director of Product Mgmt, Attunity Moderated By: Paresh Motiwala CDC and Data Warehouse Automation
  2. 2. Technical Assistance If you require assistance during the session, type your inquiry into the question pane on the right side. Maximize your screen with the zoom button on the top of the presentation window. Please fill in the short evaluation following the session. It will appear in your web browser
  3. 3. 3 Thank You Empower users with new insights through familiar tools while balancing the need for IT to monitor and manage user created content. Deliver access to all data types across structured and unstructured sources. attunity.com Attunity, a leader in data integration and management software, helps move, transform and analyze data efficiently in SQL Server/Azure environments. idera.commicrosoft.com IDERA’s award-winning SQL Server database solutions and multi-platform database, application and cloud monitoring tools ensure your business never slows down.
  4. 4. Access to online training and content Enjoy discounted event rates Join Local Chapters and Virtual Chapters Get advance notice of member exclusives PASS is a not-for-profit organization which offers year-round learning opportunities to data professionals Membership is free, join today at www.sqlpass.org JOIN PASS
  5. 5. Save on PASS Summit 2016 Registration! • The world’s largest gathering of SQL Server & BI professionals • Learn from the world’s top data experts, in over 190 technical sessions • More than 4000 attendees from all over the world • Meet the Microsoft engineering team! Save $200 right now using discount code 24HOP200! $2,195 until September 18, 2016 www.passsummit.com
  6. 6. BIO Jen Underwood • Founder Impact Analytix, LLC • 20 “hands-on” data warehousing, reporting, and predictive analytics • Former Microsoft Principal Program Manager, SQL Server BI www.jenunderwood.com @idigdata /in/idigdata
  7. 7. BIO Kevin Petrie • Senior Director at Attunity • 20 years experience in tech, including marketing, big data services, strategy, and journalism • Frequent speaker and blogger • Writes for various technology publications • Bookworm, outdoor fitness nut, husband, and father of three boys @KevinPetrieTech /in/kpetrie www.attunity.com
  8. 8. BIO Reza Khan • Director of Product Management at Attunity • 20 years of experience in technology and technical support • Extensive expertise in cloud and hybrid solutions • Frequent speaker @RezaK929 /in/rezak929 www.attunity.com
  9. 9. Enabling Real-time Analytics CDC and Data Warehouse Automation Jen Underwood, Founder, Impact Analytics Kevin Petrie, Senior Director, & Reza Khan, Director of Product Mgmt, Attunity
  10. 10. Agenda • Current data warehouse integration trends • Hybrid BI design patterns and technologies • Data warehouse development automation • Just-in-time and real-time BI/analytics technologies • Real-world user success stories • Data integration tips and best practices
  11. 11. Enterprise challenges today
  12. 12. “It took one year to deliver a change request… when it was finally released, it was no longer required.” - DW manager at Fortune 500 insurer
  13. 13. Industry trends Historically ~2% of BI apps in cloud Cloud growing over ~30% YoY As more apps move to cloud, more data gravity challenges for BI leaders
  14. 14. According to IDC, Amazon had over 237% YoY growth Microsoft states Azure had over 120% YoY growth Industry trends Source: Indeed Job Trends August 2016
  15. 15. DBAs/developers need to adjust to growing hybrid reporting environments Heterogeneous + Hybrid data Industry trends Source: Tableau June 2016
  16. 16. Source: Dave McCrory http://blog.mccrory.me/2010/12/07/data-gravity-in-the-clouds/ Hybrid data gravity challenges
  17. 17. Laws regarding data security and storage location a critical factor Even with fast networks and caching, distance, bandwidth and latency = HUGE challenges Need to test querying hybrid data in a timely manner, without timeouts and delays Analytics needs to defy data gravity Where is your data in the cloud?
  18. 18. Work across Cloud environments Target DB On EC2 Redshift S3 Proprietary Data Warehouse Online File Storage Online RDBMS service( i.e. database and servers) Any customer installed RDBMS on Elastic Cloud Compute Server (EC2) RDS EMR Managed Hadoop (Hortonworks) SQL DW Target DB On VM SQL DB Cloud SQL Target DB On Compute MySQL database as a service
  19. 19. Heterogeneous hybrid data sources RDBMS Oracle SQL Server DB2 LUW DB2 iSeries DB2 z/OS MySQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Actian Vector Actian Matrix Hortonworks Cloudera MapR Pivotal Hadoop IMS/DB SQL M/P Enscribe RMS VSAM Legacy AWS RDS Salesforce Cloud RDBMS Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Pivotal DB (Greenplum) Pivotal HAWQ Actian Vector Actian Matrix Sybase IQ Hortonworks Cloudera MapR Pivotal Hadoop MongoDB NoSQL AWS RDS/Redshift/EC2 Google Cloud SQL Google Cloud Dataproc Azure SQL Data Warehouse Azure SQL Database Cloud Kafka Message Broker targets sources
  20. 20. Modern Logical Architecture Data Virtualization and/or Logical Data Warehouse Visual Data Discovery Tools Traditional BI Reporting Data Science and Analytics Excel Intelligence Data Warehouse Data Lake NoSQL Relational Databases In-Memory Databases Cluster Compute APIs and Streaming Data Ingestion SaaS Apps, Line of Business Apps, IoT and Public Data Sources APIs and Streaming Queries CDC/Replication ETL/ELT APIs and Queries
  21. 21. Workflow Management and Monitoring Data warehouse automation Landing Staging Warehouse Marts Discover, Import or Create Model Automate Mapping and ETL Complex Transformations Change Propagation Automate Design with Best Practices p r e - b u i l t d e s i g n p a t t e r n s
  22. 22. Data warehouse automation Model-driven Data Warehouse Automation • Eliminates manual coding of ETL • Integrated Change- Data-Capture (CDC) from source systems • Enables agile development approach • Simple to install, implement and manage Data Model DesignData Loader Real-time CDC StagingLanding Data Warehouse Automated DDL | ETL Data Transformation | Population | Management Data Marts Import | Create | Modify
  23. 23. “Comparing prior EDW efforts is like comparing apples to oranges. We went from taking between 10-14 days to build a data mart down to just a single day.” - Data Architect, Leading national furniture manufacturer
  24. 24. Demo
  25. 25. Agile automated integration Centralized control • Any to any mobility • On premises, in cloud or hybrid • Microsoft Azure SQL DB and DW Wide range of sources and targets • All major DB, EDW, Hadoop, file and cloud platforms High speed data transfer • Data compression • Parallel/concurrent data transfer • Configurable batch sizes
  26. 26. Agile automated integration No manual coding or scripting Automated end-to-end Optimized and configurable
  27. 27. Replication architecture
  28. 28. Enterprise-class change data capture Flexible and optimized CDC options • Transactions applied in real-time and in order • Changes applied in optimized batches • Integration with data warehouse native loaders to ingest and merge • Message encoded streaming of changes (for Kafka message broker)
  29. 29. Optimized for Cloud data transfer High speed data transfer across data centers and Cloud • Support for Amazon Web Services, Microsoft Azure and Google Cloud • Uses compression to reduce data and remove sparse records • Separates data in parallel streams for optimal transport over network File Channel File Channel Replication server Replication server Compress – Parallelize - Encrypt Merge- Decompress
  30. 30. Demo
  31. 31. Order history Name SSN Date Philip Wenger cm61ba906fd 2/28/2005 Denny Usher ox7ff654ae6d 3/18/2005 Alicia Hodge i2y36cg776rg 4/10/2005 Alta Levy nx290pldo90l 4/27/2005 Dionne Hardin ypo85ba616rj 5/12/2005 Kristy Flowers bns51ra806fd 5/22/2005 Order history Name SSN Date Philip Wenger cm61ba906fd 2/28/2005 Denny Usher ox7ff654ae6d 3/18/2005 Alicia Hodge i2y36cg776rg 4/10/2005 Alta Levy nx290pldo90l 4/27/2005 Customer data Product data Order History Stretch to cloud Query Microsoft Azure  Denny Usher ox7ff654ae6d 3/18/2005 Query across realms SQL Server 2016 • Stretch cold tables from on-premises SQL Server to Azure cloud database • Online cold data • Query Remote • No app changes • Always encrypted • Row level security
  32. 32. Query across realms SQL Server PolyBase • Queries against relational data and ‘semi-structured’ data in HDFS or Azure • Use existing T-SQL skills and BI tools to gain insights from different types of data
  33. 33. Query across realms Elastic Queries • Transact-SQL query that spans multiple databases • Use existing T-SQL skills and BI tools
  34. 34. Data streaming into Kafka • Broaden your Big Data ecosystem • Move high data volumes from many sources to one or more Big Data targets through Kafka message brokers • Enable massive data consolidation: • Transactional systems + external data sources => new use cases IoT, real time analytics, etc.
  35. 35. CDC Data Streaming into Kafka MSG n 2 1 MSG MSG Data Streaming Transaction logs In memory optimized metadata management and data transport JSON data format Bulk Load MSG n 2 1 MSG MSG Data Streaming JSON data format Message broker Batch data Message broker
  36. 36. Data integration tips and best practices Best Practices – build it right • Leverage proven methodologies and design patterns • Model driven or data driven: Inmon, Kimball, Vault Agile – rapidly iterate and adapt your DW • Align business and IT to deliver better quality • Deliver value in iterations faster to accommodate changes Automate Workflow End to End • Orchestrate all EDW, data marts ETL tasks as single process • Populate multiple data marts in parallel • Monitor, control tasks and notify users of events
  37. 37. Data integration tips and best practices Ensure Data Integrity for Accurate Reporting • Profile data and remediate issues • Validate and repair data before loading Improve data quality • Configure, enforce rules before loading • Automatically discover issues – values, formats, data ranges, duplicates • Define and divert exceptions and remediation tasks Trace Lineage and Analyze Change Impact
  38. 38. Data economics – the bottom line Cost – Value + Use data more efficiently Reduce unused data burden Convert more data to insight, faster $$ $
  39. 39. Hybrid reporting empowers the business to make smarter decisions and generate more value from all of your data Leverage automation to expedite agile data warehouse development For real-time analytics, defy data gravity for by securely replicating, connecting or remotely querying data Unlock the value of data in real-time
  40. 40. Enabling Real-time Analytics What your peers are achieving with automation Kevin Petrie, Senior Director, Attunity
  41. 41. Use Cases We Help Enterprises Address •Data Ingest / Streaming Ingest with Kafka •EDW Offload and Optimization with Hadoop •Hadoop Data Usage and Workload Analytics Hadoop & Big Data •Real-time Data Warehousing with CDC •Data Warehouse Automation •Performance Management and Optimization Data Warehousing •Enterprise Data Replication •Query Offload for Live Reporting •Zero Downtime Migration Database •Enable Analytics in the Cloud (DW, Hadoop) •Migrate to the Cloud (AWS, Azure, Google) •Hybrid Cloud Infrastructure and Applications Cloud
  42. 42. In the real world: Tangerine Bank Leading Retail Bank In Canada Legacy Oracle BI/DW, CRM, WMS APS Results • Accelerated analytical insights • Reduced data • Lower latency • Cut cost
  43. 43. Results • 80% reduction in implementation costs • 95% faster to generate ETL code • 1,000s of lines of commands • 75% quicker from design to production • 3 months instead of 12 months • 500% increase in agility • Implements changes 12 times a year instead of 2 per year • 40% fewer skilled resources needed In the real world: International Insurance Company Accelerates Complex Transformations “We had initially planned on 45 days of ETL coding. With automation we had it completed in 2 days. Senior Information Management Analyst
  44. 44. In the real world: Fortune 100 auto manufacturer 4500 applications DB2 MF SQL Oracle Automated data lake consolidation Results • Consolidating on Hadoop Data Lake with Kafka data brokers • Integrating all sources and targets, minimizing labor and cost
  45. 45. “ Attunity’s intuitive user interface eliminated development, consulting, and professional services costs, resulting in an estimated annual savings of nearly $60,000. Jeremy Kayne, COO DomainPower Attunity Replicate
  46. 46. “ Without Attunity, we could not have done the project since it would have been too costly for us in terms of development work. We would have been forced to give up on our real-time data propagation requirement.Christian Phan-Trong, Architecture Director Swiss Life France Attunity Replicate
  47. 47. “ By using near real-time data from Attunity [Replicate] together with our dashboard tool, we were able to see ten years of data from our ERP system for the first time – all in one chart. Our business teams were excited about the possibilities, once they realized that they could look at data in ways they’d never seen before. Our customers have also been thrilled about gaining better insight into inventory and delivery information.Roland Sutherland, Information Systems Manager SMART Modular Technologies Attunity Replicate
  48. 48. Demo With Reza Khan, Director Product Mgmt, Attunity
  49. 49. Questions?
  50. 50. Coming up next! Indexing for Beginners Kathi Kellenberger
  51. 51. Thank You for Attending Follow @pass24hop Share your thoughts with #pass24hop & #sqlpass

×