Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Db2 analytics accelerator on ibm integrated analytics system technical overview


Published on

The new generation of the accelerator runs on IBM Integrated Analytics System and on IBM Z processors

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Db2 analytics accelerator on ibm integrated analytics system technical overview

  1. 1. © 2017 IBM Corporation Db2 Analytics Accelerator on IBM Integrated Analytics System Technical Overview November 2017 Daniel Martin –
  2. 2. © 2017 IBM Corporation2 Disclaimer IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
  3. 3. © 2017 IBM Corporation3 Agenda • Analytics Accelerator Version 7 • New architecture • New engine • API Compatibility to previous generations • Deployment on IBM Integrated Analytics System • Sister deployment on IBM Z • Platform Comparison
  4. 4. © 2017 IBM Corporation4 Db2 Analytics Accelerator Version 7.1 Deployment Options In Version 7.1, Db2 acceleration can be implemented within different operating environments: • On an on-premises appliance • On a software appliance installed on the z14 mainframe Both new options will offer • the same functionality • the same API • the same implementation This provides: • Coexistence and combination of deployment options, fully transparent for Db2 applications • Flexibility in moving data for query acceleration as workload demands grow or change • Consistency and efficiency in managing different Db2 Analytics Accelerator environments Next Generation Technology: Two deployment options Current Technology: Appliance
  5. 5. © 2017 IBM Corporation5 Hardware Appliance Uniform experience, simultaneous use, and easy transition between different implementations Common analytics engine across all the platforms: Db2 Warehouse Summary: One API – One implementation – Two deployment options Deployment on IBM Z
  6. 6. © 2017 IBM Corporation6 Db2 Analytics Accelerator Version 7.1: Shared Container
  7. 7. © 2017 IBM Corporation7 Version 7.1 - New Architecture Virtual or physical server CPU Memory Docker Supported OS Storage (local, SAN, NAS) (Clustered) Filesystem Docker container IBM Analytics Engine Authentication Accelerator server Workload Monitoring Systems Manager Additional future functionality Infrastructure Management
  8. 8. © 2017 IBM Corporation8 Powered by Db2 with BLU Acceleration •Huge potential for faster ingest for incremental updates, and thereby less HTAP query delay! •IBM’s premier analytics engine across many products •Latest analytics technology innovations •Improved SQL compatibility and performance •Higher degree of concurrent users and queries •Roadmap: Spark, ML, Notebooks, DSM, … In-memory column processing with dynamic movement of data from storage Multi-core and SIMD parallelism (Single instruction Multiple Data) Patented compression technique preserves order so data can be used without decompressing Skips unnecessary processing of irrelevant data The new engine is replaced internally – external interfaces will stay the same. The same Db2 subsystem can be connected to the existing and new generation.
  9. 9. © 2017 IBM Corporation9 SQL Coverage: Comparing Accelerator V7 with existing V5 Technology Data Type support Accelerator V7 (on IAS or Z) Accelerator V5 (on PDA) EBCDIC MBCS, GRAPHIC Supported natively Converted to UTF 8 TIMESTAMP(12) Supported natively Truncated to precision 6 FOR BIT DATA subtype Supported natively for EBCDIC, UNICODE, ASCII Supported for EBCDIC only DECFLOAT On the Roadmap Not supported BINARY On the Roadmap Not supported
  10. 10. © 2017 IBM Corporation10 SQL Coverage: Comparing Accelerator V7 with existing V5 Technology SQL support Accelerator V7 (on IAS or Z) Accelerator V5 (on PDA) correlated subqueries All types supported including table expressions with sideway references Only a small subset supported Recursive SQL Supported Not supported TIMESTAMP value 24:00:00 Supported natively Mapped to 23:59:59.999999 Scalar functions Improved support Some not supported when using specific datatypes, e.g. MIN/MAX, DAY, LAST_DAY, BIT*, TIMESTAMP_ISO, VARIANCE/STDDEV/… with UNIQUE clause, HEX() function Supported Not supported Mixed Encodings Adding EBCDIC tables when UNICODE tables already added is supported (SQL joins with data in Unicode and EBCDIC not possible) Only supported to add UNICODE tables after EBCDIC table has already been added (required to set AQT_ENABLE_MULTIPLE_ENCODINGS environment variable) CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_DATE Supported with improved accuracy (no longer dependent on time synchronization) Supported Local Date Exit format Not supported Supported
  11. 11. © 2017 IBM Corporation11 Db2 Analytics Accelerator Version 7.1, Deployment on IBM Integrated Analytics System
  12. 12. © 2017 IBM Corporation12 NPS® 8000 Series TwinFin™ with i-Class™ Advanced Analytics NPS® 10000 Series TwinFin™ 2003 2006 2009 2010 PureData System for Analytics N2000 PureData System for Analytics N3000 IBM Integrated Analytics System (PDA & PDOA Convergence) Convergence - Combining the best of Netezza and DB2 Appliances 2012 2014 2017 BCU for AIX BW 7000 BW 7050 BW 7100 BW 7100 ISAS 7600 ISAS 7700 ISAS 7710 PDOA 1.0 PDOA 1.1
  13. 13. © 2017 IBM Corporation13 Db2 Analytics Accelerator for z/OS Version 7.1, deployment on IBM Integrated Analytics System (IIAS) • Next generation hardware appliance • A full solution that provides all components out of the box –including optimized hardware and software • All components provided by IBM in a balanced, performance-optimized configuration • HW, which includes the rack, the physical servers and the storage • SW stack including the Docker host operating system as well as the Docker container and the infrastructure management • IBM Power hardware for the appliance, balanced and optimized for price/performance
  14. 14. © 2017 IBM Corporation14 Hardware Building blocks (Server/Storage) Compute Node  Power8 S822L  Murano Chipset  Two 12-core Sockets @ 3.42 GHz  512 Gigabytes of DDR3 memory  Four dual port 16Gb FibreChannel HBAs  Two Ethernet NICs, each with two 1Gb/s two 10Gb/s ports  Two 600GB SAS hard drives  Flash Storage  IBM FlashSystem 900  Dual Flash Controllers  5.7 Terabyte MicroLatency Flash Storage Modules  Two-Dimensional RAID 5 and spare for extremely high reliability and availability  One, Two, or Three units per rack
  15. 15. © 2017 IBM Corporation15 IBM Integrated Analytics System Configurations M4001-003 1/3 Rack M4001-006 2/3 Rack M4001-010 Full Rack Servers 3 5 7 Cores 72 120 168 Memory 1.5 TB 2.5 TB 3.5 TB User capacity (Assumes 4x compression) 64 TB 128 TB 192 TB IBM Power 8 S822L 24 core server 3.42GHz, IBM FlashSystem 900 Mellanox 10G Ethernet switches
  16. 16. © 2017 IBM Corporation16 Integrated Analytics System (1/3 rack) - Docker container  One container per physical server  Head Node vs Data Node  3 sizes:  1/3 Rack (3 servers)  2/3 Rack (5 servers)  Full Rack (7 servers) Head node (node0101) node0102 node0103  LDAP server and Accelerator server is active on one node only at any given time
  17. 17. © 2017 IBM Corporation17 Scaling up/out with Accelerator on Z versus deployment on IIAS CP CP CP …4-35IFLs… MLN MLN MLN MLN MLN MLN IDAA on Z: One dashDB Node dashDB Head Node MLN MLN MLN CP CP CP dashDB Node 2 MLN MLN MLN MLN CP CP CP dashDB Node 3 MLN MLN MLN MLN CP CP CP …24CPcores… …24CPcores… …24CPcores… IDAA on IIAS: 3 dashDB Nodes Cluster Filesystem
  18. 18. © 2017 IBM Corporation18 Internal Network-Architecture  Fully redundant topology  Active-active technology  Scales to 8 Racks  Each node uses 4x 10GbE ports to form a single bonded interface. (Linux mode-4 LACP bonding) 1 0 fbond Mlnx 2410 Switch “fabsw01a” 1 0 1 0 1 0 node0102 Mlnx 2410 Switch “fabsw01b” 1 0 fbond 1 0 1 0 1 0 node0103 1 0 fbond 1 0 1 0 1 0 node0101 1 0 1 0 1 0 1 0 node0105 fbond 1 0 1 0 1 0 1 0 node0107 fbond1 0 1 0 1 0 1 0 node0106 fbond 2x 40 1 0 fbond 1 0 1 0 1 0 node0104  Fully redundant network using 1 GbE  Active-active dual star topology  Scales to 8 Racks  Allows resilient management and monitoring of all rack components Management Network Mlnx 8831 Switch “mgtsw01a” Mlnx 8831 Switch “mgtsw01b” 1 mbond 1 node0101 Fab Switch 1 mbond 1 node0102 1 mbond 1 node0103 1 mbond 1 node0104 1 mbond 1 node0105 1 mbond 1 node0106 1 mbond 1 node0107 1 1 TMS900 1 1 V5000 1 1 Term Svr1 1 FC Switch1 PDUs 1  Fully redundant 16G FC  Forms a Per Rack SAN and defines a “Fault Domain”  Allows high speed multi-pathed access to all storage elements from all processing nodes within the “Fault Domain” Fibre Channel SAN Data Fabric Network
  19. 19. © 2017 IBM Corporation19 High Availability – Hardware Elements Robust Hardware Elements • Power8 • IBM Flash System 900 • Completely resilient storage subsystem for the appliance • two load sharing power supplies, redundant fans, and two separate storage controllers • RAID5 layout across Flash elements within each module, and then again RAID5 layout across the Flash modules Redundant Hardware Elements • Data Fabric, Management Network and Storage Fibre Channel Network • pairs of switches are used to provide complete failover redundancy • Processing Nodes • Partitions of failed node are reassigned evenly to the surviving nodes within the same rack
  20. 20. © 2017 IBM Corporation20 High Availability – Node Failover / Failback  Processing Nodes – Organized into Highly Available clusters to provide continuous operation in the event of failure of one of the nodes  First attempt to power recycle failed node and make it usable again  If recycle failed, the data partitions of the failed node are reassigned to the surviving nodes within the same rack  The system will experience an outage (up to 30 mins if reboot required) while a failover is performed
  21. 21. © 2017 IBM Corporation21 Encryption of „data at rest“  Addresses risk of stolen flash cards / flash card replacements (e.g. after failures)  Integrated Analytics System provides Encryption through IBM Flash 900 built-in encryption  XTS-AES 256-bit symmetric used built-in  Used by Accelerator on IAS w/o external key management. Flash Controller is key-aware so system not protected in case of FlashSystem lost complete.  Ext. key management leveraging separate key-management-server is on the roadmap Attention: Data Studio does not display the encryption w/o external key management as encrypted disk although there’s no data stored in clear. Starting using external key management, encryption will be detected by Data Studio
  22. 22. © 2017 IBM Corporation22 Db2 Analytics Accelerator Version 7.1, deployment on IBM Z
  23. 23. © 2017 IBM Corporation23 • A software appliance running on IBM Z • Packages the SW stack into an IBM Secure Service Container to deliver a fully self-managed appliance running in a SSC LPAR that can be deployed in minutes • Integrates seamlessly into the customer’s IBM Z environment and leverages known LPAR-, memory and CPU management procedures, including call home support for enterprise hardware components. • Uses customer-provided storage to hold the Accelerator-side data • Scales smoothly with the assignment of available processor cores, initially addressing sizes comparable up to 1/2 rack PDA (N3001-005) appliances Deployment on IBM Z
  24. 24. © 2017 IBM Corporation24 Accelerator on IBM Z – Optimized for Multiple Instances Usage  Accelerator on IBM Z instance = 1 node = 1 LPAR  Single instance typically 8 – 35 IFLs  Single instance typically 0.5 – 8 TB memory  Different workloads in different instances  Optimize instance for workload  Instance size determined by individual workload requirements • not “sum of all”  IFLs may be shared – dev/test environments  Isolation of independent / competing workloads DB 1 Accelerator on IBM Z 1 DB2 z/OS DB2 z/OS DB2 z/OS Setup only symbolic DB 3 Accelerator on IBM Z 3 DB 2 Accelerator on IBM Z 2
  25. 25. © 2017 IBM Corporation25 Key advantages • Out-of-the-box experience • Workload Optimized System • Wide set of Analytics use cases • Proven technology with client references cross-industry • Out-of-the-box experience • Workload Optimized System • Optimized for True HTAP • Evolving set of Analytics use cases • Download & Go experience • Homogeneity within IBM Z: common resources, operation • Evolving set of Analytics use cases Workload Size • Very good scale-out • Very good scale-out • Optimized for very large query throughput and load performance • Good scale-up to full drawer Db2 Analytics Accelerator on Pure Data for Analytics N3001 (Netezza Technology) Db2 Analytics Accelerator on Integrated Analytics System M4001 Db2 Analytics Accelerator on IBM Z Key Characteristics of the deployment options
  26. 26. © 2017 IBM Corporation Thank You