;   The Gap!                                 Pini Cohen                                     EVP                           ...
Agenda•   Major Trends and Issues•   Development and SOA•   ESM BSM CMDB•   DBMS and DATA•   Platforms – Servers•   Client...
Mini Agenda DBMS• NoSQL• DBMS appliances• From RT’s – about DBA organization,  consolidation, etc.• CDC tools• DBA staffin...
The balance is changing                                                                                        Cloud appli...
Big Data• Big Data is a term applied to data sets whose size is beyond the ability of  commonly used software tools to cap...
What do we expect from DBMS?• Availability• Consistency• Scalability                                                 Sourc...
Brewers (CAP) Theorem• It is impossible for a distributed computer  system to simultaneously provide all three of  the fol...
What It Means                                                     Source: Scalebasehttp://guyharrison.squarespace.com/blog...
Dealing With CAP• Drop Partition Tolerance  – Run everything on one machine.  – This is, of course, not very scalable.• Dr...
Dealing With CAP• Drop Consistency  – Welcome to the “Eventually Consistent” term.     • At the end – everything will work...
NoSQL Types• Key/Value    – A big hash table    – Examples: Voldemort, Amazon Dynamo• Big Table    – Big table, column fam...
NO-SQL                                    Source: ScalebasePini Cohen’s work Copyright 2011 @STKI http://browsertoolkit.co...
Pros/Cons• Pros:   –   Performance   –   BigData   –   Most solutions are open source   –   Data is replicated to nodes an...
DBMS market new frontiers:                                        Native Cloud Service         Hadoop     Cassandra       ...
Database.comPini Cohen’s work Copyright 2011 @STKI                                 15Do not remove source or attribution f...
There are some NoSQL projects          out there…                                                                         ...
NoSQL Market Forecast 2011-2015     http://www.marketresearchmedia.com/2010/11/11/nosql-market/      Pini Cohen’s work Cop...
Coming Soon - Oracle Database 11g on Amazon Relational Database Service     Pini Cohen’s work Copyright 2011 @STKI     Do ...
Can we live with NoSQL               limitations?• Facebook has dropped Cassandra• “..we found Cassandras eventual consist...
Meanwhile, back on “Earth”, Growing    Market- special purpose DBMS machines•   Teradata•   Oracle Exadata•   EMC Greenplu...
Information ManagementThe Netezza TwinFin™ Appliance                                 Slice of User Data              Disk ...
Information ManagementAppliance family for data life-cycle management                            Skimmer                  ...
DBMS appliances• First impression from initial testing of DBMS appliances:• Importexport was not trivial. More effort then...
DBA organization• In Oracle centric organizations MSSQL will be in sub-group. In  such organizations MSSQL sub-team will h...
From RT• Users are deploying DBMS in virtual server environment in some  cases• About half of application upgrades will ca...
Automation• Many production error are caused by change  management issues so organizations have  development tools and pro...
DBMS consolidation projects• Upgrade applicationDBMS to the same version and put  DBMS on one DBMS with several instances ...
Local Trends• Data Compression can save up to 30% and  even more.• All DBMS in packagesapplications developed  in outsourc...
DBA organization• DBA’s are divided in larger organizations to several groups:   –   Production DBS   –   Testing and Chan...
Attunity – Microsoft OEM deal     Pini Cohen’s work Copyright 2011 @STKI                                 30     Do not rem...
Attunity CDC Suite for SSIS                             an Operational Data Replication solution       Complete solution f...
DBMS Support Ratios• Number of developers (in the Open) supported by DBA FTE        Per FTE                              #...
Israel Market Positioning – DBMS Open –                             General Usage                                         ...
Market Status and Recommendations• Users are using these integrators (support, maintenance) in DBMS  open area:• Oracle•  ...
STKI’s take on DBMS•   Automation Automation Automation•   Workflow, Self-service•   Focus on change management•   Standar...
Thank you                                                     Pini Cohen Pini Cohen’s work Copyright 2011 @STKI           ...
Upcoming SlideShare
Loading in...5
×

Summit 2011 infra_dbms

685

Published on

DBMS, Data, NoSQL, DBMS consolidation

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
685
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Summit 2011 infra_dbms"

  1. 1. ; The Gap! Pini Cohen EVP pini@stki.infoPini Cohen’s work Copyright 2011 @STKIDo not remove source or attribution from any graphic or portion of graphic
  2. 2. Agenda• Major Trends and Issues• Development and SOA• ESM BSM CMDB• DBMS and DATA• Platforms – Servers• Clients• Storage Source: http://astonguild.org.uk/files/NEW_MENU_FRONT_RGB%5B1%5D.jpg Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  3. 3. Mini Agenda DBMS• NoSQL• DBMS appliances• From RT’s – about DBA organization, consolidation, etc.• CDC tools• DBA staffing ratios• Ratings Pini Cohen’s work Copyright 2011 @STKI 3 Do not remove source or attribution from any graphic or portion of graphic
  4. 4. The balance is changing Cloud application have different needs Infra is betterNeed for more Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  5. 5. Big Data• Big Data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.• Examples include web logs, RFID, sensor networks, social networks, Internet text and documents, Internet search indexing, call detail records, genomics, astronomy, biological research, military surveillance, medical records, photography archives, video archives, and large scale eCommerce. (wikipedia) Source: http://fortunewallstreet.files.wordpress.com/2010/12/matrix.jpg Pini Cohen’s work Copyright 2011 @STKI 5 Do not remove source or attribution from any graphic or portion of graphic
  6. 6. What do we expect from DBMS?• Availability• Consistency• Scalability Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  7. 7. Brewers (CAP) Theorem• It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: – Consistency (all nodes see the same data at the same time) – Availability (node failures do not prevent survivors from continuing to operate) – Partition Tolerance (the system continues to operate despite arbitrary message loss) Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI http://en.wikipedia.org/wiki/CAP_theorem Do not remove source or attribution from any graphic or portion of graphic
  8. 8. What It Means Source: Scalebasehttp://guyharrison.squarespace.com/blog/2010/6/13/consistency-models-in-non-relational-databases.html Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  9. 9. Dealing With CAP• Drop Partition Tolerance – Run everything on one machine. – This is, of course, not very scalable.• Drop Availability – If a partition fail, everything waits until the data is consistent again. – This can be very complex to handle over a large number of nodes. Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  10. 10. Dealing With CAP• Drop Consistency – Welcome to the “Eventually Consistent” term. • At the end – everything will work out just fine - And hi, sometimes this is a good enough solution – When no updates occur for a long period of time, eventually all updates will propagate through the system and all the nodes will be consistent – For a given accepted update and a given node, eventually either the update reaches the node or the node is removed from service – Known as BASE (Basically Available, Soft state, Eventual consistency), as opposed to ACID Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  11. 11. NoSQL Types• Key/Value – A big hash table – Examples: Voldemort, Amazon Dynamo• Big Table – Big table, column families – Examples: Hbase, Cassandra• Document based – Collections of collections – Examples: CouchDB, MongoDB• Graph databases – Based on graph theory – Examples: Neo4J• Each solves a different problem Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  12. 12. NO-SQL Source: ScalebasePini Cohen’s work Copyright 2011 @STKI http://browsertoolkit.com/fault-tolerance.pngDo not remove source or attribution from any graphic or portion of graphic
  13. 13. Pros/Cons• Pros: – Performance – BigData – Most solutions are open source – Data is replicated to nodes and is therefore fault-tolerant (partitioning) – Dont require a schema – Can scale up and down• Cons: – Code change – No framework support – Not ACID – Eco system (BI, Backup) – There is always a database at the backend – Some API is just too simple Source: Scalebase Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  14. 14. DBMS market new frontiers: Native Cloud Service Hadoop Cassandra Database.com Xeround Redis Voldemort Amazon Simple DB FathomDB VoltDB MySQL, Memcached Cluster Ed Amazon RDS NoSQL Cloud Enabled SQL MySQL, PostGress, Microsoft SQL Azure XMLDB Gemstone Current Object DB Clustrix IT Microsoft Oracle SQL Server DB2 Traditional Pini Cohen’s work Copyright 2011 @STKI Source: http://xeround.com/14 Do not remove source or attribution from any graphic or portion of graphic
  15. 15. Database.comPini Cohen’s work Copyright 2011 @STKI 15Do not remove source or attribution from any graphic or portion of graphic
  16. 16. There are some NoSQL projects out there… Source: NoSQL Databases: Providing Extreme Scale and Flexibility By Matthew D. Sarrel Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  17. 17. NoSQL Market Forecast 2011-2015 http://www.marketresearchmedia.com/2010/11/11/nosql-market/ Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  18. 18. Coming Soon - Oracle Database 11g on Amazon Relational Database Service Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  19. 19. Can we live with NoSQL limitations?• Facebook has dropped Cassandra• “..we found Cassandras eventual consistency model to be a difficult pattern to reconcile for our new Messages infrastructure”• Facebook has selected HBase (Columnar DBMS) . http://www.facebook.com/notes/facebook-engineering/the-underlying-technology-of-messages/454991608919 Pini Cohen’s work Copyright 2011 @STKI 19 Do not remove source or attribution from any graphic or portion of graphic
  20. 20. Meanwhile, back on “Earth”, Growing Market- special purpose DBMS machines• Teradata• Oracle Exadata• EMC Greenplum• HP• Microsoft- Fast Track and Parallel Data Warehouse Pini Cohen’s work Copyright 2011 @STKI 20 Do not remove source or attribution from any graphic or portion of graphic
  21. 21. Information ManagementThe Netezza TwinFin™ Appliance Slice of User Data Disk Enclosures Swap and Mirror partitions High speed data streaming SQL Compiler Query Plan SMP Hosts Optimize Admin S-Blades™ (with FPGA-based Processor & Database Accelerator) streaming DB logic High-performance database engine streaming joins, aggregations, sorts, etc.21 © 2010 IBM Corporation
  22. 22. Information ManagementAppliance family for data life-cycle management Skimmer TwinFin Cruiser Dev & Test System Data Warehouse Queryable Archiving High Performance Analytics Back-up / DR 1 TB to 10 TB 1 TB to 1.5 PB 100 TB to 10 PB22 © 2010 IBM Corporation
  23. 23. DBMS appliances• First impression from initial testing of DBMS appliances:• Importexport was not trivial. More effort then expected.• Unprecedented performance boost. Examples (empty machine): – From 7-8 hours to 20 seconds – From 3 hours to 1 hour• With heavy loaded machine performance boost is lower• Heavy IO load gets the most performance boost Pini Cohen’s work Copyright 2011 @STKI 23 Do not remove source or attribution from any graphic or portion of graphic
  24. 24. DBA organization• In Oracle centric organizations MSSQL will be in sub-group. In such organizations MSSQL sub-team will have less resources that might cause to SLA issues• DBMS departments suffer constantly from shortage of personal resource. In many case they will agree to support new project with extra personal (since it Capex and not Opex ..) but will use the help in “old” applications• Oracle RAC is superior technology but require more effort and skills and in some cases (bigger –complicated DBMS) might even lead to downtime issues. Users are hoping that Exadata will help with RAC stability. Pini Cohen’s work Copyright 2011 @STKI 24 Do not remove source or attribution from any graphic or portion of graphic
  25. 25. From RT• Users are deploying DBMS in virtual server environment in some cases• About half of application upgrades will cause some serious work from the DBA part (performance, application issues, etc.)• Users indicated the data compression tools and procedures (in the DBMS level) has saved about 30% of space• There is no “Black Box DBMS” in every productproject installed in the organization the DBMS has to “put his hands on”• Users believe that the market is before big changes – technologies and players Pini Cohen’s work Copyright 2011 @STKI 25 Do not remove source or attribution from any graphic or portion of graphic
  26. 26. Automation• Many production error are caused by change management issues so organizations have development tools and procedures for change management: – Moving DBMS changes from one environment to the other with labels that are related to the SCM tool – What are the differences between two DBMS – Automatic partition management tools – Even RAC management tools – DBA’s should not type directly to the DMBS/ They should uses just scripts that where tested in all environments Pini Cohen’s work Copyright 2011 @STKI 26 Do not remove source or attribution from any graphic or portion of graphic
  27. 27. DBMS consolidation projects• Upgrade applicationDBMS to the same version and put DBMS on one DBMS with several instances (up to 100 applications)• Benefits: – From no on do upgrades, patches on one DBMS – Saving licenses• Issues: – All application should have the same upgrade cycle (from DBMS perspective) and all should agree on down time slots – Some technical issues (all application should have the same “init parms”) Pini Cohen’s work Copyright 2011 @STKI 27 Do not remove source or attribution from any graphic or portion of graphic
  28. 28. Local Trends• Data Compression can save up to 30% and even more.• All DBMS in packagesapplications developed in outsourcing needs intervention from the local DBA team. “Black Box DBMS” model is not working. Pini Cohen’s work Copyright 2011 @STKI 28 Do not remove source or attribution from any graphic or portion of graphic
  29. 29. DBA organization• DBA’s are divided in larger organizations to several groups: – Production DBS – Testing and Change Management – Application – ADBA – In larger organizations the production has sub-team for performance• Example of DBA organization: Non-Prod 20% Prod Source: STKI ADBA 55% 25% Pini Cohen’s work Copyright 2011 @STKI 29 Do not remove source or attribution from any graphic or portion of graphic
  30. 30. Attunity – Microsoft OEM deal Pini Cohen’s work Copyright 2011 @STKI 30 Do not remove source or attribution from any graphic or portion of graphic
  31. 31. Attunity CDC Suite for SSIS an Operational Data Replication solution Complete solution for real-time and efficient heterogeneous data replication and integration, integrated with SQL Server. 1. Low impact on data server using log-based CDC technology 2. Efficiency and reduced resources, processing only incremental changes 3. Fully integrated with SSIS (2000, 2005) for ease of use 4. Monitoring and control console to manage replication processes 5. Capitalize on existing investments in SQL Server licenses1. SQL Server2. Oracle3. DB2/4004. DB2 on z/OS5. NonStop SQL/MP Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  32. 32. DBMS Support Ratios• Number of developers (in the Open) supported by DBA FTE Per FTE # of Applications New 25 percentile 11 Median 19 75 percentile 28 Source: STKI Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  33. 33. Israel Market Positioning – DBMS Open – General Usage Oracle MicrosoftLocal Support IBM This analysis should be used with its supporting documents Israeli Market Presence Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  34. 34. Market Status and Recommendations• Users are using these integrators (support, maintenance) in DBMS open area:• Oracle• Microsoft• Veracity Many smallgood integrators in this area• Matrix• Valinor• Priority• IBM• Inspire many other exist Pini Cohen’s work Copyright 2011 @STKI Do not remove source or attribution from any graphic or portion of graphic
  35. 35. STKI’s take on DBMS• Automation Automation Automation• Workflow, Self-service• Focus on change management• Standardization• Watch for NoSQL development• DBMS Consolidation Pini Cohen’s work Copyright 2011 @STKI 35 Do not remove source or attribution from any graphic or portion of graphic
  36. 36. Thank you Pini Cohen Pini Cohen’s work Copyright 2011 @STKI 36 Do not remove source or attribution from any graphic or portion of graphic

×