HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg

1,490 views

Published on

Learn the evolution and consolidation of Bloomberg's core infrastructure around fewer, faster, and simpler systems, and the role HBase plays within that effort. You'll also hear about HBase modifications to accommodate the "medium data" use case and get a preview of what's to come.

Published in: Software
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,490
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Welcome everyone! Today is going to be fantastic and we have quite the agenda for you. In the interest of time, lets dive right in.
  • Welcome everyone! Today is going to be fantastic and we have quite the agenda for you. In the interest of time, lets dive right in.
  • HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg

    1. 1. 1 hbasecon.com
    2. 2. HBASEATBLOOMBERG// THE EVOLUTION OF BLOOMBERG DATA SYSTEMSMEDIUM DATA NEEDS FOR THE FINANCIAL INDUSTRY MAY // 07 // 2015
    3. 3. HBASEATBLOOMBERG// BLOOMBERG 3 Leading Data and Analytics provider to the financial industry
    4. 4. HBASEATBLOOMBERG// DATA IS OUR BUSINESS 4
    5. 5. HBASEATBLOOMBERG// September 28: Full Workshop at Bloomberg September 30: Showcase at Strata Hadoop Call for papers at: bloomberglabs.com/data-science DATA SCIENCE FOR SOCIAL GOOD: GOVERNMENT INNOVATION, PUBLIC HEALTH, ENVIRONMENT, EDUCATION
    6. 6. HBASEATBLOOMBERG// 6 • We have a “medium data” problem… • Speed and availability are paramount • Hundreds of thousands of users with expensive requests We’ve built many systems to address DATA MANAGEMENT TODAY
    7. 7. HBASEATBLOOMBERG// DATA MANAGEMENT CHALLENGES 7 • Single security analytics on Big Iron • Replication of Systems and Data • Complexity kills Top 500 Supercomputer list, 2013 >96% Linux. 100% of top 40.
    8. 8. HBASEATBLOOMBERG// DATA MANAGEMENT TOMORROW 8 • Simplicity and performance • Benefit from external developments • Retain our independence • Details matter
    9. 9. HBASEATBLOOMBERG// THE PREMISE 9 • Can apply big data techniques to our medium data problem, by addressing gaps in existing open systems • HBase is a good bet • Part of a broader whole • The Biggest community wins
    10. 10. HBASEATBLOOMBERG// CHALLENGES Our requirements from HBase are: • Read performance – fast with low variability • High availability • Operational simplicity • Efficient use of good hardware • Expressive power Bloomberg has been investing in all these aspects of HBase
    11. 11. HBASEATBLOOMBERG// WE’VE MADE THAT BET 11
    12. 12. HBASEATBLOOMBERG// WE’RE NOT THE ONLY ONES 12 Google Cloud Bigtable
    13. 13. HBASEATBLOOMBERG// AIMING HIGHER We can make things better by working together Let’s be the gold standard
    14. 14. HBASEATBLOOMBERG// 14
    15. 15. >>>>>>>>>>>>>> CALL TO ACTION
    16. 16. HBASEATBLOOMBERG// FURTHER BOLSTER RELIABILITY 16 Great strides such as HBASE-10070 but more to do • Improved reconciliation of state between Master, META and ZK • More determinism in Admin/Master operations
    17. 17. HBASEATBLOOMBERG// BENEFIT FROM MODERN HARDWARE 17 • 32 cores - 256GB RAM – SSD - untapped potential • CPU load max 20% , inadequate throughput • Multi-RS administratively painful • Much better story with memory
    18. 18. HBASEATBLOOMBERG// IMPROVE MULTI-TENANCY 18 • Mixed workloads challenging • interactive vs batch • read vs write • different read access patterns • Many solutions in progress • Administrative simplicity is key
    19. 19. HBASEATBLOOMBERG// SPARK INTEGRATION 19 • Analytical frameworks need a distributed database • Columnar file format != column database • Integrate with HBase to move towards the universal database
    20. 20. HBASEATBLOOMBERG// ANALYTICS: EFFICIENCY 20 • Choice of row and columnar storage engines • Expose primitives for efficiency: • Column pruning • Predicate pushdowns • Data locality
    21. 21. HBASEATBLOOMBERG// THE FUTURE IS BRIGHT 21 • The state of the “Hadoop Database” union is strong – Increasing adoption – Strong foundation – Great community • Prominent role in the data & analytics platform of the future • Let’s go create the future
    22. 22. >>>>>>>>>>>>>> THANK YOU
    23. 23. 23 hbasecon.com

    ×