Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2014 MapR Technologies 1© 2014 MapR Technologies
Zeta Architecture
© 2014 MapR Technologies 2
Agenda
• Current State
– History
– Moving Forward
• The Next Enterprise Architecture
• Business...
© 2014 MapR Technologies 3© 2014 MapR Technologies
Current State
© 2014 MapR Technologies 4
Study History to Prepare for the Future
• A data center was built
• The servers were statically...
© 2014 MapR Technologies 5
Understanding the Why’s
• Isolation of resources
– Assists in troubleshooting
– Prevents the an...
© 2014 MapR Technologies 6
Issues with Isolated Workloads
• Segregated servers lead to under utilized hardware
– Wasted ca...
© 2014 MapR Technologies 7
Goals Moving Forward
• Leverage all existing hardware
• Create isolation in a different way
• I...
© 2014 MapR Technologies 8© 2014 MapR Technologies
The Next “Last” Enterprise Architecture
© 2014 MapR Technologies 9
The Next Generation Enterprise Architecture
• Dynamic compute resources
• Common storage platfo...
© 2014 MapR Technologies 10
Technologies That Work
© 2014 MapR Technologies 11
We Will Call This Architecture…
© 2014 MapR Technologies 12
What’s in a Name
• The letter Z is the last letter in the English
alphabet, but Zeta is not th...
© 2014 MapR Technologies 13
Origin Story of the Zeta Architecture
• Cultivated by Jim Scott
– Created the pretty diagrams
...
© 2014 MapR Technologies 14
Zeta Architecture at Google
© 2014 MapR Technologies 15© 2014 MapR Technologies
Concrete Implementations
© 2014 MapR Technologies 16
Web Server Logs
• Web server generates logs
• Land on local disk
– Logs periodically rotated
•...
© 2014 MapR Technologies 17
Web Server Logs
• Web server generates logs
• Land on DFS
– Logs still rotate
– Logs now toler...
© 2014 MapR Technologies 18
Advertising Platform
© 2014 MapR Technologies 19
Advertising Platform - Simplified
© 2014 MapR Technologies 20
Advertising Platform on Zeta
© 2014 MapR Technologies 21© 2014 MapR Technologies
Business Implications
© 2014 MapR Technologies 22
Integration of Existing Systems
• Use standards like NFS to connect existing
systems
• Pluggab...
© 2014 MapR Technologies 23
Rethink the Data Center
• All Servers
– Run Mesos
– Participate in the Distributed File System...
© 2014 MapR Technologies 24
Simplified Architecture
• Less moving parts
– Less things to go wrong
• Better resource utiliz...
© 2014 MapR Technologies 25
Business Continuity
• Resilience
– Redundancy
– High Availability
– Spare Capacity
• Recovery
...
© 2014 MapR Technologies 26
Platform-wide Security and Compliance
• Authentication, Authorization, Auditing
– Users and jo...
© 2014 MapR Technologies 27
Net Benefit
• Reduced operating expenses (OPEX)
– Better utilization of available capacity and...
© 2014 MapR Technologies 28
Recap
• Saves valuable time and money
• Enables stronger business continuity capabilities
• Go...
© 2014 MapR Technologies 29
Go Forth and Implement the Zeta Architecture
© 2014 MapR Technologies 30
Q&A
@kingmesal maprtech
jscott@mapr.com
Engage with us!
MapR
maprtech
mapr-technologies
Upcoming SlideShare
Loading in …5
×

Zeta Architecture: The Next Generation Big Data Architecture

22,266 views

Published on

The Zeta Architecture is a high-level enterprise architectural construct which enables simplified business processes and defines a scalable way to increase the speed of integrating data into the business. The result? A powerful, data-centric enterprise.

Published in: Technology
  • ----- http://www.ajkobeshoes.com ------- there are many of NBA star basketball shoes for sale in our company,include the air jordan basketball shoes,lebron james XIII basketball shoes, kevin durant basketball shoes,kobe new basketball shoes, curry basketball shoes, cheap price,high quality, welcome you to patronize our website: www.ajkobeshoes.com ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ (please copy and paste this Url into your browser:)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I like this presentation and so that I write a long comment: https://www.linkedin.com/pulse/next-generation-big-data-architecture-yu-liu?published=u
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I like this presentation and it makes a quite good summary of recent trends of data center or enterprise level architecture. Similar concepts can be found like Mesosphere's Data Center Operating System, Couldfundry or HDP/YARN form Hortonworks. However, a question to the "Zeta Architecture" is: as Google has already invented/deployed all the things in the "Zeta Architecture" why it is still called "the next(last) generation" architecture? To my understanding, the core part of this "Zeta Architecture" is a layer of DC (clusters) resource management that provides a maintainable and reliable multi-tenant on-prime or could platform. In Page.9, all the listed items except "real-time application support" are directly dominated by the resource management of data-center. Actually for the "real-time application support" and "integration of existing systems" the DC-level resource management layer also contributes a lot and makes things being pretty easier. Many vendors including Google (Borg, Omega, and Heracles), Microsoft (Applo), Mesosphere (mesos), CouldFundry (Diego), Hortonworks (I don't like YARN because it's too restricted to Hadoop-like and not really suitable for various of distributed systems) and etc., they have already invented or even been providing services/products for DC resource management. So that, the "Zeta Architecture" is probably not the next generation but current generation architecture :) If someone is thinking "it will be the next generation ..." then you are already behind the time. The real next generation architecture for data center in my mind should have the following properties: 1. better Resource Provisioning and Scheduling Algorithms for both back-end analyst tasks and also user-facing, latency-sensitive services; 2. better Energy-Efficient hardware and software stacks; 3. better Service Fault Tolerance including static and dynamic fault tolerance strategies and Byzantine fault tolerance 4. better storage system that provides low-latency access to large-scale datasets (much better than MapR-FS and conceptually it should like RAMCloud)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Zeta Architecture: The Next Generation Big Data Architecture

  1. 1. © 2014 MapR Technologies 1© 2014 MapR Technologies Zeta Architecture
  2. 2. © 2014 MapR Technologies 2 Agenda • Current State – History – Moving Forward • The Next Enterprise Architecture • Business Implications • Concrete Implementations
  3. 3. © 2014 MapR Technologies 3© 2014 MapR Technologies Current State
  4. 4. © 2014 MapR Technologies 4 Study History to Prepare for the Future • A data center was built • The servers were statically partitioned • If we want to break the cycle we have to break the partitions and become dynamic
  5. 5. © 2014 MapR Technologies 5 Understanding the Why’s • Isolation of resources – Assists in troubleshooting – Prevents the analytics team from impacting production • Maximum throughput of an application – Guaranteed volume (maximum): compute, memory and storage • Business Continuity – We know exactly what is backed up, when, and where – Difficult to perfect and to test
  6. 6. © 2014 MapR Technologies 6 Issues with Isolated Workloads • Segregated servers lead to under utilized hardware – Wasted capacity and energy • Complicated processes to move data to the required processing servers – Operational impact, including extra monitoring – Time delays moving data (not real-time) – Troubleshooting time when there are issues • Difficult to thoroughly test DEV vs. QA vs. Production – Environments have different shapes and sizes – They will not have identical configurations
  7. 7. © 2014 MapR Technologies 7 Goals Moving Forward • Leverage all existing hardware • Create isolation in a different way • Improve production operational processes • Fix process of moving from DEV to QA to Production • Support real-time business continuity
  8. 8. © 2014 MapR Technologies 8© 2014 MapR Technologies The Next “Last” Enterprise Architecture
  9. 9. © 2014 MapR Technologies 9 The Next Generation Enterprise Architecture • Dynamic compute resources • Common storage platform • Real-time application support • Flexible programming models • Deployment management • Solution based approach • Applications to operate a business * This is a pluggable architecture
  10. 10. © 2014 MapR Technologies 10 Technologies That Work
  11. 11. © 2014 MapR Technologies 11 We Will Call This Architecture…
  12. 12. © 2014 MapR Technologies 12 What’s in a Name • The letter Z is the last letter in the English alphabet, but Zeta is not the last letter of the Greek alphabet – But this is the last generalized architecture you will need. • Sixth letter of the Greek alphabet – Hexagon represents the 6 surrounding pieces • Zeta represents the number 7 – 7 total components in this architecture – Components work with a global resource manager
  13. 13. © 2014 MapR Technologies 13 Origin Story of the Zeta Architecture • Cultivated by Jim Scott – Created the pretty diagrams – Put a nice name on it – Documented the concepts • Not really a new concept – Google pretty much pioneered these technology concepts – They have never really discussed it cohesively in this way
  14. 14. © 2014 MapR Technologies 14 Zeta Architecture at Google
  15. 15. © 2014 MapR Technologies 15© 2014 MapR Technologies Concrete Implementations
  16. 16. © 2014 MapR Technologies 16 Web Server Logs • Web server generates logs • Land on local disk – Logs periodically rotated • Shipped to other servers • Run jobs on logs
  17. 17. © 2014 MapR Technologies 17 Web Server Logs • Web server generates logs • Land on DFS – Logs still rotate – Logs now tolerant of a server failure prior to rotation – Logs are instantaneously available for computation • Run jobs on logs – Data locality
  18. 18. © 2014 MapR Technologies 18 Advertising Platform
  19. 19. © 2014 MapR Technologies 19 Advertising Platform - Simplified
  20. 20. © 2014 MapR Technologies 20 Advertising Platform on Zeta
  21. 21. © 2014 MapR Technologies 21© 2014 MapR Technologies Business Implications
  22. 22. © 2014 MapR Technologies 22 Integration of Existing Systems • Use standards like NFS to connect existing systems • Pluggable security models fit into your companies current standards • Database that have been tested to work – MySQL, Vertica, Sybase IQ – ElasticSearch, SOLR • May work, but not tested – Oracle, DB2, SQL Server, PostgreSQL • Applications in this architecture can still use them • If they start supporting these technologies things change
  23. 23. © 2014 MapR Technologies 23 Rethink the Data Center • All Servers – Run Mesos – Participate in the Distributed File System • Dynamic Allocation of Resources – Spin up more web servers – Custom Business Applications – Big Data Analytics • Data Locality – No more shipping data – Store and process the data where it was created
  24. 24. © 2014 MapR Technologies 24 Simplified Architecture • Less moving parts – Less things to go wrong • Better resource utilization – Scale any application up or down on demand • Common deployment model (new isolation model) – Repeatability between environments (dev, qa, production) • Shared file system – Get at the data anywhere in the cluster – Simplifies business continuity
  25. 25. © 2014 MapR Technologies 25 Business Continuity • Resilience – Redundancy – High Availability – Spare Capacity • Recovery – Snapshots – Disaster Recovery • Contingency – Protect against the unforeseen – Multisite Capability Production WAN Production Research Datacenter 1 Datacenter 2 WAN EC2
  26. 26. © 2014 MapR Technologies 26 Platform-wide Security and Compliance • Authentication, Authorization, Auditing – Users and jobs – All tiers • Data protection – Wire-level encryption between servers – Masking • Regulatory Compliance – Automatic expiration of “old” data – Data locality supported by distributed file system
  27. 27. © 2014 MapR Technologies 27 Net Benefit • Reduced operating expenses (OPEX) – Better utilization of available capacity and data center space • Reduced capital expenses (CAPEX) – Less total hardware needed • Improves time to market – Streamlined deployments – Environments become consistent and predictable • Delivers a competitive advantage – Via platform scaling – Performance improvements
  28. 28. © 2014 MapR Technologies 28 Recap • Saves valuable time and money • Enables stronger business continuity capabilities • Google has been doing this for years – Real-time is the crux of everything Google does • Time for the rest of us to operate at Google scale – The technologies are there and they play together nicely – Process changes must occur internally to achieve this architecture • This approach will become the “traditional” way of thinking – Don’t get beat to it by your competitors
  29. 29. © 2014 MapR Technologies 29 Go Forth and Implement the Zeta Architecture
  30. 30. © 2014 MapR Technologies 30 Q&A @kingmesal maprtech jscott@mapr.com Engage with us! MapR maprtech mapr-technologies

×