• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Building for operations - Reinhardt Quelle
 

Building for operations - Reinhardt Quelle

on

  • 1,075 views

 

Statistics

Views

Total Views
1,075
Views on SlideShare
668
Embed Views
407

Actions

Likes
0
Downloads
11
Comments
0

4 Embeds 407

http://openopsiq.com 365
http://openopsiq.wordpress.com 39
https://twitter.com 2
http://webcache.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Building for operations - Reinhardt Quelle Building for operations - Reinhardt Quelle Presentation Transcript

    • Building Applications for Operability Reinhardt Quelle Operations Architect rquelle@cisco.com rquelle@* V16e © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 1
    • scr © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Great stuff, but not that’s not what my team does… © 2011 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 3
    • We host and manage SaaS Applications Cisco Social Jabber Millions of Meetings for 10s of Millions of Users totaling Billions of Minutes each month 7x24x365 WebEx © 2011-2012 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) 4
    • DevOps Damon Edwards (DTO) John Willis (now at Dell) suggest “CAMS” as useful mnemonic. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • DevOps > Tools •Automation © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • DevOps > Tools •Automation •Metrics © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • DevOps > Tools •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • DevOps > Tools •Culture •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • DevOps > Tools •Culture / Collaboration •Automation •Metrics •Sharing © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Collaboration Requires: • shared goals • communication • empathy • © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Yay, We’re going to do DevOps! http://www.realclearworld.com/blog/2009/01/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • When you start, its more like this: http://www.news.com.au/world/are-barack-obama-and-vladamir-putin-holding-highlevel-talks-or-are-they-on-an-awkward-date/storyfndir2ev-1226665803294 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • At a more personal level: remember that first date? http://www.hercampus.com/love/dating-hooking/9-first-date-disasters-how-deal-them © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Where do we begin? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Perspective helps (a lot) http://www.navigonlp.com/blog/stop-look-look-again/boat-land-perspective-2/ © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • I come from an Operations background © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Green = Good © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Illustrating the Ops perspective to Engineering http://cdn.twentytwowords.com/wp-content/uploads/Rhinos-Perspective-e1369734532511-634x447.jpg © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • A quick illustration: A real, albeit deceased, product (RIP) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Developer’s View Compute Node resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
    • Operations’ View Compute Node /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
    • Servers run in "clusters" Compute Node Compute Node Compute Node inetd config inetd config config sshd config rsyslog config LVM /etc/snmp/* config LVM ... ... ntp.conf ... DR:BD ... © 2013 Cisco and/or its affiliates. All rights reserved. config ... App start/stop orchestration config ntp.conf .... sshd config rsyslog config ... ntp.conf ... .... DR:BD ... ntp.conf App config config ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App config B Reinhardt Quelle (rquelle@cisco.com) sshd rsyslog smb.conf config A start/stop orchestration /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B config resolv.conf smb.conf config A /etc/passwd inetd /etc/* resolv.conf DR:BD /etc/sysconfig/* /etc/logrotate/* LVM /etc/snmp/* ... config B start/stop orchestration config LVM ... App sshd smb.conf config A /etc/passwd inetd /etc/logrotate/* resolv.conf smb.conf /etc/sysconfig/* /etc/* .... .... resolv.conf DR:BD config rsyslog /etc/snmp/* Compute Node /etc/passwd inetd /etc/* /etc/* rsyslog /etc/sysconfig/* /etc/logrotate/* /etc/logrotate/* sshd Compute Node /etc/passwd /etc/passwd /etc/sysconfig/* start/stop orchestration ... config A App config B start/stop orchestration config A config B
    • Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN
    • Which means they may have to share data: Compute Node Compute Node Compute Node inetd sshd /etc/sysconfig/* config config inetd config sshd config LVM rsyslog ... ... ntp.conf ... DR:BD ... config ntp.conf /etc/sysconfig/* /etc/snmp/* config A App start/stop orchestration inetd sshd /etc/sysconfig/* config config rsyslog config ... ... ntp.conf rsyslog config ... .... DR:BD ntp.conf ... App ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... config A App config B start/stop orchestration config smb.conf ... config B rsyslog /etc/logrotate/* resolv.conf smb.conf config A config LVM /etc/snmp/* resolv.conf DR:BD sshd /etc/sysconfig/* config /etc/* LVM .... /etc/passwd inetd /etc/logrotate/* /etc/* ... config B /etc/passwd /etc/logrotate/* smb.conf ... start/stop orchestration sshd resolv.conf smb.conf App config LVM .... .... resolv.conf DR:BD /etc/snmp/* config LVM config inetd /etc/* /etc/* rsyslog /etc/sysconfig/* Compute Node /etc/passwd /etc/logrotate/* /etc/logrotate/* /etc/snmp/* Compute Node /etc/passwd /etc/passwd ... config A App config B start/stop orchestration config A config B start/stop orchestration spare SAN LUN LUN LUN LUN Note: there are *much* better ways to share data! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • We have several front-end servers Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config rsyslog config LVM ... sshd config rsyslog config LVM ... config A Service X config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* rsyslog config LVM ... .... sshd config rsyslog config ... ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config sshd config rsyslog config ... ntp.conf ... sshd ntp.conf ... rsyslog config ... /etc/snmp/* .... config rsyslog /etc/sysconfig/* config sshd config ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B cluster manager /etc/passwd inetd /etc/* smb.conf PPSD config A config B LVM config resolv.conf DR:BD Service Y /etc/logrotate/* /etc/* .... config B Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* resolv.conf DR:BD Service X config B /etc/* .... smb.conf /etc/passwd inetd LVM /etc/snmp/* resolv.conf ... ntp.conf ... Service Y config B /etc/logrotate/* /etc/* DR:BD .... resolv.conf config A Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd /etc/logrotate/* config ... ... Postpath Unified Node (PUN) sshd DR:BD smb.conf Service Y config ... /etc/snmp/* /etc/* ntp.conf ... /etc/sysconfig/* config resolv.conf ntp.conf smb.conf inetd config LVM .... resolv.conf config B sshd rsyslog /etc/snmp/* /etc/* .... config A config /etc/logrotate/* ... DR:BD /etc/snmp/* /etc/* Service X inetd /etc/logrotate/* ... DR:BD /etc/passwd /etc/sysconfig/* config config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN
    • ...which also have to be redundant. Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd sshd /etc/passwd /etc/sysconfig/* config config inetd sshd config /etc/logrotate/* config config resolv.conf /etc/sysconfig/* ... inetd ... sshd ntp.conf config smb.conf rsyslog Service X Service Y DR:BD config B config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog config B ... ntp.conf Service X config A /etc/snmp/* config B Postpath Unified Node (PUN) Service Y sshd config rsyslog config LVM ... Service X config B sshd config rsyslog config LVM /etc/sysconfig/* config ntp.conf ... ntp.conf ... sshd config /etc/snmp/* config A config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf smb.conf .... DR:BD ntp.conf ... rsyslog ... config B config rsyslog ntp.conf ... config ... config B ... /etc/logrotate/* .... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... config A PPSD config B cluster manager config LVM /etc/snmp/* ... PPSD config rsyslog /etc/sysconfig/* config sshd /etc/* DR:BD ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN config A config B /etc/passwd inetd /etc/logrotate/* smb.conf config A cluster manager sshd resolv.conf DR:BD Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* .... config A /etc/passwd inetd LVM /etc/snmp/* ... PPSD ... Service X config B Postpath Unified Node (PUN) config smb.conf ... Service Y config A /etc/logrotate/* resolv.conf config A cluster manager config B /etc/* smb.conf PPSD config A /etc/sysconfig/* config LVM /etc/snmp/* resolv.conf DR:BD Service X /etc/passwd inetd /etc/logrotate/* /etc/* .... /etc/logrotate/* smb.conf rsyslog config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... config resolv.conf smb.conf config A /etc/passwd inetd sshd ntp.conf config LVM ... Service X inetd ... .... resolv.conf smb.conf config A ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM resolv.conf config DR:BD /etc/sysconfig/* LVM .... /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd DR:BD /etc/snmp/* ... config A /etc/sysconfig/* config /etc/logrotate/* .... /etc/logrotate/* config ... LVM config A config /etc/passwd LVM config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... DR:BD inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
    • And that means load balancers to direct traffic Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd /etc/logrotate/* sshd config sshd /etc/snmp/* config config resolv.conf /etc/sysconfig/* LVM ... inetd ... sshd ntp.conf config smb.conf config A Service Y DR:BD config B sshd /etc/snmp/* config config resolv.conf ... inetd ... sshd ntp.conf config smb.conf /etc/* rsyslog resolv.conf config B ... ntp.conf Service X config A /etc/snmp/* config B Load Balancer Postpath Unified Node (PUN) Service Y config A Service X config B config rsyslog config LVM ... /etc/sysconfig/* config sshd config rsyslog config ... resolv.conf ntp.conf ntp.conf ... config B Service X .... sshd config rsyslog ntp.conf ... Service X config B .... resolv.conf config B ... ntp.conf ... sshd rsyslog ntp.conf ... config ... config rsyslog /etc/sysconfig/* config sshd /etc/snmp/* config ... /etc/logrotate/* /etc/snmp/* /etc/* .... /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... resolv.conf DR:BD ntp.conf ... smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN Reinhardt Quelle (rquelle@cisco.com) LUN config A config B /etc/passwd inetd LVM config resolv.conf DR:BD config B /etc/logrotate/* /etc/* .... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config LVM /etc/snmp/* config A /etc/passwd inetd ... cluster manager Service Y DR:BD ... config A Postpath Unified Node (PUN) config resolv.conf DR:BD Service Y smb.conf PPSD config A config B /etc/* ... config A smb.conf /etc/logrotate/* /etc/* resolv.conf DR:BD config A /etc/sysconfig/* config LVM /etc/snmp/* /etc/* .... /etc/snmp/* config ... LVM /etc/passwd inetd /etc/logrotate/* LVM /etc/snmp/* /etc/logrotate/* smb.conf config B ... Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/passwd /etc/sysconfig/* ... Postpath Unified Node (PUN) /etc/sysconfig/* config .... smb.conf /etc/passwd inetd sshd ntp.conf config config resolv.conf rsyslog ... config A inetd ... .... smb.conf Service X ... /etc/* ... config A Service Y DR:BD config B LVM /etc/logrotate/* config ... LVM ... config A /etc/snmp/* config DR:BD /etc/sysconfig/* LVM .... config Front-End Server /etc/* /etc/passwd DR:BD /etc/snmp/* /etc/sysconfig/* rsyslog .... /etc/logrotate/* config ... LVM Service X config /etc/passwd DR:BD config /etc/logrotate/* Front-End Server /etc/* rsyslog .... rsyslog inetd /etc/logrotate/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config
    • There are multiple clusters Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config sshd config inetd config config LVM config A Service X /etc/logrotate/* sshd ntp.conf config ... config A Service X Postpath Unified Node (PUN) config A Postpath Unified Node (PUN) Postpath Unified Node (PUN) sshd config rsyslog /etc/* .... resolv.conf config B ... ntp.conf Postpath Unified Node (PUN) config ntp.conf PPSD config B DR:BD cluster manager DR:BD ... /etc/snmp/* /etc/* .... resolv.conf ntp.conf ... ntp.conf smb.conf ... smb.conf ... config A PPSD ... .... smb.conf config B /etc/logrotate/* config config LVM /etc/* resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... config A /etc/passwd inetd ... /etc/* smb.conf ... config config LVM config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd rsyslog /etc/snmp/* resolv.conf ntp.conf ... DR:BD smb.conf config B /etc/logrotate/* ... .... config A Service Y /etc/passwd inetd config LVM /etc/* resolv.conf config A ... config A Service X config B /etc/sysconfig/* config sshd rsyslog /etc/snmp/* .... cluster manager /etc/snmp/* ... config A Service Y DR:BD config B config A /etc/logrotate/* ... /etc/* resolv.conf PPSD inetd config LVM /etc/snmp/* ... /etc/logrotate/* config ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* ... DR:BD config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/logrotate/* config config LVM /etc/passwd /etc/sysconfig/* smb.conf resolv.conf config B ... config A Service X config B /etc/sysconfig/* config sshd .... config resolv.conf LVM ... /etc/passwd inetd rsyslog sshd ntp.conf config smb.conf Service Y config B ... rsyslog ... config A DR:BD .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/* config ... LVM resolv.conf ... /etc/snmp/* smb.conf rsyslog config B ... /etc/snmp/* config /etc/logrotate/* smb.conf Service X config Front-End Server /etc/* LVM /etc/sysconfig/* .... ... config A Service Y DR:BD config B /etc/sysconfig/* config sshd rsyslog config resolv.conf inetd /etc/passwd /etc/* config ... LVM ... DR:BD /etc/snmp/* smb.conf rsyslog LVM .... /etc/logrotate/* sshd ntp.conf config ... DR:BD /etc/sysconfig/* config /etc/passwd config resolv.conf inetd /etc/snmp/* Front-End Server /etc/* rsyslog .... ... inetd /etc/logrotate/* /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/logrotate/* PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config DR:BD rsyslog ... .... inetd DR:BD ... /etc/snmp/* Service Y DR:BD ntp.conf config A Service X resolv.conf ntp.conf config A Postpath Unified Node (PUN) config A Service Y config inetd Postpath Unified Node (PUN) config config inetd .... ntp.conf Postpath Unified Node (PUN) config config config ... inetd /etc/sysconfig/* config sshd config rsyslog config /etc/logrotate/* /etc/logrotate/* /etc/snmp/* /etc/snmp/* /etc/* /etc/* .... ... LVM ... .... ... DR:BD LVM resolv.conf ntp.conf /etc/passwd /etc/sysconfig/* config sshd .... LVM DR:BD resolv.conf config B Postpath Unified Node (PUN) /etc/* ... ... inetd rsyslog /etc/snmp/* .... LVM DR:BD config A Service Y config B /etc/passwd config /etc/* .... ntp.conf config A Service X /etc/logrotate/* /etc/snmp/* /etc/* resolv.conf ... DR:BD ... resolv.conf ntp.conf resolv.conf ntp.conf smb.conf smb.conf smb.conf ... ... ntp.conf smb.conf ... PPSD /etc/* resolv.conf ... config A config B /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* /etc/snmp/* ... /etc/snmp/* ... config A config B ... smb.conf /etc/passwd /etc/sysconfig/* config sshd rsyslog /etc/logrotate/* ... Service Y config B /etc/passwd config LVM config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD /etc/logrotate/* config ... Service Y DR:BD config B ... /etc/passwd inetd rsyslog config A Service X smb.conf ... config B /etc/passwd /etc/sysconfig/* smb.conf LVM ... config A smb.conf Service X .... config resolv.conf sshd ntp.conf config rsyslog .... config B ... Service Y DR:BD config B ... inetd /etc/* config ... LVM resolv.conf DR:BD /etc/snmp/* smb.conf ... config A ... /etc/logrotate/* sshd ntp.conf config rsyslog .... config B ... /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* config ... LVM config A /etc/logrotate/* config rsyslog .... config resolv.conf inetd /etc/logrotate/* sshd ntp.conf config config B ... /etc/sysconfig/* smb.conf Service X LVM Front-End Server /etc/* config /etc/passwd config resolv.conf rsyslog /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* config ... inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* LVM /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog Load Balancer LUN ... config A PPSD config B cluster manager config A config A PPSD config B cluster manager PPSD config B cluster manager smb.conf ... config A config A PPSD config B cluster manager config B cluster manager spare SAN LUN LUN LUN Webmail RPC Front-End Server ADM Front-End Server Front-End Server /etc/passwd inetd /etc/passwd /etc/sysconfig/* config inetd config config LVM ... DR:BD ... inetd rsyslog config resolv.conf smb.conf Service X Service Y DR:BD config B config LVM ... DR:BD /etc/snmp/* ... ... config A config B ... inetd config resolv.conf sshd ntp.conf config smb.conf rsyslog LVM Service X ntp.conf ... config A Service Y DR:BD config B config A Postpath Unified Node (PUN) config B ... config config Service Y config A Service X config B /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/snmp/* Service X ntp.conf config ... DR:BD ... /etc/snmp/* config B /etc/* config ... Service Y DR:BD .... ... config A resolv.conf config B ... ntp.conf ... config A Service X config B /etc/snmp/* config config /etc/snmp/* /etc/sysconfig/* config sshd config config /etc/logrotate/* /etc/snmp/* /etc/* /etc/* .... LVM ... DR:BD ... .... LVM ... DR:BD ... resolv.conf ntp.conf smb.conf smb.conf ... ... resolv.conf ntp.conf smb.conf ... ... config A PPSD config B config A PPSD config B cluster manager config A PPSD config B cluster manager smb.conf ... config A PPSD config B cluster manager config A config B cluster manager spare SAN LUN © 2013 Cisco and/or its affiliates. All rights reserved. LUN LUN config A config B /etc/passwd inetd rsyslog /etc/logrotate/* .... resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/* ... ... config A config B /etc/passwd inetd rsyslog /etc/logrotate/* LVM DR:BD resolv.conf ntp.conf Service Y Postpath Unified Node (PUN) config .... LVM cluster manager config A smb.conf smb.conf PPSD /etc/logrotate/* smb.conf LVM resolv.conf /etc/passwd /etc/sysconfig/* config sshd /etc/* .... resolv.conf ntp.conf inetd rsyslog /etc/logrotate/* /etc/* ... ... config A config B Postpath Unified Node (PUN) /etc/passwd inetd rsyslog /etc/logrotate/* LVM /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd DR:BD .... inetd rsyslog .... smb.conf /etc/passwd inetd rsyslog ... /etc/* ... config B DR:BD /etc/snmp/* smb.conf Service X ... /etc/logrotate/* config ... .... config A /etc/snmp/* config LVM /etc/passwd /etc/sysconfig/* /etc/* resolv.conf /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/logrotate/* config ... LVM config /etc/logrotate/* sshd /etc/snmp/* /etc/passwd /etc/sysconfig/* sshd ntp.conf config config A config Front-End Server /etc/* .... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd LUN LUN Reinhardt Quelle (rquelle@cisco.com)
    • ... and geo-redundant data centers. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* config A sshd ntp.conf config rsyslog .... resolv.conf config B ... config A Service X resolv.conf config B ... config B Postpath Unified Node (PUN) config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config config Postpath Unified Node (PUN) config config LVM config LVM config config config B Postpath Unified Node (PUN) inetd config config config LVM config LVM /etc/logrotate/* config LUN LUN Webmail config LVM ADM inetd smb.conf ... ... config A config /etc/sysconfig/* inetd sshd ntp.conf config Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config /etc/snmp/* resolv.conf config A Service X Service X config B inetd Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) config LVM config inetd config Service Y DR:BD config B resolv.conf config A Service X config B inetd Service X config B inetd config /etc/* .... ... config A resolv.conf config B ... ntp.conf smb.conf LVM config inetd inetd /etc/sysconfig/* config /etc/logrotate/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* config /etc/snmp/* /etc/* rsyslog .... LVM ... config /etc/* rsyslog .... LVM ... config .... ... resolv.conf ntp.conf ... DR:BD resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* config DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/logrotate/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... DR:BD smb.conf config A config B ... config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/snmp/* /etc/* ... ... PPSD cluster manager Service Y config B /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config rsyslog .... config A /etc/passwd /etc/sysconfig/* config sshd /etc/* ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM resolv.conf config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config B inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf .... /etc/snmp/* config ... LVM ... config A config LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... /etc/passwd config A PPSD cluster manager sshd ntp.conf config /etc/* config B ... config A Service X ... ... DR:BD ... inetd ... resolv.conf ntp.conf smb.conf /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... smb.conf /etc/snmp/* config LVM resolv.conf config A config B cluster manager /etc/snmp/* config ... LVM config A Service X /etc/logrotate/* config rsyslog .... ... ... PPSD Load Balancer /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config sshd /etc/snmp/* /etc/* LVM DR:BD ... config B sshd ntp.conf config Postpath Unified Node (PUN) /etc/logrotate/* config rsyslog .... ntp.conf smb.conf config A PPSD cluster manager inetd rsyslog inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/passwd /etc/sysconfig/* config sshd /etc/* ... ... DR:BD ... config A config B cluster manager config B Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf PPSD inetd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B /etc/passwd /etc/logrotate/* config rsyslog .... ... DR:BD smb.conf config A config B config A Service X config B Postpath Unified Node (PUN) /etc/* rsyslog .... resolv.conf ntp.conf ntp.conf ... config A /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... inetd /etc/logrotate/* sshd /etc/snmp/* config PPSD resolv.conf config B ... smb.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config LVM DR:BD Service Y config B /etc/passwd /etc/sysconfig/* config sshd rsyslog config A Service X config B /etc/passwd cluster manager ... config A Service Y DR:BD config B .... ... config A Service Y config A Service X ntp.conf /etc/* smb.conf ... config A /etc/snmp/* config ... LVM resolv.conf config B ... smb.conf Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... ... config A Service Y DR:BD config B ntp.conf sshd ntp.conf config /etc/* config ... LVM ... config A config B ... inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... ADM /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config A config B rsyslog .... /etc/snmp/* smb.conf Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config B LUN Front-End Server inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config A PPSD config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD ntp.conf ... DR:BD smb.conf PPSD cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog .... ... resolv.conf ntp.conf ... ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B inetd /etc/logrotate/* sshd /etc/* rsyslog .... resolv.conf PPSD cluster manager config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* /etc/* ... ... ... config A config B ... Postpath Unified Node (PUN) SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf smb.conf /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD smb.conf SAN Load Balancer /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config rsyslog .... ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config B /etc/logrotate/* smb.conf resolv.conf ... /etc/* rsyslog .... ntp.conf smb.conf config A PPSD cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM config B ... config A Service X /etc/logrotate/* config /etc/* ... ... DR:BD ... .... inetd rsyslog .... ... config A smb.conf Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A config B cluster manager config B /etc/snmp/* ... ... /etc/* config ... Service Y DR:BD config B config A /etc/sysconfig/* config sshd /etc/snmp/* config LVM resolv.conf PPSD config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... ... ... config A PPSD sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf ... config A config B cluster manager config A Service X ntp.conf /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd ntp.conf ... DR:BD smb.conf PPSD rsyslog .... ... resolv.conf ntp.conf ... DR:BD ... resolv.conf Service Y Postpath Unified Node (PUN) /etc/passwd inetd /etc/snmp/* ... config B config config LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM config B ... config A Service X Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config B /etc/logrotate/* sshd ntp.conf config smb.conf config A Service Y /etc/passwd inetd /etc/snmp/* /etc/* rsyslog .... resolv.conf config A config B config B /etc/logrotate/* sshd /etc/snmp/* /etc/* ... ... PPSD Service Y DR:BD config B /etc/sysconfig/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd .... ... config A rsyslog .... config resolv.conf ... ... /etc/* config ... ... /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD inetd config A Service X ntp.conf config A Service X /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog cluster manager resolv.conf config B ... /etc/snmp/* smb.conf rsyslog sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X /etc/passwd inetd Service Y DR:BD config B sshd ntp.conf config LVM ... config A Service Y config A Service X ntp.conf inetd .... ... config A inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... smb.conf ... config A /etc/snmp/* /etc/passwd /etc/sysconfig/* config sshd rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog LVM smb.conf Service X sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf /etc/* config ... Service Y DR:BD config B inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf LVM Service X config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
    • Which means data must be replicated - somehow. GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd .... resolv.conf ntp.conf smb.conf config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) Service Y config B PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager spare spare SAN SAN LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... LVM config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM DR:BD config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog rsyslog ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server Service X /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN sshd ntp.conf Service Y config B spare inetd .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Load Balancer /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
    • But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config config A config A Service X config B Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd /etc/snmp/* config config rsyslog /etc/logrotate/* rsyslog DR:BD Postpath Unified Node (PUN) config config /etc/logrotate/* config LUN LUN Webmail config .... LVM ... resolv.conf resolv.conf ntp.conf ... ADM smb.conf ... PPSD ... config A inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B Postpath Unified Node (PUN) Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config config LVM config LVM Postpath Unified Node (PUN) config DR:BD config LVM config A Service X config resolv.conf Postpath Unified Node (PUN) inetd config config config config LVM /etc/logrotate/* config /etc/snmp/* /etc/* rsyslog .... LVM ... DR:BD config .... ... resolv.conf ntp.conf ... smb.conf DR:BD ntp.conf ... smb.conf ... smb.conf ... config A PPSD sshd /etc/snmp/* config resolv.conf ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* LVM resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config rsyslog .... ... resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... ... config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* LVM DR:BD smb.conf config A config B ... Postpath Unified Node (PUN) PPSD config B cluster manager ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server /etc/sysconfig/* smb.conf rsyslog Service Y DR:BD inetd sshd ntp.conf config ... DR:BD /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A inetd DR:BD ... /etc/snmp/* smb.conf ntp.conf Service X config A config B ... Service Y DR:BD /etc/sysconfig/* sshd ntp.conf config ... DR:BD smb.conf rsyslog resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... /etc/sysconfig/* smb.conf /etc/* rsyslog LVM ... config A resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) config resolv.conf ntp.conf smb.conf smb.conf ... .... Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf config A config B config A config B /etc/passwd /etc/* rsyslog LVM ... PPSD cluster manager Service X /etc/logrotate/* resolv.conf DR:BD ... config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf config A config B Service Y /etc/passwd /etc/logrotate/* resolv.conf ... config A config B Postpath Unified Node (PUN) /etc/passwd LVM ... config A config B ... /etc/* config PPSD cluster manager Service X smb.conf ... /etc/logrotate/* /etc/snmp/* /etc/logrotate/* smb.conf LVM ... config A ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... smb.conf /etc/passwd inetd /etc/snmp/* inetd /etc/* ... config B ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... config B ... /etc/sysconfig/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD /etc/snmp/* config Front-End Server /etc/* rsyslog .... inetd smb.conf Service X config /etc/logrotate/* sshd /etc/snmp/* config /etc/logrotate/* config ... LVM ... config A config B ... inetd /etc/logrotate/* config Front-End Server /etc/* LVM /etc/passwd config resolv.conf .... /etc/passwd /etc/sysconfig/* config sshd rsyslog .... inetd /etc/* config ... LVM resolv.conf Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/logrotate/* sshd ntp.conf config rsyslog .... ... config A config Front-End Server /etc/* /etc/sysconfig/* LVM /etc/passwd config resolv.conf /etc/* ADM Front-End Server /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM /etc/passwd config resolv.conf LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... LVM config A config B ... /etc/logrotate/* LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM /etc/passwd config resolv.conf sshd ntp.conf config ... DR:BD inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail Front-End Server /etc/passwd inetd /etc/logrotate/* config config LVM LUN ADM Front-End Server /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. ntp.conf smb.conf spare SAN LVM .... resolv.conf Service Y config B spare DR:BD /etc/* ... config A config B ... config A Service X config B /etc/sysconfig/* config /etc/logrotate/* sshd rsyslog .... resolv.conf ntp.conf /etc/snmp/* config ... Service Y DR:BD config B config A /etc/passwd /etc/sysconfig/* config /etc/* ... ... PPSD cluster manager config A Service X ntp.conf Service Y config B /etc/passwd inetd /etc/snmp/* config LVM DR:BD ... config A Service X config B /etc/logrotate/* sshd rsyslog .... ntp.conf /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config ... Postpath Unified Node (PUN) smb.conf config A .... inetd rsyslog .... smb.conf config A Service Y /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf /etc/sysconfig/* config config B cluster manager resolv.conf /etc/passwd inetd resolv.conf ... PPSD config B config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog .... config B ... config A Service X /etc/snmp/* config ... sshd ntp.conf config ... ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD cluster manager Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config B Postpath Unified Node (PUN) /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B config A Service X config B /etc/logrotate/* sshd /etc/snmp/* config resolv.conf PPSD ... config A /etc/sysconfig/* config /etc/* rsyslog .... ... ... DR:BD inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog cluster manager ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd inetd resolv.conf config B ... ... config A Service Y Service Y DR:BD config B .... ... config A smb.conf ... Service X config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf config A sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Billing Mgt Servers config A PPSD config B cluster manager LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Service X ntp.conf ... DR:BD smb.conf ... config B cluster manager spare LUN RPC Front-End Server sshd /etc/snmp/* /etc/* rsyslog ... DR:BD smb.conf config A PPSD sshd /etc/snmp/* config .... LVM ntp.conf ... DR:BD ... /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A config B cluster manager inetd /etc/logrotate/* config LVM resolv.conf PPSD sshd /etc/snmp/* /etc/* rsyslog .... ... ... DR:BD ... config A /etc/passwd /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* config LVM ntp.conf smb.conf config B cluster manager inetd config B Postpath Unified Node (PUN) /etc/passwd /etc/sysconfig/* config /etc/* rsyslog .... resolv.conf config A PPSD config B cluster manager ... SAN LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* config /etc/* ... ... config A PPSD Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD ... config A config B cluster manager SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd /etc/logrotate/* rsyslog .... smb.conf ... PPSD config /etc/* ntp.conf ... DR:BD smb.conf ... config B cluster manager sshd /etc/snmp/* ... resolv.conf ntp.conf ... smb.conf config A PPSD config LVM ... resolv.conf ntp.conf ... DR:BD ... config rsyslog .... LVM ... resolv.conf ntp.conf smb.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM resolv.conf config A config B cluster manager sshd /etc/snmp/* /etc/* config .... ... ... PPSD config /etc/* config LVM DR:BD ... sshd /etc/snmp/* /etc/* rsyslog .... ntp.conf smb.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/snmp/* /etc/* resolv.conf config A config B inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd ... ... PPSD /etc/snmp/* config ... Service Y DR:BD config B ... config LVM DR:BD Service X smb.conf ... /etc/logrotate/* sshd rsyslog /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... Postpath Unified Node (PUN) cluster manager /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog
    • But wait! There’s more! GSS Webmail RPC Front-End Server ADM Front-End Server inetd config sshd ntp.conf config /etc/snmp/* rsyslog Service X rsyslog config A Service X sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X inetd sshd ntp.conf config rsyslog config B ... /etc/snmp/* config A Service X /etc/logrotate/* sshd ntp.conf config rsyslog config B ... config A Service X Service Y DR:BD config B ntp.conf Security Appliances config A config B ... config A Postpath Unified Node (PUN) inetd Postpath Unified Node (PUN) /etc/logrotate/* sshd config rsyslog sshd config rsyslog /etc/snmp/* smb.conf smb.conf ... smb.conf ... config A PPSD config B PPSD config B smb.conf PPSD config B cluster manager config /etc/* .... LVM ... DR:BD ... resolv.conf resolv.conf ntp.conf LUN LUN smb.conf ... smb.conf ... PPSD config B cluster manager ... config A ADM inetd config /etc/sysconfig/* inetd sshd ntp.conf config smb.conf config A Service Y DR:BD /etc/sysconfig/* inetd sshd ntp.conf config smb.conf rsyslog LVM resolv.conf config B ... config A Service X resolv.conf config B config config LVM config DR:BD config LVM config A Service X resolv.conf config inetd config LVM config config /etc/logrotate/* sshd /etc/snmp/* config config LVM .... ... resolv.conf ntp.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A config A PPSD config B cluster manager config B cluster manager SAN Identity Servers LUN LUN LUN Webmail LUN RPC Front-End Server Front-End Server Front-End Server ... /etc/sysconfig/* smb.conf LVM config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM ... config A resolv.conf config B ... ntp.conf Service X config A ... inetd Front-End Server /etc/snmp/* resolv.conf ntp.conf Service X config A config B Service Y DR:BD Front-End Server inetd ... /etc/sysconfig/* smb.conf resolv.conf ntp.conf Service X config A Service Y DR:BD config B ... ... /etc/snmp/* /etc/sysconfig/* smb.conf rsyslog LVM resolv.conf ntp.conf Service X config A config A Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf smb.conf ... ... config Service Y config A Service X config B inetd /etc/sysconfig/* config sshd config .... /etc/snmp/* ntp.conf inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd config .... config ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... ntp.conf ... /etc/snmp/* inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... /etc/* .... ... config A resolv.conf config B ... ntp.conf inetd /etc/sysconfig/* config sshd config /etc/snmp/* config .... ntp.conf ... inetd /etc/sysconfig/* config sshd config config .... /etc/snmp/* ntp.conf ... SAN config .... LUN LUN /etc/logrotate/* /etc/snmp/* ntp.conf ... /etc/* rsyslog config .... ... resolv.conf DR:BD ntp.conf ... smb.conf smb.conf ... PPSD ... config A PPSD config B cluster manager config A config B cluster manager spare LUN Reinhardt Quelle (rquelle@cisco.com) config LVM ... ... config A config B SAN LUN /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager inetd /etc/* rsyslog LVM ... ... LUN LUN LUN config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B ... Service X spare LUN /etc/snmp/* config ... Service Y DR:BD smb.conf config A config B Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager Service Y /etc/passwd /etc/logrotate/* LVM ... ... config A config B config A config B Postpath Unified Node (PUN) /etc/passwd resolv.conf DR:BD smb.conf PPSD cluster manager Service X /etc/* rsyslog LVM ... ... config A config B /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B Service Y Postpath Unified Node (PUN) /etc/* rsyslog LVM ... PPSD cluster manager config B /etc/passwd /etc/logrotate/* resolv.conf DR:BD ... config A config B config A Service X Postpath Unified Node (PUN) /etc/* rsyslog smb.conf PPSD cluster manager config A config B /etc/passwd /etc/logrotate/* LVM ... ... config A config B Service Y Postpath Unified Node (PUN) /etc/sysconfig/* config sshd resolv.conf DR:BD smb.conf PPSD cluster manager config A config B /etc/passwd /etc/* rsyslog LVM ... ... Service X /etc/logrotate/* resolv.conf DR:BD smb.conf config A config B config A config B Postpath Unified Node (PUN) /etc/* rsyslog resolv.conf ... Service Y /etc/passwd /etc/logrotate/* LVM ... config A config B Postpath Unified Node (PUN) /etc/passwd /etc/* config PPSD cluster manager config A config B ... /etc/logrotate/* /etc/snmp/* Service X smb.conf ... Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/logrotate/* smb.conf LVM ... config A smb.conf /etc/passwd inetd /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config rsyslog .... ... config B /etc/snmp/* inetd /etc/* smb.conf Service X ... .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... ... config A config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* config B ... /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LVM ... config A config B ... config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog .... /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... /etc/* config ... LVM ... config A config config LVM /etc/logrotate/* smb.conf rsyslog .... /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* /etc/sysconfig/* DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* ADM /etc/logrotate/* sshd rsyslog .... /etc/snmp/* config B ... Service Y DR:BD config B ... /etc/logrotate/* config ... .... inetd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config /etc/* Identity Servers LUN RPC /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog .... inetd /etc/logrotate/* config ... LUN Front-End Server config /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD /etc/passwd config resolv.conf sshd ntp.conf config inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* ... LUN Webmail /etc/passwd inetd /etc/logrotate/* config config LVM DR:BD LUN ADM /etc/passwd /etc/sysconfig/* config Front-End Server /etc/* rsyslog © 2013 Cisco and/or its affiliates. All rights reserved. config resolv.conf ... ... config B cluster manager spare SAN LVM /etc/snmp/* /etc/* rsyslog .... ... DR:BD smb.conf config A PPSD /etc/sysconfig/* config /etc/* LVM ntp.conf ... DR:BD ... config A config B cluster manager config B /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... resolv.conf ntp.conf smb.conf PPSD sshd /etc/snmp/* config LVM resolv.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/* rsyslog .... ... DR:BD smb.conf config A config B Monitoring Servers ntp.conf config A Service X /etc/sysconfig/* config spare DR:BD .... resolv.conf /etc/passwd inetd /etc/logrotate/* sshd /etc/snmp/* /etc/* rsyslog .... resolv.conf ntp.conf /etc/* ... config A config B ... ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* sshd /etc/* ... ... PPSD cluster manager /etc/snmp/* config ... Service Y DR:BD config B smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd /etc/snmp/* config LVM DR:BD ... config A Service X ntp.conf ... config A Service X config B inetd /etc/logrotate/* sshd rsyslog .... ntp.conf smb.conf config A config B /etc/passwd /etc/logrotate/* smb.conf LVM config B ... /etc/sysconfig/* config resolv.conf sshd ntp.conf config smb.conf config A Service Y .... inetd rsyslog .... ... config B /etc/snmp/* ... ... /etc/* ... config A Service Y DR:BD config B ntp.conf Postpath Unified Node (PUN) resolv.conf ... PPSD cluster manager resolv.conf /etc/sysconfig/* config config LVM /etc/snmp/* config ... LVM DR:BD /etc/logrotate/* smb.conf rsyslog /etc/passwd /etc/snmp/* config ... sshd ntp.conf config .... config B ... config A Service X inetd ... DR:BD smb.conf config A config B cluster manager inetd smb.conf /etc/* rsyslog .... ntp.conf ... PPSD Service Y DR:BD config B /etc/logrotate/* sshd /etc/snmp/* config ... DR:BD ... config B cluster manager config A Service X /etc/logrotate/* /etc/sysconfig/* config /etc/passwd config resolv.conf ... ... /etc/* ... config A Front-End Server /etc/* rsyslog .... /etc/snmp/* config ... Postpath Unified Node (PUN) resolv.conf ntp.conf smb.conf config A PPSD config B Load Balancer /etc/sysconfig/* config sshd /etc/snmp/* config LVM DR:BD /etc/logrotate/* smb.conf /etc/sysconfig/* config /etc/* LVM resolv.conf ... config A config B cluster manager sshd ntp.conf config LVM inetd /etc/logrotate/* config Front-End Server /etc/* /etc/passwd /etc/sysconfig/* inetd rsyslog /etc/passwd /etc/sysconfig/* config sshd rsyslog .... ... ... /etc/passwd inetd /etc/logrotate/* config rsyslog .... ... ... DR:BD smb.conf PPSD sshd /etc/snmp/* config LVM ntp.conf ... ... config A Service Y config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/* rsyslog .... ... resolv.conf ntp.conf smb.conf config A Service X /etc/passwd inetd /etc/logrotate/* config /etc/* rsyslog .... resolv.conf config A config B ... config A config B Postpath Unified Node (PUN) /etc/sysconfig/* config sshd /etc/snmp/* /etc/* ... ... PPSD inetd /etc/logrotate/* sshd /etc/snmp/* config LVM DR:BD ntp.conf smb.conf /etc/passwd /etc/sysconfig/* config Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog config A Service X config B Postpath Unified Node (PUN) cluster manager resolv.conf config B ... ... config A Service Y /etc/passwd /etc/sysconfig/* config Service Y DR:BD config B .... ... config A smb.conf ... config A Service X inetd config A Service X ntp.conf /etc/* config ... LVM config B ... /etc/snmp/* config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf rsyslog smb.conf Postpath Unified Node (PUN) sshd ntp.conf config .... ... config A Service Y DR:BD config B ntp.conf inetd Front-End Server /etc/passwd inetd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... .... ... config A /etc/snmp/* ADM Front-End Server /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* LUN RPC /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd config resolv.conf ... ... /etc/* config ... LVM config B rsyslog .... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* config LVM DR:BD /etc/logrotate/* /etc/passwd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog /etc/passwd config resolv.conf ... ... rsyslog inetd /etc/logrotate/* sshd /etc/snmp/* config .... rsyslog config B cluster manager LUN Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM LUN Webmail Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Service X LUN Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers config A PPSD config B cluster manager spare LUN RPC Front-End Server Service X ntp.conf SAN Webmail sshd /etc/snmp/* .... ... ... config A PPSD config config /etc/* LVM DR:BD smb.conf config A config B cluster manager sshd rsyslog /etc/snmp/* .... ntp.conf Security Appliances /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf ... config A PPSD config B inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... smb.conf ... config A PPSD cluster manager config config LVM DR:BD ntp.conf config A /etc/passwd /etc/sysconfig/* config config B Postpath Unified Node (PUN) /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* ... ... resolv.conf ntp.conf ... config A config B cluster manager Postpath Unified Node (PUN) .... LVM DR:BD resolv.conf ... /etc/sysconfig/* config /etc/* ... ... smb.conf ... config A PPSD cluster manager config config .... LVM DR:BD ntp.conf smb.conf ... config A config B cluster manager LUN inetd ntp.conf Service Y config B spare Load Balancer .... resolv.conf config A Service X config B /etc/passwd inetd /etc/logrotate/* sshd rsyslog /etc/snmp/* /etc/* ... ... resolv.conf ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config .... LVM DR:BD resolv.conf SAN Monitoring Servers /etc/* ... config A config B ... smb.conf config A Service Y config B /etc/passwd inetd config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD ntp.conf Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config config /etc/* ... ... resolv.conf ntp.conf sshd rsyslog /etc/snmp/* .... LVM DR:BD resolv.conf cluster manager config config /etc/* ... ... sshd rsyslog /etc/snmp/* .... LVM DR:BD /etc/logrotate/* config /etc/* .... ntp.conf inetd config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B Postpath Unified Node (PUN) /etc/passwd inetd config A Service X config B Postpath Unified Node (PUN) /etc/sysconfig/* config /etc/logrotate/* config /etc/* resolv.conf inetd config A Service Y config B /etc/passwd /etc/sysconfig/* config config A Service X config B /etc/passwd /etc/sysconfig/* config config A Service Y config B /etc/passwd inetd /etc/snmp/* PPSD config A Service X config B ... ... /etc/snmp/* config ... Service Y DR:BD config B ... Postpath Unified Node (PUN) config Service X smb.conf ... config LVM DR:BD /etc/logrotate/* smb.conf config A ntp.conf smb.conf ... /etc/logrotate/* sshd rsyslog /etc/passwd /etc/sysconfig/* config resolv.conf sshd ntp.conf config LVM smb.conf /etc/sysconfig/* config inetd rsyslog ... /etc/passwd inetd .... ... ... .... resolv.conf smb.conf Service Y Postpath Unified Node (PUN) /etc/snmp/* config /etc/* ... config A ... config B config DR:BD /etc/snmp/* config ... LVM resolv.conf LVM /etc/logrotate/* smb.conf smb.conf Service X /etc/sysconfig/* config sshd Front-End Server /etc/* inetd .... ... config A Service Y DR:BD config B ntp.conf /etc/passwd inetd rsyslog /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf Front-End Server /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf ADM /etc/passwd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B ntp.conf inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... /etc/* config ... LVM resolv.conf config B ... /etc/snmp/* inetd /etc/logrotate/* config Front-End Server /etc/* rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf .... ... config A Service Y DR:BD config B sshd ntp.conf config /etc/* config ... LVM inetd /etc/sysconfig/* config sshd /etc/snmp/* config /etc/passwd /etc/sysconfig/* ... ... RPC /etc/passwd inetd /etc/logrotate/* config rsyslog .... config resolv.conf LVM DR:BD /etc/logrotate/* smb.conf Front-End Server /etc/* config /etc/passwd /etc/sysconfig/* inetd /etc/sysconfig/* config sshd /etc/snmp/* Front-End Server /etc/* rsyslog .... config resolv.conf ... ... config A inetd /etc/logrotate/* sshd /etc/snmp/* config Front-End Server /etc/passwd /etc/sysconfig/* config /etc/logrotate/* config Front-End Server /etc/* LVM Front-End Server /etc/passwd /etc/sysconfig/* config sshd DR:BD Webmail Front-End Server /etc/passwd inetd rsyslog Billing Mgt Servers
    • “Turtles all the way down” Even individual systems are not like your developer’s workstation: physical hostname service hostname hostname eth0 physical /etc/passwd inetd config sshd config NIC /etc/sysconfig/* /etc/logrotate/* rsyslog LVM eth0 inetd /etc/snmp/* / /etc/* config config ... sshd config rsyslog config LVM hd ntp.conf ... NIC /etc/sysconfig/* ... /etc/snmp/* / /etc/* .... eth1 mgt eth2 iSCSI, NFS resolv.conf smb.conf DR:BD ntp.conf ... ... smb.conf ... Service Y eth0:1 cluster /etc/logrotate/* .... resolv.conf DR:BD /etc/passwd / config A config B Service Y config A config B /var.. SAN inst1 © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) inst2
    • Deployment Environment is Complex • Many nodes in cluster = many “points” to monitor, including systems and services • Dependencies on external services (DNS, Active Directory, SAN) • Dependencies within application stack • Sequencing of operations critical © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • So, lets just use PaaS! $ cf deploy appname Container App A © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Uh,Yeah. HTTPS HTTP Internet LB/ SSL CF Router UAM REST Server Ubuntu 12.04 Ubuntu 12.04 NATS HM Cloud Controller Ubuntu 12.04 C/C: NATS/ HTTP HTTP/S Data/Msg Bus Container Container App A App B Container Container App A' DEA App B' DEA 1 DEA 1 Ubuntu DEA 1 12.04 Ubuntu 12.04 Ubuntu 12.04 © 2013 Cisco and/or its affiliates. All rights reserved. VHOST A VHOST B RabbitMQ RabbitMQ CentOS 6.4 CentOS 6.4 Reinhardt Quelle (rquelle@cisco.com) Keysp. A Keysp. B Cassandra Cassandra CentOS 6.4 Cassandra CentOS 6.4 Cassandra CentOS 6.4 CentOS 6.4
    • I’m not going to discuss application architecture in detail • Designing for Always-on Services: • We don’t ever stop serving customers (intentionally, anyway) • Avoiding the Fallacies of Distributed Computing: Emphasized by load and geography • Embracing Failure: At scale, something is ALWAYS breaking • Designing for the real world Circuit breakers, exponential fallback, dynamic configuration • The need for loosely coupled services: Multi-service synchronized updates are extraordinarily costly Read Web Operations (Alspaw, et al), Release It! (Nygaard) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Running Bits in Production 1 Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Ow! A wall! Alerts 8 How did we end up at that wall? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Note: “Monitoring” is multifaceted Where do we want the bits to run? Deployment Targets 2 How do they get there? Packaging/Distribution 3 How are they supposed to behave? Configuration 4 How do we start and stop them? Command/Control (aka Orchestration) 5 Are we sure they are running (or ready to run)? Health Checks and Verification 6 When will we run into a wall? Metrics 7 Was that a wall we just hit? Alerts 8 What just happened there? Logs © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) Monitoring 1
    • 1 Where does a package run? Service ≠ Server • Multiple servers may run same service • A single server may run multiple services even several of the same kind (esp. 32bit) © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • One Service, Multiple Hosts LB Front-End Server Front-End Server /etc/passwd inetd config /etc/sysconfig/* /etc/passwd inetd config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/logrotate/* sshd config rsyslog config LVM /etc/snmp/* ... /etc/* .... ... ntp.conf /etc/snmp/* /etc/* resolv.conf DR:BD /etc/sysconfig/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... Service X © 2013 Cisco and/or its affiliates. All rights reserved. config A config B smb.conf ... Service Y config A config B Reinhardt Quelle (rquelle@cisco.com) Service X config A config B Service Y config A config B
    • Multiple Services, One Host Application Compute Node /etc/passwd inetd config sshd config rsyslog config LVM /etc/sysconfig/* ... /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... App Instance © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com) App Instance config A' config B'
    • Deployment Target Choices Where bits run: Shared system May depend on: Virtual Machine Balancing Resources Dedicated server Failover Strategies Security or Isolation Changes over time “Refactoring your Infrastructure” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • 2 Packaging/Distribution • How is software Bundled, Transported and Unpacked/Installed • Goal is to get a specific version of bits extracted onto a system in a predictable way with its dependencies © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Distribution Choices • Recommended: • versioned and signed OS package (RPM, Deb) from controlled repository • puppet “code” is in package, too - puppetfile is source • Less desirable: • direct version control repository (svn, git) checkout • scp tarball • Note there are different “units of deployment” • Images, Packages, Applications © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Configuration 3 • How is this instance supposed to behave? • How is this server supposed to be configured? • May be multiple instances on a given server • Configuration is often changed over time • Consider “static” and “dynamic” configuration © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Standard OS Platform Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • + Application Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com)
    • Fail Postpath Unified Node (PUN) /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ntp.conf ... smb.conf ... PPSD © 2013 Cisco and/or its affiliates. All rights reserved. config A config B Reinhardt Quelle (rquelle@cisco.com) uh, oh...
    • Application Configuration • Unique per application instance • Does not overwrite system/environment • Application switches: apache2 -f nginx -c postfix -c Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle (rquelle@cisco.com)
    • Various classes of configuration • Static (ish) • Environments: Dev, QA, Staging, Production • Data Centers: SJC, DFW, NRT, AMS • Clusters: ISJ1, AC • Use Puppet/Chef/cfengine • Dynamic • Cluster membership • Service hosts: DB, Logs, Metrics, Identity • Use Zookeeper/Noah/Eureka/DNS - MX, GSS/Anycast © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Common Practice: Configuration Management Compute Node configuration daemon Machine Spec config sshd config rsyslog ... DR:BD ... /etc/sysconfig/* config LVM Cluster Spec -* - TOSCA - CloudFormation - Heat /etc/passwd inetd /etc/logrotate/* /etc/snmp/* /etc/* + Environmental Data - Puppet Hiera - Chef Databags .... resolv.conf configuration master ntp.conf smb.conf ... - Puppet Modules - Chef Cookbooks App config A config B Compute Node configuration daemon /etc/passwd inetd config sshd config rsyslog config LVM ... DR:BD ... /etc/sysconfig/* /etc/logrotate/* /etc/snmp/* /etc/* .... resolv.conf ntp.conf smb.conf ... App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) config A config B
    • Need a new Dev/QA test environment? Compute Node /etc/passwd configuration daemon inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf -* - TOSCA - CloudFormation - Heat Machine Spec DR:BD mstr A Cluster Spec smb.conf ... App + Environmental Data ntp.conf ... config A config B Compute Node /etc/passwd configuration daemon - Puppet Hiera - Chef Databags inetd /etc/sysconfig/* config /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* - Puppet Modules - Chef Cookbooks .... resolv.conf DR:BD ntp.conf ... smb.conf ... App config A config B Environmental Data Compute Node - Puppet Hiera - Chef Databags configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD mstr B ... ntp.conf smb.conf ... App config A config B Compute Node configuration daemon /etc/passwd inetd config /etc/sysconfig/* /etc/logrotate/* sshd config rsyslog config LVM ... /etc/snmp/* /etc/* .... resolv.conf DR:BD ... ntp.conf smb.conf ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com) App config A config B
    • Make applications configurationfriendly apache2 -f /path/to/config nginx -c /path/to/config postfix -c /path/to/config Select Config apachectl configtest nginx -t postfix check Verify Config Reload Changed Config Dump Config © 2013 Cisco and/or its affiliates. All rights reserved. apachectl graceful nginx reload postfix reload postconf Reinhardt Quelle (rquelle@cisco.com)
    • Make configuration cooperative and configuration management friendly: logrotate include /etc/logrotate.d monit include /etc/monit.d sudo #include /etc/sudoers.d Sensu © 2013 Cisco and/or its affiliates. All rights reserved. “Deep Merge” of JSON config Reinhardt Quelle (rquelle@cisco.com)
    • Command and Control (aka “orchestration”) 4 • Starting and stopping services usually requires specific sequence • By definition, cluster has multiple systems on which processes must be controlled • • • Sequencing must be enforced, repeatable Dependencies must be managed Ad-hoc processes may be required on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Many tools available • Fabric • Capistrano • MCollective • RunDeck • And many, many more © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • So now we’re running... Sure about that? © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Are We Alive? 5 • • • • • Health Checks provide specific status Often variable depth: • shallow/frequent, deep/less frequent Many Consumers • Cluster managers, load balancers, process monitors Personal favorite - “Pre-Flight checks” Call on demand © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Now you can run, but... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Where are we headed? 6 • • • Metrics help us understand trends and predict future... • Operations team can deal with server, environment or give us historical context for some event. Collected by central tools for every device in environment Monitoring OS ? Network Equipment App © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Application Metrics • Metrics used in development for profiling often continue to be interesting in production • Highly specific to a specific service: number of open connections quota usage message duration message size distribution AD Lookup response time, uptime message distribution across users/ folders cache hit/miss rate MAPI operation count © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Example Application Metrics Apache $ apache2ctl status memcached $ echo -ne "statsrn" | nc -i1 localhost 11211 ... STAT curr_connections 2 STAT total_connections 71072406 STAT connection_structures 11 STAT cmd_get 58419832 STAT cmd_set 4127175 STAT get_hits 53793350 STAT get_misses 4626482 ... squid [calh@photocache3 ~] $ /usr/sbin/squidclient -p 80 cache_object://localhost/5min ... client_http.all_median_svc_time = 0.000911 seconds client_http.miss_median_svc_time = 0.042766 seconds client_http.nm_median_svc_time = 0.000000 seconds client_http.nh_median_svc_time = 0.024508 seconds client_http.hit_median_svc_time = 0.000911 seconds ... mysql mysql> SHOW SLAVE STATUS G ... Connect_retry: 60 Master_Log_File: db1-bin.439 Read_Master_Log_Pos: 374103193 Relay_Log_File: dbslave-relay.125 Relay_Log_Pos: 854427197 Replicate_ignore_db: Last_errno: 0 Last_error: Skip_counter: 0 Exec_master_log_pos: 587702364 Relay_log_space: 854431912 ... © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Again, many tools available • JMX • NewRelic • statsd (extremely light-weight and easy to use) • snmp (highly standard, but high barrier to use and configuration for application layer) • collectd + graphite © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • 7 Alerts • Actionable • May be derived from other sources, or raised directly. • Have to be routed to be useful - thank you, Sean! • The application teams will get these! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • 8 What have you been up to? • • Logging key for debugging, tracing, compliance, etc. Logs should be collected off-box for multiple constituents, including: Operations, Support, InfoSec/Compliance, Engineering • Maximum utility when levels manageable at runtime without application restart • Logstash, Splunk... no excuses © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Logs vs. Stats • Logs can be fall-back facility for monitoring of health and metrics But this is less than desirable - Shifts development to ops, usually late in the game - Can be brittle Usually less efficient © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Logging in Production • Clustered, parallel processes require host and process IDs • Activity flows across services - how do we correlate? • • Audit logs important - who did what when? • Passwords must never be logged Keep privacy in mind - if logs have private data, we can’t easily take them out of production. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • “Wow, that’s a lot we have to build!” © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Or not. © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • If applications provide standard management interfaces... Logging Metrics App Health Process Mgt Config © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • ...they can “plug in” to existing infrastructure Log Host OS Logging Logging Metrics Health Process Mgt Stats Host Metrics Logging Alert Manager Metrics App Health Health Process Mgt Cluster/Orchestration Manager CM Config Config Configuration Manager Config Config VCS Config Engineering © 2013 Cisco and/or its affiliates. All rights reserved. Operations Reinhardt Quelle (rquelle@cisco.com)
    • Operation’s Asks • Make Operability an integral part of product requirements and designs • Engage your operations team as soon as possible to let us help you leverage our platform and services • Talk with us about HA architectures: we don’t have a magic wand! © 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle (rquelle@cisco.com)
    • Thank you! www.cisco.com/careers © 2010 Cisco and/or its affiliates. All rights reserved. 75