Behind the Wizard’s Curtain: Scalability and Security at Zuora (Subscribed13)


Published on

Ever wonder what's in the Zuora cloud? Join us and learn how Zuora has built a scalable and secure cloud based subscription billing management service. Hear from scalability, security and operations engineers and have your questions answered.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Behind the Wizard’s Curtain: Scalability and Security at Zuora (Subscribed13)

  1. 1. Behind the Wizard’s Curtain: Scalability and Security at Zuora Subscribed 2013 Thomas Fou Information Security & Compliance Levon Stepanian Performance Engineering &
  2. 2. Key 9 •  Key 9 •  A Day in the Life of Zuora •  Behind the Curtain: The Zuora Cloud & Platform •  Zuora’s Investment in Performance •  Security & Compliance •  Q/A AGENDA
  3. 3. Key 9 Key 9 •  9 Keys to Subscription Success •  Key 9: Mission Critical, Reliable, Scalable & Secure “Subscription businesses want a reliable ‘enterprise-grade’ system with services that are built on a secured, mission-critical, and scalable infrastructure. The SaaS Subscription Infrastructure must have reliable 7x24x365 operations, regular new feature deployments, and well-formed, predictable business continuity”
  4. 4. Key 9 Disclaimer Pt. 1 •  This  session  may  contain  product  features  that  are  currently  under   development.   •  This  session/overview  of  new  technology  represents  no  commitment   from  Zuora  to  deliver  these  features  in  generally  available  product.   •  Customers  who  purchase  services  should  make  the  purchase   decisions  based  upon  features  that  are  currently  available.     •  Technical  feasibility  and  market  demand  will  affect  final  delivery.   •  Pricing  and  packaging  for  any  new  technologies  or  features  discussed   or  presented  have  not  been  determined.  
  5. 5. Key 9 Disclaimer Pt. 2 Zuora Confidential Not for distribution beyond the intended audience at Subscribed 2013  
  6. 6. Key 9 A Day in the Life of Zuora •  Monthly Synchronous Transaction Volumes –  SOAP and REST APIs •  Subscription Management –  550K created –  1M amended –  Half a Billion API calls a month (15M/day) –  > 80% are queries •  Top 3 Objects: Subscriptions, Products, RatePlans
  7. 7. Key 9 A Day in the Life of Zuora •  Monthly Asynchronous Transaction Volume (Aug/2013) •  39M  total  asynchronous  transac0ons   •  ~50%  during  1st  day  &  last  4  days  of  month   •  Transac0on  mix  –  small  &  large   •  Monthly  varia0on   6M   1M   6K   ?K   336M   15M   INVOICE  
  8. 8. Key 9 Performance Data •  Benchmark Data –  375 orders/sec for a single tenant •  2012 Amazon Cyber Monday Peak Rate: 306 items/sec –  150+ payment authorizations/sec for a single tenant •  Production Data –  50% of our tenants -> 70K invoices per hour –  Tenant generating 1.7M invoices in a single bill run
  9. 9. Key 9 L B AMQ   DB  (M)   Behind the Curtain: The Zuora Cloud L B Billing  &   Payment   Servers   UI/API   Server   Global  (S)   PDFGen   Servers   Web   Server   Zuora  for   Salesforce   Servers   Messaging  Infrastructure   File   Storage   Global  (M)   Tenant   Shard   Tenant   Shard  (M)   Tenant   Shard  (M)   Tenant   Shard  (S)   File   Storage   AMQ   DB  (S)   (M)aster/(S)lave   RO  Replicas  (Not  Shown)   F W Security   Appliance  
  10. 10. Key 9 L B AMQ   DB  (M)   Scaling Zuora L B Billing  &   Payment   Servers   UI/API   Server   Global  (S)   PDFGen Servers   Web   Server   Zuora  for   Salesforce   Servers   Messaging  Infrastructure   File   Storage   Global  (M)   Tenant   Shard   Tenant   Shard  (M)   Tenant   Shard  (M)   Tenant   Shard  (S)   File   Storage   AMQ   DB  (S)   F W Security   Appliance  
  11. 11. Key 9 Infrastructure Scalability –  Enterprise Ready Tier 1 Data Center •  Switch SuperNAP (Las Vegas) •  High density, state of the art infrastructure •  Super beefy hardware, storage and networking gear •  7x24x365 resource monitoring and alerting 10   90   Avg.  Produc0on     Capacity  U0liza0on   uMlized   idle   –  Plenty of standby spare capacity to accommodate growth •  Max utilization ~ 30%
  12. 12. Key 9 Zuora’s Investment in Performance •  Bottlenecks are everywhere! •  Zuora’s massive & continual investment in performance –  Search & Destroy philosophy adopted by all teams –  Refactoring/optimizing code –  Production-like environment profiling/analysis –  Better aligning s/w and h/w architectures –  Investment in state of the art technology
  13. 13. Key 9 Customer Facing Performance Improvements •  Rating & Billing Engine (RBE) TurboBooster •  1.6X to 25X Bill Run speedup in production •  Optimizing & minimizing # of queries •  More charges/subscription = Larger speedups •  Zuora for Salesforce 360 TurboSync •  Up to 50X 360 Sync speedup in production •  Exploiting parallel pipelines, Bulk Salesforce APIs •  More objects to sync = Larger speedups
  14. 14. Key 9 Customer Facing Performance Improvements •  Zuora for Salesforce 360 TurboSync 50X  
  15. 15. Key 9 ?   Customer Facing Performance Improvements •  Orders/sec Capacity Improvements •  Continuous infrastructure improvements •  Code re-factoring, optimizations Orders/sec   (Peak  Cap.)  
  16. 16. Key 9 Customer Testimonials •  “After directly engaging with Zuora Engineering on a looming requirement to support a large increase in scale, I was reassured by how quickly they moved to support the stated 100tps for creation of hosted payment methods. I was impressed that Zuora Engineering then applied the 100tps requirement to the other API calls, in anticipation of downstream increases in volume. If a difficult requirement arises in the future, I will have no hesitation in directly engaging Zuora Engineering again, knowing that it will be a job well done.” – Architect •  "We're really excited about Zuora's improvements to Bill Run execution times. As a business that bills hundreds of thousands of transactions in each bill run, it's important for to generate invoices rapidly and get paid as quickly as possible" said Rick Moore, VP Finance & Business Operations at "The latest performance improvements have significantly reduced our scheduled bill run times by over 50%--that's a huge improvement, and attests to the fact that as our business grows, Zuora continues to scale to accommodate that growth.” •  “We named it TurboSync because of how fast it copied 3million records into our system. What would normally would have taken days instead took a few hours!” said Cathy MacDonald, Executive Vice President of IT at Xplornet Communications Inc. Anonymous     Zuora  Customer  
  17. 17. Key 9 Zuora Compliance •  PCI Level 1 Compliant •  SSAE16 SOC 1 Type 2 Compliant •  TRUSTe Certified •  US-EU Safe Harbor
  18. 18. Key 9 Zuora Security •  Physical Security –  World-class primary and backup datacenters –  Switch SuperNAP – PCI and SSAE16 SOC1/SOC2/SOC3 –  CoreSite – SSAE16 SOC1 Compliant •  Network Security –  Production environment completely separate –  Firewall and network zone segregation –  Two-factor authentication remote access •  Application Security –  HTTPS for all incoming/outgoing data transfer –  CC data encrypted using AES-256 SafeNet FIPS certified hardware encryption –  Application security testing
  19. 19. Key 9 Zuora Security •  Vulnerability Management –  Qualys Internal/External Network Scans –  WhiteHat Security Application Scans –  Coalfire Web Application Penetration Testing –  Monitor CVE, NIST, vendor vulnerability lists –  Apply critical patches monthly
  20. 20. Key 9 Zuora Data Flow
  21. 21. Key 9 How To Reduce PCI Scope •  Each entity responsible for how it uses data •  Limit where PCI data is stored, processed, transmitted •  Segment cardholder data network from other networks •  Use effective encryption •  Implement strong key management practices •  Limit scope of Cardholder Data Environment (CDE) •  Zuora Hosted Payment Method (HPM)
  22. 22. Key 9 End - QA
  23. 23. Key 9 Appendix
  24. 24. Key 9 Scalability Case Study •  Customer Profile –  Leading global news company –  Top UK newspaper publisher •  Subscription Launch –  Digital paper (smartphones, tablets, online) –  Access to breaking news (24x7) –  Fantasy soccer –  Apps for streaming soccer matches
  25. 25. Key 9 Scalability Case Study •  Performance Requirement –  100 transactions per second customer acquisition rate •  360,000 customers an hour! •  End to end testing uncovered bottlenecks –  Internal and External to Zuora •  Outcome –  Configuration tuning –  Horizontally scaled back-end servers –  Enhanced monitoring + alerting around launch dates