Neustar Cassandra Experience


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Neustar Cassandra Experience

  1. 1. Neustar Cassandra Experience11/30/11Chirag Patel © Neustar, Inc. / Proprietary and Confidential
  2. 2. Who is Neustar?2 © Neustar, Inc. / Proprietary and Confidential
  3. 3. UltraDNS Overview» Cloud based DNS Services » Managed External DNS » Basic DNS » Traffic Management Services » UltraDDI - Managed Internal DNS, DHCP & IPAM » Ultra Internet Gateway – Recursive DNS » SiteProtect – DDOS Mitigation Service
  4. 4. Existing design» External applications inject DNS data » UI / SOAP-API / AXFR / IXFR » Millions of record changes per day» Oracle as persistent store » Millions of zones » Hundreds of Millions of records» Oracle replicates this data into our DNS Server nodes » 15 nodes globally distributed» DNS Server answers DNS queries » 17 Billion queries per day
  5. 5. Existing design Injecting Applications Oracle Master Databases Oracle Replicate Oracle Replicate Oracle Replicate Database Database Database DNS Server DNS Server DNS Server
  6. 6. DNSSEC Requirements» What is DNSSEC? » IETF specifications for securing DNS information » Authentication of DNS answers» Support for adding DNS Security into existing infrastructure» Typical record increase of 8x per zone» Support for versioning existing and new DNS data» Support for storing DNS data before signing» DNSSEC &non-DNSSEC injecting applications» DNS Server will still read data from Oracle» Separate Beta environment for DNSSEC
  7. 7. Why Cassandra?» Write optimized» Eventually consistent» Highly scalable» Highly available even with heavy write loads» Easy to understand data model» Excellent high level, easy to use client libraries available (hector, pycassa)» Lot of resources available online » Mailing list, incl. Jonathan Ellis are extremely responsive
  8. 8. Migration to Cassandra Injecting Applications Signer Cassandra Cluster Oracle Master Databases Oracle Replicate Oracle Replicate Oracle Replicate Database Database Database DNS DNS DNS Server Server Server Node N
  9. 9. Proof of Concept» Consulting from Riptano (Now DataStax)» 2 day data design brain storming» Used Cassandra v0.7» Used Hector Cassandra client v0.7
  10. 10. Cassandra Data Design Zone Indexes Zone Account Index Zone Zone Zone Serialize ID Key Name Name VID d Zone Record Indexes ZoneVersion Zone_RTy Index Record Zone Zone pe Key ID Name VID Record ZoneRecordChanges Zone RecordID Serialized Zone VID Record Change Name Record ID Type
  11. 11. Searching Example Zone Indexes Zone V2 <Zone Data V2> 1 Type_Primary_google.c om V1 <Zone Data V1> Type_Primary_yahoo.c V2 <Zone Data V2> om V1 <Zone Data V1> Record Indexes Record Name_a1_R2 R2 A R2 <R2 Data> Name_a3_R3 R3 R3 <R3 Data> TTL_0086400_R2 R2 TTL_0096400_R3 R3
  12. 12. Versioning Example Zone V3 <Zone Data V3> V3 <Empty> V2 <Zone Data V2> V2 <Empty> V1 <Zone Data V1> V1 <Empty> ZoneRecordChanges Record V3 R4 R4 <R4 Data> R2 Del R3 <R3 Data> V2 R3 Add R1 Del V1 R1 Add R2 Add Del R4, Add Del R3, Add R2 R1 V3 V2 V2 D R3 R3 R1 R4 R2 R2
  13. 13. Performance Comparison» Significant improvement in Write times » Shotgun across multiple Cassandra nodes in a datacenter » Insertions times went from minutes to seconds» No wait between writes » Oracle replication would choke on large inserts
  14. 14. Problems faced» Versioning» Indexing/Searching» Pagination» Sync issues with Oracle» Training QA» Training OPS
  15. 15. Lessons learned» Difficult to sync Oracle & Cassandra together» Versioning is difficult, need to take different approach» Pagination might require to cache data in memory » Redis or memcached» Searching needs to be well thought of before designing data model» Need to involve Operations & QA team sooner for more feedback
  16. 16. Future Plans» Deploy Cassandra in main production environment» Simplify existing data model» Common Interface to interact with Oracle & Cassandra» Use Cassandra to store smaller set of isolated data.» Replace Oracle fully with Cassandra in one injecting application
  17. 17. Questions?Contact for details