Real-time data analytics with Cassandra at iland
•
0 likes•1,059 views
This presentation, from the 2014 Datastax Cassandra Tech Day in Houston, TX, is a quick overview on how iland Internet Solutions leverages Cassandra as its sole data storage platform for performance metrics as well as configuration across multiple-data centers in the US, EU and Asia. iland exposes these metrics to its customers trough its ECS portal application which is covering a wealth of functionality including offering visibility into resource consumption, billing, performance, alerts, the impact of change and other key areas. The platform also provides predictive analytics that help companies monitor performance, achieve consistency and anticipate growth requirements.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Real-time data analytics with Cassandra at iland
Similar to Real-time data analytics with Cassandra at iland (20)
Recently uploaded
Recently uploaded (20)
Real-time data analytics with Cassandra at iland
- 1. Real-time data analytics with C* at iland Cassandra Tech Day @ Houston, TX! October 14th, 2014! Julien Anguenot (@anguenot) !
- 2. Agenda • iland, iland platform and why C*?! • real-time data & domain constraints! • quick overview of iland C* deployment
- 4. Julien Anguenot @ Cassandra Tech Day Houston 2014 iland Internet Solutions 19 8 8 3 Years delivering Years of delivering IT Services Years cloud infrastructure & disaster recovery expertise IISO 27001 & SSAE16 global data centers Cloud-based Specializations: Production; Test & Dev; DR 4
- 5. Julien Anguenot @ Cassandra Tech Day Houston 2014 iland platform essentially • data warehouse running across multiple data-centers! • monitoring (resource consumption / performance)! • billing! • alerting! • predictive analytics! • cloud management! • cloud services (backups, DR, etc.)! • desktop and mobile applications (iland portal app) 5
- 6. Julien Anguenot @ Cassandra Tech Day Houston 2014 6
- 7. Julien Anguenot @ Cassandra Tech Day Houston 2014 7
- 8. Julien Anguenot @ Cassandra Tech Day Houston 2014 8
- 9. Julien Anguenot @ Cassandra Tech Day Houston 2014 9
- 10. Julien Anguenot @ Cassandra Tech Day Houston 2014 10
- 11. Why did we choose Cassandra? • MySQL and MongoDB attempts been big fails! • write latency (constant-time writes)! • distributed nature (multi-data centers)! • scalability, reliability, performance, availability! • sharding makes things complicated! • no master - slave means no SPOF! • simplicity! Julien Anguenot @ Cassandra Tech Day Houston 2014 11
- 12. real-time data & domain constraints
- 13. Julien Anguenot @ Cassandra Tech Day Houston 2014 Constraints • write latency! • precision (used for billing)! • availability! • multi-data center! • tens of thousands of VMs ! • agent-less! • pull (vs push) 13
- 14. Julien Anguenot @ Cassandra Tech Day Houston 2014 Pipeline • collection of real-time data! • store! • aggregation! • rollups! • processing! • alerting! • reporting! • querying 14
- 15. Julien Anguenot @ Cassandra Tech Day Houston 2014 Data sources • VMware infrastructure stack (each location)! • vCloud Director, vCenter, vShield Manager! • network statistics! • AMQP, Syslog-ng, Web Services! • Salesforce! • Veeam, Zerto, more … 15
- 16. Julien Anguenot @ Cassandra Tech Day Houston 2014 Real-time metrics! • 20 seconds samples! • dozens of performance counters per entity (VM, VNIC, etc.)! • time series! • (timestamp, value)! • metadata! • entity (network, vm, etc.)! • unit, etc.! • then 1min, 1h, 1d, 1w and 1m historical rollups 16
- 17. Overview of iland C* deployment
- 18. Julien Anguenot @ Cassandra Tech Day Houston 2014 C* iland’s cluster (1/2) • one (1) C* cluster ! • 6 data centers - 25+ nodes - one (1) keyspace 18
- 19. Julien Anguenot @ Cassandra Tech Day Houston 2014 C* iland’s cluster (2/2) • each DC (1 or 2 racks of 3 nodes)! • 3 nodes per location for writes! • replication Factor (RF) = 3 ! • 3 nodes per location for API reads (Dallas, TX, London, UK, Singapore)! • each node! • VM (vCenter powered) - Ubuntu 14.04 LTS w/ Cassandra base dpkg! • 16GB of RAM / 16 vCPUs! • currently: ~ 1TB of data per node! • not using SSD (yet) 19
- 20. Asia Data centers and replication (1/2) Julien Anguenot @ Cassandra Tech Day Houston 2014 20 US LA,CA Reston, VA Dallas, TX Singapore EU London,UK Manchester,UK
- 21. Data centers and replication (2/2) iland core platform iland core platform Julien Anguenot @ Cassandra Tech Day Houston 2014 21 iland ReST API C* W iland ReST API C* R C* W C* R C* R only deployed in: Dallas, TX - London, UK - Singapore
- 22. US API EU API Asia API EU portal Asia portal Citrix Netscaler (US) Julien Anguenot @ Cassandra Tech Day Houston 2014 Access to data (read / write) 22 US portal https://my.ilandcloud.com/
- 23. Julien Anguenot @ Cassandra Tech Day Houston 2014 At the application level? • History! • POC: C* 1.2 + Astyanax (2013/02)! • V1: C* 2.0 + Astyanax w/ CQL3 over thrift (2013/06) ! • Current version: C* 2.0 + DataStax CQL Java driver! • Java / JEE cluster (Wildfly AS)! • AMQP / RabbitMQ! • Redis to cache up “hot” data (read latency)! • Python & DataStax CQL driver for cluster / data upgrades 23
- 24. Slides ! @ ! http://www.slideshare.net/anguenot/cassandra-tech-dayhou2014 http://www.iland.com/enterprise-cloud-services- portal/! @ilandcloud! @anguenot