SlideShare a Scribd company logo
1 of 63
Download to read offline
Leveraging C* for real-time multi-dc public cloud analytics
Julien Anguenot
VP Software Engineering
@anguenot
1 iland cloud story & use case
2 data & domain constraints
3 deployment, hardware, configuration and architecture
overview
4 lessons learned
5 future platform extensions
3
iland cloud story & use case
Who are we?
• public, private, DRaaS, BaaS cloud provider
• Cisco CMSP
• VMware Vspp for 7+ years
• 20+ years in business
• HQ in Houston, TX
• http://www.iland.com
4
Yet another cloud provider? Well, …
5
• performance and stability
• custom SLA
• compliance
• security
• DRaaS
• global datacenter footprint: US, UK and Singapore
• dedicated support staff!
• iland cloud platform, Web management console and API
The iland cloud platform
6
iland cloud platform essentially
• data warehouse running across multiple data-centers
• monitoring (resource consumption / performance)
• billing (customers and internal use)
• alerting
• predictive analytics
• cloud management
• cloud services (backups, DR, etc.)
• desktop and mobile management consoles
• API
• Cassandra powered!
7
The iland cloud Web management console
8
9
10
11
12
13
14
15
16
17
So, why did we do all this?
• Initial motivations (v1)
• vendor software (VMware vCloud Director) lacking:
• performance analytics (real-time and historical)
• billing
• alerts
• cross datacenter visibility
• more private cloud type transparency
• abstract ourselves from vendors and integrate an
umbrella of heterogeneous services
• modern UX and good looking UI
18
19
data and domain constraints
Constraints
20
• write latency
• high throughput
• precision (used for billing)
• availability
• multi-data center
• scalability: tens of thousands of VMs
• agent-less
• pull/poll vs push
• high latency environs (multi-dc)
Pipeline
21
• collection of real-time data
• store
• aggregation
• correlation
• rollups (historical)
• processing
• alerting
• billing
• reporting
• querying
Real-time collected perf counters
22
• 20 seconds samples
• compute, storage, network
• 15+ perf counters collected
• ~50 data points per minute and per VM
• time series
• (timestamp, value)
• metadata
• unit
• interval
• etc.
• 1 year TTL
VM CPU 20 seconds perf counters
23
Group Name Type
CPU USAGE AVERAGE
CPU USAGE_MHZ AVERAGE
CPU READY SUMMATION
VM memory 20 seconds perf counters
24
Group Name Type
MEM ACTIVE AVERAGE
MEM CONSUMED AVERAGE
MEM VM_MEM_CTRL SUMMATION
VM network 20 seconds perf counters
25
Group Name Type
NET RECEIVED AVERAGE
NET TRANSMITTED AVERAGE
NET USAGE AVERAGE
VM disk 20 seconds perf counters
26
Group Name Type
DISK READ AVERAGE
DISK WRITE AVERAGE
DISK MAX_TOTAL_LATENCY LATEST
DISK USAGE AVERAGE
DISK PROVISIONED LATEST
DISK USED LATEST
DISK NUMBER_WRITE_AVERAGED AVERAGE
DISK NUMBER_READ_AVERAGED AVERAGE
More counters collected for 3rd party services
27
VM to time serie bindings
28
• binding on VM UUID
• serie UUID
• <VM_UUID>:disk:numberReadAveraged:average
• Simple, fast and easy to construct at application level.
29
30
VM containment and aggregation of real-time samples
31
• what’s this?
• resource pool / vs instance-based $$
• 20 seconds samples aggregated
from VM to VDC top level
• separated tables
Historical rollups and intervals
32
• VM, VAPP, VDC, ORG and network
• 1 minute (TTL = 1 year)
• 1 hour (used for billing)
• 1 day
• 1 week
• 1 month
• separated tables
• new performance counter types created
• TTL > 3 years for 1h samples for compliance & billing reasons
• application level responsibilities
1 minute rollups processing
33
• processed to trigger alerts (usage, billing)
• processed to compute real-time billing
1 hour rollups processing
34
• processed for final billing computation
• leveraging salesforce.com collected data
Data sources essentially
35
• compute
• storage
• network
• Management
• users
• cloud configuration
• salesforce.com
• 3rd party services: backups, DR, etc.
• pluggable: add / upgrade / remove services
Cassandra is the sole record keeper
36
37
deployment, configuration, hardware
and architecture overview
iland cloud platform foundation
38
• Cisco UCS
• VMware ESXi
• VMware vSphere (management)
• our Cassandra cluster runs on the exact same base
foundation as our customer public clouds.
39
Simplified architecture (each DC)
HAProxy Apache
KeyCloak
Wildfly AS
Postgres
Wildfly AS
Resteasy API
Wildfly AS
cluster
Apache
Lucene
NFS
Apache
Cassandra
Compute
Storage
Network
+ 3rd parties
Salesforce
iland cloud
Cassandra ring
API
AngularJS / API
Redis
Sentinel
AMQP
syslog-ng
Cassandra version history
40
• late 2014: 2.1.x
• early 2014: 2.0.x w/ Java CQL driver
• late 2013: 2.0 beta w/ Astanyax (CQL3) (v1)
• empty cluster
• early 2013: 1.2.x w/ Astanyax (initial proto)
iland’s cassandra cluster overall
41
• 6 datacenters
• 1 (one) keyspace
• 27 nodes
• 1.5 to 2TB per node (TTL)
42
Reston, VALA,CA
Dallas, TX
US
Singapore
Asia
London,UK
Manchester,UK
EU
Each DC
43
• 1 or 2 C* rack(s) of 3 Cassandra nodes
• endpoint_snitch: RackInferringSnitch
• RF=3
44
Each node
45
• VM
• Ubuntu 14.04 LTS
• Apache Cassandra Open Source distribution
• 32GB of RAM
• 16 CPUs
• 3 disks: system, commit logs, data
Hardware
46
• Cisco UCS B200 M3
• not very expensive
• Disks
• Initially 10K SAS disks
• now hybrid array (accelerated SSD)
• reads off SSD (75/25)
• boot time
• maintenance ops
• Cassandra CPU and RAM intensive.
• No need to get crazy on disks initially
• C* really runs well on non-SSD
Network
47
• 1G and 10G lines (currently switching all to 10G)
• Cassandra chatty but performs well in high latency
environs
• network usage is pretty much constant
• 25 Mb/s in between DC:
• default C* 2.1 outbound throttle
• Increase when streaming node is needed
• Permanent VPN in between DC (no C* SSL)
Network
48
ultimately an API for everything and everywhere
49
50
C* W
iland ReST API
iland core platform iland core platform
iland ReST API
C* R C* RC* W
C* R only deployed in: Dallas, TX - London, UK - Singapore
51
Lessons learned
Tuning Cassandra node: JVM
52
• Java 8
• MAX_HEAP_SIZE=“8G”
• HEAP_NEWSIZE=“2G”
• Still using CMS but eager to switch to G1 w/ latest
Cassandra version.
• no magic bullet
• test and monitor
• 2.0.x to 2.1.x: had to revisit drastically
Tuning Cassandra node: some config opts
53
• concurrent_writes / concurrent_reads
• nodetool tpstats
• concurrent_compactors
• nodetool compactionstats
• ++
• auto_snapshot
• batch_size_warn_threshold_in_kb
• monitor
• no magic bullet
• test and monitor
Minimize C* reads (with Redis in our case)
54
• writes are great / reads are good
• application level optimizations
• 16G of cached data in every DC
• very little in Redis. Bindings and alerts
• in-memory only (no save on disk)
Migration
55
• went live with 2.1.1 because of UDT
• suggest waiting for at least 5 or 6 dot releases
• 2.0.x / 2.1.x
• have to re-tune the whole cluster
• new features can be an issue initially (drivers)
• Python driver very slow for data migration
Don’t’s
56
• secondary indexes (or make sure you know what you’re doing)
• IN operator
• don’t forget TTL
• no easy way around range deletes
• complex “relational” type of models
Do’s
57
• design simple data model
• queries driven data model
• writes are cheap: duplicate data to accommodate queries
• prepared statements
• batches
• minimize reads from C*
• UDT
#pain
58
• bootstrapping new DC
• streaming very hard to complete OK w/ 2.0
• temp node tuning during streaming
• Cassandra 2.2 should help with bootstrap resume
• repairs
• very long and costly op
• incremental repairs broken until late 2.1.x
59
future platform extensions
Issue with in-app server aggregations and rollups
60
• JEE container works great but…
• lack of traceability / monitoring around jobs
• separation of concerns
• need to minimize reads against Cassandra
• in-memory computation
• code base growing fast (200k+ Java loc)
Spark for aggregations and rollups
61
• tackling issues in previous slides
• multiple new use cases:
• for instance, heavy throughput data for network
analysis
• machine learning
• Kafka & Spark Streaming
• currently experimenting
Multiple Keyspaces
62
• compliance / data isolation
• lower network traffic
Thank you

More Related Content

What's hot

Understanding Cassandra internals to solve real-world problems
Understanding Cassandra internals to solve real-world problemsUnderstanding Cassandra internals to solve real-world problems
Understanding Cassandra internals to solve real-world problems
Acunu
 

What's hot (20)

Lightweight Transactions in Scylla versus Apache Cassandra
Lightweight Transactions in Scylla versus Apache CassandraLightweight Transactions in Scylla versus Apache Cassandra
Lightweight Transactions in Scylla versus Apache Cassandra
 
Advanced Operations
Advanced OperationsAdvanced Operations
Advanced Operations
 
Live traffic capture and replay in cassandra 4.0
Live traffic capture and replay in cassandra 4.0Live traffic capture and replay in cassandra 4.0
Live traffic capture and replay in cassandra 4.0
 
Scylla Summit 2016: Scylla at Samsung SDS
Scylla Summit 2016: Scylla at Samsung SDSScylla Summit 2016: Scylla at Samsung SDS
Scylla Summit 2016: Scylla at Samsung SDS
 
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...
 
Building and running cloud native cassandra
Building and running cloud native cassandraBuilding and running cloud native cassandra
Building and running cloud native cassandra
 
Webinar: How to Shrink Your Datacenter Footprint by 50%
Webinar: How to Shrink Your Datacenter Footprint by 50%Webinar: How to Shrink Your Datacenter Footprint by 50%
Webinar: How to Shrink Your Datacenter Footprint by 50%
 
Oracle to Cassandra Core Concepts Guid Part 1: A new hope
Oracle to Cassandra Core Concepts Guid Part 1: A new hopeOracle to Cassandra Core Concepts Guid Part 1: A new hope
Oracle to Cassandra Core Concepts Guid Part 1: A new hope
 
GumGum: Multi-Region Cassandra in AWS
GumGum: Multi-Region Cassandra in AWSGumGum: Multi-Region Cassandra in AWS
GumGum: Multi-Region Cassandra in AWS
 
How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)
 
PagerDuty: Span the WAN? Yes you can!
PagerDuty: Span the WAN? Yes you can!PagerDuty: Span the WAN? Yes you can!
PagerDuty: Span the WAN? Yes you can!
 
Safer restarts, faster streaming, and better repair, just a glimpse of cassan...
Safer restarts, faster streaming, and better repair, just a glimpse of cassan...Safer restarts, faster streaming, and better repair, just a glimpse of cassan...
Safer restarts, faster streaming, and better repair, just a glimpse of cassan...
 
Building Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and KafkaBuilding Event Streaming Architectures on Scylla and Kafka
Building Event Streaming Architectures on Scylla and Kafka
 
Understanding Cassandra internals to solve real-world problems
Understanding Cassandra internals to solve real-world problemsUnderstanding Cassandra internals to solve real-world problems
Understanding Cassandra internals to solve real-world problems
 
Macy's: Changing Engines in Mid-Flight
Macy's: Changing Engines in Mid-FlightMacy's: Changing Engines in Mid-Flight
Macy's: Changing Engines in Mid-Flight
 
RedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis codeRedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis code
 
Cassandra CLuster Management by Japan Cassandra Community
Cassandra CLuster Management by Japan Cassandra CommunityCassandra CLuster Management by Japan Cassandra Community
Cassandra CLuster Management by Japan Cassandra Community
 
Scylla Summit 2016: Analytics Show Time - Spark and Presto Powered by Scylla
Scylla Summit 2016: Analytics Show Time - Spark and Presto Powered by ScyllaScylla Summit 2016: Analytics Show Time - Spark and Presto Powered by Scylla
Scylla Summit 2016: Analytics Show Time - Spark and Presto Powered by Scylla
 
Building Scalable, Real Time Applications for Financial Services with DataStax
Building Scalable, Real Time Applications for Financial Services with DataStaxBuilding Scalable, Real Time Applications for Financial Services with DataStax
Building Scalable, Real Time Applications for Financial Services with DataStax
 
Target: Performance Tuning Cassandra at Target
Target: Performance Tuning Cassandra at TargetTarget: Performance Tuning Cassandra at Target
Target: Performance Tuning Cassandra at Target
 

Viewers also liked

Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
DataStax
 

Viewers also liked (20)

Optimizing IAM with Single Sign-On From the Cloud to On-Premise
Optimizing IAM with Single Sign-On From the Cloud to On-PremiseOptimizing IAM with Single Sign-On From the Cloud to On-Premise
Optimizing IAM with Single Sign-On From the Cloud to On-Premise
 
CrowdStrike: Real World DTCS For Operators
CrowdStrike: Real World DTCS For OperatorsCrowdStrike: Real World DTCS For Operators
CrowdStrike: Real World DTCS For Operators
 
Carlos Santa María - Hiperconvergencia, el futuro del Data Center - semanainf...
Carlos Santa María - Hiperconvergencia, el futuro del Data Center - semanainf...Carlos Santa María - Hiperconvergencia, el futuro del Data Center - semanainf...
Carlos Santa María - Hiperconvergencia, el futuro del Data Center - semanainf...
 
Cassandra Summit 2014: Novel Multi-Region Clusters — Cassandra Deployments Sp...
Cassandra Summit 2014: Novel Multi-Region Clusters — Cassandra Deployments Sp...Cassandra Summit 2014: Novel Multi-Region Clusters — Cassandra Deployments Sp...
Cassandra Summit 2014: Novel Multi-Region Clusters — Cassandra Deployments Sp...
 
NGCC 2016 - Support large partitions
NGCC 2016 - Support large partitionsNGCC 2016 - Support large partitions
NGCC 2016 - Support large partitions
 
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
 
3800 die-bonder overview
3800 die-bonder overview3800 die-bonder overview
3800 die-bonder overview
 
Highly available, scalable and secure data with Cassandra and DataStax Enterp...
Highly available, scalable and secure data with Cassandra and DataStax Enterp...Highly available, scalable and secure data with Cassandra and DataStax Enterp...
Highly available, scalable and secure data with Cassandra and DataStax Enterp...
 
Cassandra Summit 2015: Real World DTCS For Operators
Cassandra Summit 2015: Real World DTCS For OperatorsCassandra Summit 2015: Real World DTCS For Operators
Cassandra Summit 2015: Real World DTCS For Operators
 
Securing Cassandra
Securing CassandraSecuring Cassandra
Securing Cassandra
 
Multi-Region Cassandra Clusters
Multi-Region Cassandra ClustersMulti-Region Cassandra Clusters
Multi-Region Cassandra Clusters
 
Cassandra multi-datacenter operations essentials
Cassandra multi-datacenter operations essentialsCassandra multi-datacenter operations essentials
Cassandra multi-datacenter operations essentials
 
Lessons Learned from Real-World Deployments of Java EE 7 at JavaOne 2014
Lessons Learned from Real-World Deployments of Java EE 7 at JavaOne 2014Lessons Learned from Real-World Deployments of Java EE 7 at JavaOne 2014
Lessons Learned from Real-World Deployments of Java EE 7 at JavaOne 2014
 
DataStax - Analytics on Apache Cassandra - Paris Tech Talks meetup
DataStax - Analytics on Apache Cassandra - Paris Tech Talks meetupDataStax - Analytics on Apache Cassandra - Paris Tech Talks meetup
DataStax - Analytics on Apache Cassandra - Paris Tech Talks meetup
 
Cassandra Operations at Netflix
Cassandra Operations at NetflixCassandra Operations at Netflix
Cassandra Operations at Netflix
 
An Introduction to Priam
An Introduction to PriamAn Introduction to Priam
An Introduction to Priam
 
Multi Data Center Strategies
Multi Data Center StrategiesMulti Data Center Strategies
Multi Data Center Strategies
 
Ficstar Software: Cassandra Installation to Optimization
Ficstar Software: Cassandra Installation to OptimizationFicstar Software: Cassandra Installation to Optimization
Ficstar Software: Cassandra Installation to Optimization
 
DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The Sequel
 
Apache Cassandra and DataStax Enterprise Explained with Peter Halliday at Wil...
Apache Cassandra and DataStax Enterprise Explained with Peter Halliday at Wil...Apache Cassandra and DataStax Enterprise Explained with Peter Halliday at Wil...
Apache Cassandra and DataStax Enterprise Explained with Peter Halliday at Wil...
 

Similar to iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter public cloud analytics

Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
DataStax
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack
Nitin Mehta
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview
Dimas Prasetyo
 

Similar to iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter public cloud analytics (20)

BigData Developers MeetUp
BigData Developers MeetUpBigData Developers MeetUp
BigData Developers MeetUp
 
Devops kc
Devops kcDevops kc
Devops kc
 
Server 2016 sneak peek
Server 2016 sneak peekServer 2016 sneak peek
Server 2016 sneak peek
 
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
 
Performance Analysis: new tools and concepts from the cloud
Performance Analysis: new tools and concepts from the cloudPerformance Analysis: new tools and concepts from the cloud
Performance Analysis: new tools and concepts from the cloud
 
The impact of cloud NSBCon NY by Yves Goeleven
The impact of cloud NSBCon NY by Yves GoelevenThe impact of cloud NSBCon NY by Yves Goeleven
The impact of cloud NSBCon NY by Yves Goeleven
 
Introduction to HPC & Supercomputing in AI
Introduction to HPC & Supercomputing in AIIntroduction to HPC & Supercomputing in AI
Introduction to HPC & Supercomputing in AI
 
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
 
Accelerated SDN in Azure
Accelerated SDN in AzureAccelerated SDN in Azure
Accelerated SDN in Azure
 
Ambedded - how to build a true no single point of failure ceph cluster
Ambedded - how to build a true no single point of failure ceph cluster Ambedded - how to build a true no single point of failure ceph cluster
Ambedded - how to build a true no single point of failure ceph cluster
 
"Clouds on the Horizon Get Ready for Drizzle" by David Axmark @ eLiberatica 2009
"Clouds on the Horizon Get Ready for Drizzle" by David Axmark @ eLiberatica 2009"Clouds on the Horizon Get Ready for Drizzle" by David Axmark @ eLiberatica 2009
"Clouds on the Horizon Get Ready for Drizzle" by David Axmark @ eLiberatica 2009
 
Monitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECSMonitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECS
 
Containers orchestrators: Docker vs. Kubernetes
Containers orchestrators: Docker vs. KubernetesContainers orchestrators: Docker vs. Kubernetes
Containers orchestrators: Docker vs. Kubernetes
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack
 
Quick-and-Easy Deployment of a Ceph Storage Cluster
Quick-and-Easy Deployment of a Ceph Storage ClusterQuick-and-Easy Deployment of a Ceph Storage Cluster
Quick-and-Easy Deployment of a Ceph Storage Cluster
 
Cassandra training
Cassandra trainingCassandra training
Cassandra training
 
How to Build a Multi-DC Cassandra Cluster in AWS with OpsCenter LCM
How to Build a Multi-DC Cassandra Cluster in AWS with OpsCenter LCMHow to Build a Multi-DC Cassandra Cluster in AWS with OpsCenter LCM
How to Build a Multi-DC Cassandra Cluster in AWS with OpsCenter LCM
 
Swift at Scale: The IBM SoftLayer Story
Swift at Scale: The IBM SoftLayer StorySwift at Scale: The IBM SoftLayer Story
Swift at Scale: The IBM SoftLayer Story
 
A Dataflow Processing Chip for Training Deep Neural Networks
A Dataflow Processing Chip for Training Deep Neural NetworksA Dataflow Processing Chip for Training Deep Neural Networks
A Dataflow Processing Chip for Training Deep Neural Networks
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview
 

More from DataStax Academy

Cassandra on Docker @ Walmart Labs
Cassandra on Docker @ Walmart LabsCassandra on Docker @ Walmart Labs
Cassandra on Docker @ Walmart Labs
DataStax Academy
 
Cassandra Adoption on Cisco UCS & Open stack
Cassandra Adoption on Cisco UCS & Open stackCassandra Adoption on Cisco UCS & Open stack
Cassandra Adoption on Cisco UCS & Open stack
DataStax Academy
 
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & PythonCassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
DataStax Academy
 
Standing Up Your First Cluster
Standing Up Your First ClusterStanding Up Your First Cluster
Standing Up Your First Cluster
DataStax Academy
 
Real Time Analytics with Dse
Real Time Analytics with DseReal Time Analytics with Dse
Real Time Analytics with Dse
DataStax Academy
 
Introduction to Data Modeling with Apache Cassandra
Introduction to Data Modeling with Apache CassandraIntroduction to Data Modeling with Apache Cassandra
Introduction to Data Modeling with Apache Cassandra
DataStax Academy
 
Enabling Search in your Cassandra Application with DataStax Enterprise
Enabling Search in your Cassandra Application with DataStax EnterpriseEnabling Search in your Cassandra Application with DataStax Enterprise
Enabling Search in your Cassandra Application with DataStax Enterprise
DataStax Academy
 
Advanced Data Modeling with Apache Cassandra
Advanced Data Modeling with Apache CassandraAdvanced Data Modeling with Apache Cassandra
Advanced Data Modeling with Apache Cassandra
DataStax Academy
 

More from DataStax Academy (20)

Forrester CXNYC 2017 - Delivering great real-time cx is a true craft
Forrester CXNYC 2017 - Delivering great real-time cx is a true craftForrester CXNYC 2017 - Delivering great real-time cx is a true craft
Forrester CXNYC 2017 - Delivering great real-time cx is a true craft
 
Introduction to DataStax Enterprise Graph Database
Introduction to DataStax Enterprise Graph DatabaseIntroduction to DataStax Enterprise Graph Database
Introduction to DataStax Enterprise Graph Database
 
Introduction to DataStax Enterprise Advanced Replication with Apache Cassandra
Introduction to DataStax Enterprise Advanced Replication with Apache CassandraIntroduction to DataStax Enterprise Advanced Replication with Apache Cassandra
Introduction to DataStax Enterprise Advanced Replication with Apache Cassandra
 
Cassandra on Docker @ Walmart Labs
Cassandra on Docker @ Walmart LabsCassandra on Docker @ Walmart Labs
Cassandra on Docker @ Walmart Labs
 
Cassandra 3.0 Data Modeling
Cassandra 3.0 Data ModelingCassandra 3.0 Data Modeling
Cassandra 3.0 Data Modeling
 
Cassandra Adoption on Cisco UCS & Open stack
Cassandra Adoption on Cisco UCS & Open stackCassandra Adoption on Cisco UCS & Open stack
Cassandra Adoption on Cisco UCS & Open stack
 
Data Modeling for Apache Cassandra
Data Modeling for Apache CassandraData Modeling for Apache Cassandra
Data Modeling for Apache Cassandra
 
Coursera Cassandra Driver
Coursera Cassandra DriverCoursera Cassandra Driver
Coursera Cassandra Driver
 
Production Ready Cassandra
Production Ready CassandraProduction Ready Cassandra
Production Ready Cassandra
 
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & PythonCassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
Cassandra @ Netflix: Monitoring C* at Scale, Gossip and Tickler & Python
 
Cassandra @ Sony: The good, the bad, and the ugly part 1
Cassandra @ Sony: The good, the bad, and the ugly part 1Cassandra @ Sony: The good, the bad, and the ugly part 1
Cassandra @ Sony: The good, the bad, and the ugly part 1
 
Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2Cassandra @ Sony: The good, the bad, and the ugly part 2
Cassandra @ Sony: The good, the bad, and the ugly part 2
 
Standing Up Your First Cluster
Standing Up Your First ClusterStanding Up Your First Cluster
Standing Up Your First Cluster
 
Real Time Analytics with Dse
Real Time Analytics with DseReal Time Analytics with Dse
Real Time Analytics with Dse
 
Introduction to Data Modeling with Apache Cassandra
Introduction to Data Modeling with Apache CassandraIntroduction to Data Modeling with Apache Cassandra
Introduction to Data Modeling with Apache Cassandra
 
Cassandra Core Concepts
Cassandra Core ConceptsCassandra Core Concepts
Cassandra Core Concepts
 
Enabling Search in your Cassandra Application with DataStax Enterprise
Enabling Search in your Cassandra Application with DataStax EnterpriseEnabling Search in your Cassandra Application with DataStax Enterprise
Enabling Search in your Cassandra Application with DataStax Enterprise
 
Bad Habits Die Hard
Bad Habits Die Hard Bad Habits Die Hard
Bad Habits Die Hard
 
Advanced Data Modeling with Apache Cassandra
Advanced Data Modeling with Apache CassandraAdvanced Data Modeling with Apache Cassandra
Advanced Data Modeling with Apache Cassandra
 
Advanced Cassandra
Advanced CassandraAdvanced Cassandra
Advanced Cassandra
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 

Recently uploaded (20)

Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 

iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter public cloud analytics

  • 1. Leveraging C* for real-time multi-dc public cloud analytics Julien Anguenot VP Software Engineering @anguenot
  • 2. 1 iland cloud story & use case 2 data & domain constraints 3 deployment, hardware, configuration and architecture overview 4 lessons learned 5 future platform extensions
  • 3. 3 iland cloud story & use case
  • 4. Who are we? • public, private, DRaaS, BaaS cloud provider • Cisco CMSP • VMware Vspp for 7+ years • 20+ years in business • HQ in Houston, TX • http://www.iland.com 4
  • 5. Yet another cloud provider? Well, … 5 • performance and stability • custom SLA • compliance • security • DRaaS • global datacenter footprint: US, UK and Singapore • dedicated support staff! • iland cloud platform, Web management console and API
  • 6. The iland cloud platform 6
  • 7. iland cloud platform essentially • data warehouse running across multiple data-centers • monitoring (resource consumption / performance) • billing (customers and internal use) • alerting • predictive analytics • cloud management • cloud services (backups, DR, etc.) • desktop and mobile management consoles • API • Cassandra powered! 7
  • 8. The iland cloud Web management console 8
  • 9. 9
  • 10. 10
  • 11. 11
  • 12. 12
  • 13. 13
  • 14. 14
  • 15. 15
  • 16. 16
  • 17. 17
  • 18. So, why did we do all this? • Initial motivations (v1) • vendor software (VMware vCloud Director) lacking: • performance analytics (real-time and historical) • billing • alerts • cross datacenter visibility • more private cloud type transparency • abstract ourselves from vendors and integrate an umbrella of heterogeneous services • modern UX and good looking UI 18
  • 19. 19 data and domain constraints
  • 20. Constraints 20 • write latency • high throughput • precision (used for billing) • availability • multi-data center • scalability: tens of thousands of VMs • agent-less • pull/poll vs push • high latency environs (multi-dc)
  • 21. Pipeline 21 • collection of real-time data • store • aggregation • correlation • rollups (historical) • processing • alerting • billing • reporting • querying
  • 22. Real-time collected perf counters 22 • 20 seconds samples • compute, storage, network • 15+ perf counters collected • ~50 data points per minute and per VM • time series • (timestamp, value) • metadata • unit • interval • etc. • 1 year TTL
  • 23. VM CPU 20 seconds perf counters 23 Group Name Type CPU USAGE AVERAGE CPU USAGE_MHZ AVERAGE CPU READY SUMMATION
  • 24. VM memory 20 seconds perf counters 24 Group Name Type MEM ACTIVE AVERAGE MEM CONSUMED AVERAGE MEM VM_MEM_CTRL SUMMATION
  • 25. VM network 20 seconds perf counters 25 Group Name Type NET RECEIVED AVERAGE NET TRANSMITTED AVERAGE NET USAGE AVERAGE
  • 26. VM disk 20 seconds perf counters 26 Group Name Type DISK READ AVERAGE DISK WRITE AVERAGE DISK MAX_TOTAL_LATENCY LATEST DISK USAGE AVERAGE DISK PROVISIONED LATEST DISK USED LATEST DISK NUMBER_WRITE_AVERAGED AVERAGE DISK NUMBER_READ_AVERAGED AVERAGE
  • 27. More counters collected for 3rd party services 27
  • 28. VM to time serie bindings 28 • binding on VM UUID • serie UUID • <VM_UUID>:disk:numberReadAveraged:average • Simple, fast and easy to construct at application level.
  • 29. 29
  • 30. 30
  • 31. VM containment and aggregation of real-time samples 31 • what’s this? • resource pool / vs instance-based $$ • 20 seconds samples aggregated from VM to VDC top level • separated tables
  • 32. Historical rollups and intervals 32 • VM, VAPP, VDC, ORG and network • 1 minute (TTL = 1 year) • 1 hour (used for billing) • 1 day • 1 week • 1 month • separated tables • new performance counter types created • TTL > 3 years for 1h samples for compliance & billing reasons • application level responsibilities
  • 33. 1 minute rollups processing 33 • processed to trigger alerts (usage, billing) • processed to compute real-time billing
  • 34. 1 hour rollups processing 34 • processed for final billing computation • leveraging salesforce.com collected data
  • 35. Data sources essentially 35 • compute • storage • network • Management • users • cloud configuration • salesforce.com • 3rd party services: backups, DR, etc. • pluggable: add / upgrade / remove services
  • 36. Cassandra is the sole record keeper 36
  • 38. iland cloud platform foundation 38 • Cisco UCS • VMware ESXi • VMware vSphere (management) • our Cassandra cluster runs on the exact same base foundation as our customer public clouds.
  • 39. 39 Simplified architecture (each DC) HAProxy Apache KeyCloak Wildfly AS Postgres Wildfly AS Resteasy API Wildfly AS cluster Apache Lucene NFS Apache Cassandra Compute Storage Network + 3rd parties Salesforce iland cloud Cassandra ring API AngularJS / API Redis Sentinel AMQP syslog-ng
  • 40. Cassandra version history 40 • late 2014: 2.1.x • early 2014: 2.0.x w/ Java CQL driver • late 2013: 2.0 beta w/ Astanyax (CQL3) (v1) • empty cluster • early 2013: 1.2.x w/ Astanyax (initial proto)
  • 41. iland’s cassandra cluster overall 41 • 6 datacenters • 1 (one) keyspace • 27 nodes • 1.5 to 2TB per node (TTL)
  • 43. Each DC 43 • 1 or 2 C* rack(s) of 3 Cassandra nodes • endpoint_snitch: RackInferringSnitch • RF=3
  • 44. 44
  • 45. Each node 45 • VM • Ubuntu 14.04 LTS • Apache Cassandra Open Source distribution • 32GB of RAM • 16 CPUs • 3 disks: system, commit logs, data
  • 46. Hardware 46 • Cisco UCS B200 M3 • not very expensive • Disks • Initially 10K SAS disks • now hybrid array (accelerated SSD) • reads off SSD (75/25) • boot time • maintenance ops • Cassandra CPU and RAM intensive. • No need to get crazy on disks initially • C* really runs well on non-SSD
  • 47. Network 47 • 1G and 10G lines (currently switching all to 10G) • Cassandra chatty but performs well in high latency environs • network usage is pretty much constant • 25 Mb/s in between DC: • default C* 2.1 outbound throttle • Increase when streaming node is needed • Permanent VPN in between DC (no C* SSL)
  • 49. ultimately an API for everything and everywhere 49
  • 50. 50 C* W iland ReST API iland core platform iland core platform iland ReST API C* R C* RC* W C* R only deployed in: Dallas, TX - London, UK - Singapore
  • 52. Tuning Cassandra node: JVM 52 • Java 8 • MAX_HEAP_SIZE=“8G” • HEAP_NEWSIZE=“2G” • Still using CMS but eager to switch to G1 w/ latest Cassandra version. • no magic bullet • test and monitor • 2.0.x to 2.1.x: had to revisit drastically
  • 53. Tuning Cassandra node: some config opts 53 • concurrent_writes / concurrent_reads • nodetool tpstats • concurrent_compactors • nodetool compactionstats • ++ • auto_snapshot • batch_size_warn_threshold_in_kb • monitor • no magic bullet • test and monitor
  • 54. Minimize C* reads (with Redis in our case) 54 • writes are great / reads are good • application level optimizations • 16G of cached data in every DC • very little in Redis. Bindings and alerts • in-memory only (no save on disk)
  • 55. Migration 55 • went live with 2.1.1 because of UDT • suggest waiting for at least 5 or 6 dot releases • 2.0.x / 2.1.x • have to re-tune the whole cluster • new features can be an issue initially (drivers) • Python driver very slow for data migration
  • 56. Don’t’s 56 • secondary indexes (or make sure you know what you’re doing) • IN operator • don’t forget TTL • no easy way around range deletes • complex “relational” type of models
  • 57. Do’s 57 • design simple data model • queries driven data model • writes are cheap: duplicate data to accommodate queries • prepared statements • batches • minimize reads from C* • UDT
  • 58. #pain 58 • bootstrapping new DC • streaming very hard to complete OK w/ 2.0 • temp node tuning during streaming • Cassandra 2.2 should help with bootstrap resume • repairs • very long and costly op • incremental repairs broken until late 2.1.x
  • 60. Issue with in-app server aggregations and rollups 60 • JEE container works great but… • lack of traceability / monitoring around jobs • separation of concerns • need to minimize reads against Cassandra • in-memory computation • code base growing fast (200k+ Java loc)
  • 61. Spark for aggregations and rollups 61 • tackling issues in previous slides • multiple new use cases: • for instance, heavy throughput data for network analysis • machine learning • Kafka & Spark Streaming • currently experimenting
  • 62. Multiple Keyspaces 62 • compliance / data isolation • lower network traffic