Pollfish is a survey platform which provides access to millions of targeted users. Pollfish allows easy distribution and targeting of surveys through existing mobile apps. (https://www.pollfish.com/). At pollfish we use Cassandra for difference use cases, eg. for application data store to maximize write throughput when appropriate and for our analytics project to find insights in application generated data. As a medium to accomplish our success so far, we use the Datastax's DSE 4.6 environment which integrates Appache Cassadra, Spark and a hadoop compatible file system (CFS). We will discuss how we started, how the journey was and the impressions gained so far along with some tips learned the hard way. This is a result of joint work of an excellent team here at Pollfish.

1. Stavros Kontopoulos
Software Engineer, MSc
Cassandra at Pollfish.com

3. What we do at Pollfish?
• Target mobile users with surveys through our android/ ios sdk which is installed via thousands
of mobile apps. Developers benefit from completed surveys, companies also may run a survey
campaign in real time. Now analytics/ML pipeline…
Why Apache Cassandra?
• Store time series about system events, user activities, survey results and much more..
• Amazing write throughput, take advantage of idempotent writes with proper resolution.
• Decent read speed throughput and low latency.
• Integrates with spark to implement our analytics, insights pipeline.
• A new business model.
A flashback… why I am here…

4. Our Tech Team- Disciplinary oriented
• Front End Development - UI Design • Back-end, data engineer(s) • Data Scientist • DevOps

5. Pollfish High Level Architecture
Mobile Users (~600K active per day)
APP SERVER 1
APP SERVER N
Other systems
(PostGres, Geo, Redis…)
DSE
Cassandra/Spark Cluster
Survey Customers

6. Why Datastax
Start up program and support when you go to production
Many tools for development and maintenance: Pig, hive, shark, CFS, OpsCenter.
Clear setup and support for real time data storage and analytics in the same cluster.
Can be extended for other workloads like Search (Solr).
Supports multiple DCs easily for other purposes staging, backup etc.
Product version we used: DSE 4.5.1 (spark 0.9.1), 4.5.3, 4.6.0 (Spark 1.1.0)

7. Our Cassandra Cluster- Setup
2 DataCenters (Cassandra Analytics):
• 1 for real-time data storage, read/write path.
• 1 for analytics nodes (Spark and Hadoop enabled), read/write path ETL, machine learning.
• Use the DSE setup with DSESimpleSnitch for mixed workloads multi-DC clusters.
3 Nodes per DC (details next), planning for more.
Data is written at Cassandra DC and replicated to Analytics DC for all keyspaces needed.
2 seed nodes, one per DC.

8. Our Cassandra Cluster- Setup

9. Our Cassandra Cluster- Setup

10. Our Cassandra Cluster- Infrastructure Details
Cassandra DC:
• Node 0,1,2: Standard_A4 (8 cores, 14 GB memory), 15DISKs in RAID 0
Analytics DC:
• Node 0,1,2: Standard_D13 (8 cores, 56 GB memory), 15DISKs in RAID 0
A bit about disks (ideally JBOD s but we can only have network disks)…
FS type: Ext4
Analytics disks per node:
Azure Os disk (/dev/sda1 , mp: /): 29GB
Azure VM tmp disk (/dev/sdb1, mp: /mnt/resource): 400GB - for spark temp files…
data_file_directories: /mnt/dsedata/lib/cassandra/data (2,9T)
commitlog_directory: /var/lib/cassandra/commitlog (197GB disk)

11. A few tips towards production…
Follow as a start:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html
http://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-Best-Practices.pdf
• NTP is a must for all servers, Cassandra needs that for node data synchronization and when you
do analytics
• Proper limits:
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited
• Optimum blockdev --setra settings for RAID (OK): All value must be set to 128KB for RA.
• No swap

12. A few Cassandra optimization tips
For Cassandra DC:
• concurrent_reads: 64
• concurrent_writes: 32
• Adjust according to the load and node IOPS/throughput..
Heap size adjustments: Datastax does a good job to automatically handle this… In general
for production 8GB for heap is ok… to avoid gc pauses… found in bigger jvms. We use 4GB in
smaller Cassandra nodes and 8GB in Analytics.
• In case you have a load with short lived objects adjust accordingly the parNew size.
Your tools: jconsole,jvisualvm,hprof, jstack, enable GC reports at logging level…
JVM_OPTS=”$JVM_OPTS -XX:+PrintGCDetails”
JVM_OPTS=”$JVM_OPTS -XX:+PrintGCDateStamps”
JVM_OPTS=”$JVM_OPTS -XX:+PrintHeapAtGC“
…

13. Managing nodes
When you have a running car how you change the tire?
Adding a new clean node to the cluster fast :
• Add the following line to /etc/dse/cassandra/cassandra-env.sh :
• JVM_OPTS=”$JVM_OPTS -Dcassandra.join_ring=false“
• Start the service
• Revert change and execute nodetool join
• After joining finishes execute a node rebalance from OpsCenter.
• Fallback: remove data and commit log dirs. And start again.
Stopping a node:
nodetool drain -h host name
sudo service dse stop

14. Integration with Spark- Use cases
Commmon ETL case:
Mobile User profile (ML):
HDFS Compatible (CFS)
Cassandra Raw Data
Spark/Cassandra
Spark/Cassandra
HDFS Compatible (CFS)
Cassandra User profile data
Customers Amazon S3

15. Integration with Spark- Our job framework
• We have built an analytics framework to run spark jobs on top of Cassandra.
Executes jobs remotely through maven.
• SparkContext is created on a non-dse node. Needed some tricks to make it work, now we move
to a more dse official approach with spark submit. Note: We started with 0.9.1 DSE version
where no spark submit was available.
• The framework provides an API to run jobs in production and use common functionality at the
Data science side for code re-use.

16. Datastax Cassandra Opscenter
• Facilitates: monitoring, health check and daily operations at the cluster level.
• We use it for: node rebalance, repairs,snapshotting, latency throughput checks along with low
level tools like iotop, ioping etc.

17. Datastax Cassandra Opscenter (Pending cluster operations)

18. Datastax Cassandra Opscenter (Keyspace Details)

19. Datastax Cassandra Opscenter (Last 10 days)

20. The Good and the Bad with Cassandra and DSE
good
• Easy to start with,
enough documentation.
• Reliable performance so far.
• Easily scalable,
easy to add/remove nodes.
• Support.
bad
• Bugs at the Cassandra level and the
Spark level. Need active follow up of the
lists for Spark, Cassandra and DSE tools.
Upgrade maybe the only solution and not
a piece of case.
• Gets tricky to optimize depending on your
load specifically when you don’t have
time to measure everything upfront and
you are in production already.

21. The Good and the Bad with Cassandra and DSE (Some issue examples)
Apply backpressure gently when overloaded with writes,
https://issues.apache.org/jira/browse/CASSANDRA-7937
Reproduce by writing large volume of data with large column values eg. 300K, crawling scenario...
Spark Cassandra Connector issues (eg. java driver not closing connection with cluster)...

22. As a developer some cool stuff I mess with
Currently I am extending the framework I have built to deliver spark jobs on top of Cassandra.
• IT with Cassandra embedded connector.
• Cassandra Schema Design.
• ETL use cases.
Organizing dev env:
Optimizing development environment with vagraant, automatic installation - avoid upgrade
headaches, setup and IT level testing. Introduced CI with Jenkins, Artifactory etc…

23. thank you