Coursera's Adoption of Cassandra

•

5 likes•1,044 views

Like many startups, Coursera began its data storage journey with MySQL, a familiar and industry-proven database. As Coursera's user base grew from several thousand to many millions, we found that MySQL provided limited availability and restricted our ability to scale easily. New product initiatives and requirements provided a perfect opportunity to revisit our choice of core workhorse database. After evaluating several NoSQL databases, including MongoDB, DynamoDB and HBase, we elected to transition to Cassandra . Cassandra's relative maturity, masterless architecture (for availability), tunable consistency, and stable low-latency performance made it a clear winner for our needs. Learn more about what it takes to transition from SQL to Cassandra in this talk.

Technology

Biography
Daniel Chia
@DanielJHChia
Software Engineer, Infrastructure Team
2© 2015. All Rights Reserved.

© 2015. All Rights Reserved.
1 Introduction
2 Want We Want From Our Database
3 MySQL Limitations
4 Cassandra - What and Why
5 Looking Back

6© 2015. All Rights Reserved.
Web
iOS
Android

Database Wants
7© 2015. All Rights Reserved.

Consistently Fast Latencies
8© 2015. All Rights Reserved.

Availability
9© 2015. All Rights Reserved.

Scalability
10© 2015. All Rights Reserved.

Other Niceties
• Operational ease
• Multi-region capability
11© 2015. All Rights Reserved.

Coursera Tech Stack
• 100% AWS
• MySQL + Cassandra
• Service-oriented
12© 2015. All Rights Reserved.

RDS Challenges
• Normalized data model Unpredictable query performance
• Scaling by sharding not ideal
• Single master limitation
13© 2015. All Rights Reserved.

C*
• Columnar model
• Tunable consistency
• Fast
• Horizontally scalable
• Great community
14© 2015. All Rights Reserved.

15© 2015. All Rights Reserved.
Looking Back

Cassandra - Initial Pain Points
• Can’t execute arbitrary queries
• Filtering, sorting, etc.
• Can’t be abused as an OLAP database
• Worries about ‘eventual’ consistency
16© 2015. All Rights Reserved.

SQL NoSQL Mindset Shift
• Build in-house Cassandra expertise
• Data modeling still important
• Know your queries
17© 2015. All Rights Reserved.

Cassandra ≠ [database XYZ]
“But if you judge a fish by its ability to climb a tree,
it will live its whole life believing that it is stupid.”
-Albert Einstein
18© 2015. All Rights Reserved.

Enrollment Example
• Learners enroll into a course
• learner (many-to-many) course
• Need to track this membership
19© 2015. All Rights Reserved.

MySQL
CREATE TABLE `courses_learners` (
`id` INT(11) NOT NULL auto_increment,
`course_id` INT(11) NOT NULL,
`learner_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `c_l` (`learner_id`, `course_id`),
CONSTRAINT `ref1` FOREIGN KEY (`course_id`)
CONSTRAINT `ref2` FOREIGN KEY (`learner_id`)
)
20© 2015. All Rights Reserved.

Cassandra
CREATE TABLE courses_by_learner (
learner_id uuid,
course_id uuid,
PRIMARY KEY (learner_id, course_id)
)
24© 2015. All Rights Reserved.

Helpful Things
• Data modeling consulting
• Monitoring
• Data access layer for common use cases
25© 2015. All Rights Reserved.

Gotchas
• Lots of truly ad-hoc queries is hard
• Don’t use C* directly to explore your data. (Spark?)
• Sorting, filtering can be hard
• Consider Solr / ElasticSearch
• Or even MySQL depending on load / importance
26© 2015. All Rights Reserved.

Many applications are initially developed using relational database technology before upsizing to Cassandra as they mature. This presentation will examine the indicators that it is time to start considering a migration away from relational and factors to consider when making the decision to move. We will then discuss different approaches for implementing the migration and how to plan, estimate and manage the work along with key risks and gotchas you may encounter.

Cassandra summit 2015 - Simplifying Streaming Analytics

Brenden Matthews

Using Apache Cassandra, Apache Spark and Apache Kafka is a powerful combination to realize streaming analytics, from online recommendations to metrics applications. We will have a look at this setup in the context of popular architectures such as Lambda and Kappa and discuss requirements such as latency and processing guarantees in a practical setup. This talk includes a demo of a streaming analytics application using the Mesosphere DCOS as a deployment environment.

When and how to migrate from a relational database to Cassandra

Ben Slater

Many applications are initially developed using relational database technology before “upsizing” to Cassandra as they mature. This presentation examines the indicators that it is time to start considering a migration away from relational and factors to consider when making the decision to move. It then discusses different approaches for implementing the migration and how to plan, estimate and manage the work along with key risks and gotchas you may encounter.

DOES SFO 2016 - Rich Jackson & Rosalind Radcliffe - The Mainframe DevOps Team...

Gene Kim

Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, Cloudera

HostedbyConfluent

Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments

Maxis Alchemize imug 2017

BrandonWilhelm4

Couchbase Meetup Jan 2016

Michael Kehoe

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.

Designing Resilient Application Platforms with Apache Cassandra - Hayato Shim...

jaxLondonConference

Presented at JAX London 2013 All too often I have observed infrastructure designs for deploying Java applications come as an afterthought by businesses, technical analysts, and application developers. Choices of technologies are frequently made with no final deployment infrastructures being discussed. The talk will cover the design considerations on building resilient applications, and application deployment platforms across multiple data centres, and how organisations can leverage technologies such as Apache Cassandra to achieve this.

SkySQL MariaDB 云数据组件

YUCHENG HU

Northwestern Mutual Journey – Transform BI Space to Cloud

Databricks

The volume of available data is growing by the second (to an estimated 175 zetabytes by 2025), and it is becoming increasingly granular in its information. With that change every organization is moving towards building a data driven culture. We at Northwestern Mutual share similar story of driving towards making data driven decisions to improve both efficiency and effectiveness. Legacy system analysis revealed bottlenecks, excesses, duplications etc. Based on ever growing need to analyze more data our BI Team decided to make a move to more modern, scalable, cost effective data platform. As a financial company, data security is as important as ingestion of data. In addition to fast ingestion and compute we would need a solution that can support column level encryption, Role based access to different teams from our datalake. In this talk we describe our journey to move 100’s of ELT jobs from current MSBI stack to Databricks and building a datalake (using Lakehouse). How we reduced our daily data load time from 7 hours to 2 hours with capability to ingest more data. Share our experience, challenges, learning, architecture and design patterns used while undertaking this huge migration effort. Different sets of tools/frameworks built by our engineers to help ease the learning curve that our non-Apache Spark engineers would have to go through during this migration. You will leave this session with more understand on what it would mean for you and your organization if you are thinking about migrating to Apache Spark/Databricks.

Big Data as easy as 1, 2, 3, ... 4 ... with KNIME

Rosaria Silipo

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...

HostedbyConfluent

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it. When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost? Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know: - architecture and real-life examples of the flexible and highly available data collection platform - custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced - easy way to receive data from thousands of servers and many cloud services - how to archive data at low cost You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

ETL in data warehouse as a service enviornment

Sanil Mhatre

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...

HostedbyConfluent

When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.

Couchbase Connect 2016

Michael Kehoe

A Walkthrough of InfluxCloud 2.0 by Tim Hall

InfluxData

intern

Shivang Singh

Meetup core concepts-erick-ramirez-20150729

Erick Ramirez

SQL Server Deployments made easy with DACPAC

Sanil Mhatre

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

HostedbyConfluent

Kafka Connectors are used extensively in data migration solutions, serving as a middle tier when migrating data across databases. In addition, microservice architectures also use Kafka Connectors heavily when communicating with one another while still operating independently on their own data stores. In this talk, we cover these use cases in more detail along with a deep dive into the architecture of the source and sink Kafka Connectors for Cosmos DB.

Bootstrap SaaS startup using Open Source Tools

botsplash.com

(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...

Severalnines

Over the past few years, VidaXL has become a European market leader in the online retail of slow moving consumer goods. When a company achieved over 50% year over year growth for the past 9 years, there is hardly enough time to overhaul existing systems. This means existing systems will be stretched to the maximum of their capabilities, and often additional performance will be gained by utilizing a large variety of datastores.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Using Apache Spark for Predicting Degrading and Failing Parts in Aviation

Databricks

When the Cloud is a Rockin: High Availability in Apache CloudStack

John Burwell

CloudStack currently provides a variety bespoke high availability mechanisms for resources such as virtual machines, hosts, and virtual routers. Each of these implementations duplicates the HA check/recovery cycle, as well as, concurrency, persistence, and clustering required manage high available for any CloudStack resource. The High Availability Resource Management Service has been developed to consolidate these concerns -- providing a robust, extensible HA mechanism. Using this service, plugins only need to define health check, activity check, and fence operations.

Production Ready Cassandra (Beginner)

Coursera's Adoption of Cassandra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Coursera's Adoption of Cassandra

Similar to Coursera's Adoption of Cassandra (20)

More from DataStax Academy

More from DataStax Academy (20)

Recently uploaded

Recently uploaded (20)

Coursera's Adoption of Cassandra