Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward

•Download as PPTX, PDF•

2 likes•3,878 views

To support 6 million on-demand rides per day, a lot has to happen in near-real time. Latency translates into missed rides and monetary losses. Grab relies data streaming in Apache Kafka, with Scylla to tie it all together. This presentation details how Grab uses Scylla as a high throughput, low-latency aggregation store to combine multiple Kafka streams in near real-time, highlighting impressive characteristics of Scylla and how it fared against other databases in Grab’s exhaustive evaluations.

Software

Agenda
▪ Background
▪ Motivation for using ScyllaDB
▪ ScyllaDB use at Grab
▪ Conclusion
▪ Q&A

Introduction to Grab
▪ One of the most frequently used mobile platforms in
Southeast Asia
▪ Multiple services (Multi-million users per day and growing!)
• Transport
• Food
• Payment
• Shopping
• Package Delivery
• ...

Why ScyllaDB
▪ Grab Tech
• Microservices (Golang)
• Kafka for streaming
• Why? Loose coupling and Scale! doh! :)

Why ScyllaDB
● Stream Processing
● Transforms
● Aggregates
● Joins
● ...

Why ScyllaDB
▪ Required a State store (metadata store) which has the
following characteristics:
• Ability to handle very high throughput
• Ability to handle bursts (doh!)
• Ability to scale out (i.e, handle hockey stick like growth)
• Very low latencies (both write and reads) - Near RealTime
• Low operational overhead
• Cost efficient

Number of Use cases:
▪ Fraud detection
▪ Stream Aggregation &
▪ Stream Statistics
▪ ...
ScyllaDB Usage

▪ Background
https://www.scmp.com/week-asia/business/article/2154770/who-pays-when-indonesian-ride-
sharing-fraud-goes-full-throttle
https://www.techrepublic.com/article/seven-ride-sharing-scams-to-watch-out-for/
ScyllaDB Usage - Fraud Detection

▪ Use Kafka streams to do real-time flagging of rides
▪ Might require combining multiple streams together
▪ Requires
• Low latency
• Ability to handle high throughput
ScyllaDB Usage - Fraud Detection

ScyllaDB Usage - Stream Statistics
▪ Teams like to find out:
• Counts of … (eg: count of rides in the past month, etc)
• Raw stats particular to a key for help in debugging
• Business users - require stats on city/country data

ScyllaDB Usage - Stream Statistics
▪ Requires a fast store for raw stats
▪ Ability to build Time series on top of it.
• Hello KairosDB :) ...

ScyllaDB Usage - Numbers
▪ One 15 node cluster (version 2.2)
▪ Running on AWS - EC2
▪ Single DC (may be multi-DC in the future)
▪ Overall ~15TB data across all nodes

ScyllaDB at Grab So far..
▪ Overall
• Great experience
• Cost effective
• Very responsive team
• As good as advertised :)
• Growing within the company…

ScyllaDB at Grab So far..
▪ Some hiccups
• nodetool repair - Writes and read timeouts every time when the run
finishes (even with -pr)
• Wish nodes can join the cluster faster
• Right now takes a really long time (approx 1TB data on each node across all
keyspaces) - takes ~2.5 hours to add a node
• Better error logging

Thank You
Any Questions ?
Please stay in touch
aravindvelamur@gmail.com

Grab is one of the most frequently used mobile platforms in Southeast Asia, providing the everyday services that matter most to consumers. Its users commute, eat, arrange shopping deliveries, and pay with one e-wallet. Grab relies on the combination of Apache Kafka and Scylla for a very critical use case -- instantaneously detecting fraudulent transactions that might occur across approximately more than six million on-demand rides per day taking place in eight countries across Southeast Asia. Doing this successfully requires many things to happen in near-real time. Join our webinar for this fascinating real-time big data use case, and learn the steps Grab took to optimize their fraud detection systems using the Scylla NoSQL database along with Apache Kafka.

Augury: Real-Time Insights for the Industrial IoT

ScyllaDB

Augury stores and serves time-series features from massive streams of IoT data, both for real-time insights, and offline learning and analytics. Learn about Augury’s needs and constraints, their solution evaluation and architecture, and fundamental practices for efficient data modeling, plus get a glimpse into the next-gen architecture at Augury, with a view on time-series feature storage and serving.

Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud

ScyllaDB

Scylla Summit 2016: ScyllaDB, Present and Future

ScyllaDB

Scylla Summit 2022: An Odyssey to ScyllaDB and Apache Kafka

ScyllaDB

Will LaForest is the Public Sector CTO for Confluent. In his current position, Will evangelizes the benefits of Apache Kafka, event-driven data in motion architecture, and open-source software is addressing mission challenges in the Government. He has spent 25 years wrangling data at massive scale. His technical career spans diverse areas from software engineering, NoSQL, data science, cloud computing, machine learning, and building statistical visualization software but began with code slinging at DARPA as a teenager. Will holds degrees in mathematics and physics from the University of Virginia. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

GPS Insight on Using Presto with Scylla for Data Analytics and Data Archival

ScyllaDB

GPS Insight is a leader in fleet vehicle management using IoT. Internally they use a combination of SQL and NoSQL big data technologies, including distributed SQL data analytics via Presto, an open-source query engine developed by Facebook. Learn how to set up, configure, and use Presto with Scylla for supporting ad hoc non-partition key queries for analytics and data scientists. Plus hear how to use Presto for a Data Archival approach with csv files on S3 or similar storage appliance.

How ReversingLabs Serves File Reputation Service for 10B Files

ScyllaDB

ReversingLabs is on a mission to deliver threat intelligence to their users by providing complete visibility and insight into every destructive object. To deliver on their commitment, they migrated to Scylla to handle thousands of updates per second in their processing engines. In their talk, they will go over their requirements and show how they tuned the system to handle requests from their API frontend.

Why you need benchmarks Finding the right database solution for your use case can be an arduous journey. The database deployment touches aspects of throughput performance, latency control, high availability and data resilience. You will need to decide on the infrastructure to use: Cloud, on-premise or a hybrid solution. Data models also have an impact on finding the right fit for the use case. Once you establish a requirements set, the next step is to test your use case against the databases of choice. In this workshop, we will discuss the different data points you need to collect in order to get the most realistic testing environment. We will cover: Data model impact on performance and latency Client behavior related to database capabilities Failover and high availability testing Hardware selection and cluster configuration impact We will show 2 benchmarking tools you can use to test and benchmark your clusters to identify the optimal deployment scenario for your use case. Attend this virtual workshop if you are: Looking to minimize the cost of your database deployment Making a database decision based on performance and scale data Planning to emulate your workload on a pre-production system where you can test, fail fast and learn.

Lessons Learned - Monitoring the Data Pipeline at Hulu

DataWorks Summit

This document summarizes lessons learned about monitoring a data pipeline at Hulu. It discusses how the initial monitoring approach had some issues from the perspectives of users and detecting problems. A new approach is proposed using a graph data structure to provide contextual troubleshooting that connects any issues to their impacts on business units and user needs. This approach aims to make troubleshooting easier by querying the relationships between different components and resources. Small independent services would also be easier to create and maintain within this approach.

Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration

ScyllaDB

Patience with Apache Cassandra’s volatile latencies was wearing thin at Rakuten, a global online retailer serving 1.5B worldwide members. The Rakuten Catalog Platform team architected an advanced data platform – with Cassandra at its core – to normalize, validate, transform, and store product data for their global operations. However, while the business was expecting this platform to support extreme growth with exceptional end-user experiences, the team was battling Cassandra’s instability, inconsistent performance at scale, and maintenance overhead. So, they decided to migrate. Join this webinar to hear a firsthand account of: How specific Cassandra challenges were impacting the team and their product How they determined whether migration would be worth the effort What processes they used to evaluate alternative databases What their migration required from a technical perspective Strategies (and lessons learned) for your own database migration

Introducing Scylla Cloud

ScyllaDB

ScyllaDB recently launched our Scylla Cloud database as a service, which combines the speed and power of the Scylla NoSQL database with the ease of a fully managed cloud service. Scylla Cloud relieves your team of day-to-day cluster management so you can focus on creating modern, interactive applications that respond to queries in milliseconds. Join us for an overview of Scylla Cloud, including a live demo of how to launch and connect to a cluster, how to create and query a table, and how to run a few operations, all in minutes.

Kiwi.com Reaches Cruising Altitude with Scylla

ScyllaDB

Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB

ScyllaDB

One cloud is hard enough, am I right? Now everyone expects that you can deploy containerized applications "everywhere" and things will "just work." Our customer sure did! Join Miles Ward, CTO, and Jenn Viau, Staff Solutions Architect, at SADA on a detailed, data-filled exploration of the complexities and constraints of modern multi-cloud and hybrid scenarios, rooted in the pursuit of almighty uptime and SLO adherence. They'll show what worked, and what didn't, in a detailed architectural review, as well as demonstrate (and perf test live!) components of the final production system. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2022: Migrating SQL Schemas for ScyllaDB: Data Modeling Best Pr...

ScyllaDB

To maximize the benefits of ScyllaDB, you must adapt the structure of your data. Data modeling for ScyllaDB should be query-driven based on your access patterns – a very different approach than normalization for SQL tables. In this session, you will learn how tools can help you migrate your existing SQL structures to accelerate your digital transformation and application modernization. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Ebay: DB Capacity planning at eBay

DataStax Academy

Bass Chorng is a principal capacity engineer at eBay who specializes in database performance, availability, and scalability. He established eBay's database capacity team in 2003. eBay uses both NoSQL and RDBMS databases including Cassandra, MongoDB, CouchBase, and Oracle. eBay sees over 400 billion database calls per day across 2000 NoSQL nodes and 450 Oracle nodes while hosting 800 million active items and 120 million active users. Capacity planning involves analyzing traffic, utilization, forecasting growth, and converting resource needs into costs. It requires knowledge of the platform, bottlenecks, and new technologies.

Scylla Summit 2019 Keynote - Avi Kivity

ScyllaDB

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Anant Corporation

During this week's Cassandra Lunch we covered packaged and DIY methods for Lucene-based indexes on Cassandra; as well as some pros and cons for using Lucene Based Indexes on Cassandra. Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-23-lucene-based-indexes-on-cassandra/ Cassandra Lunch Recording: https://youtu.be/Z0NXWmZAB8s Additional Resources: https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search/searchTOC.html https://github.com/instaclustr/cassandra-lucene-index https://github.com/strapdata/elassandra https://github.com/tjake/Solandra Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Follow Us and Reach Us At: Anant: https://www.anant.us/ Cassandra.Link: https://cassandra.link/ Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

How SkyElectric Uses Scylla to Power Its Smart Energy Platform

ScyllaDB

SkyElectric uses Scylla to power its smart energy platform. Scylla provides better performance, scalability, and lower latency than their previous MySQL database. With Scylla, SkyElectric has seen average write latency of 1.4ms and read latency of under 1ms, which is 10x faster throughput than MySQL. While Scylla has been easy to operate and support responsive upgrades and repairs, SkyElectric hopes to see improvements in data changelog, faster node joining, and backup/restore processes.

Scylla Summit 2022: Stream Processing with ScyllaDB

ScyllaDB

Palo Alto Networks processes terabytes of events each day. One of their many challenges is to understand which of those events (which might come from various different sensors) actually describe the same story but from many different viewpoints. Traditionally, such a system would need some sort of a database to store the events, and a message queue to notify consumers about new events that arrived into the system. They wanted to mitigate the cost and operational overhead of deploying yet another stateful component to their system, and designed a solution that uses ScyllaDB as the database for the events *and* as a message queue that allows our consumers to consume the correct events each time. Join this talk with Daniel Belenky, Principal Software Engineer, Palo Alto Networks where he will walk you through their process. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

How to Build a Scylla Database Cluster that Fits Your Needs

ScyllaDB

Sizing a database cluster makes or breaks your application. Too small and you could sustain spikes in usage and recover from a node loss or an operational slowdown. Too big and your cluster will cost more and waste valuable human resources. Since different workloads have different requirements, successful sizing of your application should be optimized for both throughput and latency performance. However, in many cases, the requirements for each contradicts each other. In this webinar, we explain how to remediate the contradicting forces and build a sustainable cluster to meet both performance and resiliency requirements.

About VisualDNA Architecture @ Rubyslava 2014

Michal Harish

Michal Hariš provides an overview of the evolution of VisualDNA's data architecture over the past 3 years. Originally, 10 people managed a single MySQL table holding 50M user profiles. They transitioned to using Cassandra and Hadoop to address scalability issues. Currently, they have a 120 person team using a lambda architecture with Java, Scala, Hadoop, Cassandra, Kafka, Redis, R and AngularJS. Real-time processing of 8.5k events/second is done alongside batch pipelines and machine learning. They have learned lessons around system design, testing, and remote collaboration while addressing challenges such as globally distributed APIs and bottlenecks in their data pipeline.

Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

ScyllaDB

Disney+ Hotstar is the fastest growing branch of Disney+. Join Disney+ Hotstar Architect Vamsi Subhash and senior data engineer Balakrishnan Kaliyamoorthy to learn… How Disney+ Hotstar architected their systems to handle massive data loads Why they chose to replace both Redis and Elasticsearch Their requirements for massively scalable data infrastructure and evolving data models How they migrated their data to Scylla Cloud, ScyllaDB’s fully managed NoSQL database-as-a-service, without suffering downtime

Implementing a Distributed NoSQL Database in a Persistent Distributed Ledger ...

ScyllaDB

IOTA uses Scylla to store unlimited amounts of data generated by IoT devices on the IOTA network. Scylla is an open source NoSQL database that provides shared-nothing architecture and LSM-based storage optimized for IOTA's needs. Key features of using Scylla include its ability to handle unlimited amounts of time series data from devices, dynamically add or remove nodes from the Scylla cluster supporting IOTA, and power the IOTA Tangle Explorer.

Expedia Group: Our Migration Journey from Apache Cassandra to ScyllaDB

ScyllaDB

Personalization Journey: From Single Node to Cloud Streaming

Databricks

In the online gaming industry we receive a vast amount of transactions that need to be handled in real time. Our customers get to choose from hundreds or even thousand options, and providing a seamless experience is crucial in our industry. Recommendation systems can be the answer in such cases but require handling loads of data and need to utilize large amounts of processing power. Towards this goal, in the last two years we have taken down the road of machine learning and AI in order to transform our customer’s daily experience and upgrade our internal services.

Scylla Summit 2018: Cassandra and ScyllaDB at Yahoo! Japan

ScyllaDB

Yahoo! JAPAN is one of the most successful internet service companies in Japan. Their NoSQL Team's Takahiro Iwase and Murukesh Mohanan have been testing out ScyllaDB, comparing it with Cassandra on multiple parameters: performance (both throughout and latency), reliability and ease of use. They will discuss the motivations behind their search for a successor of Cassandra that can handle exceedingly heavy traffic, and their evaluation of ScyllaDB in this regard.

Cloud native data platform

Li Gao

Cloud Native Data Platform at Fitbit - Fitbit collects 100 TB of user data daily from 30 million users across fitness trackers, smartwatches, and apps for internal teams like data science, research, and customer support as well as enterprise wellness programs. - The data platform includes MySQL, Kafka, Cassandra, S3, EMR, Presto/Spark and supports both batch and real-time workflows across multiple AWS accounts for compliance. - Key challenges included diverse user needs, multiple compliance requirements, and a lean team. The multi-tenant architecture in AWS with fine-grained S3 buckets and IAM roles helps address these challenges.

Introduction to Kafka

Akash Vacher

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

ScyllaDB

The RCP/Rakuten Catalog Platform has been growing at a brisk speed over the last couple of years. Our original backbone was Cassandra. However, as they continued their growth, they internally started realizing that it was not suitable for our next stage of growth. As such, they started looking into ScyllaDB as a better ROI solution as well as a much more stable backend. The migration itself was challenging since this has to be done for a production live data processing pipeline with minimal impact on customers. In this talk, Hitesh Shah, Engineering Manager at Rakuten USA will dive deeper into challenges and takeaways. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

What's hot

Workshop - How to benchmark your database

ScyllaDB

Lessons Learned - Monitoring the Data Pipeline at Hulu

DataWorks Summit

Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration

ScyllaDB

Introducing Scylla Cloud

ScyllaDB

Kiwi.com Reaches Cruising Altitude with Scylla

ScyllaDB

Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB

ScyllaDB

Scylla Summit 2022: Migrating SQL Schemas for ScyllaDB: Data Modeling Best Pr...

ScyllaDB

Ebay: DB Capacity planning at eBay

DataStax Academy

Scylla Summit 2019 Keynote - Avi Kivity

ScyllaDB

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Anant Corporation

How SkyElectric Uses Scylla to Power Its Smart Energy Platform

ScyllaDB

Scylla Summit 2022: Stream Processing with ScyllaDB

ScyllaDB

How to Build a Scylla Database Cluster that Fits Your Needs

ScyllaDB

About VisualDNA Architecture @ Rubyslava 2014

Michal Harish

Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

ScyllaDB

Implementing a Distributed NoSQL Database in a Persistent Distributed Ledger ...

ScyllaDB

Expedia Group: Our Migration Journey from Apache Cassandra to ScyllaDB

ScyllaDB

Personalization Journey: From Single Node to Cloud Streaming

Databricks

Scylla Summit 2018: Cassandra and ScyllaDB at Yahoo! Japan

ScyllaDB

Cloud native data platform

Li Gao

What's hot (20)

Workshop - How to benchmark your database

Lessons Learned - Monitoring the Data Pipeline at Hulu

Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration

Introducing Scylla Cloud

Kiwi.com Reaches Cruising Altitude with Scylla

Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB

Scylla Summit 2022: Migrating SQL Schemas for ScyllaDB: Data Modeling Best Pr...

Ebay: DB Capacity planning at eBay

Scylla Summit 2019 Keynote - Avi Kivity

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

How SkyElectric Uses Scylla to Power Its Smart Energy Platform

Scylla Summit 2022: Stream Processing with ScyllaDB

How to Build a Scylla Database Cluster that Fits Your Needs

About VisualDNA Architecture @ Rubyslava 2014

Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

Implementing a Distributed NoSQL Database in a Persistent Distributed Ledger ...

Expedia Group: Our Migration Journey from Apache Cassandra to ScyllaDB

Personalization Journey: From Single Node to Cloud Streaming

Scylla Summit 2018: Cassandra and ScyllaDB at Yahoo! Japan

Cloud native data platform

Similar to Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward

Introduction to Kafka

Akash Vacher

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

ScyllaDB

Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks

Databricks

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Garindra Prahandono

ScaleDB Technical Presentation

Ivan Zoratti

Scalable complex event processing on samza @UBER

Shuyi Chen

The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real time data needs for various Uber products. This platform has been in production for almost a year and it has proven to be very flexible to solve many use cases. In this talk, we will share in detail the design and architecture of the platform, and how we employ Samza, Kafka, and Siddhi at scale. This slides was presented at Stream Processing Meetup @ LinkedIn on June 15 2016.

How QBerg scaled to store data longer, query it faster

MariaDB plc

The continuous increase in terms of services and countries to which QBerg delivers its services requires an ever-increasing load of resources. During the last year QBerg has reached a critical point, storing so much transactional data that standard relational databases were unable to meet the SLAs, or support the features, required by customers. As an example, they had to cap web analytics to running on a maximum of four months of history. The introduction of MariaDB ColumnStore, flanked by existing MariaDB Server databases, not only will allow them to store multiple years’ worth of historical data for analytics – it decreased overall processing time by one order of magnitude right off the bat. The move to a unified platform was incremental, using MariaDB MaxScale as both a router and a replicator. QBerg is now able to replicate full InnoDB schemas to MariaDB ColumnStore and incrementally update big tables without impacting the performance of ongoing transactions.

XStream: stream processing platform at facebook

Aniket Mokashi

XStream is Facebook's unified stream processing platform that provides a fully managed stream processing service. It was built using the Stylus C++ stream processing framework and uses a common SQL dialect called CoreSQL. XStream employs an interpretive execution model using the new Velox vectorized SQL evaluation engine for high performance. This provides a consistent and high efficiency stream processing platform to support diverse real-time use cases at planetary scale for Facebook.

Rubyslava + PyVo #48

Jozef Képesi

Skypicker is a flight ticket search and booking engine that processes hundreds of terabytes of airline data monthly and sells thousands of tickets daily in Europe, Russia, and China. It uses PostgreSQL extensively to power its API and databases. The document discusses Skypicker's PostgreSQL infrastructure, which includes 5 database clusters with over 0.5TB of memory each that handle 20 million updates per hour. It also covers optimizations made to scale the databases, such as table partitioning, cascading replication, and query tuning. Skypicker is currently preparing for a large increase in data and is hiring database masters and developers.

Nebula Graph nMeetup in Shanghai - Meet with Graph Technology Enthusiasts

Nebula Graph

This is a speech given by Nebula Graph during the offline meetup with a bunch of graph technology enthusiasts. The slides mainly include the following info: 1. A brief introduction to the graph theory and graph database category 2. The Nebula Graph team's thoughts on the graph technology and the development of the graph database industry in recent years, including advantages and challenges 3. The architecture of Nebula Graph based on the thoughts 4. Q&A

Scylla Summit 2018: Kiwi.com Migration to Scylla - The Why, the How, the Fail...

ScyllaDB

At Kiwi.com we never stop innovating our product and our architecture. Over the past couple of years, we saw a significant rise in technology requirements both globally and internally and had already tried several database solutions. The transformation went from small applications to complex microservices architectures. We first migrated to Cassandra from a big PostgreSQL cluster to get better performance and scalability, but our demands never stopped growing. That is why we decided to go with Scylla. In this talk, I will cover how our team approached testing of Scylla, the migration plan, how it impacts our business and how it influenced our high-level architecture of the application and infrastructure. It has a significant impact on disaster recovery and availability of our overall system.

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

MariaDB plc

Copart is a global vehicle auction company that has been using MariaDB to power its databases and applications. It migrated from its legacy AS400 database to MariaDB due to its scalability, lower costs, and support for advanced features like encryption. Copart has implemented MariaDB in various configurations including Galera clusters, master-slave, and multi-instance setups to power its production and non-production environments. It continues to explore new MariaDB capabilities and configurations to improve performance, reduce costs, and meet its expanding international operations.

Trending with Purpose

Jason Dixon

This talk evalutes some easy ways to extract useful trending and capacity planning out of your existing monitoring investment. Using Nagios performance data, we examine simple behaviors with PNP4Nagios and graduate on to more insightful analytics with Graphite. With metrics in hand we look at the questions that IT /should/ be asking, such as: * What sort of data should I trend? * Why do I need to trend it? * How do Operational or Engineering trends relate to Business or Transactional monitoring? * How does this data impact our customer relationship and/or their bottom-line? Finally, we look at creative ways to get profiling data out of your production systems with a minimum amount of effort from your development team.

Data Analytics Service Company and Its Ruby Usage

SATOSHI TAGOMORI

This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.

ShareChat's Journey Migrating 100TB of Data to ScyllaDB with NO Downtime

ScyllaDB

We at ShareChat want to describe our journey of migrating our NoSQL use-cases from different databases to cloud agnostic | performant | cost effective ScyllaDB. We had this mammoth task of migrating all the data from existing databases to ScyllaDB with NO downtime. We built a framework which had minimal impact on the application in terms of latency and downtime during the migration process and that has the ability to fallback to the source table in case of any inconsistency during migration. Our framework solved for multiple use-cases including both counter and non-counter use-cases across 3 languages used in ShareChat (NodeJS, GoLang, Java) through shared library. We built recovery and auditing mechanisms to maintain consistency across source and destination tables. There are 3 main components. For the existing data, we have a analytical job based on apache beam which moves the data between source and destination tables. For the live traffic, we have a generic driver which applications use to perform dual writes in both source and destination tables. For any failure scenarios be it in dual writes or any consistency issues, we have a job to re-sync the inconsistent data. During this entire process, we faced many challenges and ended up pushing ScyllaDB to its limit. We tried different consistency levels, compaction strategies, optimised data models and were finally able to move 100 TB of raw data from multiple databases to ScyllaDB with minimal inconsistency.

使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化－曾書庭

台灣資料科學年會

The document discusses using Elasticsearch and Kibana for big data search and visualization. It describes some problems with traditional request-response models like long cycles and engineers being bottlenecks. It then outlines requirements for the solution like being easy to use, fast, scalable, and solving 80% of problems. Elasticsearch and Kibana are presented as a solution, with Elasticsearch indexing and storing large amounts of data and Kibana providing interactive visualizations without coding. Examples of how it has been used at Gogolook are provided, and future plans like logging all user events are discussed.

Scale Relational Database with NewSQL

PingCAP

Big Data, Fast Data @ PayPal (YOW 2018)

Sid Anand

PayPal has seen tremendous growth in recent years, processing over 7.8 billion payments transactions annually for over 227 million active customer accounts across 200+ markets and currencies. To support this scale, PayPal's data infrastructure includes over 2,000 database instances, 116 billion database calls per day, and over 74 petabytes of total storage. PayPal continues enhancing its data infrastructure to meet growing analytics and machine learning needs through technologies like Kafka, Hadoop, graph databases and real-time OLAP engines.

AWS Redshift Introduction - Big Data Analytics

Keeyong Han

Redshift is a scalable SQL database in AWS that can store up to 1.6PB of data across multiple servers. It uses a columnar data storage model that makes adding or removing columns fast. Data is uploaded from S3 using SQL COPY commands and queried using standard SQL. The document provides recommendations for getting started with Redshift, such as performing daily full refreshes initially and then implementing incremental update mechanisms to enable more frequent updates.

Aggregations at Scale for ShareChat —Using Kafka Streams and ScyllaDB

ScyllaDB

ShareChat is a social media app with ~180 MAU and 50M DAU. We capture and aggregate various engagement metrics, viz. likes, views, shares, comments, etc., at a post level to curate better content for our users. In terms of numbers for the engagement metrics, we have writes and reads happening at a scale of 55k-60k ops/sec and 290k-300k ops/sec, respectively. With these engagement metrics directly impacting users, we need a datastore that would offer lower latencies and is highly available, resilient, and scalable. It would be better if we could achieve all of these at an optimal cost. This is to learn how we accomplished the abovementioned criteria by using in-house Kafka streams and ScyllaDB.

Similar to Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward (20)

Introduction to Kafka

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at Databricks

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

ScaleDB Technical Presentation

Scalable complex event processing on samza @UBER

How QBerg scaled to store data longer, query it faster

XStream: stream processing platform at facebook

Rubyslava + PyVo #48

Nebula Graph nMeetup in Shanghai - Meet with Graph Technology Enthusiasts

Scylla Summit 2018: Kiwi.com Migration to Scylla - The Why, the How, the Fail...

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

Trending with Purpose

Data Analytics Service Company and Its Ruby Usage

ShareChat's Journey Migrating 100TB of Data to ScyllaDB with NO Downtime

使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化－曾書庭

Scale Relational Database with NewSQL

Big Data, Fast Data @ PayPal (YOW 2018)

AWS Redshift Introduction - Big Data Analytics

Aggregations at Scale for ShareChat —Using Kafka Streams and ScyllaDB

More from ScyllaDB

Optimizing NoSQL Performance Through Observability

ScyllaDB

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Event-Driven Architecture Masterclass: Challenges in Stream Processing

ScyllaDB

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

ScyllaDB

Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...

ScyllaDB

Developer Data Modeling Mistakes: From Postgres to NoSQL

ScyllaDB

See where an RDBMS-pro’s intuition leads him astray – and learn practical tips for the data modeling transition ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. However, developers new to high-performance NoSQL intuitively shoot themselves in the foot with respect to things like table design, query design, indexing, and partitioning. Watch where our experienced Postgres developer intuitively falls into traps that hurt performance and scalability. And learn with him as our database performance expert offers friendly guidance on navigating all the unexpected behaviors that tend to trip up RDBMS experts. This webinar focuses on common data modeling and querying mistakes that occur when developers move from SQL to NoSQL. For example: - Understanding query first design principles - Planning for schema evolution - Steering clear of common pitfalls and anti-patterns - Assessing data access patterns This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

What Developers Need to Unlearn for High Performance NoSQL

ScyllaDB

See where an RDBMS-pro’s intuition leads him astray – and learn practical tips for the transition ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. However, developers new to high-performance NoSQL intuitively shoot themselves in the foot with respect to things like table design, query design, indexing, and partitioning. Watch where our experienced Postgres developer intuitively falls into traps that hurt performance and scalability. And learn with him as our database performance expert offers friendly guidance on navigating all the unexpected behaviors that tend to trip up RDBMS experts. Our first webinar of this series will cover common mistakes with practices such as: - Translating the data model to NoSQL - Optimizing table design - Optimizing query performance - Planning for partitioning This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Low Latency at Extreme Scale: Proven Practices & Pitfalls

ScyllaDB

Expert tips on how to maximize your database performance at scale Untangle the complexity of achieving database performance at scale. Join this webinar to discover commonly overlooked ways to get predictable low latency, even at extreme scale. Our Solution Architects will walk you through the strategies and pitfalls learned by working on thousands of real-world distributed database projects, many reaching 1M OPS with single-digit MS latencies. In addition to offering clear recommendations, we’ll also explain the process behind how we arrived at them – so you can benefit from the lessons learned by other teams. We’ll cover how to: - Design and deploy a large-scale distributed database cluster - Optimize your clients’ interactions with it - Expand the cluster horizontally and globally - Ensure it survives whatever disasters the world throws at it

Dissecting Real-World Database Performance Dilemmas

ScyllaDB

Tackling your own database performance challenges is serious business. For a change of pace, let’s have some fun learning from other teams’ performance predicaments. Join us for an interactive session where we dissect four specific database performance challenges faced by teams considering or using ScyllaDB. For each dilemma, we'll: - Examine the context and technical requirements - Talk about potential solutions and cover the pros and cons of each - Disclose what approach the team took, and how it worked out About the speaker: Felipe is an IT specialist with years of experience on distributed systems and open-source technologies. He is one of the co-authors of "Database Performance at Scale", an Open Access, freely available publication for individuals interested on improving database performance. At ScyllaDB, he works as a Solution Architect.

Beyond Linear Scaling: A New Path for Performance with ScyllaDB

ScyllaDB

Linear scaling (sometimes near linear scaling) is often mentioned in several benchmarks, articles and product comparisons as proof that a given technology and algorithmic optimizations perform better than another. But is that really what performance is all about, and should you even care? This webinar discusses performance beyond linear scalability, including what typically matters more when running high throughput and low latency workloads at scale. We'll cover how ScyllaDB offers unparalleled performance and share our insights on: - The hidden aspects of linear scaling - When linear scaling matters most and when it’s simply irrelevant - Often overlooked considerations for optimizing and measuring distributed systems performance Watch now to learn from our experience (and lessons learned) in building the fastest NoSQL database in the world.

Dissecting Real-World Database Performance Dilemmas

ScyllaDB

Navigating Complex Database Performance Hurdles Tackling your own database performance challenges is serious business. For a change of pace, let’s have some fun learning from other teams’ performance predicaments. Join us for an interactive session where we dissect 4 specific database performance challenges faced by teams considering or using ScyllaDB. For each dilemma: - The presenters will describe the context and technical requirements - Together, we’ll talk about potential solutions and cover the pros and cons of each - Finally, we’ll disclose what approach the team took, and how it worked out Throughout the event, we’ll have opportunities to win ScyllaDB swag and prizes! Come prepared to engage in lively discussions and gain valuable insight into database performance strategies.

Database Performance at Scale Masterclass: Workload Characteristics by Felipe...

ScyllaDB

Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...

ScyllaDB

Database Performance at Scale Masterclass: Driver Strategies by Piotr Sarna

ScyllaDB

Replacing Your Cache with ScyllaDB

ScyllaDB

This document discusses replacing external caching solutions with using the internal caching capabilities of ScyllaDB. It provides examples of companies that improved performance, reduced costs and complexity by moving from Redis or Elasticsearch with an external cache to using ScyllaDB's embedded cache instead. The document also outlines some of the advantages of ScyllaDB's cache like improved latency, coherency with the database and observability compared to external caching layers.

Powering Real-Time Apps with ScyllaDB_ Low Latency & Linear Scalability

ScyllaDB

Discover how your team can achieve low latency at the extreme scale that your data-intensive applications require. We’ll walk you through an example of how ScyllaDB scales linearly to achieve 1M and then 2M OPS – with <1ms P99 latency. We’ll cover how this works on a sample realtime app (an ML feature store), share best practices for performance, and talk about the most important tradeoffs you’ll need to negotiate. Join us to learn: - Why and how to ensure your database takes full advantage of your cloud infrastructure - What architectural considerations matter most for high throughput and low latency - Key factors to consider when selecting a high-performance database

7 Reasons Not to Put an External Cache in Front of Your Database.pptx

ScyllaDB

This document discusses the pros and cons of placing an external cache in front of a database. It introduces Tomasz Grabiec and Tzach Livyatan from ScyllaDB and describes ScyllaDB's optimized internal caching design. External caches can increase latency and costs while ignoring the database's context and workload knowledge. ScyllaDB embeds its cache to minimize overhead and ensure data and query awareness. The document shares customer examples that improved performance and reduced costs by moving from cached databases to ScyllaDB.

Getting the most out of ScyllaDB

ScyllaDB

Expert tips on how to maximize your database potential If you’re considering or getting started with ScyllaDB, you’re probably intrigued by its potential to achieve high throughput and predictable low latency at a reasonable cost. So how do you ensure that you’re maximizing that potential for your team’s specific workloads and use case? This webinar offers practical advice for navigating the various decision points you’ll face as you assess whether ScyllaDB is a good fit for your team and later roll it out into production. We’ll cover the most critical considerations, tradeoffs, and recommendations related to: - Infrastructure selection - ScyllaDB configuration - Client-side setup - Data modeling

NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration

ScyllaDB

NoSQL Database Migration Masterclass - Session 3: Migration Logistics

ScyllaDB

NoSQL Data Migration Masterclass - Session 1 Migration Strategies and Challenges

ScyllaDB

More from ScyllaDB (20)

Optimizing NoSQL Performance Through Observability

Event-Driven Architecture Masterclass: Challenges in Stream Processing

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...

Developer Data Modeling Mistakes: From Postgres to NoSQL

What Developers Need to Unlearn for High Performance NoSQL

Low Latency at Extreme Scale: Proven Practices & Pitfalls

Dissecting Real-World Database Performance Dilemmas

Beyond Linear Scaling: A New Path for Performance with ScyllaDB

Dissecting Real-World Database Performance Dilemmas

Database Performance at Scale Masterclass: Workload Characteristics by Felipe...

Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...

Database Performance at Scale Masterclass: Driver Strategies by Piotr Sarna

Replacing Your Cache with ScyllaDB

Powering Real-Time Apps with ScyllaDB_ Low Latency & Linear Scalability

7 Reasons Not to Put an External Cache in Front of Your Database.pptx

Getting the most out of ScyllaDB

NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration

NoSQL Database Migration Masterclass - Session 3: Migration Logistics

NoSQL Data Migration Masterclass - Session 1 Migration Strategies and Challenges

Recently uploaded

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

Łukasz Chruściel

Revolutionizing Visual Effects Mastering AI Face Swaps.pdf

Undress Baby

The quest for the best AI face swap solution is marked by an amalgamation of technological prowess and artistic finesse, where cutting-edge algorithms seamlessly replace faces in images or videos with striking realism. Leveraging advanced deep learning techniques, the best AI face swap tools meticulously analyze facial features, lighting conditions, and expressions to execute flawless transformations, ensuring natural-looking results that blur the line between reality and illusion, captivating users with their ingenuity and sophistication. Web:- https://undressbaby.com/

Artificia Intellicence and XPath Extension Functions

Octavian Nadolu

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

What is Augmented Reality Image Tracking

pavan998932

Hand Rolled Applicative User ValidationCode Kata

Philip Schwarz

Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>? The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory. Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away. The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

Need for Speed: Removing speed bumps from your Symfony projects ⚡️

Łukasz Chruściel

No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception. In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed. We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Green Software Development

GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)

Alina Yurenko

SWEBOK and Education at FUSE Okinawa 2024

Hironori Washizaki

socradar-q1-2024-aviation-industry-report.pdf

SOCRadar

SOCRadar's Aviation Industry Q1 Incident Report is out now! The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers. SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Google

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-fusion-buddy-review AI Fusion Buddy Review: Key Features ✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini ✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique! ✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs! ✅Fully automated AI articles bulk generation! ✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more. ✅With one keyword or URL, generate complete websites, landing pages, and more… ✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7. ✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches. ✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all! ✅Save over $5000 per year and kick out dependency on third parties completely! ✅Brand New App: Not available anywhere else! ✅ Beginner-friendly! ✅ZERO upfront cost or any extra expenses ✅Risk-Free: 30-Day Money-Back Guarantee! ✅Commercial License included! See My Other Reviews Article: (1) AI Genie Review: https://sumonreview.com/ai-genie-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIFusionBuddyReview, #AIFusionBuddyFeatures, #AIFusionBuddyPricing, #AIFusionBuddyProsandCons, #AIFusionBuddyTutorial, #AIFusionBuddyUserExperience #AIFusionBuddyforBeginners, #AIFusionBuddyBenefits, #AIFusionBuddyComparison, #AIFusionBuddyInstallation, #AIFusionBuddyRefundPolicy, #AIFusionBuddyDemo, #AIFusionBuddyMaintenanceFees, #AIFusionBuddyNewbieFriendly, #WhatIsAIFusionBuddy?, #HowDoesAIFusionBuddyWorks

Orion Context Broker introduction 20240604

Fermin Galan

Recently uploaded (20)

Vitthal Shirke Java Microservices Resume.pdf

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

Revolutionizing Visual Effects Mastering AI Face Swaps.pdf

Artificia Intellicence and XPath Extension Functions

Mobile App Development Company In Noida | Drona Infotech

May Marketo Masterclass, London MUG May 22 2024.pdf

What is Augmented Reality Image Tracking

Hand Rolled Applicative User ValidationCode Kata

Energy consumption of Database Management - Florina Jonuzi

Need for Speed: Removing speed bumps from your Symfony projects ⚡️

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)

SWEBOK and Education at FUSE Okinawa 2024

socradar-q1-2024-aviation-industry-report.pdf

openEuler Case Study - The Journey to Supply Chain Security

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

GraphSummit Paris - The art of the possible with Graph Technology

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Orion Context Broker introduction 20240604

Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward

1. Grab & Scylla: Driving SEA Forward Aravind Velamur Srinivasan Software Engineer, Grab Technology Corp

2. Presenter bio ▪ Lead Software Engineer at Grab Technologies • Streaming Platform • Data Platform Infrastructure ▪ Worked at Uber Inc prior to Grab ▪ Masters in Computer Science from University of Virginia (UVa) ▪ Contact: aravindvelamur@gmail.com

3. Agenda ▪ Background ▪ Motivation for using ScyllaDB ▪ ScyllaDB use at Grab ▪ Conclusion ▪ Q&A

4. Background

5. Introduction to Grab ▪ One of the most frequently used mobile platforms in Southeast Asia ▪ Multiple services (Multi-million users per day and growing!) • Transport • Food • Payment • Shopping • Package Delivery • ...

6. Motivation

7. Why ScyllaDB ▪ Grab Tech • Microservices (Golang) • Kafka for streaming • Why? Loose coupling and Scale! doh! :)

8. Why ScyllaDB ● Stream Processing ● Transforms ● Aggregates ● Joins ● ...

9. Why ScyllaDB ▪ Required a State store (metadata store) which has the following characteristics: • Ability to handle very high throughput • Ability to handle bursts (doh!) • Ability to scale out (i.e, handle hockey stick like growth) • Very low latencies (both write and reads) - Near RealTime • Low operational overhead • Cost efficient

10. Why ScyllaDB

11. ScyllaDB USAGE

12. Number of Use cases: ▪ Fraud detection ▪ Stream Aggregation & ▪ Stream Statistics ▪ ... ScyllaDB Usage

13. ▪ Background https://www.scmp.com/week-asia/business/article/2154770/who-pays-when-indonesian-ride- sharing-fraud-goes-full-throttle https://www.techrepublic.com/article/seven-ride-sharing-scams-to-watch-out-for/ ScyllaDB Usage - Fraud Detection

14. ▪ Use Kafka streams to do real-time flagging of rides ▪ Might require combining multiple streams together ▪ Requires • Low latency • Ability to handle high throughput ScyllaDB Usage - Fraud Detection

15. ScyllaDB Usage - Stream Statistics

16. ScyllaDB Usage - Stream Statistics ▪ Teams like to find out: • Counts of … (eg: count of rides in the past month, etc) • Raw stats particular to a key for help in debugging • Business users - require stats on city/country data

17. ScyllaDB Usage - Stream Statistics ▪ Requires a fast store for raw stats ▪ Ability to build Time series on top of it. • Hello KairosDB :) ...

18. ScyllaDB Usage - Numbers ▪ One 15 node cluster (version 2.2) ▪ Running on AWS - EC2 ▪ Single DC (may be multi-DC in the future) ▪ Overall ~15TB data across all nodes

19. ScyllaDB Usage - Numbers

20. Conclusion

21. ScyllaDB at Grab So far.. ▪ Overall • Great experience • Cost effective • Very responsive team • As good as advertised :) • Growing within the company…

22. ScyllaDB at Grab So far.. ▪ Some hiccups • nodetool repair - Writes and read timeouts every time when the run finishes (even with -pr) • Wish nodes can join the cluster faster • Right now takes a really long time (approx 1TB data on each node across all keyspaces) - takes ~2.5 hours to add a node • Better error logging

23. Thank You Any Questions ? Please stay in touch aravindvelamur@gmail.com

Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward

Similar to Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward (20)

More from ScyllaDB

More from ScyllaDB (20)

Recently uploaded

Recently uploaded (20)

Scylla Summit 2018: Grab and Scylla: Driving Southeast Asia Forward