Scaling Hike Messenger to 15M Users

•

13 likes•8,658 views

Rajat Bansal, CTO of Hike, discusses how the company grew its use of MongoDB from a single replica set on EC2 to multiple replica sets and sharded clusters to support scaling. He explains that MongoDB's ease of setup and administration allowed Hike to grow quickly with a small team. Outages taught Hike about optimizing for array operations, managing disk I/O, and protecting systems from full collection scans. Bansal emphasizes putting mechanisms in place to safeguard production systems and notes that proactive monitoring and production support help avoid issues.

1
Rajat Bansal
Chief Technology Ofﬁcer
Scaling to Millions, FAST!

2
- The Hike Journey!
- Why MongoDB?!
- MongoDB Use Cases:Overview and Deep Dive!
- Lessons learned & Next Steps
Agenda

3
The fastest growing !
Made in India IM App!

15
Why MongoDB?
Storage decisions governed by CAP Theorem

16
Why MongoDB?
CConsistency
Availability Partition Tolerance
Cost
Availability 
of Talent
Production Support
A P

17
Cost
- Start Small, Grow Big, FAST, REALLY FAST!
- Started with 1 replica set on EC2!
- Grew to multiple replica sets, sharded clusters

18
Availability
- Small team of 3 people writing the entire server!
- Easy Ramp-up and management!
- Low setup and administration requirements

19
Production Support
- Cost of downtime is very high. !
- Small team needed support in downtime. !
- Decision to take Production Support proved
life-saving in outages

20
User Proﬁle Store!
!! - 3000 reads / 500 writes per sec!
!
Temporary Message Store!
!! - 1000 reads / 3900 writes per sec!
!! !
Other Miscellaneous usage: Grid FS etc
HIKE’s MongoDB Use-cases

21
Ofﬂine Message Store!
!! - App Level Sharding!
!! - 4 Mongo instances with 32 DB each!
!! - Horizontally scalable upto 128 instances when needed!
!! - Tested upto 30K Ops in simulated environment !
!! - Protected by “Redis” layer to reduce queries!
!! - Latencies < 1ms
HIKE MongoDB Architecture

Primary!
Secondary!
Secondary!
Mongod-1
Primary!
Secondary!
Secondary!
Mongod-2
Primary!
Secondary!
Secondary!
Mongod-3
Primary!
Secondary!
Secondary!
Mongod-4
Shard Manager
32 dbs each
App Layer

23
User Proﬁle Store

- Replica Set (1 primary, 2 Secondary)

- Writes to Primary

- Reads from Secondary

- Latencies < 1ms
HIKE MongoDB Architecture

24
mongoDB (Happy State < 1ms)
0.65ms
0.80ms

27
Outage 1!
!! - Latencies went over the roof “1ms —> 1000ms”!
!! - What went wrong: Lot of operations on “Arrays”!
!! - “Production Support” to the rescue! !
!!
“Adding and modifying array entries can require a scan of
much or all of each array being updated, resulting in slow
operations"
HIKE Learnings

29
Outage 2!
!! - Latencies increased 20-50X!
!! - What went wrong: !
!! ! ! - Disk I/O was bottleneck!
!! ! ! - “ReadAhead” was high!
!! !
!“Read/Write Throughput Exceeds I/O”
HIKE Learnings

31
Outage 3!
!! - MongoDB crashed!
!! - Adhoc Script doing fullTableScan !
!! - Need to protect your systems “noTableScan” !
!
!
“Protect your production systems. Use the mechanisms
available”! !
HIKE Learnings

32
- Proactive Health Checks
- Production Support Helps
- Put Mechanisms to safeguard production
HIKE Learnings

What's hot

Bond with pidilite case study 1

AyushiGupta996429

Hike messenger

yadavprachi

SALES AND DISTRIBUTION CASE STUDY Balaji wafer

rutikaingle1

Project on marketing study of parlePayal Awere

Pidilite Channelmgmtankushmit

Asian paints Sales and Distribution strategy

Sowjanya Jangam

Flipkart and Myntra M&A

Arpana Masih

“Comparative Analysis Of Frooti And It’s Competitors In Rasayani”

abhijit055

Maggi noodles Case Study

Daksh Raj Chopra

Distribution & Channel Management, Promotion Decisions OF ITC Limited

Reyaz Jafar

Main ppt on coca colaSuravarapu Mohana Vamsi

Ppt on coca cola

Orissa university of agricultural and technology

Report on Pepsico India Market Research Analysis

Ashish Pandey

Aashirvaad aata 4 p's

MeghnaJaiswal5

strategic analysis of ITC

Vinni Kaur

Rural Marketing DaburVivek Upadhyay

Case Study: Alibaba's Expansion

Jean-Baptiste Bard

E commerce industrySwayam Dey

Project report affect on buying behaviour of branding

Tripureshwar Sah

A project report on analysis on customer of big bazaar

Babasab Patil

What's hot (20)

Bond with pidilite case study 1

Hike messenger

SALES AND DISTRIBUTION CASE STUDY Balaji wafer

Project on marketing study of parle

Pidilite Channelmgmt

Asian paints Sales and Distribution strategy

Flipkart and Myntra M&A

“Comparative Analysis Of Frooti And It’s Competitors In Rasayani”

Maggi noodles Case Study

Distribution & Channel Management, Promotion Decisions OF ITC Limited

Main ppt on coca cola

Ppt on coca cola

Report on Pepsico India Market Research Analysis

Aashirvaad aata 4 p's

strategic analysis of ITC

Rural Marketing Dabur

Case Study: Alibaba's Expansion

E commerce industry

Project report affect on buying behaviour of branding

A project report on analysis on customer of big bazaar

Similar to Scaling Hike Messenger to 15M Users

Scaling and Transaction Futures

MongoDB

Ops Jumpstart: Admin 101

MongoDB

Kafka at Peak Performance

Todd Palino

Big Data means big hardware, and the less of it we can use to do the job properly, the better the bottom line. Apache Kafka makes up the core of our data pipelines at many organizations, including LinkedIn, and we are on a perpetual quest to squeeze as much as we can out of our systems, from Zookeeper, to the brokers, to the various client applications. This means we need to know how well the system is running, and only then can we start turning the knobs to optimize it. In this talk, we will explore how best to monitor Kafka and its clients to assure they are working well. Then we will dive into how to get the best performance from Kafka, including how to pick hardware and the effect of a variety of configurations in both the broker and clients. We’ll also talk about setting up Kafka for no data loss.

MongoFr : MongoDB as a log Collector

Pierre Baillet

MongoDB Days Silicon Valley: Jumpstart: Ops/Admin 101

MongoDB

Presented by Achille Brighton, Principal Consulting Engineer, MongoDB Experience level: Introductory New to MongoDB? We'll provide an overview of installation, high availability through replication, scale out through sharding, and options for monitoring and backup. No prior knowledge of MongoDB is assumed. This session will jumpstart your knowledge of MongoDB operations, providing you with context for the rest of the day's content.

Operating MongoDB in the Cloud

MongoDB

Rakuten Ichiba_Rakuten Technology Conference 2016

Rakuten Group, Inc.

●Overall introduction of Ichiba Introduction ●Redis Cluster in Rakuten Ichiba How we use Redis Cluster in Rakuten Ichiba ●R Framework The challenge of updating a legacy system sharing code between multiple teams, using an in-house developed library for the Rakuten Ichiba Frontend side. ●Rakuten Catalog Platform- Classification Approach for 280,000,000 Ichiba items - 1. Taxonomy Strategy(Analyze, Adoption) 2. Rakuten Catalog Platform Classification Ichiba Item data -> Taxonomy(Taxonomy(Genre/Tag/Attribute) management/development) -> Catalog(Product Master) -> Data governance system -> Data Processing Unit -> Auto classification(Item information/Image) ●How to reconstruct a million-user app Describes why we decided to rewrite our app, what difficulties we faced and how we create the new structure to ensure it's flexible, stable and maintainable. https://tech.rakuten.co.jp/

NoSQL into E-Commerce: lessons learned

La FeWeb

How to serve 2500 Ad requests per second

Miguel Sousa Filipe

SAPO's text advertisement system is a service that provides contextualized text-ads on webpages or searchpages developed in house by SAPO for 4 years now. The current implementation serves at peak +90M ad requests per day; It peaks at ~ 2500 requests per second. Average latency is 33 milisecs 99.9% of requests are served under 150 milisecs The goal of this presentation is to broadly describe how such a system works and focus on the following topics: - functionality/features - performance - availability & reliability - technologies and software used.

MongoDB Europe 2016 - Powering Microservices with Docker, Kubernetes, and Kafka

MongoDB

Organisations are building their applications around microservice architectures because of the flexibility, speed of delivery, and maintainability they deliver. This session introduces you to technologies such as Docker, Kubernetes & Kafka which are driving the microservices revolution. Learn about containers and orchestration – and most importantly how to exploit them for stateful services such as MongoDB.

Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

MongoDB

This session introduced technologies such as Docker, Kubernetes, and Kafka, which are driving the microservices revolution. Learn about containers and orchestration – and most importantly, how to exploit them for stateful services such as MongoDB. What You Will Learn: * Why organizations are choosing to use microservice architectures, what benefits they deliver, and when they should – or shouldn't – be used. * Technologies that are used to build microservices – including containers, orchestration, and messaging systems. * Why MongoDB is a good fit for microservices and what special steps need to be taken to make them work well together.

Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

MongoDB

High Frequency Trading and NoSQL database

Peter Lawrey

20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02

Francisco Gonçalves

Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB

MongoDB

Speaker: Andrew Morgan, Principal Product Marketing Manager, MongoDB Level: 100 (Beginner) Track: Microservices Organizations are building their applications around microservice architectures because of the flexibility, speed of delivery, and maintainability they deliver. Want to try out MongoDB on your laptop? Execute a single command and you have a lightweight, self-contained sandbox; another command removes all trace when you're done. Replicate your complete application for your development, test, operations, and support teams. This session introduces you to technologies such as Docker, Kubernetes, and Kafka, which are driving the microservices revolution. Learn about containers and orchestration – and most importantly, how to exploit them for stateful services such as MongoDB. What You Will Learn: - Why organizations are choosing to use microservice architectures, what benefits they deliver, and when they should - or shouldn't - be used. - Technologies that are used to build microservices – including containers, orchestration and messaging systems. - Why MongoDB is a good fit for microservices and what special steps need to be taken to make them work well together.

Circonus: Design failures - A Case Study

Heinrich Hartmann

Get More Out of MongoDB with TokuMX

Tim Callaghan

Benchmarking, Load Testing, and Preventing Terrible Disasters

MongoDB

"Have you ever crossed your fingers before performing an upgrade or switching storage engines, because you weren't quite sure what would happen? Have you ever been bitten by a slight change in behavior that turned out to be unexpectedly significant for your workload? At Parse we have developed a workflow that lets us repeatedly capture and replay real production workloads offline. This has allowed us to confidently perform upgrades across a large fleet with a minimum amount of canarying, and has helped us load test a variety of storage engines with real workloads so we can compare and understand the performance tradeoffs. In this talk we will cover best practices for upgrades and migrations, and we will walk through how to use our open-sourced tooling to demonstrate how you can do the same. We will also share some fun war stories about various disasters found and averted *before* putting them into production thanks to offline benchmarking."

Bandwidth, Throughput, Iops, And Flops

billmenger

Richardrodger nodeday-2014-final

Richard Rodger

This talk is an appeal to server-side JavaScript developers to make use of this time of change - Node.js is going to become the primary server-side platform for most developers. We can move forward from the old way of building web apps as large inter-locking co-dependent code bases. The Node.js module system has shown us the way. It's the first step. Now, we need to use the beauty of Node modules to help us build robust, scalable apps. This approach is called the Micro-Services Architecture. It's more than just having some services with HTTP end-points. It's about taking this to the extreme. Everything is a service, and no service is larger than 100 lines of code. We've been using this approach for most of our projects for the last 18 months and it works really well. We get to drop loads of project management ceremony. There will be some customer war stories.

Similar to Scaling Hike Messenger to 15M Users (20)

Scaling and Transaction Futures

Ops Jumpstart: Admin 101

Kafka at Peak Performance

MongoFr : MongoDB as a log Collector

MongoDB Days Silicon Valley: Jumpstart: Ops/Admin 101

Operating MongoDB in the Cloud

Rakuten Ichiba_Rakuten Technology Conference 2016

NoSQL into E-Commerce: lessons learned

How to serve 2500 Ad requests per second

MongoDB Europe 2016 - Powering Microservices with Docker, Kubernetes, and Kafka

Powering Microservices with Docker, Kubernetes, Kafka, & MongoDB

High Frequency Trading and NoSQL database

20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02

Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB

Circonus: Design failures - A Case Study

Get More Out of MongoDB with TokuMX

Benchmarking, Load Testing, and Preventing Terrible Disasters

Bandwidth, Throughput, Iops, And Flops

Richardrodger nodeday-2014-final

More from MongoDB

MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas

MongoDB

During this talk we'll navigate through a customer's journey as they migrate an existing MongoDB deployment to MongoDB Atlas. While the migration itself can be as simple as a few clicks, the prep/post effort requires due diligence to ensure a smooth transfer. We'll cover these steps in detail and provide best practices. In addition, we’ll provide an overview of what to consider when migrating other cloud data stores, traditional databases and MongoDB imitations to MongoDB Atlas.

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!

MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...

MongoDB

MongoDB Kubernetes operator and MongoDB Open Service Broker are ready for production operations. Learn about how MongoDB can be used with the most popular container orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications. A demo will show you how easy it is to enable MongoDB clusters as an External Service using the Open Service Broker API for MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB

MongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...

MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data

MongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB SoCal 2020: MongoDB Atlas Jump Start

MongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]

MongoDB

Our clients have unique use cases and data patterns that mandate the choice of a particular strategy. To implement these strategies, it is mandatory that we unlearn a lot of relational concepts while designing and rapidly developing efficient applications on NoSQL. In this session, we will talk about some of our client use cases, the strategies we have adopted, and the features of MongoDB that assisted in implementing these strategies.

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2

MongoDB

Encryption is not a new concept to MongoDB. Encryption may occur in-transit (with TLS) and at-rest (with the encrypted storage engine). But MongoDB 4.2 introduces support for Client Side Encryption, ensuring the most sensitive data is encrypted before ever leaving the client application. Even full access to your MongoDB servers is not enough to decrypt this data. And better yet, Client Side Encryption can be enabled at the "flick of a switch". This session covers using Client Side Encryption in your applications. This includes the necessary setup, how to encrypt data without sacrificing queryability, and what trade-offs to expect.

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...

MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!

MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset

MongoDB

When you need to model data, is your first instinct to start breaking it down into rows and columns? Mine used to be too. When you want to develop apps in a modern, agile way, NoSQL databases can be the best option. Come to this talk to learn how to take advantage of all that NoSQL databases have to offer and discover the benefits of changing your mindset from the legacy, tabular way of modeling data. We’ll compare and contrast the terms and concepts in SQL databases and MongoDB, explain the benefits of using MongoDB compared to SQL databases, and walk through data modeling basics so you feel confident as you begin using MongoDB.

MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart

MongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...

MongoDB

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++

MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...

MongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive

MongoDB

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang

MongoDB

Virtual assistants are becoming the new norm when it comes to daily life, with Amazon’s Alexa being the leader in the space. As a developer, not only do you need to make web and mobile compliant applications, but you need to be able to support virtual assistants like Alexa. However, the process isn’t quite the same between the platforms. How do you handle requests? Where do you store your data and work with it to create meaningful responses with little delay? How much of your code needs to change between platforms? In this session we’ll see how to design and develop applications known as Skills for Amazon Alexa powered devices using the Go programming language and MongoDB.

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...

MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...

MongoDB

Il n’a jamais été aussi facile de commander en ligne et de se faire livrer en moins de 48h très souvent gratuitement. Cette simplicité d’usage cache un marché complexe de plus de 8000 milliards de $. La data est bien connu du monde de la Supply Chain (itinéraires, informations sur les marchandises, douanes,…), mais la valeur de ces données opérationnelles reste peu exploitée. En alliant expertise métier et Data Science, Upply redéfinit les fondamentaux de la Supply Chain en proposant à chacun des acteurs de surmonter la volatilité et l’inefficacité du marché.

More from MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data

MongoDB SoCal 2020: MongoDB Atlas Jump Start

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset

MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...

Recently uploaded

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

SOFTTECHHUB

As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

20 Comprehensive Checklist of Designing and Developing a Website

Pixlogix Infotech

Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

GridMate - End to end testing is a critical piece to ensure quality and avoid...

ThomasParaiso2

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

Mind map of terminologies used in context of Generative AI

Kumud Singh

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

Recently uploaded (20)

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Full-RAG: A modern architecture for hyper-personalization

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

A tale of scale & speed: How the US Navy is enabling software delivery from l...

UiPath Test Automation using UiPath Test Suite series, part 5

20240607 QFM018 Elixir Reading List May 2024

Essentials of Automations: The Art of Triggers and Actions in FME

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

20 Comprehensive Checklist of Designing and Developing a Website

Removing Uninteresting Bytes in Software Fuzzing

Artificial Intelligence for XMLDevelopment

How to Get CNIC Information System with Paksim Ga.pptx

DevOps and Testing slides at DASA Connect

Climate Impact of Software Testing at Nordic Testing Days

Generative AI Deep Dive: Advancing from Proof of Concept to Production

GridMate - End to end testing is a critical piece to ensure quality and avoid...

20240609 QFM020 Irresponsible AI Reading List May 2024

20240605 QFM017 Machine Intelligence Reading List May 2024

Mind map of terminologies used in context of Generative AI

National Security Agency - NSA mobile device best practices

Scaling Hike Messenger to 15M Users

1. 1 Rajat Bansal Chief Technology Ofﬁcer Scaling to Millions, FAST!

2. 2 - The Hike Journey! - Why MongoDB?! - MongoDB Use Cases:Overview and Deep Dive! - Lessons learned & Next Steps Agenda

3. 3 The fastest growing ! Made in India IM App!

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12

13. 13 AND TODAY

14. 14

15. 15 Why MongoDB? Storage decisions governed by CAP Theorem

16. 16 Why MongoDB? CConsistency Availability Partition Tolerance Cost Availability  of Talent Production Support A P

17. 17 Cost - Start Small, Grow Big, FAST, REALLY FAST! - Started with 1 replica set on EC2! - Grew to multiple replica sets, sharded clusters

18. 18 Availability - Small team of 3 people writing the entire server! - Easy Ramp-up and management! - Low setup and administration requirements

19. 19 Production Support - Cost of downtime is very high. ! - Small team needed support in downtime. ! - Decision to take Production Support proved life-saving in outages

20. 20 User Proﬁle Store! !! - 3000 reads / 500 writes per sec! ! Temporary Message Store! !! - 1000 reads / 3900 writes per sec! !! ! Other Miscellaneous usage: Grid FS etc HIKE’s MongoDB Use-cases

21. 21 Ofﬂine Message Store! !! - App Level Sharding! !! - 4 Mongo instances with 32 DB each! !! - Horizontally scalable upto 128 instances when needed! !! - Tested upto 30K Ops in simulated environment ! !! - Protected by “Redis” layer to reduce queries! !! - Latencies < 1ms HIKE MongoDB Architecture

22. Primary! Secondary! Secondary! Mongod-1 Primary! Secondary! Secondary! Mongod-2 Primary! Secondary! Secondary! Mongod-3 Primary! Secondary! Secondary! Mongod-4 Shard Manager 32 dbs each App Layer

23. 23 User Proﬁle Store - Replica Set (1 primary, 2 Secondary) - Writes to Primary - Reads from Secondary - Latencies < 1ms HIKE MongoDB Architecture

24. 24 mongoDB (Happy State < 1ms) 0.65ms 0.80ms

25. 25 mongoDB (1 Year Timeline)

26. 26 mongoDB (Outage 1) Outage 1

27. 27 Outage 1! !! - Latencies went over the roof “1ms —> 1000ms”! !! - What went wrong: Lot of operations on “Arrays”! !! - “Production Support” to the rescue! ! !! “Adding and modifying array entries can require a scan of much or all of each array being updated, resulting in slow operations" HIKE Learnings

28. 28 mongoDB (Outage 2) Outage 2

29. 29 Outage 2! !! - Latencies increased 20-50X! !! - What went wrong: ! !! ! ! - Disk I/O was bottleneck! !! ! ! - “ReadAhead” was high! !! ! !“Read/Write Throughput Exceeds I/O” HIKE Learnings

30. 30 mongoDB (Outage 3) Outage 3

31. 31 Outage 3! !! - MongoDB crashed! !! - Adhoc Script doing fullTableScan ! !! - Need to protect your systems “noTableScan” ! ! ! “Protect your production systems. Use the mechanisms available”! ! HIKE Learnings

32. 32 - Proactive Health Checks - Production Support Helps - Put Mechanisms to safeguard production HIKE Learnings

33. http://hike.in @hikeapp 33

Scaling Hike Messenger to 15M Users

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scaling Hike Messenger to 15M Users

Similar to Scaling Hike Messenger to 15M Users (20)

More from MongoDB

More from MongoDB (20)

Recently uploaded

Recently uploaded (20)

Scaling Hike Messenger to 15M Users