SlideShare a Scribd company logo

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

•Download as PPT, PDF•
•
Rick Copeland
Rick Copeland

Zarkov is a real-time analytics platform that uses MongoDB, Python, Gevent, and ZeroMQ to aggregate and analyze event data. It handles high volumes of event ingestion and uses a lightweight map/reduce framework to incrementally calculate statistics. The system was developed at SourceForge.net to monitor user and project activity in real-time and has proven capable of replacing other analytics solutions.

Technology
1 of 22
Realtime Analytics using MongoDB, Python, Gevent, and ZeroMQ Rick Copeland @rick446 [email_address]
SourceForge s MongoDB ,[object Object],[object Object],[object Object]
The Problem We’re Trying to Solve ,[object Object],[object Object],[object Object],[object Object]
Introducing Zarkov 0.0.1 ,[object Object],[object Object],[object Object]
Zarkov Architecture MongoDB BSON over ZeroMQ Journal Greenlet Commit Greenlet Write-ahead log Write-ahead log Aggregation Greenlet
Technologies ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
“Wow, it’s really fast; can it replace…” ,[object Object],[object Object],[object Object],“Probably, but it’ll take some work….”
Moving towards production.... ,[object Object],[object Object],[object Object],[object Object],[object Object]
Zarkov’s already doing a lot… ,[object Object],[object Object],[object Object],[object Object],[object Object]
Quick Map/reduce Refresher ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Quick Map/reduce Refresher ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Parallel
Zarkov Map/Reduce Architecture map_in_#.bson Query Map Sort Reduce Commit map_out_#.bson reduce_in.bson Job Mgr
Zarkov Map/Reduce ,[object Object],[object Object],[object Object],[object Object]
Zarkov Web Service ,[object Object],[object Object],[object Object],[object Object],[object Object]
Zarkov Deployment at SF.net
Lessons learned at
MongoDB Tricks ,[object Object],[object Object],class IdGen ( object): @classmethod def get_ids(cls, inc = 1): obj = cls.query.find_and_modify( query={ '_id': 0}, update ={ '$inc': dict(inc =inc), }, upsert= True, new = True) return range(obj .inc - inc, obj.inc)
MongoDB Pitfalls ,[object Object],[object Object],[object Object],[object Object]
Open Source Ming http://sf.net/projects/merciless/ MIT License Allura http://sf.net/p/allura/ Apache License Zarkov http://sf.net/p/zarkov/ Apache License
Future Work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Rick Copeland @rick446 [email_address]
Credits ,[object Object]

Recommended

Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.io
Rick Copeland
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMR
Shinji Tanaka
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
Bartosz Konieczny
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
Http capturing
Http capturingHttp capturing
Http capturing
Eric Ahn
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
Bram Vogelaar
 
2015 555 kharchenko_ppt
2015 555 kharchenko_ppt2015 555 kharchenko_ppt
2015 555 kharchenko_ppt
Maxym Kharchenko
 
Html5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webglHtml5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webgl
Kilian Valkhof
 

Recommended

Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.io
Rick Copeland
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMR
Shinji Tanaka
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
Bartosz Konieczny
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
Http capturing
Http capturingHttp capturing
Http capturing
Eric Ahn
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
Bram Vogelaar
 
2015 555 kharchenko_ppt
2015 555 kharchenko_ppt2015 555 kharchenko_ppt
2015 555 kharchenko_ppt
Maxym Kharchenko
 
Html5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webglHtml5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webgl
Kilian Valkhof
 
Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7
Masahiro Nagano
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
Ismael Celis
 
Commit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - extCommit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - ext
Maxym Kharchenko
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
Дэв Тим Афс
 
Cross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App EngineCross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App Engine
Andy McKay
 
PSGI and Plack from first principles
PSGI and Plack from first principlesPSGI and Plack from first principles
PSGI and Plack from first principles
Perl Careers
 
Presto overview
Presto overviewPresto overview
Presto overview
Shixiong Zhu
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
Bartosz Konieczny
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
GlynnForrest
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
Odoo
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
benwaine
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
Alejandro E Brito Monedero
 
Serverless Ballerina
Serverless BallerinaServerless Ballerina
Serverless Ballerina
Ballerina
 
All you need to know about the JavaScript event loop
All you need to know about the JavaScript event loopAll you need to know about the JavaScript event loop
All you need to know about the JavaScript event loop
Saša Tatar
 
Asynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time MessagingAsynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time Messaging
Steve Rhoades
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
Workhorse Computing
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
Locaweb
 
Visualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LVVisualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LV
Maxym Kharchenko
 
Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
PyData
 
Big Data Monitoring Cockpit
Big Data Monitoring CockpitBig Data Monitoring Cockpit
Big Data Monitoring Cockpit
Stefan Bergstein
 

More Related Content

What's hot

Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7
Masahiro Nagano
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
Odoo
 

What's hot (20)

Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
 
Commit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - extCommit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - ext
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
 
Cross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App EngineCross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App Engine
 
PSGI and Plack from first principles
PSGI and Plack from first principlesPSGI and Plack from first principles
PSGI and Plack from first principles
 
Presto overview
Presto overviewPresto overview
Presto overview
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
 
Serverless Ballerina
Serverless BallerinaServerless Ballerina
Serverless Ballerina
 
All you need to know about the JavaScript event loop
All you need to know about the JavaScript event loopAll you need to know about the JavaScript event loop
All you need to know about the JavaScript event loop
 
Asynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time MessagingAsynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time Messaging
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
 
Visualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LVVisualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LV
 

Viewers also liked

Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
PyData
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
MongoDB
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
fcrippa
 
Ing. Economica
Ing. EconomicaIng. Economica
Ing. Economica
Ger Bless
 
Apuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computoApuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computo
iliana52
 
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Joel A. Gómez Treviño
 
La clase de educacion fisica
La clase de educacion fisicaLa clase de educacion fisica
La clase de educacion fisica
Anto Anzevino
 

Viewers also liked (20)

Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
 
Big Data Monitoring Cockpit
Big Data Monitoring CockpitBig Data Monitoring Cockpit
Big Data Monitoring Cockpit
 
Super Fast Gevent Introduction
Super Fast Gevent IntroductionSuper Fast Gevent Introduction
Super Fast Gevent Introduction
 
gunicorn introduction
gunicorn introductiongunicorn introduction
gunicorn introduction
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
 
How to monitor MongoDB
How to monitor MongoDBHow to monitor MongoDB
How to monitor MongoDB
 
presentación para el Cecec2013
presentación para el Cecec2013 presentación para el Cecec2013
presentación para el Cecec2013
 
Ensayo 7
Ensayo 7Ensayo 7
Ensayo 7
 
Processo Administrativo Interno ARSESP
Processo Administrativo Interno ARSESPProcesso Administrativo Interno ARSESP
Processo Administrativo Interno ARSESP
 
Ing. Economica
Ing. EconomicaIng. Economica
Ing. Economica
 
Apuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computoApuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computo
 
Demande A Dieu
Demande A DieuDemande A Dieu
Demande A Dieu
 
Innovación educativa con recursos abiertos
Innovación educativa con recursos abiertosInnovación educativa con recursos abiertos
Innovación educativa con recursos abiertos
 
100 a 189 exercícios de portugues
100 a 189 exercícios de portugues100 a 189 exercícios de portugues
100 a 189 exercícios de portugues
 
Paper calculo
Paper calculoPaper calculo
Paper calculo
 
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
 
Metacognicion
MetacognicionMetacognicion
Metacognicion
 
Summon: The Path of Least Resistance
Summon: The Path of Least ResistanceSummon: The Path of Least Resistance
Summon: The Path of Least Resistance
 
La clase de educacion fisica
La clase de educacion fisicaLa clase de educacion fisica
La clase de educacion fisica
 

Similar to Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rick Copeland
 
Barcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop PresentationBarcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop Presentation
Norberto Leite
 
Behm Shah Pagerank
Behm Shah PagerankBehm Shah Pagerank
Behm Shah Pagerank
gothicane
 
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile AppsPratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Axway Appcelerator
 
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NApache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Yahoo Developer Network
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014
MongoDB
 

Similar to Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ (20)

CouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 HourCouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 Hour
 
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomy
 
Barcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop PresentationBarcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop Presentation
 
Behm Shah Pagerank
Behm Shah PagerankBehm Shah Pagerank
Behm Shah Pagerank
 
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
 
Complex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBoxComplex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBox
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
 
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile AppsPratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
 
MAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxMAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptx
 
Plproxy
PlproxyPlproxy
Plproxy
 
Map Reduce
Map ReduceMap Reduce
Map Reduce
 
Monitoring Spark Applications
Monitoring Spark ApplicationsMonitoring Spark Applications
Monitoring Spark Applications
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
 
Three Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big DataThree Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big Data
 
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NApache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
 
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with easeGDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014
 

More from Rick Copeland

Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
Rick Copeland
 

More from Rick Copeland (10)

Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)
 
Schema Design at Scale
Schema Design at ScaleSchema Design at Scale
Schema Design at Scale
 
Building Your First MongoDB Application
Building Your First MongoDB ApplicationBuilding Your First MongoDB Application
Building Your First MongoDB Application
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
 
Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
 
Chef on Python and MongoDB
Chef on Python and MongoDBChef on Python and MongoDB
Chef on Python and MongoDB
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForgeAllura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForge
 
MongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDBMongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDB
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
panagenda
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
Stephen Perrenod
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureECS 2024 Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
Femke de Vroome
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
Srushith Repakula
 

Recently uploaded (20)

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureECS 2024 Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
The Metaverse: Are We There Yet?
The Metaverse: Are We There Yet?The Metaverse: Are We There Yet?
The Metaverse: Are We There Yet?
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

Editor's Notes

  1. Can record many more than 4k events per second  345M events per day (single-thread, VM on a laptop) – we get a lot of traffic, but not that much  MR makes this much lower if calculated continuously, still hundreds of events even with MR locking