SlideShare a Scribd company logo

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

•Download as PPT, PDF•
•
Rick Copeland
Rick Copeland

Zarkov is a real-time analytics platform that uses MongoDB, Python, Gevent, and ZeroMQ to aggregate and analyze event data. It handles high volumes of event ingestion and uses a lightweight map/reduce framework to incrementally calculate statistics. The system was developed at SourceForge.net to monitor user and project activity in real-time and has proven capable of replacing other analytics solutions.

Technology
1 of 22
Realtime Analytics using MongoDB, Python, Gevent, and ZeroMQ Rick Copeland @rick446 [email_address]
SourceForge s MongoDB ,[object Object],[object Object],[object Object]
The Problem We’re Trying to Solve ,[object Object],[object Object],[object Object],[object Object]
Introducing Zarkov 0.0.1 ,[object Object],[object Object],[object Object]
Zarkov Architecture MongoDB BSON over ZeroMQ Journal Greenlet Commit Greenlet Write-ahead log Write-ahead log Aggregation Greenlet
Technologies ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
“Wow, it’s really fast; can it replace…” ,[object Object],[object Object],[object Object],“Probably, but it’ll take some work….”
Moving towards production.... ,[object Object],[object Object],[object Object],[object Object],[object Object]
Zarkov’s already doing a lot… ,[object Object],[object Object],[object Object],[object Object],[object Object]
Quick Map/reduce Refresher ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Quick Map/reduce Refresher ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Parallel
Zarkov Map/Reduce Architecture map_in_#.bson Query Map Sort Reduce Commit map_out_#.bson reduce_in.bson Job Mgr
Zarkov Map/Reduce ,[object Object],[object Object],[object Object],[object Object]
Zarkov Web Service ,[object Object],[object Object],[object Object],[object Object],[object Object]
Zarkov Deployment at SF.net
Lessons learned at
MongoDB Tricks ,[object Object],[object Object],class IdGen ( object): @classmethod def get_ids(cls, inc = 1): obj = cls.query.find_and_modify( query={ '_id': 0}, update ={ '$inc': dict(inc =inc), }, upsert= True, new = True) return range(obj .inc - inc, obj.inc)
MongoDB Pitfalls ,[object Object],[object Object],[object Object],[object Object]
Open Source Ming http://sf.net/projects/merciless/ MIT License Allura http://sf.net/p/allura/ Apache License Zarkov http://sf.net/p/zarkov/ Apache License
Future Work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Rick Copeland @rick446 [email_address]
Credits ,[object Object]

Recommended

Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioRick Copeland
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRShinji Tanaka
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
Bartosz Konieczny
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
Http capturing
Http capturingHttp capturing
Http capturing
Eric Ahn
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
Bram Vogelaar
 
2015 555 kharchenko_ppt
2015 555 kharchenko_ppt2015 555 kharchenko_ppt
2015 555 kharchenko_ppt
Maxym Kharchenko
 
Html5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webglHtml5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webgl
Kilian Valkhof
 

Recommended

Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioRick Copeland
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRShinji Tanaka
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
Bartosz Konieczny
 
RestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message QueueRestMQ - HTTP/Redis based Message Queue
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
Http capturing
Http capturingHttp capturing
Http capturing
Eric Ahn
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
Bram Vogelaar
 
2015 555 kharchenko_ppt
2015 555 kharchenko_ppt2015 555 kharchenko_ppt
2015 555 kharchenko_ppt
Maxym Kharchenko
 
Html5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webglHtml5, css3, canvas, svg and webgl
Html5, css3, canvas, svg and webgl
Kilian Valkhof
 
Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Masahiro Nagano
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
Ismael Celis
 
Commit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - extCommit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - ext
Maxym Kharchenko
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
Дэв Тим Афс
 
Cross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App EngineCross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App Engine
Andy McKay
 
PSGI and Plack from first principles
PSGI and Plack from first principlesPSGI and Plack from first principles
PSGI and Plack from first principles
Perl Careers
 
Presto overview
Presto overviewPresto overview
Presto overviewShixiong Zhu
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
Bartosz Konieczny
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
GlynnForrest
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Odoo
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
benwaine
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
Alejandro E Brito Monedero
 
Serverless Ballerina
Serverless BallerinaServerless Ballerina
Serverless Ballerina
Ballerina
 
All you need to know about the JavaScript event loop
All you need to know about the JavaScript event loopAll you need to know about the JavaScript event loop
All you need to know about the JavaScript event loop
Saša Tatar
 
Asynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time MessagingAsynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time Messaging
Steve Rhoades
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
Workhorse Computing
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
Locaweb
 
Visualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LVVisualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LV
Maxym Kharchenko
 
Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
PyData
 
Big Data Monitoring Cockpit
Big Data Monitoring CockpitBig Data Monitoring Cockpit
Big Data Monitoring Cockpit
Stefan Bergstein
 

More Related Content

What's hot

Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Masahiro Nagano
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
Ismael Celis
 
Commit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - extCommit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - ext
Maxym Kharchenko
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
Дэв Тим Афс
 
Cross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App EngineCross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App Engine
Andy McKay
 
PSGI and Plack from first principles
PSGI and Plack from first principlesPSGI and Plack from first principles
PSGI and Plack from first principles
Perl Careers
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
Bram Vogelaar
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
Bartosz Konieczny
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
GlynnForrest
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Odoo
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
benwaine
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
Alejandro E Brito Monedero
 
Serverless Ballerina
Serverless BallerinaServerless Ballerina
Serverless Ballerina
Ballerina
 
All you need to know about the JavaScript event loop
All you need to know about the JavaScript event loopAll you need to know about the JavaScript event loop
All you need to know about the JavaScript event loop
Saša Tatar
 
Asynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time MessagingAsynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time Messaging
Steve Rhoades
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
Workhorse Computing
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
Locaweb
 
Visualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LVVisualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LV
Maxym Kharchenko
 

What's hot (20)

Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
 
Commit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - extCommit2015 kharchenko - python generators - ext
Commit2015 kharchenko - python generators - ext
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Nginx-lua
Nginx-luaNginx-lua
Nginx-lua
 
Cross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App EngineCross Domain Web
Mashups with JQuery and Google App Engine
Cross Domain Web
Mashups with JQuery and Google App Engine
 
PSGI and Plack from first principles
PSGI and Plack from first principlesPSGI and Plack from first principles
PSGI and Plack from first principles
 
Presto overview
Presto overviewPresto overview
Presto overview
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
 
Serverless Ballerina
Serverless BallerinaServerless Ballerina
Serverless Ballerina
 
All you need to know about the JavaScript event loop
All you need to know about the JavaScript event loopAll you need to know about the JavaScript event loop
All you need to know about the JavaScript event loop
 
Asynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time MessagingAsynchronous PHP and Real-time Messaging
Asynchronous PHP and Real-time Messaging
 
Lies, Damn Lies, and Benchmarks
Lies, Damn Lies, and BenchmarksLies, Damn Lies, and Benchmarks
Lies, Damn Lies, and Benchmarks
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
 
Visualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LVVisualizing ORACLE performance data with R @ #C16LV
Visualizing ORACLE performance data with R @ #C16LV
 

Viewers also liked

Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
PyData
 
Big Data Monitoring Cockpit
Big Data Monitoring CockpitBig Data Monitoring Cockpit
Big Data Monitoring Cockpit
Stefan Bergstein
 
Super Fast Gevent Introduction
Super Fast Gevent IntroductionSuper Fast Gevent Introduction
Super Fast Gevent Introduction
Walter Liu
 
gunicorn introduction
gunicorn introductiongunicorn introduction
gunicorn introduction
Adam Lowry
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeMongoDB
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQfcrippa
 
How to monitor MongoDB
How to monitor MongoDBHow to monitor MongoDB
How to monitor MongoDB
Server Density
 
presentación para el Cecec2013
presentación para el Cecec2013 presentación para el Cecec2013
presentación para el Cecec2013
Maryel Mendiola
 
Processo Administrativo Interno ARSESP
Processo Administrativo Interno ARSESPProcesso Administrativo Interno ARSESP
Processo Administrativo Interno ARSESP
Cid Tomanik
 
Ing. Economica
Ing. EconomicaIng. Economica
Ing. EconomicaGer Bless
 
Apuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computoApuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computoiliana52
 
Demande A Dieu
Demande A DieuDemande A Dieu
Demande A Dieu
guest2f53b8
 
Innovación educativa con recursos abiertos
Innovación educativa con recursos abiertosInnovación educativa con recursos abiertos
Innovación educativa con recursos abiertos
Maryel Mendiola
 
Paper calculo
Paper calculoPaper calculo
Paper calculo
Roger Carlos Perez Castro
 
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Joel A. Gómez Treviño
 
Summon: The Path of Least Resistance
Summon: The Path of Least ResistanceSummon: The Path of Least Resistance
Summon: The Path of Least Resistance
daveyp
 
La clase de educacion fisica
La clase de educacion fisicaLa clase de educacion fisica
La clase de educacion fisicaAnto Anzevino
 

Viewers also liked (20)

Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...Python and pandas as back end to real-time data driven applications by Giovan...
Python and pandas as back end to real-time data driven applications by Giovan...
 
Big Data Monitoring Cockpit
Big Data Monitoring CockpitBig Data Monitoring Cockpit
Big Data Monitoring Cockpit
 
Super Fast Gevent Introduction
Super Fast Gevent IntroductionSuper Fast Gevent Introduction
Super Fast Gevent Introduction
 
gunicorn introduction
gunicorn introductiongunicorn introduction
gunicorn introduction
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
 
How to monitor MongoDB
How to monitor MongoDBHow to monitor MongoDB
How to monitor MongoDB
 
presentación para el Cecec2013
presentación para el Cecec2013 presentación para el Cecec2013
presentación para el Cecec2013
 
Ensayo 7
Ensayo 7Ensayo 7
Ensayo 7
 
Processo Administrativo Interno ARSESP
Processo Administrativo Interno ARSESPProcesso Administrativo Interno ARSESP
Processo Administrativo Interno ARSESP
 
Ing. Economica
Ing. EconomicaIng. Economica
Ing. Economica
 
Apuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computoApuntes de operación de un equipo de computo
Apuntes de operación de un equipo de computo
 
Demande A Dieu
Demande A DieuDemande A Dieu
Demande A Dieu
 
Innovación educativa con recursos abiertos
Innovación educativa con recursos abiertosInnovación educativa con recursos abiertos
Innovación educativa con recursos abiertos
 
100 a 189 exercícios de portugues
100 a 189 exercícios de portugues100 a 189 exercícios de portugues
100 a 189 exercícios de portugues
 
Paper calculo
Paper calculoPaper calculo
Paper calculo
 
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
Encuentro Legislativo con la Sociedad sobre Tecnologías de la Información y C...
 
Metacognicion
MetacognicionMetacognicion
Metacognicion
 
Summon: The Path of Least Resistance
Summon: The Path of Least ResistanceSummon: The Path of Least Resistance
Summon: The Path of Least Resistance
 
La clase de educacion fisica
La clase de educacion fisicaLa clase de educacion fisica
La clase de educacion fisica
 

Similar to Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

CouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 HourCouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 Hour
Peter Friese
 
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rick Copeland
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomy
Dongmin Yu
 
Barcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop PresentationBarcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop PresentationNorberto Leite
 
Behm Shah Pagerank
Behm Shah PagerankBehm Shah Pagerank
Behm Shah Pagerankgothicane
 
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
NAVER D2
 
Complex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBoxComplex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBox
bobmcwhirter
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
Ran Silberman
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
Ran Silberman
 
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile AppsPratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Axway Appcelerator
 
MAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxMAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptx
HARIKRISHNANU13
 
Plproxy
PlproxyPlproxy
Plproxy
Joshua Drake
 
Map Reduce
Map ReduceMap Reduce
Map Reduce
Prashant Gupta
 
Monitoring Spark Applications
Monitoring Spark ApplicationsMonitoring Spark Applications
Monitoring Spark Applications
Tzach Zohar
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Databricks
 
Three Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big DataThree Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big Data
Dynamical Software, Inc.
 
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NApache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NYahoo Developer Network
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
leesjensen
 
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with easeGDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
KAI CHU CHUNG
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014MongoDB
 

Similar to Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ (20)

CouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 HourCouchDB Mobile - From Couch to 5K in 1 Hour
CouchDB Mobile - From Couch to 5K in 1 Hour
 
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomy
 
Barcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop PresentationBarcelona MUG MongoDB + Hadoop Presentation
Barcelona MUG MongoDB + Hadoop Presentation
 
Behm Shah Pagerank
Behm Shah PagerankBehm Shah Pagerank
Behm Shah Pagerank
 
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
 
Complex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBoxComplex Made Simple: Sleep Better with TorqueBox
Complex Made Simple: Sleep Better with TorqueBox
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
 
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile AppsPratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
Pratik Patel: Titanium as Platform: Feature-Rich, Database-Driven Mobile Apps
 
MAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptxMAP REDUCE IN DATA SCIENCE.pptx
MAP REDUCE IN DATA SCIENCE.pptx
 
Plproxy
PlproxyPlproxy
Plproxy
 
Map Reduce
Map ReduceMap Reduce
Map Reduce
 
Monitoring Spark Applications
Monitoring Spark ApplicationsMonitoring Spark Applications
Monitoring Spark Applications
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
 
Three Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big DataThree Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big Data
 
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NApache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
 
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with easeGDG Devfest 2019 - Build go kit microservices at kubernetes with ease
GDG Devfest 2019 - Build go kit microservices at kubernetes with ease
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014
 

More from Rick Copeland

Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)
Rick Copeland
 
Schema Design at Scale
Schema Design at ScaleSchema Design at Scale
Schema Design at Scale
Rick Copeland
 
Building Your First MongoDB Application
Building Your First MongoDB ApplicationBuilding Your First MongoDB Application
Building Your First MongoDB Application
Rick Copeland
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rick Copeland
 
Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
Rick Copeland
 
Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
Rick Copeland
 
Chef on Python and MongoDB
Chef on Python and MongoDBChef on Python and MongoDB
Chef on Python and MongoDB
Rick Copeland
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rick Copeland
 
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForgeAllura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Rick Copeland
 
MongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDBMongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDB
Rick Copeland
 

More from Rick Copeland (10)

Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)
 
Schema Design at Scale
Schema Design at ScaleSchema Design at Scale
Schema Design at Scale
 
Building Your First MongoDB Application
Building Your First MongoDB ApplicationBuilding Your First MongoDB Application
Building Your First MongoDB Application
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
 
Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
 
Chef on Python and MongoDB
Chef on Python and MongoDBChef on Python and MongoDB
Chef on Python and MongoDB
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForgeAllura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForge
 
MongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDBMongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDB
 

Recently uploaded

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 

Recently uploaded (20)

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 

Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ

Editor's Notes

  1. Can record many more than 4k events per second  345M events per day (single-thread, VM on a laptop) – we get a lot of traffic, but not that much  MR makes this much lower if calculated continuously, still hundreds of events even with MR locking