This document summarizes the scaling of a fantasy strategy game called Utopia Kingdoms from 4 users to over 90,000 users. It discusses the technology stack including Python, CherryPy, Amazon SimpleDB, and frontend technologies. It also describes some performance issues encountered with the database and solutions attempted, such as migrating from SimpleDB to MongoDB and implementing caching, queueing actions, and optimizing database queries. The importance of ongoing measurement and monitoring to identify bottlenecks is emphasized.
In a real life almost any project deals with the
tree structures. Different kinds of taxonomies,
site structures etc require modeling of
hierarchy relations.
Typical approaches used
● Model Tree Structures with Child References
● Model Tree Structures with Parent References
● Model Tree Structures with an Array of Ancestors
● Model Tree Structures with Materialized Paths
● Model Tree Structures with Nested Sets
In a real life almost any project deals with the
tree structures. Different kinds of taxonomies,
site structures etc require modeling of
hierarchy relations.
Typical approaches used
● Model Tree Structures with Child References
● Model Tree Structures with Parent References
● Model Tree Structures with an Array of Ancestors
● Model Tree Structures with Materialized Paths
● Model Tree Structures with Nested Sets
While known for its first-class JSON handling for Java, Jackson is not limited to JSON: with no fewer than 9 supported data formats it can be used for reading and writing data in almost any data format. This talk offers introduction to reading and writing XML and CSV using Jackson.
RubyMotion is great for quickly prototyping apps but it lacks the data modelling tools that Xcode provides. Luckily, using Core Data with RubyMotion can actually be easier and quicker with a little help from some 3rd party libraries.
MongoDB + Java - Everything you need to know Norberto Leite
Learn everything you need to know to get started building a MongoDB-based app in Java. We'll explore the relationship between MongoDB and various languages on the Java Virtual Machine such as Java, Scala, and Clojure. From there, we'll examine the popular frameworks and integration points between MongoDB and the JVM including Spring Data and object-document mappers like Morphia.
Slick: Bringing Scala’s Powerful Features to Your Database Access Rebecca Grenier
This talk will teach you how to use Slick in practice, based on our experience at EatingWell Media Group. Slick is a totally different (and better!) relational database mapping tool that brings Scala’s powerful features to your database interactions, namely: static-checking, compile-time safety, and compositionality.
Here at EatingWell, we have learned quite a bit about Slick over the past two years as we transitioned from a PHP website to Scala. I will share with you tips and tricks we have learned, as well as everything you need to get started using Slick in your Scala application.
I will begin with Slick fundamentals: how to get started making your connection, the types of databases it can access, how to actually create table objects and make queries to and from them. We will using these fundamentals to demonstrate the powerful features inherited from the Scala language itself: static-checking, compile-time safety, and compositionality. And throughout I will share plenty of tips that will help you in everything from getting started to connection pooling options and configuration for use at scale.
REST/JSON/CoreData Example Code - A TourCarl Brown
This is a talk given by Carl Brown at the 2/28/2013 CocoaCoders meeting in Austin (actually Round Rock) TX.
It describes github/carlbrown/SeismicJSON - an MIT-licensed project Carl wrote to illustrate REST JSON CoreData and NSOperations based loosely on the functionality of Apple's old SeismicXML Sample Code.
While known for its first-class JSON handling for Java, Jackson is not limited to JSON: with no fewer than 9 supported data formats it can be used for reading and writing data in almost any data format. This talk offers introduction to reading and writing XML and CSV using Jackson.
RubyMotion is great for quickly prototyping apps but it lacks the data modelling tools that Xcode provides. Luckily, using Core Data with RubyMotion can actually be easier and quicker with a little help from some 3rd party libraries.
MongoDB + Java - Everything you need to know Norberto Leite
Learn everything you need to know to get started building a MongoDB-based app in Java. We'll explore the relationship between MongoDB and various languages on the Java Virtual Machine such as Java, Scala, and Clojure. From there, we'll examine the popular frameworks and integration points between MongoDB and the JVM including Spring Data and object-document mappers like Morphia.
Slick: Bringing Scala’s Powerful Features to Your Database Access Rebecca Grenier
This talk will teach you how to use Slick in practice, based on our experience at EatingWell Media Group. Slick is a totally different (and better!) relational database mapping tool that brings Scala’s powerful features to your database interactions, namely: static-checking, compile-time safety, and compositionality.
Here at EatingWell, we have learned quite a bit about Slick over the past two years as we transitioned from a PHP website to Scala. I will share with you tips and tricks we have learned, as well as everything you need to get started using Slick in your Scala application.
I will begin with Slick fundamentals: how to get started making your connection, the types of databases it can access, how to actually create table objects and make queries to and from them. We will using these fundamentals to demonstrate the powerful features inherited from the Scala language itself: static-checking, compile-time safety, and compositionality. And throughout I will share plenty of tips that will help you in everything from getting started to connection pooling options and configuration for use at scale.
REST/JSON/CoreData Example Code - A TourCarl Brown
This is a talk given by Carl Brown at the 2/28/2013 CocoaCoders meeting in Austin (actually Round Rock) TX.
It describes github/carlbrown/SeismicJSON - an MIT-licensed project Carl wrote to illustrate REST JSON CoreData and NSOperations based loosely on the functionality of Apple's old SeismicXML Sample Code.
Performance optimization techniques for Java codeAttila Balazs
The presentation covers the the basics of performance optimizations for real-world Java code. It starts with a theoretical overview of the concepts followed by several live demos
showing how performance bottlenecks can be diagnosed and eliminated. The demos include some non-trivial multi-threaded examples
inspired by real-world applications.
Introduction to Gatling performance testing tool and how we used it for testing Zonky's REST API. Example of running distributed performance tests in AWS Fargate with real-time monitoring with Logstash/ElasticSearch/Kibana stack.
These are the slides I presented at the Nosql Night in Boston on Nov 4, 2014. The slides were adapted from a presentation given by Steve Francia in 2011. Original slide deck can be found here:
http://spf13.com/presentation/mongodb-sort-conference-2011
OSMC 2019 | How to improve database Observability by Charles JudithNETWAYS
Delivering a database service is not a simple job but to ensure that everything is working correctly your platform needs to be observable. In this talk, I’ll talk about how we make the MySQL/MariaDB databases observable. We’ll talk about the RED, USE methods, and the golden signals. You’ll discover how we dealt with the following questions “We think the database is slow”. This talk will allow you to make your databases discoverable with open source solutions.
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez
En esta sesión voy a contar las decisiones técnicas que tomamos al desarrollar QuestDB, una base de datos Open Source para series temporales compatible con Postgres, y cómo conseguimos escribir más de cuatro millones de filas por segundo sin bloquear o enlentecer las consultas.
Hablaré de cosas como (zero) Garbage Collection, vectorización de instrucciones usando SIMD, reescribir en lugar de reutilizar para arañar microsegundos, aprovecharse de los avances en procesadores, discos duros y sistemas operativos, como por ejemplo el soporte de io_uring, o del balance entre experiencia de usuario y rendimiento cuando se plantean nuevas funcionalidades.
MongoDB: Advantages of an Open Source NoSQL DatabaseFITC
Save 10% off ANY FITC event with discount code 'slideshare'
See our upcoming events at www.fitc.ca
OVERVIEW
The presentation will present an overview of the MongoDB NoSQL database, its history and current status as the leading NoSQL database. It will focus on how NoSQL, and in particular MongoDB, benefits developers building big data or web scale applications. Discuss the community around MongoDB and compare it to commercial alternatives. An introduction to installing, configuring and maintaining standalone instances and replica sets will be provided.
Presented live at FITC's Spotlight:MEAN Stack on March 28th, 2014.
More info at FITC.ca
PyGrunn2013 High Performance Web Applications with TurboGearsAlessandro Molina
Users are getting more and more used to fast websites, a second or two is way too much before they leave the page. Since version 2.2 TurboGears has focused on providing more tools to create faster web applications and improving its speed constantly. The latest 2.3 version, the first to support Python3 is up to 4x faster than the previous and provides a great toolset to make fast pages. The talk will focus on showcasing the tools provided by the framework to increase speed of your web applications and provide some tips and tricks to get maximum speed from the framework itself.
Enroll for expert level Online MongoDB Training by certified experts, Learn MongoDB 3.6 training course with live tutorial videos. Attend demo for free & you will find Spiritsofts is the best Institute within reasonable fee, updated course material.
We Provides the best learning Environment. Obtain all the training by our expert professionals which is having working experience from Top IT companies.
The Training in is every thing we explained based on real time scenarios, it works which we do in companies.
PGConf APAC 2018 - High performance json postgre-sql vs. mongodbPGConf APAC
Speakers: Dominic Dwyer & Wei Shan Ang
This talk was presented in Percona Live Europe 2017. However, we did not have enough time to test against more scenario. We will be giving an updated talk with a more comprehensive tests and numbers. We hope to run it against citusDB and MongoRocks as well to provide a comprehensive comparison.
https://www.percona.com/live/e17/sessions/high-performance-json-postgresql-vs-mongodb
QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez
In this session I will show you the technical decisions we made when building QuestDB, the open source, Postgres compatible, time-series database, and how we can achieve a million row writes per second without blocking or slowing down the reads.
Similar to Utopia Kindgoms scaling case: From 4 to 50K users (20)
4. Utopia Kingdoms
● Fantasy strategy game
● Build your own Kingdom
● Create armies and attack other Kingdoms
● Join other Kingdoms in an Alliance
● Manage resources
● Available in Facebook and Kongregate
http://www.facebook.com/UtopiaKigdomsGame
http://www.kongregate.com/games/JoltOnline/utopia-kingdoms
9. Some points of interest
(will discuss them later)
●
Your resources (population, gold, food, etc)
grows with time
●
You actions (build something, attack a player)
typically takes some time
●
Players are ranked against the rest
●
You can add friends and enemies
11. Measurement tools
● OS tools
● Task manager (top)
●
IO Monitor (iostat)
● Monitoring tools (Munin, Nagios)
● Logs
●
Needs to find a good compromise detailed/relevance
● Profiling
12. You've got to love profiling
●
Generate profiles with cProfile module
Profile whole application with
python -m cProfile -o file.prof my_app.py
(not very useful in a web app)
●
If you're using a framework, profile only your
functions to reduce noise
14. Analyzing profile
●
gprof2dot
●
Using dot, convert to graph
gprof2dot -f pstats file.prof | dot -Tpng -o file.png
●
Good for workflows
●
RunSnakeRun
●
Good for cumulative times
18. All static should be out of python
●
Use a good web server to serve all static
content (Static HTML, CSS, JavaScript code)
●
Some options
●
Apache
●
Nginx
●
Cherokee
●
Amazon S3
20. Example
●
Asking for friends/enemies to DB
●
Costly request in SimpleDB (using SQL statement)
●
On each request
●
Cache the friends on memcache for 1 hour
●
Invalidate the cache if adding/removing
friends or enemies
21. Caching caveats
●
Cache only after knowing there is a problem
●
Do not trust in cache for storage
●
Take a look on size of cached data
●
Choosing a good cache time can be difcult /
Invalidate cache can be complex
●
Some data is too dynamic to be cached
22. Caching is not just memcached
●
More options available:
●
Get on memory on start
●
File cache
●
Cache client side
23. Parse templates just once
●
The template rendering modules have options
to parse the templates just once
●
Be sure to activate it in production
●
In development, you'll most likely want to
parse them each time
●
Same apply to regex, specially complex ones
25. Rankings
●
Sort players on the DB is slow when you grow
the number of players
●
Solution:
●
Independent ranking server (operates just in
memory)
●
Works using binary trees
●
Small Django project, communicates using xmlrpc
●
Inconvenient:
●
Data is not persistent, if the rankings server goes
down, needs time to reconstruct the rankings
26. Database pulling - Resources
●
There was a process just taking care of the
growth of resources.
●
It goes element by element, and increasing the
values
●
It pulls the DB constantly, even when the user has
their values to maximum
●
Increment the resources of a user just the next
time is accessed (by himself or by others)
●
No usage of DB when the user is not in use
●
The request already reads from DB the user
27. Database pulling - Actions
●
Lots of actions are delayed. Recruit a unit,
buildings, raids...
●
A process check each user if an action has to
be done NOW.
●
Tons of reads just to check “not now”
●
Great delay in some actions, as they are not
executed in time
28. Database pulling - Actions
●
Implement a queue to execute the actions at
the proper time:
●
Beanstalk (allows deferred extraction)
●
A process listen to this queue and performs the
action, independently from request servers.
●
The process can be launched in a diferent
machine.
●
Multiple process can extract actions faster.
30. Amazon SimpleDB
●
Key – Value storage
●
Capable of SQL queries
●
Store a dictionary (schemaless, multiple
columns)
●
All the values are strings
●
Access through boto module
●
Pay per use
31. Problems with SimpleDB
●
Lack of control
●
Can't use local copy
– In development, you must access Amazon servers (slow
and costly)
●
Can't backup except manually
●
Can't analyze or change DB (e.g. can't define
indexes)
●
Can't monitor DB
32. Problems with SimpleDB
●
Bad tool support
●
Slow and high variability (especially on SQL
queries)
●
Sometime, the queries just timeout and had to be
repeated.
34. MongoDB
●
NoSQL
●
Schemaless
●
Fast
●
Allow complex queries
●
Retain control (backups, measure queries, etc)
●
Previous experience using it from ChampMan
35. Requisites of the migration
●
Low-level approach
●
Objects are basically dictionaries
●
Be able to save dirty fields (avoid saving
unchanged values)
●
Log queries to measure performance
36. MongoSpell
●
Thin wrap over pymongo
●
Objects are just dictionary-like elements
●
Minimal schema
●
Fast!
●
Able to log queries
●
It will probably be released soon as Open
Source
42. Scalable vs Efcient
Scalable Efficient
●
Can support more ●
Can support more
users adding more users with the same
elements elements
Work on both to achieve your goals