This document summarizes the Yatoto social crowdfunding platform. It provides details on its business model, which incorporates elements from lottery, Facebook, eBay, charities and Kickstarter. It discusses user and financial metrics from 2013-2016. It also describes the technical architecture including the Ruby on Rails backend, use of Docker for deployment, monitoring with Zabbix and error tracking with Errbit. Development practices like pair programming, code reviews and CI/CD are outlined.

1. YATOTO
SOCIAL CROWD FUNDING PLATFORM
THE TECHNICAL PART
You can find the most recent version online here
http://yatoto.gudaso .com

2. BUSINESS
Lottery - Different kind of rewards
Facebook - Pages, Events, Friends, Chat,
Main/private newsfeed
Ebay - Bidding on offers
Charities - Donations
Kickstater - Projects
Gamification - A lot of tricks to engage the users

3. NUMBERS (JUL 2013 - SEP 2016)
More than 250 000EUR for rewards
More than 100 000 Users Watched 9 Millon ads
Launched in Bulgaria and the United States
4 Designs changed
3 Devs, 1 frontend, graphic designer
1.3M Publications, 90M Stream Entries
800 Accomplished Missions for ~ 1 month

4. BACKEND
Easy to use backend interface.

5. Google Analytics on steroids
Play with A/B Testing
Smart Feature flapping
Track every event on the website
KNOWING OUR USERS

6. Ruby On Rails, PostgreSQL, Elasticsearch, Redis,
Memcached, Message bus implementation on
Sidekiq
Ansible provisioning with multistage deployments
via Capistrano/Docker
Monitoring via Zabbix + Uptime Robot
Zero downtime deployment within our CI/CD
process
TECHNOLOGY

7. THE APPLICATION
We started with Ruby On Rails with PostgreSQL, Elasticsearch, Redis, Memcached.
At first we use QEMU Virtual Machine with 9GB of RAM and it works great with +100k users. Living on
a virtual machine allow us to know every CPU cycle and every disk IO.
We expect 1m of users, thats why we scale on 3 servers with dockers.
We have 50Milion user stream entries. For our Newsfeed stream processing we use a pipeline over
Sidekiq. We have to send a lot of notifications in almost real time fashion.

8. THE FRONTEND
For the frontend we use Twitter bootsrap, and for the backend we used Zurb Foundation. Both were
good, +1 for Zurb for its usability on mobile devices.
We monitor the usage of every html element on the website with google analytics, because of this
we know how the users interact with the website.
We decided to convert Yatoto to SPA with React. We made a small feature with React, then we
remove it. The performance, and the SEO were poor, and we don't have human power to maintain it.
We kept the real time chat.

9. THE MOBILE APP
With Ionic 2, we shipped simple Yatoto application.

10. DEVELOPMENT

11. Always stay on the latest versions of everything.
With Rails we start at 4.0.0.rc2 and end at 5.0.1.
Adopt new technology like Nodejs/React
Learn to delete code and write it from scratch
Low Technical Depth
THE DEVELOPER HAPPINESS

12. 2 weeks project Yatoto a er 3+ years
YES IT IS MONOLITH
We have to be fast because we deliver entire new functionality every week or 1/3/6 months. We
change the product a lot! It is faster to refactor than micro services architecture.
We wrote Facebook, EBay, Quora, Kickstarter, GA, Games and Meme generator functionality sadly we
forgot to double LinkedIn.
Compared to the we done it easer and faster.Biggest Monolith

13. METHODOLOGY
Scrum with a lot of XP
The team do pair programming, have coding standard.
Collective code ownership and simple design.
Code reviews were more effective than writing tests.
BDD, CI/CD, Small releases

14. SPECIFICATION
Each functionality is made in mock and polished there before it enters the project

15. PROCESS
3400+ Tickets
Sometimes we were lucky to work with bigger than one week milestones.

16. The tests 50min = 36min rspec + 13min cucumber
165 rubygems, (330 with dependencies)
external services: geoip, epay/banking, sending
sms
we publish open source gems for most of our
external services.
The Commits from 9 developers and 3 others
THE CODE
Comits Developer
3781 I. B.
2303 D. N.
1721 R. B.
969 A. D.
923 A. Y.
879 I. T.
463 V. S.
323 V. N.
309 N.
65 The CEO :)
8 M.
0 K.

17. DEPLOYMENT

18. Initially was hosted on one virtual machine,
4CPU, 3GB RAM, serving > 100+K users
every ActiveRecord N+1 was noticeable
For the USA we launch 3 servers, expecting a lot of
traffic
no one came...
ORCHESTRATION WITH ANSIBLE

19. Network
Host (Security, Accounts, ntp, basic packages)
Mail
Docker
Webserver
Zabbix/ZabbixAgents
Errbit
Memcached
Redis
Elasticsearch
Postgres
Samba
EACH WEBSITE IS ORCHESTRATED WITH ANSIBLE WITH THOSE PLAYBOOKS
A er the initial effort of setup the Ansible provisioning there were no need for Sysadmin, the
"configuration" was documented in the playbooks. We are stable and fast.

20. OUR SYSADMIN WELCOME BROCHURE
When he saw it, he run away - true story! Here is the title part of the brochure.

21. generates rails_best_practices, brakeman, rdoc,
states machines diagrams once a day
tried jenkins, gitlabci, cruisecontrol.rb, drone
A er successful build the CI deploys on testing.
fails if we introduce library with strange license.
We deploy with the Vincent Driessen's branching
model. The development team uses test
environments, and the business has stable
staging environment.
CI/CD

22. SPEED & AVAILABILITY

23. DEVELOPERS KNOW THE SPEED OF THE WEBSITE IN TEST/PRODUCTION
Web pages above 100ms are considered to be slow and those urls are redirected to separate
application server dedicated to process them without hurting the overall performance. When the
pages are fixed they are returned to the fast application servers.
We are faster in discovering problems than Newrelic services.

24. ERRBIT
Monitor the errors from staging and testing environment so we know when someone hits the error.
The errors are redirected to rocket chat so that instantly knew about them

25. MONITORING
Zabbix installed on another data center is monitoring the website availability, the speed of the web
pages and alerts via push notifications/chat the developers. UptimeRobot is monitoring Zabbix
Logcheck is reporting everthing unusual

26. OPTIMIZATIONS
Assets - Use separate IP (assets.yatoto.com) for faster loading
Cache - Heavy caching on memcached and Nginx
Return HTML ASAP - later process the request in the background
Fight SPAM - Process the mail logs to find fake email accounts
Dragonfly behind Nginx with big cache
page preloading

27. SATISFACTION
The project is huge, and we have been able to maintain and develop it during the years. There were
no noticeable downtimes. Maintaining the project was with very little effort because of the
automation and the tests. We have played with a lot of technologies