Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (8)
Similar to Weightlifting at SimplySocial
Similar to Weightlifting at SimplySocial (20)
More from Bogdan Gaza
Recently uploaded
Recently uploaded (20)
Weightlifting at SimplySocial
- 2. @bogdan
- 4. + =
- 5. + =
- 6. + =?
- 8. + =
- 9. ARCHITECTURE api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs
- 10. LIFE OF A REQUEST api.gosimplysocial.com app.gosimplysocial.com API 01 API 01 API N... APP 01 APP 01 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs API 02 APP 02
- 11. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 12. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 13. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST 2M x
- 14. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 15. USE OF PYTHON api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs
- 16. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 17. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 18. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 19. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 20. CAN I HEAR A BOOO FOR ?
- 21. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 22. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 23. PYTHON ON APPSERVERS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC DEPLOYMENT Metrics Logging
- 24. USE OF PYTHON METRICS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging
- 25. INTRODUCING OSSIE METRICS GATHERING REPORTING STATS THROUGH A SIMPLE WEB INTERFACE INTERACTION WITH RUNNING PROcesses BACKEND BY STATSD and GRAPHITE
- 26. INTRODUCING OSSIE import ossie stat = ossie.Client("arcade.random_worker") stat.gauge("a_metric", 256) stat.timer("random.metric", 1.28) with stat.timer("random.metric.with"): # do something that requires an amount of time # ossie client will report the right time
- 28. USE OF PYTHON logging AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging
- 29. logginG FULL JSON logging EVEN nginx logs json powered by fluentd + S3 FUTURE: Logs can be searched by elasticsearch + kibana
- 30. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 33. INTRODUCING ARCADE queue based workers framework Convention over configuration simple deployment detailed logging and monitoring
- 34. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL
- 35. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL queue python workers job types
- 36. INTRODUCING ARCADE facebook/ !"" __init__.py !"" common # !"" __init__.py # !"" crawler/ # !"" exception.py # !"" facebook_model.py # !"" helper.py # !"" indexer/ # !"" ... !"" config # !"" __init__.py # !"" config.yml # $"" env # !"" __init__.py # !"" crawler.py # !"" ... ... !"" tests # !"" __init__.py # !"" fixtures # # !"" crawler_feed.yml # # !"" ... # # $"" stream_routing.yml # !"" test_crawler_feed.py # !"" ... # $"" test_stream_worker.py $"" workers !"" __init__.py !"" crawler.py !"" ...
- 37. from arcade.worker import TaskWorker from arcade.helper import retry, task # [...] class FacebookCrawler(TaskWorker): def __init__(self, dependencies): super(FacebookCrawler, self).__init__(dependencies) @task("default", "crawl") def crawl_feed(self, data): """ Job type: * uid * last_crawl """ feed = CrawlerFeedModel(self) feed.crawl(data) def before_each(self, data): # [...] def run(dependencies): try: FacebookCrawler(dependencies).run() except: dependencies.get("exception").capture_exception()
- 38. AND MANY MORE: continuous integration RELIABLE ALERTING CLOUDWATCH to STATSD REAL USER MONITORING
- 39. Q/A
- 40. Thanks!