Your SlideShare is downloading. ×
Weightlifting at SimplySocial
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Weightlifting at SimplySocial

325

Published on

How SimplySocial uses Python

How SimplySocial uses Python

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
325
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Weightlifting@
  • 2. @bogdan
  • 3. @SIMPLYSOCIAL
  • 4. + =
  • 5. + =
  • 6. + =?
  • 7. + =
  • 8. ARCHITECTUREapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  • 9. LIFE OF A REQUESTapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 01 API N...APP 01 APP 01 APP N...BACKEND01BACKEND02BACKENDN...DBsAPI 02APP 02
  • 10. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 11. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 12. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST2M x
  • 13. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 14. USE OF PYTHONapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  • 15. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 16. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 17. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 18. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 19. CAN I HEAR ABOOO FOR ?
  • 20. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 21. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 22. PYTHON ON APPSERVERSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICDEPLOYMENTMetricsLogging
  • 23. USE OF PYTHONMETRICSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  • 24. INTRODUCING OSSIEMETRICS GATHERINGREPORTING STATS THROUGH A SIMPLEWEB INTERFACEINTERACTION WITH RUNNING PROcessesBACKEND BY STATSD and GRAPHITE
  • 25. INTRODUCING OSSIEimport ossiestat = ossie.Client("arcade.random_worker")stat.gauge("a_metric", 256)stat.timer("random.metric", 1.28)with stat.timer("random.metric.with"):# do something that requires an amount of time# ossie client will report the right time
  • 26. USE OF PYTHONloggingAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  • 27. logginGFULL JSON loggingEVEN nginx logs jsonpowered by fluentd + S3FUTURE: Logs can be searched byelasticsearch + kibana
  • 28. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 29. AWS EC2Ubuntu 12.04ChefFABRICSUPERVISORDMetricsSUPERVISORDWorkersLogging
  • 30. AWS EC2Ubuntu 12.04ChefFABRICWORKERSMetricsSUPERVISORDWorkersLogging
  • 31. INTRODUCING ARCADEqueue based workers frameworkConvention over configurationsimple deploymentdetailed logging and monitoring
  • 32. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWL
  • 33. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWLqueue pythonworkersjob types
  • 34. INTRODUCING ARCADEfacebook/!"" __init__.py!"" common#   !"" __init__.py#   !"" crawler/#   !"" exception.py#   !"" facebook_model.py#   !"" helper.py#   !"" indexer/#   !"" ...!"" config#   !"" __init__.py#   !"" config.yml#   $"" env#   !"" __init__.py#   !"" crawler.py#   !"" ......!"" tests#   !"" __init__.py#   !"" fixtures#   #   !"" crawler_feed.yml#   #   !"" ...#   #   $"" stream_routing.yml#   !"" test_crawler_feed.py#   !"" ...#   $"" test_stream_worker.py$"" workers!"" __init__.py!"" crawler.py!"" ...
  • 35. from arcade.worker import TaskWorkerfrom arcade.helper import retry, task# [...]class FacebookCrawler(TaskWorker):def __init__(self, dependencies):super(FacebookCrawler, self).__init__(dependencies)@task("default", "crawl")def crawl_feed(self, data):"""Job type:* uid* last_crawl"""feed = CrawlerFeedModel(self)feed.crawl(data)def before_each(self, data):# [...]def run(dependencies):try:FacebookCrawler(dependencies).run()except:dependencies.get("exception").capture_exception()
  • 36. AND MANY MORE:continuous integrationRELIABLE ALERTINGCLOUDWATCH to STATSDREAL USER MONITORING
  • 37. Q/A
  • 38. Thanks!

×