Your SlideShare is downloading. ×
0
Weightlifting@
@bogdan
@SIMPLYSOCIAL
+ =
+ =
+ =?
+ =
ARCHITECTUREapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACK...
LIFE OF A REQUESTapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 01 API N...APP 01 APP 01 APP N...BACKEND01BACKEND0...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLI...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLI...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLI...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLI...
USE OF PYTHONapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BAC...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
CAN I HEAR ABOOO FOR ?
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
PYTHON ON APPSERVERSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICDEPLOYMENTMetricsLogging
USE OF PYTHONMETRICSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
INTRODUCING OSSIEMETRICS GATHERINGREPORTING STATS THROUGH A SIMPLEWEB INTERFACEINTERACTION WITH RUNNING PROcessesBACKEND B...
INTRODUCING OSSIEimport ossiestat = ossie.Client("arcade.random_worker")stat.gauge("a_metric", 256)stat.timer("random.metr...
USE OF PYTHONloggingAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
logginGFULL JSON loggingEVEN nginx logs jsonpowered by fluentd + S3FUTURE: Logs can be searched byelasticsearch + kibana
api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUS...
AWS EC2Ubuntu 12.04ChefFABRICSUPERVISORDMetricsSUPERVISORDWorkersLogging
AWS EC2Ubuntu 12.04ChefFABRICWORKERSMetricsSUPERVISORDWorkersLogging
INTRODUCING ARCADEqueue based workers frameworkConvention over configurationsimple deploymentdetailed logging and monitoring
INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWL
INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWLqueue pythonworkersjob types
INTRODUCING ARCADEfacebook/!"" __init__.py!"" common#   !"" __init__.py#   !"" crawler/#   !"" exception.py#   !"" faceboo...
from arcade.worker import TaskWorkerfrom arcade.helper import retry, task# [...]class FacebookCrawler(TaskWorker):def __in...
AND MANY MORE:continuous integrationRELIABLE ALERTINGCLOUDWATCH to STATSDREAL USER MONITORING
Q/A
Thanks!
Weightlifting at SimplySocial
Weightlifting at SimplySocial
Upcoming SlideShare
Loading in...5
×

Weightlifting at SimplySocial

347

Published on

How SimplySocial uses Python

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
347
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Weightlifting at SimplySocial"

  1. 1. Weightlifting@
  2. 2. @bogdan
  3. 3. @SIMPLYSOCIAL
  4. 4. + =
  5. 5. + =
  6. 6. + =?
  7. 7. + =
  8. 8. ARCHITECTUREapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  9. 9. LIFE OF A REQUESTapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 01 API N...APP 01 APP 01 APP N...BACKEND01BACKEND02BACKENDN...DBsAPI 02APP 02
  10. 10. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  11. 11. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  12. 12. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST2M x
  13. 13. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  14. 14. USE OF PYTHONapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  15. 15. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  16. 16. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  17. 17. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  18. 18. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  19. 19. CAN I HEAR ABOOO FOR ?
  20. 20. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  21. 21. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  22. 22. PYTHON ON APPSERVERSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICDEPLOYMENTMetricsLogging
  23. 23. USE OF PYTHONMETRICSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  24. 24. INTRODUCING OSSIEMETRICS GATHERINGREPORTING STATS THROUGH A SIMPLEWEB INTERFACEINTERACTION WITH RUNNING PROcessesBACKEND BY STATSD and GRAPHITE
  25. 25. INTRODUCING OSSIEimport ossiestat = ossie.Client("arcade.random_worker")stat.gauge("a_metric", 256)stat.timer("random.metric", 1.28)with stat.timer("random.metric.with"):# do something that requires an amount of time# ossie client will report the right time
  26. 26. USE OF PYTHONloggingAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  27. 27. logginGFULL JSON loggingEVEN nginx logs jsonpowered by fluentd + S3FUTURE: Logs can be searched byelasticsearch + kibana
  28. 28. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  29. 29. AWS EC2Ubuntu 12.04ChefFABRICSUPERVISORDMetricsSUPERVISORDWorkersLogging
  30. 30. AWS EC2Ubuntu 12.04ChefFABRICWORKERSMetricsSUPERVISORDWorkersLogging
  31. 31. INTRODUCING ARCADEqueue based workers frameworkConvention over configurationsimple deploymentdetailed logging and monitoring
  32. 32. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWL
  33. 33. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWLqueue pythonworkersjob types
  34. 34. INTRODUCING ARCADEfacebook/!"" __init__.py!"" common#   !"" __init__.py#   !"" crawler/#   !"" exception.py#   !"" facebook_model.py#   !"" helper.py#   !"" indexer/#   !"" ...!"" config#   !"" __init__.py#   !"" config.yml#   $"" env#   !"" __init__.py#   !"" crawler.py#   !"" ......!"" tests#   !"" __init__.py#   !"" fixtures#   #   !"" crawler_feed.yml#   #   !"" ...#   #   $"" stream_routing.yml#   !"" test_crawler_feed.py#   !"" ...#   $"" test_stream_worker.py$"" workers!"" __init__.py!"" crawler.py!"" ...
  35. 35. from arcade.worker import TaskWorkerfrom arcade.helper import retry, task# [...]class FacebookCrawler(TaskWorker):def __init__(self, dependencies):super(FacebookCrawler, self).__init__(dependencies)@task("default", "crawl")def crawl_feed(self, data):"""Job type:* uid* last_crawl"""feed = CrawlerFeedModel(self)feed.crawl(data)def before_each(self, data):# [...]def run(dependencies):try:FacebookCrawler(dependencies).run()except:dependencies.get("exception").capture_exception()
  36. 36. AND MANY MORE:continuous integrationRELIABLE ALERTINGCLOUDWATCH to STATSDREAL USER MONITORING
  37. 37. Q/A
  38. 38. Thanks!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×