Weightlifting at SimplySocial
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Weightlifting at SimplySocial

on

  • 442 views

How SimplySocial uses Python

How SimplySocial uses Python

Statistics

Views

Total Views
442
Views on SlideShare
442
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Weightlifting at SimplySocial Presentation Transcript

  • 1. Weightlifting@
  • 2. @bogdan
  • 3. @SIMPLYSOCIAL
  • 4. + =
  • 5. + =
  • 6. + =?
  • 7. + =
  • 8. ARCHITECTUREapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  • 9. LIFE OF A REQUESTapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 01 API N...APP 01 APP 01 APP N...BACKEND01BACKEND02BACKENDN...DBsAPI 02APP 02
  • 10. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 11. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 12. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST2M x
  • 13. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsLIFE OF A REQUEST
  • 14. USE OF PYTHONapi.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBs
  • 15. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 16. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 17. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 18. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 19. CAN I HEAR ABOOO FOR ?
  • 20. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 21. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 22. PYTHON ON APPSERVERSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICDEPLOYMENTMetricsLogging
  • 23. USE OF PYTHONMETRICSAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  • 24. INTRODUCING OSSIEMETRICS GATHERINGREPORTING STATS THROUGH A SIMPLEWEB INTERFACEINTERACTION WITH RUNNING PROcessesBACKEND BY STATSD and GRAPHITE
  • 25. INTRODUCING OSSIEimport ossiestat = ossie.Client("arcade.random_worker")stat.gauge("a_metric", 256)stat.timer("random.metric", 1.28)with stat.timer("random.metric.with"):# do something that requires an amount of time# ossie client will report the right time
  • 26. USE OF PYTHONloggingAWS EC2Ubuntu 12.04ChefNGINX + PHP FPMFABRICMetricsLogging
  • 27. logginGFULL JSON loggingEVEN nginx logs jsonpowered by fluentd + S3FUTURE: Logs can be searched byelasticsearch + kibana
  • 28. api.gosimplysocial.comapp.gosimplysocial.comAPI 01 API 02 API N...APP 01 APP 02 APP N...BACKEND01BACKEND02BACKENDN...DBsUSE OF PYTHON
  • 29. AWS EC2Ubuntu 12.04ChefFABRICSUPERVISORDMetricsSUPERVISORDWorkersLogging
  • 30. AWS EC2Ubuntu 12.04ChefFABRICWORKERSMetricsSUPERVISORDWorkersLogging
  • 31. INTRODUCING ARCADEqueue based workers frameworkConvention over configurationsimple deploymentdetailed logging and monitoring
  • 32. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWL
  • 33. INTRODUCING ARCADECRAWL INSIGHTSCRAWL PAGESCRAWL EVENTSARCADEFACEBOOKCRAWLqueue pythonworkersjob types
  • 34. INTRODUCING ARCADEfacebook/!"" __init__.py!"" common#   !"" __init__.py#   !"" crawler/#   !"" exception.py#   !"" facebook_model.py#   !"" helper.py#   !"" indexer/#   !"" ...!"" config#   !"" __init__.py#   !"" config.yml#   $"" env#   !"" __init__.py#   !"" crawler.py#   !"" ......!"" tests#   !"" __init__.py#   !"" fixtures#   #   !"" crawler_feed.yml#   #   !"" ...#   #   $"" stream_routing.yml#   !"" test_crawler_feed.py#   !"" ...#   $"" test_stream_worker.py$"" workers!"" __init__.py!"" crawler.py!"" ...
  • 35. from arcade.worker import TaskWorkerfrom arcade.helper import retry, task# [...]class FacebookCrawler(TaskWorker):def __init__(self, dependencies):super(FacebookCrawler, self).__init__(dependencies)@task("default", "crawl")def crawl_feed(self, data):"""Job type:* uid* last_crawl"""feed = CrawlerFeedModel(self)feed.crawl(data)def before_each(self, data):# [...]def run(dependencies):try:FacebookCrawler(dependencies).run()except:dependencies.get("exception").capture_exception()
  • 36. AND MANY MORE:continuous integrationRELIABLE ALERTINGCLOUDWATCH to STATSDREAL USER MONITORING
  • 37. Q/A
  • 38. Thanks!