Weightlifting at SimplySocial

•

0 likes•951 views

Bogdan Gaza

How SimplySocial uses Python

Technology

ARCHITECTURE
api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 02 API N...
APP 01 APP 02 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs

LIFE OF A REQUEST
api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 01 API N...
APP 01 APP 01 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs
API 02
APP 02

api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 02 API N...
APP 01 APP 02 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs
LIFE OF A REQUEST

api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 02 API N...
APP 01 APP 02 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs
LIFE OF A REQUEST
2M x

USE OF PYTHON
api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 02 API N...
APP 01 APP 02 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs

api.gosimplysocial.com
app.gosimplysocial.com
API 01 API 02 API N...
APP 01 APP 02 APP N...
BACKEND
01
BACKEND
02
BACKEND
N...
DBs
USE OF PYTHON

PYTHON ON APPSERVERS
AWS EC2
Ubuntu 12.04
Chef
NGINX + PHP FPM
FABRIC
DEPLOYMENT
Metrics
Logging

USE OF PYTHON
METRICS
AWS EC2
Ubuntu 12.04
Chef
NGINX + PHP FPM
FABRIC
Metrics
Logging

INTRODUCING OSSIE
METRICS GATHERING
REPORTING STATS THROUGH A SIMPLE
WEB INTERFACE
INTERACTION WITH RUNNING PROcesses
BACKEND BY STATSD and GRAPHITE

INTRODUCING OSSIE
import ossie
stat = ossie.Client("arcade.random_worker")
stat.gauge("a_metric", 256)
stat.timer("random.metric", 1.28)
with stat.timer("random.metric.with"):
# do something that requires an amount of time
# ossie client will report the right time

USE OF PYTHON
logging
AWS EC2
Ubuntu 12.04
Chef
NGINX + PHP FPM
FABRIC
Metrics
Logging

logginG
FULL JSON logging
EVEN nginx logs json
powered by fluentd + S3
FUTURE: Logs can be searched by
elasticsearch + kibana

AWS EC2
Ubuntu 12.04
Chef
FABRIC
SUPERVISORD
Metrics
SUPERVISORD
Workers
Logging

AWS EC2
Ubuntu 12.04
Chef
FABRIC
WORKERS
Metrics
SUPERVISORD
Workers
Logging

INTRODUCING ARCADE
queue based workers framework
Convention over configuration
simple deployment
detailed logging and monitoring

INTRODUCING ARCADE
CRAWL INSIGHTS
CRAWL PAGES
CRAWL EVENTS
ARCADE
FACEBOOK
CRAWL

INTRODUCING ARCADE
CRAWL INSIGHTS
CRAWL PAGES
CRAWL EVENTS
ARCADE
FACEBOOK
CRAWL
queue python
workers
job types

INTRODUCING ARCADE
facebook/
!"" __init__.py
!"" common
#   !"" __init__.py
#   !"" crawler/
#   !"" exception.py
#   !"" facebook_model.py
#   !"" helper.py
#   !"" indexer/
#   !"" ...
!"" config
#   !"" __init__.py
#   !"" config.yml
#   $"" env
#   !"" __init__.py
#   !"" crawler.py
#   !"" ...
...
!"" tests
#   !"" __init__.py
#   !"" fixtures
#   #   !"" crawler_feed.yml
#   #   !"" ...
#   #   $"" stream_routing.yml
#   !"" test_crawler_feed.py
#   !"" ...
#   $"" test_stream_worker.py
$"" workers
!"" __init__.py
!"" crawler.py
!"" ...

from arcade.worker import TaskWorker
from arcade.helper import retry, task
# [...]
class FacebookCrawler(TaskWorker):
def __init__(self, dependencies):
super(FacebookCrawler, self).__init__(dependencies)
@task("default", "crawl")
def crawl_feed(self, data):
"""
Job type:
* uid
* last_crawl
"""
feed = CrawlerFeedModel(self)
feed.crawl(data)
def before_each(self, data):
# [...]
def run(dependencies):
try:
FacebookCrawler(dependencies).run()
except:
dependencies.get("exception").capture_exception()

AND MANY MORE:
continuous integration
RELIABLE ALERTING
CLOUDWATCH to STATSD
REAL USER MONITORING

What's hot

How to make GAE adapt the Great Firewall

Hayato Yoshikawa

If you are convinced that actor model can drive your design of scalable and performant .NET applications, you will very likely give a chance to Akka.NET - a port of JVM Akka framework that gained popularity among .NET developers. And if you are fan of a functional programming, you will certainly try Akka.NET F# API. But which of them? There is an official one, but there is also an experimental one, called Akkling that offers typed actors and F# support for actor lifecycle, cluster sharding and reactive streams. Come to this talk and you will be convinced that Akkling is the way to go!

.NET Fest 2018. Vagif Abilov. Akka + F# = Akkling

NETFest

This session is a deep dive into techniques used by successful customers who optimized their use of AWS. Learn tricks and hear tips you can implement right away to reduce waste, choose the most efficient instance, and fine-tune your spending; often with improved performance and a better end-customer experience. We showcase innovative approaches and demonstrate easily applicable methods to save you time and money with Amazon EC2, Amazon S3, and a host of other services.

(ENT302) Cost Optimization on AWS | AWS re:Invent 2014

Amazon Web Services

Apache Spark is already a proven leader as a heavy-weight distributed processing framework. But how does one migrate an 8-years-old, single-server, MySQL-based legacy system to such new shiny frameworks? How do you accurately preserve the behavior of a system consuming Terabytes of data every day, hiding numerous undocumented implicit gotchas and changing constantly, while shifting to brand new development paradigms? In this talk we present Kenshoo’s attempt at this challenge, where we migrated a legacy aggregation system to Spark. Our solutions include heavy usage of metrics and graphite for analyzing production data; “local-mode” client enabling reuse of legacy tests suits; data validations using side-by-side execution; and maximum reuse of code through refactoring and composition. Some of these solutions use Spark-specific characteristics and features.

Spark Your Legacy (Spark Summit 2016)

Tzach Zohar

Libcloud presentation

Christophe Marchal

Nils Mohr & Jake Pearce - 100 years of flight data at British Airways. Past, ...

AWSCOMSUM

Illustrator_Sample

Saeid Saadatmand

The State of the Developer Ecosystem - .NET Conf Barcelona 2018

Carlos Mendible

Knockout extender

LearningTech

AWS Dublin Briefing - Cool AWS Use Cases

Ian Massingham

Altitude San Francisco 2018: WebAssembly Tools & Applications

Fastly

Art Data Hackathon - Klima iOS app

lab_SNG

Cypher for Gremlin

openCypher

TensorFlow on GCP

Hayato Yoshikawa

Lettuce example using scenarios outline

Karen Wiznia

Business Dashboards using Bonobo ETL, Grafana and Apache Airflow

Romain Dorgueil

Device status anomaly detection

David Tung

Asynchronous Python at Kumparan

Bayu Aldi Yansyah

ICGIS 2018 - Cloud-powered Machine Learnings on Geospactial Services (Channy ...

Channy Yun

What's hot (19)

How to make GAE adapt the Great Firewall

.NET Fest 2018. Vagif Abilov. Akka + F# = Akkling

(ENT302) Cost Optimization on AWS | AWS re:Invent 2014

Spark Your Legacy (Spark Summit 2016)

Libcloud presentation

Nils Mohr & Jake Pearce - 100 years of flight data at British Airways. Past, ...

Illustrator_Sample

The State of the Developer Ecosystem - .NET Conf Barcelona 2018

Knockout extender

AWS Dublin Briefing - Cool AWS Use Cases

Altitude San Francisco 2018: WebAssembly Tools & Applications

Art Data Hackathon - Klima iOS app

Cypher for Gremlin

TensorFlow on GCP

Lettuce example using scenarios outline

Business Dashboards using Bonobo ETL, Grafana and Apache Airflow

Device status anomaly detection

Asynchronous Python at Kumparan

ICGIS 2018 - Cloud-powered Machine Learnings on Geospactial Services (Channy ...

Viewers also liked

Selenium2 and Jenkins: Almost pain-free UI Testing

mikereedell

Web Application Testing with Selenium

Sargis Sargsyan

Understanding and measuring web performance

Bogdan Gaza

Automation Framework 042009 V2

guestb66d91

ATLAS Automation POC

aakashmc

test_automation_POC

Rafael Battesti

Understanding Selenium/RC, Webdriver Architecture and developing the page obj...

Atirek Gupta

Keyword-driven Test Automation Framework

Mikhail Subach

Viewers also liked (8)

Selenium2 and Jenkins: Almost pain-free UI Testing

Web Application Testing with Selenium

Understanding and measuring web performance

Automation Framework 042009 V2

ATLAS Automation POC

test_automation_POC

Understanding Selenium/RC, Webdriver Architecture and developing the page obj...

Keyword-driven Test Automation Framework

Similar to Weightlifting at SimplySocial

AWS re:Invent re:Cap 행사에서 발표된 강연 자료입니다. 아마존 웹서비스의 김일호 솔루션스 아키텍트가 발표한 내용입니다. 내용 요약: Hadoop과 Elastic MapReduce, Redshift, Kinesis, Data Pipeline, S3 등 다양한 서비스들을 활용하는 데이터 분석의 모범사례 및 아키텍처 설계 패턴에 대해 말씀드리고, re:Invent에서 새로 추가된 Amazon EC2 컴퓨팅 최적화 인스턴스 C4와 새로 발표된 Amazon EBS 볼륨 확장 및 성능 향상에 대해 함께 살펴볼 예정입니다.

AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호

Amazon Web Services Korea

Many people are building different workloads using serverless technologies these days, but how would a non-trivial system such as a social network look like on serverless? In this talk Yan will discuss his journey of migrating a social network startup to serverless, and how his team was able to improve performance, scalability and feature delivery using serverless technologies. Yan will discuss how serverless technologies such as Lambda are used to implement each part of their system, including search, push notifications, timeline, user recommendations, and business intelligence. If you're wondering how serverless can be used to solve a wide variety of challenges in your business, this is the talk for you.

How to build a social network on Serverless (AWS Community Summit)

Yan Cui

How to build a social network on serverless | Yan Cui

AWSCOMSUM

Amazed by AWS Series #4

Amazon Web Services Korea

How to build a social network on serverless

Yan Cui

Itb 2021 - Bulding Quick APIs by Gavin Pickin

Gavin Pickin

Funliday 是一個極度依賴景點資料的網路服務，大家從十月開始在使用 Funliday 在「景點瀏覽」輸入城市，或者是移動地圖出現的推薦景點，應該都是極速出現，跟之前兩年的速度實在是相差甚遠。這中間的差異主要是調整了演算法、儲存資料以及索引，更重要的是不要求絕對即時，這樣子可以讓計算的時間加快了 350 倍以上，這場分享就是解釋 Funliday 是如何利用上面這些技術加快計算時間。文字筆記：https://www.facebook.com/kewang.information/posts/2728954360714254 預計提到的內容包括：PostgreSQL、Redis、CDN

大解密！用 PostgreSQL 提升 350 倍的 Funliday 推薦景點計算速度

Mu Chun Wang

Getting Started with Serverless Architectures

Amazon Web Services

AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures? In this talk Yan and Scott will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.

Serverless in production, an experience report

Yan Cui

Serverless technologies like AWS Lambda has drastically simplified the task of building reactive systems - drop a file into S3 and a Lambda function would be triggered to process it, push an event into a Kinesis stream and magically it'll be processed by a Lambda function in real-time, you can even use Lambda to automate the process of auditing and securing your AWS account by automatically reacting to rule violations to your security policy. Join us in this talk to see some architectural design patterns that have emerged with AWS Lambda, and learn how to pick the right event source based on the tradeoffs you want. Amongst the many patterns we'll explore, here are a few to whet your appetite : pub-sub, cron, push-pull, saga and decoupled invocation.

Serveless design patterns

Yan Cui

How to build a social network on serverless

Yan Cui

Serverless Design Patterns

Yan Cui

Serveless design patterns (VoxxedDays Luxembourg)

Yan Cui

Serverless in production, an experience report (FullStack 2018)

Yan Cui

Serverless Design Patterns

Yan Cui

Data analytics master class: predict hotel revenue

Kris Peeters

Companies, from startups to enterprises across the globe, are looking to migrate data warehousing to the cloud to increase performance and lower costs. Data engineers, data analysts, and developers also need to access and consume this important data. The landscape is constantly evolving and there are many solutions available for enterprises of all sizes. In this workshop, we dive deep into architectural patterns, use cases, and best practices when designing an enterprise data warehouse in the cloud. We also address key issues such as data governance and democratization. At the end of this workshop, you’ll be equipped to design and implement a cloud enterprise data warehouse platform that provides the most benefit for your enterprise, data consumers, and customers.

GPSWKS401_Designing a Cloud Enterprise Data Warehouse

Amazon Web Services

20141021 AWS Cloud Taekwon - Big Data on AWS

Amazon Web Services Korea

Serverless in Production, an experience report (AWS UG South Wales)

Yan Cui

We present the Azure Cognitive Services on Spark, a simple and easy to use extension of the SparkML Library to all Azure Cognitive Services. This integration allows Spark Users to embed cloud intelligence directly into their spark computations, enabling a new generation of intelligent applications on Spark. Furthermore, we show that with our new Containerized Cognitive Services, one can embed cloud intelligence directly into the Spark cluster for ultra-low latency, on-prem, and offline applications. We show how using our Integration, one can compose these cognitive services with other services, SQL computations, and Deep Networks to create sophisticated and intelligent heterogenous applications. Moreover, we show how to redeploy these compositions as Restful Services with Spark Serving. We will also explore the architecture of these contributions which leverage HTTP on Spark, a novel integration between Spark with the widely used Hypertext Transfer Protocol (HTTP). This library can integrate any framework into the Spark ecosystem that is capable of communicating through HTTP. Finally, we demonstrate how to use these services to create a large class of intelligent applications such as custom search engines, realtime facial recognition systems, and unsupervised object detectors.

The Azure Cognitive Services on Spark: Clusters with Embedded Intelligent Ser...

Databricks

Similar to Weightlifting at SimplySocial (20)

AWS re:Invent re:Cap - 데이터 분석: Amazon EC2 C4 Instance + Amazon EBS - 김일호

How to build a social network on Serverless (AWS Community Summit)

How to build a social network on serverless | Yan Cui

Amazed by AWS Series #4

How to build a social network on serverless

Itb 2021 - Bulding Quick APIs by Gavin Pickin

大解密！用 PostgreSQL 提升 350 倍的 Funliday 推薦景點計算速度

Getting Started with Serverless Architectures

Serverless in production, an experience report

Serveless design patterns

How to build a social network on serverless

Serverless Design Patterns

Serveless design patterns (VoxxedDays Luxembourg)

Serverless in production, an experience report (FullStack 2018)

Serverless Design Patterns

Data analytics master class: predict hotel revenue

GPSWKS401_Designing a Cloud Enterprise Data Warehouse

20141021 AWS Cloud Taekwon - Big Data on AWS

Serverless in Production, an experience report (AWS UG South Wales)

The Azure Cognitive Services on Spark: Clusters with Embedded Intelligent Ser...

Recently uploaded

Intro to Passkeys and the State of Passwordless.pptx

FIDO Alliance

Discover the top CodeIgniter development companies that can elevate your project to new heights. Our blog explores the best firms known for their expertise in CodeIgniter framework development. From robust web applications to scalable solutions, these companies deliver excellence. Whether you're a startup or an enterprise, find the perfect match for your development needs on Top CSS Gallery's blog.

Top 10 CodeIgniter Development Companies

TopCSSGallery

Because observability is such a broad topic – and often something we learn on the job – it can feel like there’s too much to learn at once. But you don’t have to tackle everything and can start with the basics and build from there! Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. No matter what tooling is in place, there are still observability fundamentals that developers should know. That’s why I’ve put together a primer on the different telemetry types, when to use them, how to understand the data journey, and what to look for in time series graphs.

Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)

Paige Cruz

Introduction To Iamnobody89757 the vast expanse of the online realm, where anonymity and individuality intertwine, a phenomenon has emerged that captivates the collective curiosity – Iamnobody89757. This enigmatic entity, once a mere username, has transcended its humble origins to become a symbol of the intricate dance between privacy and expression in the digital age. Through this exploration, we delve into the origins, evolution, and far-reaching implications of this intriguing moniker, shedding light on the profound questions it raises about our digital selves. The Origins of iamnobody89757 The origins of iamnobody89757 are shrouded in the virtual mists of the wide internet. The term gained popularity on a niche discussion group that examined the most puzzling puzzles on the internet. In this instance, iamnobody89757 surfaced as a mysterious character who captivated other users with a conversation that was both mysterious and perceptive. These initial interactions were nuanced, filled with veiled references and subtle hints that painted a picture of someone—or something—with an intricate understanding of the digital domain’s darker corners. Although initially dismissed by many as just another anonymous user, the accuracy of certain predictions shared by iamnobody89757 soon captured the collective imagination. The iamnobody89757 enigma began with this shift from an unnoticed commenter to a fascinating subject, laying the groundwork for a growing tale that would weave its way through the fabric of online communities. The Birth of a Digital Persona Tracing the Roots The inception of Iamnobody89757 can be traced back to the early days of online forums and chat rooms, where users sought a delicate balance between anonymity and connectivity. In a world where personal information was a coveted commodity, this seemingly random assemblage of words and numbers became a shield, protecting the user’s identity while allowing them to engage in discourse without fear of judgment or repercussions.

“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf

Muhammad Subhan

Ruby has a lot of standard libraries from Ruby 1.8. I promote them democratically with GitHub today via default and bundled gems. So, I'm working to extract them for Ruby 3.4 continuously and future versions. It's long journey for me. After that, some versions may suddenly happen LoadError at require when running bundle exec or bin/rails, for example matrix or net-smtp. We need to learn what's difference default/bundled gems with standard libraries. In this presentation, I will introduce what's the difficult to extract bundled gems from default gems and the details of the functionality that Ruby's require and bundle exec with default/bundled gems. You can learn how handle your issue about standard libraries.

Long journey of Ruby Standard library at RubyKaigi 2024

Hiroshi SHIBATA

Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.

2024 May Patch Tuesday

Ivanti

Vector Search @ sw2con for slideshare.pptx

jbellis

Explore the latest trends and insights on JavaScript usage with Pixlogix's informative blog. Discover key statistics and facts about JavaScript's role in web development, its popularity among developers, and its impact on modern websites. Stay updated with the evolving landscape of JavaScript frameworks and libraries, and learn how they're shaping the future of web development. Gain valuable insights to enhance your JavaScript skills and stay ahead in the digital realm.

JavaScript Usage Statistics 2024 - The Ultimate Guide

Pixlogix Infotech

The Metaverse: Are We There Yet?

Mark Billinghurst

At Skynet Technologies, our team of accessibility experts performs automated, semi-automated, and manual audits of websites and web applications as per WCAG 2.2 level AA, ADA, and section 508. Based on evaluations of the accessibility compliance level of the website’s UI, design, source code, navigation, interactive elements, and overall usability, we will provide a digital accessibility evaluation report with in-depth details of potential accessibility barriers and remediation recommendations. Get a manual website WCAG audit (2.0, 2.1, 2.2 level AA) for a small website: 10 pages: $2,500 within 7 business days 30 pages: $7,500 within 14 business days 50 pages: $12,500 within 28 business days For medium websites: 100 pages: $25,000 within 6 weeks For larger websites or audits of all pages, please reach out hello@skynettechnologies.com.

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

Skynet Technologies

Intro in Product Management - Коротко про професію продакт менеджера

Mark Opanasiuk

Event-Driven Architecture Masterclass: Challenges in Stream Processing

ScyllaDB

WebRTC and SIP not just audio and video @ OpenSIPS 2024

Lorenzo Miniero

Portal Kombat : extension du réseau de propagande russe

中央社

Design and Development of a Provenance Capture Platform for Data Science

Paolo Missier

Using IESVE for Room Loads Analysis - UK & Ireland

IES VE

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

FIDO Alliance

In today's digital world, trust is key to customer relationships, but keeping it is a huge challenge. Customers are well-informed and empowered, quick to change brands if their trust is broken, even if it costs them more. This puts a lot of pressure on organizations to handle trust and safety issues with great care and transparency. The challenge, however, is real. Fragmented solutions have left privacy, legal, and security teams in a perpetual cycle of catch-up, struggling to update privacy notices, manage customer data rights, and answer lengthy security questionnaires—all while trying to prove ROI to the business. It's a thankless job, filled with repetition, tedious tasks, and constant interdepartmental coordination. Combine this with fast regulatory changes and the quick evolution of AI, and it becomes overwhelming. Join this webinar to learn more about TrustArc's new innovative solution Trust Center, the only unified, no-code online hub for trust and safety information built for privacy, security, compliance, and legal teams. Trust Center streamlines your path to compliance, shortens the pre-sales cycle, and reduces both legal and regulatory risks, saving time, effort, and cost. This webinar will review: - Why companies are building unified Trust Centers for a robust privacy program. - How unified Trust Centers streamline sales cycles, ensure regulatory compliance, and reduce operational bottlenecks. - How compliance, legal, security, GRC, and privacy teams benefit from a unified Trust Center in terms of needs, pains, and outcomes. - How TrustArc Trust Center saves time and work while reducing legal, reputational, and compliance risk by effectively managing policies, notices, terms, and disclosures, and providing real-time updates on subprocessors.

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

FIDO Alliance

Recently uploaded (20)

Intro to Passkeys and the State of Passwordless.pptx

Top 10 CodeIgniter Development Companies

Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)

“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf

Long journey of Ruby Standard library at RubyKaigi 2024

2024 May Patch Tuesday

Vector Search @ sw2con for slideshare.pptx

JavaScript Usage Statistics 2024 - The Ultimate Guide

The Metaverse: Are We There Yet?

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

Intro in Product Management - Коротко про професію продакт менеджера

Event-Driven Architecture Masterclass: Challenges in Stream Processing

WebRTC and SIP not just audio and video @ OpenSIPS 2024

Portal Kombat : extension du réseau de propagande russe

Design and Development of a Provenance Capture Platform for Data Science

Using IESVE for Room Loads Analysis - UK & Ireland

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

Weightlifting at SimplySocial

1. Weightlifting @

2. @bogdan

3. @SIMPLYSOCIAL

4. + =

5. + =

6. + =?

8. + =

9. ARCHITECTURE api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs

10. LIFE OF A REQUEST api.gosimplysocial.com app.gosimplysocial.com API 01 API 01 API N... APP 01 APP 01 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs API 02 APP 02

11. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST

12. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST

13. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST 2M x

14. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST

15. USE OF PYTHON api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs

16. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

17. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

18. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

19. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

20. CAN I HEAR A BOOO FOR ?

21. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

22. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

23. PYTHON ON APPSERVERS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC DEPLOYMENT Metrics Logging

24. USE OF PYTHON METRICS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging

25. INTRODUCING OSSIE METRICS GATHERING REPORTING STATS THROUGH A SIMPLE WEB INTERFACE INTERACTION WITH RUNNING PROcesses BACKEND BY STATSD and GRAPHITE

26. INTRODUCING OSSIE import ossie stat = ossie.Client("arcade.random_worker") stat.gauge("a_metric", 256) stat.timer("random.metric", 1.28) with stat.timer("random.metric.with"): # do something that requires an amount of time # ossie client will report the right time

27.

28. USE OF PYTHON logging AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging

29. logginG FULL JSON logging EVEN nginx logs json powered by fluentd + S3 FUTURE: Logs can be searched by elasticsearch + kibana

30. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON

31. AWS EC2 Ubuntu 12.04 Chef FABRIC SUPERVISORD Metrics SUPERVISORD Workers Logging

32. AWS EC2 Ubuntu 12.04 Chef FABRIC WORKERS Metrics SUPERVISORD Workers Logging

33. INTRODUCING ARCADE queue based workers framework Convention over configuration simple deployment detailed logging and monitoring

34. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL

35. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL queue python workers job types

36. INTRODUCING ARCADE facebook/ !"" __init__.py !"" common # !"" __init__.py # !"" crawler/ # !"" exception.py # !"" facebook_model.py # !"" helper.py # !"" indexer/ # !"" ... !"" config # !"" __init__.py # !"" config.yml # $"" env # !"" __init__.py # !"" crawler.py # !"" ... ... !"" tests # !"" __init__.py # !"" fixtures # # !"" crawler_feed.yml # # !"" ... # # $"" stream_routing.yml # !"" test_crawler_feed.py # !"" ... # $"" test_stream_worker.py $"" workers !"" __init__.py !"" crawler.py !"" ...

37. from arcade.worker import TaskWorker from arcade.helper import retry, task # [...] class FacebookCrawler(TaskWorker): def __init__(self, dependencies): super(FacebookCrawler, self).__init__(dependencies) @task("default", "crawl") def crawl_feed(self, data): """ Job type: * uid * last_crawl """ feed = CrawlerFeedModel(self) feed.crawl(data) def before_each(self, data): # [...] def run(dependencies): try: FacebookCrawler(dependencies).run() except: dependencies.get("exception").capture_exception()

38. AND MANY MORE: continuous integration RELIABLE ALERTING CLOUDWATCH to STATSD REAL USER MONITORING

39. Q/A

40. Thanks!

Weightlifting at SimplySocial

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (8)

Similar to Weightlifting at SimplySocial

Similar to Weightlifting at SimplySocial (20)

More from Bogdan Gaza

More from Bogdan Gaza (7)

Recently uploaded

Recently uploaded (20)

Weightlifting at SimplySocial