Email
Like
Liked ×
Save
Private Content
Embed
Loading embed code…
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.
Loading

+ Follow

Faбrik - TimesOpen: Sockets and Streams - Sept. 2012

by NYT-TimesOpen on Sep 14, 2012

584 views

Accessibility

View text version

Upload Details

Uploaded via SlideShare as Microsoft PowerPoint

Usage Rights

File a copyright complaint

1 Embed 2

http://www.unscatter.com

Statistics

Likes: 0
Downloads: 0
Comments: 0
Embed Views: 2
Views on SlideShare: 582
Total Views: 584

----- Meeting Notes (01.08.12 02:23) -----
GE225 – also analog computersHarvard: Milly Koss – beginning in 1950 Hopper, Mawkley – computer pioneersTech: Managing hardware/software companies in Venezuela & AustraliaUN: slow to change
Picture: Not really Harvard - Central message switch for London’s pneumatic tube system – 50 miles of tubes – ca 1930 – 1870’s thru the 30’s
Learning points:. When you touch some data, take it all. Keep the lowest level of detail
Events – like email but fasterBig deal – extended to client devices – takes some effort as I’ll describe in a few slidesMessaging Infrastructure used to be expensiveIn combination, we can:. Outsource a lot of complexity. Decouple and flatten by using messaging everywhereSo we can focus on doing the simple things well, e.g.:. Publish/Subscribe. Persisting data. Gathering data for analysis
I’ve been in ‘infrastructure’ mode – switching back to a developer’s point of viewWe’ll look later at some code that does just thatMeanwhile we’ll dive back into the innards of the fabrik
Message Broker: . Routes and load balances – simple and complex topologies. ResilientHorizontalDecouples:. Different technologies. Different rates – buffers, queues
Outsource complexity to Amazon Web Services
2 levelsWholesale Message Broker extends throughout the fabrik – think of it as encircling the world – components are pretty stable – doesn’t fail, although components mightRetail Message Broker is a “spur” – supports an app buddy and a collection of service buddies – spurs are added/subtracted as needed and may fail
We’ve got a little ruby in there tooA bit about nodejs and python: (since I am the “meat – or maybe the ham” in this evening’s lineup re node). a plus for node is that asynch programming is natural and consistent – and with coffeescript it’s actually fun – in python you have to “do” something – and your support modules may have chosen to “do” something else!. However for our purposes, some of the nodejs support modules are not as mature or robust – so we have had to “fix” them. In particular, for the most critical high-performance components, we’re likely to use python for now, though nodejs is strategic for us, because javascript is so important to our community of clients
Zooming up – and leaving out some detail – here’s an illustration of one way we are tying to create “non-stop” servicesThe red dot indicates clustering between virtual machine instances, each running in a separate zone – think datacenter – in an AWS regionMy 2 service buddies each perform the same service, and the broker load balances work to them, so they are each active
If one of the service buddies goes down, or a broker instance goes down, or a zone goes down, the other broker adjusts and requeues work to the survivorMeanwhile, we will automatically be adding more resources to compensate, and possibly shunting work away from this region until it is fully healthy againNow lets zoom up again and take a global view
These are the Amazon regions, and the zonesExcept a 3rd zone has now been added both to Northern California and Tokyo, so there are 20 zones availableEach zone has independent power and independent communications infrastructure – so think of them as datacentersOne of our goal is to be able to balance across all the zones, optimizing service and cost
Logically, the wholesale layer is unified in each region The retail layer is autoscaled based upon demand and “health” – remember the retail components are “spurs” off the cluster, sharing nothing
The regions are federated in a complete networkLet’s take another perspective on pretty much the same information
I’ve pulled the regional clusters together in this view
So that is the wholesale layer
The our own apps – publishers or analyzers or whatever – connect via load balancers
The retail layer connects via load balancer tooEach instance in this layer has an app buddy, a local broker, and service buddies, hence the “blend” of red and blue
Finally the external clients coming thru Route 53 and via another layer of load balancers into our retail layer
Every layer is supported by AWSNow let’s take a look at how we can use the fabrikOur initial target is to support the “publish/subscribe” pattern
Nothing too radical here – classic Pub/SubExcept:. One machine image. Small systems software complement. 10-20 small, testable, tunable apps in a different languages. Decoupled, simple, flat. Global, fast, resilient, autoscaled – complexity outsourcedDevelopers – the real payoff is for you which I’ll try to illustrate with code in a few minutes
Wholesale layer does not scale to the degree necessary to directly handle all retail eventsKill a couple birds at once by using AWS DynamoDB. Persist data immediately. Use dynamo to assemble detail from many sources in near real time
That’s enough of an overview of the fabrik – let’s see a demo and look at some code
Specifically we’ll look at some html that runs the demoThe publishing app is written in coffeescript and pretty simple too – anyone who’s interested can see me after and we’ll take a lookFirst the demo

Faбrik - TimesOpen: Sockets and Streams - Sept. 2012 Presentation Transcript

NYT faбrikTimesOpen – 12 September 2012
Who Architect Infrastructure Group Previous – US Army – Harvard – Tech companies – United Nations 2
What we’ll cover A story faбrik overview Code and demo 3
Takeaways Application Developer – Demand „events‟ (no polling, no thanks, been there, done that) – Demand infrastructure that scales and you don‟t have to worry about Infrastructure Engineer – Decouple, flatten, simplify – Outsource complexity 4
The Story: Harvard U, 1986+ Bad: – No Internet (pre-web) – data locked in mainframe – large central clerical staff – monolithic central systems Good: – Vision of an Information Utility – desire to innovate – lots of desktop computers (30,000) – email everywhere although over diverse networks and technologies 5
The Story: Harvard U, 1986+ Solution: – Relational database (decouple data from application) – Email backbone (decouple producers from consumers) – Event-driven desktop applications (flatten) – Identical code on mainframe (simplify) Result: – Data warehouse unlocked (before the term was coined) – Central clerical staff functions upgraded/dispersed – Old central systems replaceable and, ultimately, replaced – Happy users! 6
faбrik – Why? ? 7
NYT MissionEnhance society by creating, collecting anddistributing high quality news, information andentertainment- Distributing: publish / subscribe- Collecting: gather / analyze- High Quality: fast, reliable, accurate 8
faбrik Asynchronous Messaging Framework For client devices as well as our apps Enabled by: – Websockets – Robust message handling software – Amazon Web Services Focusing on simple, common services 9
faбrik – solve the problem! App 10
faбrik – basic App Message Broker App App 11
faбrik – basic Amazon Web Services App Message Broker • EC2 • S3 App • Identity & App Access Mgt • DynamoDB • Route 53 … 12
faбrik – basic++ App Service Buddy Buddy Message Broker “Retail” Message Broker“Wholesale” Service Buddy Other App 13
faбrik: Current Implementation Open source – Erlang/OTP – RabbitMQ – Nodejs – Sockjs (websockets +) – Python (gevent, select/epoll, tornado, futures, …) – ZeroMQ (new) Automated build/deployment 14
faбrik – active/active cluster Region Wherever Zone „a‟ Zone „b‟ Service Service Buddy Buddy „a‟ „b‟ 15
faбrik – active/active cluster Region Wherever Zone „a‟ Zone „b‟ Service Service Buddy Buddy „a‟ „b‟ 16
faбrik 17
faбrik 18
faбrik 19
faбrik – Layers 20
faбrik – Layers 21
faбrik – Layers 22
faбrik – Layers 23
24
25
Pub / Sub 26
Pub / Sub 27
Pub / Sub 28
Pub / Sub 29
Pub / Sub 30
Gather / Analyze 31
Gather / Analyze 32
Gather / Analyze 33
Gather / Analyze 34
Gather / Analyze 35
Gather / Analyze 36
Gather / Analyze 37
faбrik Demo Code App 38

Faбrik - TimesOpen: Sockets and Streams - Sept. 2012

by NYT-TimesOpen on Sep 14, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

Faбrik - TimesOpen: Sockets and Streams - Sept. 2012 Presentation Transcript