Recommended
More Related Content
Similar to Faбrik - TimesOpen: Sockets and Streams - Sept. 2012
Similar to Faбrik - TimesOpen: Sockets and Streams - Sept. 2012 (20)
Recently uploaded
Recently uploaded (20)
Faбrik - TimesOpen: Sockets and Streams - Sept. 2012
- 1. NYT faбrik TimesOpen – 12 September 2012
- 2. Who Architect Infrastructure Group Previous – US Army – Harvard – Tech companies – United Nations 2
- 3. What we’ll cover A story faбrik overview Code and demo 3
- 4. Takeaways Application Developer – Demand „events‟ (no polling, no thanks, been there, done that) – Demand infrastructure that scales and you don‟t have to worry about Infrastructure Engineer – Decouple, flatten, simplify – Outsource complexity 4
- 5. The Story: Harvard U, 1986+ Bad: – No Internet (pre-web) – data locked in mainframe – large central clerical staff – monolithic central systems Good: – Vision of an Information Utility – desire to innovate – lots of desktop computers (30,000) – email everywhere although over diverse networks and technologies 5
- 6. The Story: Harvard U, 1986+ Solution: – Relational database (decouple data from application) – Email backbone (decouple producers from consumers) – Event-driven desktop applications (flatten) – Identical code on mainframe (simplify) Result: – Data warehouse unlocked (before the term was coined) – Central clerical staff functions upgraded/dispersed – Old central systems replaceable and, ultimately, replaced – Happy users! 6
- 7. faбrik – Why? ? 7
- 8. NYT Mission Enhance society by creating, collecting and distributing high quality news, information and entertainment - Distributing: publish / subscribe - Collecting: gather / analyze - High Quality: fast, reliable, accurate 8
- 9. faбrik Asynchronous Messaging Framework For client devices as well as our apps Enabled by: – Websockets – Robust message handling software – Amazon Web Services Focusing on simple, common services 9
- 10. faбrik – solve the problem! App 10
- 11. faбrik – basic App Message Broker App App 11
- 12. faбrik – basic Amazon Web Services App Message Broker • EC2 • S3 App • Identity & App Access Mgt • DynamoDB • Route 53 … 12
- 13. faбrik – basic++ App Service Buddy Buddy Message Broker “Retail” Message Broker “Wholesale” Service Buddy Other App 13
- 14. faбrik: Current Implementation Open source – Erlang/OTP – RabbitMQ – Nodejs – Sockjs (websockets +) – Python (gevent, select/epoll, tornado, futures, …) – ZeroMQ (new) Automated build/deployment 14
- 15. faбrik – active/active cluster Region Wherever Zone „a‟ Zone „b‟ Service Service Buddy Buddy „a‟ „b‟ 15
- 16. faбrik – active/active cluster Region Wherever Zone „a‟ Zone „b‟ Service Service Buddy Buddy „a‟ „b‟ 16
- 17. faбrik 17
- 18. faбrik 18
- 19. faбrik 19
- 20. faбrik – Layers 20
- 21. faбrik – Layers 21
- 22. faбrik – Layers 22
- 23. faбrik – Layers 23
- 24. 24
- 25. 25
- 26. Pub / Sub 26
- 27. Pub / Sub 27
- 28. Pub / Sub 28
- 29. Pub / Sub 29
- 30. Pub / Sub 30
- 31. Gather / Analyze 31
- 32. Gather / Analyze 32
- 33. Gather / Analyze 33
- 34. Gather / Analyze 34
- 35. Gather / Analyze 35
- 36. Gather / Analyze 36
- 37. Gather / Analyze 37
- 38. faбrik Demo Code App 38
Editor's Notes
- ----- Meeting Notes (01.08.12 02:23) -----
- GE225 – also analog computersHarvard: Milly Koss – beginning in 1950 Hopper, Mawkley – computer pioneersTech: Managing hardware/software companies in Venezuela & AustraliaUN: slow to change
- Picture: Not really Harvard - Central message switch for London’s pneumatic tube system – 50 miles of tubes – ca 1930 – 1870’s thru the 30’s
- Learning points:. When you touch some data, take it all. Keep the lowest level of detail
- Events – like email but fasterBig deal – extended to client devices – takes some effort as I’ll describe in a few slidesMessaging Infrastructure used to be expensiveIn combination, we can:. Outsource a lot of complexity. Decouple and flatten by using messaging everywhereSo we can focus on doing the simple things well, e.g.:. Publish/Subscribe. Persisting data. Gathering data for analysis
- I’ve been in ‘infrastructure’ mode – switching back to a developer’s point of viewWe’ll look later at some code that does just thatMeanwhile we’ll dive back into the innards of the fabrik
- Message Broker: . Routes and load balances – simple and complex topologies. ResilientHorizontalDecouples:. Different technologies. Different rates – buffers, queues
- Outsource complexity to Amazon Web Services
- 2 levelsWholesale Message Broker extends throughout the fabrik – think of it as encircling the world – components are pretty stable – doesn’t fail, although components mightRetail Message Broker is a “spur” – supports an app buddy and a collection of service buddies – spurs are added/subtracted as needed and may fail
- We’ve got a little ruby in there tooA bit about nodejs and python: (since I am the “meat – or maybe the ham” in this evening’s lineup re node). a plus for node is that asynch programming is natural and consistent – and with coffeescript it’s actually fun – in python you have to “do” something – and your support modules may have chosen to “do” something else!. However for our purposes, some of the nodejs support modules are not as mature or robust – so we have had to “fix” them. In particular, for the most critical high-performance components, we’re likely to use python for now, though nodejs is strategic for us, because javascript is so important to our community of clients
- Zooming up – and leaving out some detail – here’s an illustration of one way we are tying to create “non-stop” servicesThe red dot indicates clustering between virtual machine instances, each running in a separate zone – think datacenter – in an AWS regionMy 2 service buddies each perform the same service, and the broker load balances work to them, so they are each active
- If one of the service buddies goes down, or a broker instance goes down, or a zone goes down, the other broker adjusts and requeues work to the survivorMeanwhile, we will automatically be adding more resources to compensate, and possibly shunting work away from this region until it is fully healthy againNow lets zoom up again and take a global view
- These are the Amazon regions, and the zonesExcept a 3rd zone has now been added both to Northern California and Tokyo, so there are 20 zones availableEach zone has independent power and independent communications infrastructure – so think of them as datacentersOne of our goal is to be able to balance across all the zones, optimizing service and cost
- Logically, the wholesale layer is unified in each region The retail layer is autoscaled based upon demand and “health” – remember the retail components are “spurs” off the cluster, sharing nothing
- The regions are federated in a complete networkLet’s take another perspective on pretty much the same information
- I’ve pulled the regional clusters together in this view
- So that is the wholesale layer
- The our own apps – publishers or analyzers or whatever – connect via load balancers
- The retail layer connects via load balancer tooEach instance in this layer has an app buddy, a local broker, and service buddies, hence the “blend” of red and blue
- Finally the external clients coming thru Route 53 and via another layer of load balancers into our retail layer
- Every layer is supported by AWSNow let’s take a look at how we can use the fabrikOur initial target is to support the “publish/subscribe” pattern
- Nothing too radical here – classic Pub/SubExcept:. One machine image. Small systems software complement. 10-20 small, testable, tunable apps in a different languages. Decoupled, simple, flat. Global, fast, resilient, autoscaled – complexity outsourcedDevelopers – the real payoff is for you which I’ll try to illustrate with code in a few minutes
- Wholesale layer does not scale to the degree necessary to directly handle all retail eventsKill a couple birds at once by using AWS DynamoDB. Persist data immediately. Use dynamo to assemble detail from many sources in near real time
- That’s enough of an overview of the fabrik – let’s see a demo and look at some code
- Specifically we’ll look at some html that runs the demoThe publishing app is written in coffeescript and pretty simple too – anyone who’s interested can see me after and we’ll take a lookFirst the demo