Your SlideShare is downloading. ×
Programming the cloud with Skywriting
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Programming the cloud with Skywriting


Published on

An introduction to cloud programming models and the Skywriting project. Talk originally given at Imperial College, London, on 13th May 2010. …

An introduction to cloud programming models and the Skywriting project. Talk originally given at Imperial College, London, on 13th May 2010.

More information about the Skywriting project can be found here:

Published in: Technology

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Thanks for the introduction, Eva. Well, as Eva said, my name’s Derek Murray, I’m a third year PhD student at Cambridge, and today I’m going to talk about Skywriting, which is a little bit of work I’ve been doing with these guys: Malte, Chris, Anil and my supervisor Steve Hand.Skywriting is a system for large-scale distributed computation – in this respect it’s similar to things like Google MapReduce and Microsoft’s Dryad – so that’s systems where your data or compute need is so big that you have to use a cluster in parallel to get the job done.It was the success of these systems – in particular Hadoop, the open-source MapReduce – that motivated us to start this work. What I found interesting was that people were using these things in entirely unexpected ways… taking MapReduce, which is excellent for log-processing, and running some big iterative machine learning algorithm on it. We reckoned that people were using MapReduce not because of its programming model, but despite it.So we set out to build something that combines all the advantages of previous systems, with a very flexible programming model. The result was Skywriting, so let’s see what you think…
  • All the systems we’ll discuss today use the simple notion of task parallelism. Many algorithms can be divided into tasks, which are just chunks of sequential code. The key observation is that two independent tasks can run in parallel. And when your whole job divides into a fully independent bag of tasks, it’s said to be “embarrassingly parallel”.
  • And how do you run these embarrassingly parallel jobs? Well, you give your bag of tasks to a master, which doles them out on demand to a set of workers.This is a very simple architecture to program. And it has a lot of benefits. If one of the workers crashes, fine! The master will notice and give that worker’s current task to someone else. And if a worker is a bit slower than the others, that’s also fine! Each worker pulls a new task when it has completed the last one, so even a heterogeneous pool can do useful work.
  • Embarrassing parallelism is not very interesting: it only lets you do boring things like search for aliens and brute-force people’s passwords.
  • It gets much more interesting – i.e. commercially useful – when the tasks have dependencies between them. So here, we have two tasks A and B, and a relation that says A must run before B. The usual reason for this is because A writes some output, and B wants to read it.Think of this like makefile rules. You can build up graphs out of these dependencies, and resolve them in parallel.In fact, the original name for this project was “Cloud Make”. Fortunately it changed….
  • Are you all familiar with MapReduce?Introduced by Google in 2004, MapReduce used the observation that the map() function from functional programming can run in parallel over large lists. So they broke down their huge data into chunks, and ran each through a “map task”, generating some key-value pairs that are then sorted by key in this shuffle phase, and then the values for each key are folded in parallel using a “reduce task”.This basically uses the same master-worker task farm that I showed on a previous slide, with the single constraint that all the map tasks must finish before the reduce tasks begin. Therefore it had the benefit of working at huge scale, and being very reliable.
  • A couple of years later, Microsoft, which also has a search engine, released “Dryad”, which generalisesMapReduce by allowing the user to specify a job as any directed acyclic graph. The graph has vertices – which are arbitrary sequential code in your favourite language – and channels, which could be files, in-memory FIFOs, TCP connections or whatever.Clearly you can implement MapReduce in Dryad, since it’s just a DAG. But Dryad makes things like Joins much easier, because a task can have multiple inputs.
  • So far, we can run any finite directed acyclic graph using Dryad. As the name suggests, however, Dryad is not terribly good at cyclic data flows.These turn up all the time in fields like machine learning, scientific computing and information retrieval. Take PageRank, for example, which involves repeatedly premultiplying a vector by a large sparse matrix representing the web. You keep doing this until you reach a fixpoint, and the PageRank vector has converged.At present, all you can do is submit one job after another. This is bad for a number of reasons. First of all, it’s very slow: MapReduce and Dryad are designed for batch submission, and so starting an individual job takes on the order of 30 seconds. If your iteration is shorter than that, you’re losing out on parallel speedup.It also introduces a co-dependency between the client and the cluster. Now the client, which is just some simple program that submits jobs to the cluster, has to stay running for the duration of the job, but since it’s outside the cluster, it gets none of the advantages of fault-tolerance, of data locality, of fair scheduling. Since the client now contains critical job state, it’s necessary to add all these features manually.
  • Remember our Master-worker architecture? Well, if you’ve ever tried to setup Hadoop or Dryad, you’ll know that you need to make sure all of the workers are the same, running the same operating system, on the same local network.
  • But what if all you have is a little ad-hoc cluster, with a Windows desktop, a Linux server and a Mac laptop?
  • Or, perhaps less contrived, what if your data are spread between different cloud providers. So you might have some data in Amazon S3, some in Google’s App Engine, and some in Windows Azure. Our mantra is “put the computation near the data”, and it’s not practical to shift all the data to one place.
  • And what about this? Say you have a really important task to complete, but you don’t know how long it’ll take – maybe you’re using some kind of randomised algorithm. So you fire off three copies of the same task… and eventually one finishes. At this point, you can just kill the other two.Although MapReduce and Dryad have limited support for this, it’s not first-class: you can’t do it on demand, only in response to “straggler” nodes that take much longer to complete than others.
  • I’ve spent quite a lot of slides being rather coy about what’s to come, but if you’ve read the abstract, you’ll know that Skywriting is
  • …two things. First, instead of using DAGs to describe a job, we use the most powerful thing available to us: a Turing-complete coordination language. This sounds ominous and theoretical, but actually it’s just a programming language that looks a lot like JavaScript, with all the usual control flow structures, loops, ifs, functions and so on.Since we want to run things efficiently in parallel, it has support for spawning tasks, and a way to call external code.The other main component is the distributed execution engine, which actually executes Skywriting programs in the cluster. The interesting thing about this is that a “task” is just a Skywriting function – a continuation to be more precise – which means that tasks can spawn other tasks, and thereby grow the job dynamically.
  • 1.0 – 1.2 GHz Xeon or Opteron. 1.7GB RAM, 150GB disk.
  • 50 x 50 on 50 workers.Input size is
  • Best score is 15x15 = 225 tasks, at 83 s (2.6x speedup).
  • Transcript

    • 1. Programming the cloud withSkywriting
      Derek Murray
      withMalteSchwarzkopf, Chris Smowton, Anil Madhavapeddy and Steve Hand
    • 2. Outline
      State of the art
      Skywriting by example
      Iterative algorithms
      Heterogeneous clusters
      Speculative execution
      Performance case studies
      Future directions
    • 3. Task
      Task farming
    • 4. Task farming
    • 5.
    • 6. Task farming
      runs before
    • 7. MapReduce
    • 8.
    • 9. Dryad
    • 10.
    • 11.
    • 12. Problem: iterative algorithms
      Not converged
    • 13. Problem: cluster heterogeneity
    • 14. Problem: cluster heterogeneity
    • 15. Problem: cluster heterogeneity
    • 16. Problem: speculative execution
    • 17.
    • 18. Solution: Skywriting
      Turing-complete coordination language
      Support for spawning tasks
      Interface to external code
      Distributed execution engine
      Executes tasks in parallel on a cluster
      Handles failure, locality, data motion, etc.
    • 19. Spawning a Skywriting task
      function f(arg1, arg2) { … }
      result = spawn(f, [arg1, arg2]);
    • 20. Building a task graph
      function f(x, y) { … }
      function g(x, y){ … }
      function h(x, y) { … }
      a = spawn(f, [7, 8]);
      b = spawn(g, [a, 0]);
      c = spawn(g, [a, 1]);
      d = spawn(h, [b, c]);
      return d;
    • 21. Iterative algorithm
      current = …;
      do {
      prev = current;
      a = spawn(f, [prev, 0]);
      b= spawn(f, [prev, 1]);
      c = spawn(f, [prev, 2]);
      current = spawn(g, [a, b, c]);
      done = spawn(h, [current]);
      while (!*done);
    • 22. Iterative algorithm
    • 23. Aside: recursive algorithm
      function f(x) {
      if (/* x is small enough */) {
      return /* do something with x */;
      } else {
      x_lo = /* bottom half of x */;
      x_hi = /* top half of x */;
      return [spawn(f, [x_lo]),
      spawn(f, [x_hi])];
    • 24. Executing external code
      y = exec(executor_name, { “inputs” : [x1, x2, x3], … },
      • Run Java, C, .NET and pipe-based code
      • 25. Heterogeneous cluster support
      • 26. Workers advertise “execution facilities”
      • 27. Tasks migrate to necessary facilities
    • Speculative execution
    • 28. Speculative execution
      x = …;
      a = spawn(f, [x]);
      b= spawn(f, [x]);
      c= spawn(f, [x]);
      result =waituntil(any, [a, b, c]);
      return result[“available”];
    • 29. Performance case studies
      All experiments used Amazon EC2
      m1.smallinstances, running Ubuntu 8.10
    • 30. Job creation overhead
    • 31. Smith-Waterman data flow
    • 32. Parallel Smith-Waterman
    • 33. Parallel Smith-Waterman
    • 34. Future work
      Distributed data structures
      Coping when the lists etc. get big
      Better language integration
      Compile to JVM, CLR, LLVM etc.
      Decentralised master-worker
      Run on multiple clouds
      Self-scaling clusters
      Add and remove workers as needed
    • 35. Conclusions
      Universal programming model for cloud computing
      Runs real jobs with low overhead
      Lots more still to do!
    • 36. Questions?
      • Email
      • 37.
      • 38. Project website (source code, tutorial, etc.)
      • 39.
      • 40.