No More “Sbt Assembly”: Rethinking Spark-Submit Using CueSheet: Spark Summit East talk by Jong Wook Kim
A crucial, but not-so-much-interesting part of developing Spark applications is packaging them for the deployment. Although there have been many successful notebook-style interactive environments such as Apache Zeppelin and Databricks, for any serious projects with a number of dependencies it is often the case where we build a Spark application as a separate Scala project in a version-controlled source tree. For this, the official documentation tells us to use SBT or Maven’s assembly plugin to make a fat jar and run the spark-submit script, but this process significantly slows down the development cycle of coding, ‘assembly’-ing, configuring, debugging, and deploying. This presentation will introduce CueSheet, an open-source framework built around Apache Spark that accelerates the development of Spark applications. Originally developed at Kakao Corp., CueSheet removes the need of even opening the terminal by taking care of packaging, submitting and deploying Spark applications. It thus enables developers to focus only on the important parts – programming and debugging. Without any special configuration, CueSheet makes it possible to not only launch Spark applications directly from the IDE but also debug by stepping through breakpoints. These advantages, which are made possible by a number of JVM tricks that we will briefly go over, have tremendously boosted the productivity of developing Spark applications in a number of teams at Kakao. The presentation will also discuss how it helps to nicely separate the configuration a Spark application from its code in a reusable manner.
