In Data Engineer’s Lunch #45: Apache Livy, we discussed Apache Livy, a REST API for interacting with Spark Clusters, submitting jobs and managing Spark Contexts and cached data. Accompanying Blog: https://blog.anant.us/data-engineers-lunch-45-apache-livy Accompanying YouTube: https://youtu.be/WMRN815FuQ8 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

1. Version 1.0
Apache Livy
A REST API for interacting with Spark Clusters, submitting jobs
and managing Spark Contexts and cached data.
Obioma Anomnachi
Engineer @ Anant

2. Apache Livy
● Apache Livy is a tool that enables users to interact with a Spark cluster via a REST interface
○ It allows users to submit jobs as pre-compiler jars, or snippets of code via REST
○ It also comes with a Java/Scala client API
○ It manages Spark Contexts, including allowing them to be used for multiple jobs or by multiple clients
○ Allows the sharing of caches RDDs, Dataframes, Datasets across multiple jobs
● It allows the submission of jobs without a Spark Client
● Existing Spark code needs no changes to be compatible with Livy, and existing cluster need little
modification

3. Spark Job Submission Options
● Normally, Spark interactions take place through the Spark shell (Python or Scala or R), or through
submission of batch jobs with spark-submit
○ These aren't actually very different, the spark-submit deploy mode option can allow submitted jobs to run in
client mode, which can also give access to a REPL shell
○ Either of the above types rely on access to the cluster from a limited number of sources
■ A node permanently part of the cluster
■ Temporarily adding a node to the cluster to use spark-submit
■ Using a cluster gateway
○ All three methods essentially require access to a machine that is part of the cluster, allowing users to access
implementation and configuration details
○ Difficult to deny or allow certain types of access to users in different security groups with a limited number
nodes / gateways to access
■ Makes it hard to integrate with existing security / access control solutions

4. Demo
● https://github.com/anomnaco/LivyDemo

5. Resources
● https://livy.apache.org/
● https://github.com/apache/incubator-livy

6. Extra Discussion
● Livy’s website says “Apache Livy also simplifies the interaction between Spark and application
servers, thus enabling the use of Spark for interactive web/mobile applications.”
● As a thought experiment: What use case would you use this functionality for? Where is it better to
use Livy, so that you can use Spark directly for furnishing your applications rather than loading the
results of Spark’s work into a solution more focused on being application friendly?

7. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037