Downloaded 40 times
Uploaded byDuyhai Doan
Apache zeppelin the missing component for the big data ecosystem
Duy Hai Doan presented Apache Zeppelin, an open-source web-based notebook that allows users to interact with data. Zeppelin provides a front-end GUI and display system for data analysis tools and uses interpreters to connect to back-end systems like Spark, Cassandra, and Flink. Doan demonstrated Zeppelin's notebook interface, display options, and how users can write their own interpreters to connect new systems to Zeppelin. Future plans for Zeppelin include improving usability, adding authentication and authorization, and developing more interpreters and visualizations.
More Related Content
Similar to Apache zeppelin the missing component for the big data ecosystem
Apache zeppelin the missing component for the big data ecosystem
- 1. @doanduyhai#VoxxedVienna Apache Zeppelin the missingGUI for your BigData eco-system DuyHai DOAN Apache Cassandra Evangelist
- 2. @doanduyhai Who Am I? Duy Hai DOAN Cassandra technical advocate • talks, meetups, confs • open-source devs (Achilles, …) • OSS Cassandra point of contact ☞ duy_hai.doan@datastax.com ☞ @doanduyhai 2
- 3. @doanduyhai Datastax • Founded inApril 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees • Headquarter in San Francisco Bay area • EU headquarter in London, offices in France and Germany • Datastax Enterprise = OSS Cassandra + extra features 3
- 4. What is ApacheZeppelin ? Presentation Architecture
- 5.
- 6.
- 7. @doanduyhai Zeppelin Architecture Zeppelin Server ZeppelinEngine 7 R E S TWebSocket Spark Interpreter Group Spark SparkSQL Zeppelin Interpreter Factory Tajo Interpreter Flink Interpreter Cassandra Interpreter JVM JVM JVM JVM JVM
- 8. @doanduyhai What does Zeppelinprovide ? Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON) 8
- 9.
- 10.
- 11. Zeppelin Display System Raw,Table, HTML, Angular with Scala Available graphs View modes Dynamic form Iframe export
- 12.
- 13. Interpreter to Front-EndStreaming
- 14. @doanduyhai Interpreter to front-endstreaming Zeppelin Server 14 WebSocket Interpreter JVM JVM
- 15.
- 16. Interpreter system Core interpreters Third-partiesinterpreters Interpreters conf & usage
- 17. @doanduyhai Interpreter processing lifecycle ① Receive input commands/data • as raw text • from form data ② Process the input commands/data by the external back-end ③ Format the response using Zeppelin display system ④ Send response back to the Zeppelin engine 17
- 18. @doanduyhai Core interpreters • Spark(Spark core, SparkSQL/DataFrame, PySpark) • Spark core = default (or %spark) • SparkSQL = %sql • Shell (%sh) • Markdown (%md) • AngularJS (%angular) 18
- 19. @doanduyhai Third-parties interpreters • Hive • Phoenix • Tajo • Flink • Ignite • Lens • Cassandra • Geode • PostgreSQL • Kylin • ElasticSearch 19
- 20. @doanduyhai Interpreter conf &usage https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
- 21. Writing An Interpreter HowTo Simple interpreter example (AsciiDoc) Complex interpreter example (Cassandra)
- 22. @doanduyhai Steps to writeyour own interpreter • Create a class that extends Interpreter base class • Register it in a static block • Optionnally define default config params 22 static { Interpreter.register("MyInterpreterName", MyClassName.class.getName()); } static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build()); }
- 23. @doanduyhai To register yourinterpreter as default • Edit the enum ZeppelinConfiguration.ConfVars • Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS 23
- 24. @doanduyhai To register yourinterpreter in config files • Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template • Add your interpreter FQCN in the property zeppelin.interpreters 24 <property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter, org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter, org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter </value> </property>
- 25.
- 26.
- 27. @doanduyhai Simple AsciiDoc Interpreter 27 ZeppelinServer AsciiDoc Interpreter JVMZeppelin Engine Raw Text Block Raw Text Block Converted To HTML HTML Output ① ② ③④ JVM
- 28.
- 29. @doanduyhai Cassandra Interpreter Architecture 29 Cassandra Interpreter JVM Display Resultsas HTML ① ② ⑤ Zeppelin Server JVM Raw Text Block Raw Text Block Cassandra Cassandra Java Driver ③ Async CQL statements ④Render HTML ⑥
- 30. @doanduyhai Cassandra Interpreter Commands 30 NativeCQL statements SELECT * FROM …; INSERT INTO …; … Schema commands DESCRIBE TABLE …; DESCRIBE KEYSPACE …; … Prepared statements Commands @prepare …; @bind …; @remove_prepared …; Help command HELP; Options Commands @consistency …; @retryPolicy …; @fetchSize …;
- 31.
- 32. @doanduyhai Cassandra Online InterpreterDocs 32 • http://zeppelin.incubator.apache.org/docs/interpreter/cassandra.html
- 33.
- 34. @doanduyhai Enterprise Ready • ApacheShiro authentication (ZEPPELIN-548) • Note authorization (PR #681) • Multi-tenancy 34
- 35. @doanduyhai Usability • UX improvement • Better table data support • Export data as CSV etc . (PR #725, PR #714, PR #6, PR #89) • Table pagination … 35
- 36. @doanduyhai Pluggability • Pluggable visualization • Pluggable interpreter • Repository and registry for pluggable components 36
- 37.
- 38.
- 39.