Downloaded 30 times
Uploaded byPivotalOpenSourceHub
Apache Zeppelin Meetup Christian Tzolov 1/21/16
The document outlines a unified data analytics platform integrating technologies such as Apache Zeppelin, Ambari, Geode, Spring XD, and Hawq, designed to manage and analyze large data streams. It highlights the capabilities of the platform to orchestrate processes, provide real-time analytics, and manage big data systems effectively. Key components include data lake management, in-memory data grids, and SQL compliance, facilitating high-performance applications for organizations like China Railway and Indian Railways.
More Related Content
Similar to Apache Zeppelin Meetup Christian Tzolov 1/21/16
![[DSC Europe 25] Ivan Peric - Intelligence Swarm Logic and Techno-Functional M...](https://cdn.slidesharecdn.com/ss_thumbnails/7my7c97fsduiccadgavw-2-251212103249-5a03f7c6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...](https://cdn.slidesharecdn.com/ss_thumbnails/3qtejajvsjqrzwfept2c-10-251212103250-7f2b1068-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jon Dajci - Bridging TradFi and DeFi: Building the Future of ...](https://cdn.slidesharecdn.com/ss_thumbnails/fqmhfvlbqhkihjvqvhmu-7-251211083849-6af7e325-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...](https://cdn.slidesharecdn.com/ss_thumbnails/8jjrus8ttko6qj64f58f-3-251212103250-642c6374-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Nikolay Burlutskiy - Best Practices for Building Enterprise M...](https://cdn.slidesharecdn.com/ss_thumbnails/uirvaiuvq8y1w8hzd9tx-7-251212103249-2619edb4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Aleksandra Dragicevic - AI-Boosted Research in Healthcare: Fr...](https://cdn.slidesharecdn.com/ss_thumbnails/iqwngszurf2r7pi1lnnj-4-aleksandra-dragicevic-ad-dsc-europe-conference-20-251208151905-37c3238a-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...](https://cdn.slidesharecdn.com/ss_thumbnails/tiyim1ins5jvbrvzpzla-2-251209104645-c69d3553-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragana Ilic - AI for Big Data in Astronomy.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/8palya86qaatvjhva1ms-2-dragana-ilic-ai-ilic-251208151906-652b819c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Sara Polak - The Archaeology of Innovation: AI as the Next Cr...](https://cdn.slidesharecdn.com/ss_thumbnails/7ecbscdnt8mlcuqbd2ln-2-sara-polak-ai-creative-industries-251208152533-aa1fcf54-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Branko Dzakula - From Defense to Attack: How AI Redefines Cyb...](https://cdn.slidesharecdn.com/ss_thumbnails/80bdzdxpr3ky2g0qvyk9-8-251211083048-ce5fc1ee-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dusan Nesic - Securing Tomorrow’s Infrastructure: Why Cyber-P...](https://cdn.slidesharecdn.com/ss_thumbnails/qikbszfftyowjm2q6duw-1-251211083848-8f2ead6b-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache Zeppelin Meetup Christian Tzolov 1/21/16
- 1. Unified Data Analytics Platform (withZeppelin, Ambari, Geode, SpringXD and HAWQ) by Christian Tzolov @christzolov
- 2. Whoami Christian Tzolov Technical Architectat Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member ctzolov@pivotal.io blog.tzolov.net @christzolov
- 3. Contents • DEMO • ZeppelinInterpreters • PSQL (to became JDBC in 0.6.x) • Geode • SpringXD • Apache Ambari • Zeppelin Service • Geode, HAWQ and Spring XD services • Webpage Embedder View
- 4. Demo: Twitter Streamswith SpringXD, Geode and HAWQ
- 5. Technical Stack Apache HDFSData Lake - PHD or HDP Hadoop Apache HAWQ SQL on Hadoop (OLAP) Apache Geode In-memory data grid (OLTP) Spring XD Integration and Streaming Runtime Apache Ambari Manages All Clusters Apache Zeppelin Web UI for interaction with Data Systems Hadoop/HDFS Geode HAWQ SpringXD Ambari Zeppelin
- 6. Spring XD Orchestrates andautomates all steps across multiple data stream pipelines • HTTP • Tail • File • Mail • Twitter • Gemfire • Syslog • TCP • UDP • JMS • RabbitMQ • MQTT • Kafka • Reactor TCP/UDP • Filter • Transformer • Object-to-JSON • JSON-to-Tuple • Splitter • Aggregator • HTTP Client • Groovy Scripts • Java Code • JPMML Evaluator • Spark Streaming • File • HDFS • JDBC • TCP • Log • Mail • RabbitMQ • Gemfire • Splunk • MQTT • Kafka • Dynamic Router • Counters
- 7. Apache Geode • Cache- Performance / Consistency / Resiliency • Region - Highly available, redundant, distributed Map China Railway Corporation 5,700 train stations 4.5 million tickets per day 20 million daily users 1.4 billion page views per day 40,000 visits per second Indian Railways 7,000 stations 72,000 miles of track 23 million passengers daily 120,000 concurrent users 10,000 transactions per minute
- 8. Apache HAWQ • Builtaround a Greenplum MPP DB • 100% ANSI SQL compliant: SQL-92/99/2003… • ODBC and JDBC • Hadoop Native: Parquet, HDFS and YARN • Extensible - Web Tables, PXF • TPC-DS outperforms Impala by overall 454%
- 9. Demo tweets = twittersearch--query=<keywork> | hdfs --directory=/user/zeppelin/xd/tweets geodeTap = tap:stream:tweets > gemfire-json-server --regionName=regionTweet hawqTap = tap:stream:tweets > transform --script=tweetJsonToTsv.groovy | gpfdist --table=xdsink tweetsCount = tap:stream:tweets > json-to-tuple | transform --expression='payload.id_str' | counter
- 10. SpringXD Interpreter(s) • %xd.streamand %xd.job • Multiple streams or jobs in a paragraph. • Special Deploy/Launch Semantics • Zeppelin Dynamic Forms (${…}) • Comprihensive Stream and Job DSL auto- completion (Ctrl+.)
- 11.
- 12. PSQL Interpreter • Prefix:%psql.sql • PostgreSQL, HAWQ/PXF, Greenplum … JDBC • PSQL command line shell (via %sh) • Zeppelin Dynamic Forms (${…}) • Comprihensive SQL/JDBC autocompletion (Ctrl+.)
- 13.
- 14.
- 15.
- 16. Geode Interpreter • Prefix:%geode.oql • OQL and PDX nested access (user.name) • Geode command line shell (via %sh) • Zeppelin Dynamic Forms (${…}) • Basic OQL auto-completion (Ctrl+.)
- 17.
- 18.
- 19.
- 20. Apache Ambari Zeppelin, Geode,HAWQ, SpringXD Services …
- 21.
- 22. Ambari Services • AmbariZeppelin Service: github , rpm, blog • Ambari Geode Service: github, rpm • Ambari SpringXD Service: github • Ambari HAWQ Service (Pivotal BDS dist)
- 23.
- 24.
- 25.
Editor's Notes
- #2 I’m glad to be here. be able to share few ideas about how to combine the strenght of powerfull tools like zep, xd, geode, hawq.
- #3 Mention the PSQL, Geode and SpringXD intepreters work
- #5 http://10.68.58.121:9995/#/notebook/2BC41KDMZ
- #6 http://10.68.58.121:9995/#/notebook/2BC41KDMZ
- #7 http://10.68.58.121:8080/#/main/services/HDFS/summary
- #8 It simplifies big data projects by orchestrating and automating all steps across multiple data stream pipelines—creating, deploying, and managing many pipelines in a unified, extensible, distributed way. A stream is composed from modules. Each module is deployed to a container and its channels are bound to the transport.
- #11 http://10.68.58.121:9995/#/notebook/2BC41KDMZ
- #13 add spring xd ambari container layout
- #14 %psql.sql drop table if exists mytable; create table mytable (i int); insert into mytable select generate_series(1, 100); %psql.sql select * from mytable; %psql.sql select count(*) from mytable; select * from mytable;
- #15 add slide showing the HAWQ deployment model (AMBARI)
- #17 %psql.sql drop table if exists mytable; create table mytable (i int); insert into mytable select generate_series(1, 100); %psql.sql select * from mytable; %psql.sql select count(*) from mytable; select * from mytable;
- #23 http://10.68.58.121:8080/#/main/dashboard/metrics
- #24 Zeppelin http://blog.tzolov.net/2015/08/zeppelin-service-for-ambari.html?view=sidebar https://github.com/tzolov/zeppelin-ambari-plugin https://bintray.com/big-data/rpm/zeppelin-ambari-plugin/view Geode (Credits to Steve Shangguan!) https://github.com/tzolov/ambari-gemfire/tree/geode https://bintray.com/big-data/rpm/geode-ambari-plugin/view
- #26 https://github.com/tzolov/ambari-webpage-embedder-view