Apache Zeppelin Helium and Beyond
•
8 likes•3,857 views
This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It describes Zeppelin's origins, key features like interpreters and notebook storage being pluggable, a proposed "Helium" project to extend Zeppelin into a full analytics application platform, the roadmap for further developing Zeppelin, and how to get involved with the Zeppelin community.
Report
Share
Report
Share
Download to read offline
Recommended
Apache Zeppelin, Helium and Beyond
Apache Zeppelin is an open-source web-based notebook that enables interactive data analytics. It began in 2012 as an internal tool called Zeppelin at NIF Labs and was later open-sourced. In 2014 it joined Apache Incubator and became an Apache project in 2016. Helium is a proposed next version of Zeppelin that aims to make visualizations and applications pluggable modules. This would allow users to more easily extend Zeppelin's capabilities.
Running Apache Zeppelin production
Discussion around concerns related to deploying Apache Zeppelin in production, including deployment choices, security, performance and integration.
Multi User Data science with Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It summarizes Zeppelin's features such as support for multiple programming languages and deep integration with Apache Spark. The document also outlines Zeppelin's roadmap, including goals to improve support for authentication, authorization, and collaboration features.
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
Helium makes Zeppelin fly!
Apache Zeppelin is interactive data analytics environment for large scale data processing systems. It deeply integrates to Apache spark and many other frameworks, provides beautiful interactive web-based interface, data visualization, collaborative work environment and many other nice features to make your data science lifecycle more fun and enjoyable. Helium is a framework that manages pluggable components like Visualization, Spell inside of Zeppelin. Pluggable component extends Zeppelin's capability and particularly useful when Zeppelin is being used as a collaborative data science environment. Moon will demonstrate create custom visualization, publish to Helium online registry and use them in the notebook. Also talk about how Helium framework and Helium online registry works behind the scene and future roadmap as well. You'll see not only how easy creating and publishing Helium package is but also what possibility these pluggable modules gives to Zeppelin as a data science tool and business intelligence tool.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.4.Building a Data Product using apache Zeppelin - Apache Kylin Meetup @Shanghai
The document discusses building a data product using Apache Zeppelin (incubating) that sends email notifications about new open source projects on GitHub. It outlines the steps to download data from the GitHub archive, explore and filter the data to focus on interesting companies, join additional data from the GitHub API, generate an HTML template to visualize the results, and schedule sending the email notifications.
Interactive Analytics using Apache Spark
Agenda:
• Brief overview of Spark provided spark-shell, spark-submit
• Overview of Spark ContextOverview of Zeppelin and Jupyter notebooks for Spark
• Introduction to IBM Spark Kernel
• Introduction to Cloudera Livy and Spark JobServer
Github Link:
Previous meetups:-
1) Introduction to Resilient Distributed Dataset and deep dive
Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-and-resilient-distributed-dataset-basics-and-deep-dive
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/225159947/
Video: https://www.youtube.com/watch?v=MkeRWyF1y_0
Github: https://github.com/SatyaNarayan1/spark_meetup
2) Introduction to Spark DataFrames/SQL and Deep dive
Slides: http://www.slideshare.net/sachinparmarss/deep-dive-spark-data-frames-sql-and-catalyst-optimizer
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/226419828/
Video: https://www.youtube.com/watch?v=h71MNWRv99M
Github: https://github.com/parmarsachin/spark-dataframe-demo
3) Apache Spark - Introduction to Spark Streaming and Deep dive
Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-to-spark-streaming-and-deep-dive-57671774
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/227008581/
Video:
Github: https://github.com/agsachin/spark-meetup
Looking forward to have a great interactive session. Do provide feedback.
Recommended
Apache Zeppelin, Helium and Beyond
Apache Zeppelin is an open-source web-based notebook that enables interactive data analytics. It began in 2012 as an internal tool called Zeppelin at NIF Labs and was later open-sourced. In 2014 it joined Apache Incubator and became an Apache project in 2016. Helium is a proposed next version of Zeppelin that aims to make visualizations and applications pluggable modules. This would allow users to more easily extend Zeppelin's capabilities.
Running Apache Zeppelin production
Discussion around concerns related to deploying Apache Zeppelin in production, including deployment choices, security, performance and integration.
Multi User Data science with Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It summarizes Zeppelin's features such as support for multiple programming languages and deep integration with Apache Spark. The document also outlines Zeppelin's roadmap, including goals to improve support for authentication, authorization, and collaboration features.
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
Helium makes Zeppelin fly!
Apache Zeppelin is interactive data analytics environment for large scale data processing systems. It deeply integrates to Apache spark and many other frameworks, provides beautiful interactive web-based interface, data visualization, collaborative work environment and many other nice features to make your data science lifecycle more fun and enjoyable. Helium is a framework that manages pluggable components like Visualization, Spell inside of Zeppelin. Pluggable component extends Zeppelin's capability and particularly useful when Zeppelin is being used as a collaborative data science environment. Moon will demonstrate create custom visualization, publish to Helium online registry and use them in the notebook. Also talk about how Helium framework and Helium online registry works behind the scene and future roadmap as well. You'll see not only how easy creating and publishing Helium package is but also what possibility these pluggable modules gives to Zeppelin as a data science tool and business intelligence tool.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.4.Building a Data Product using apache Zeppelin - Apache Kylin Meetup @Shanghai
The document discusses building a data product using Apache Zeppelin (incubating) that sends email notifications about new open source projects on GitHub. It outlines the steps to download data from the GitHub archive, explore and filter the data to focus on interesting companies, join additional data from the GitHub API, generate an HTML template to visualize the results, and schedule sending the email notifications.
Interactive Analytics using Apache Spark
Agenda:
• Brief overview of Spark provided spark-shell, spark-submit
• Overview of Spark ContextOverview of Zeppelin and Jupyter notebooks for Spark
• Introduction to IBM Spark Kernel
• Introduction to Cloudera Livy and Spark JobServer
Github Link:
Previous meetups:-
1) Introduction to Resilient Distributed Dataset and deep dive
Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-and-resilient-distributed-dataset-basics-and-deep-dive
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/225159947/
Video: https://www.youtube.com/watch?v=MkeRWyF1y_0
Github: https://github.com/SatyaNarayan1/spark_meetup
2) Introduction to Spark DataFrames/SQL and Deep dive
Slides: http://www.slideshare.net/sachinparmarss/deep-dive-spark-data-frames-sql-and-catalyst-optimizer
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/226419828/
Video: https://www.youtube.com/watch?v=h71MNWRv99M
Github: https://github.com/parmarsachin/spark-dataframe-demo
3) Apache Spark - Introduction to Spark Streaming and Deep dive
Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-to-spark-streaming-and-deep-dive-57671774
Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/227008581/
Video:
Github: https://github.com/agsachin/spark-meetup
Looking forward to have a great interactive session. Do provide feedback.
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
In this talk I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets and facebook posts.
In the same flow I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, Email and Slack.
I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy and Python TextBlob.
This talk will be a deep dive into how to manage complex dataflow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning and business rules.
Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in stream and
storing data into Apache Phoenix on HBase and Apache Hive as ORC tables.
In this talk, I will walk through each step in the process from ingest of each source, applying filters, performing transformations, converting types, picking and converting fields and finally storing data to Apache Phoenix on HBase.
A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x.
This is based on a few talks I have given at the Future of Data - Princeton meetup on various ingestion and processing patterns with Apache NiFi.
Speaker:
TImothy Spann, Solutions Engineer, Hortonworks
Zeppelin at Twitter
Zeppelin at Twitter - Prasad Wagle, Technical Lead in the Data Platform team - Twitter
Prasad will talk about how Zeppelin is used at Twitter, the development work they did before release and the features and enhancements they are working on to increase adoption.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
Big Data is all about being to access and process data in various formats, and from various sources. Apache Bahir provides extensions to distributed analytic platforms providing them access to different data sources. In this talk we will introduce you to Apache Bahir and its various connectors that are available for Apache Spark and Apache Flink. We will also go over the details of how to build, test and deploy an Spark Application using the MQTT data source for the new Apache Spark 2.0 Structure Streaming functionality.
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
When interacting with analytics dashboards in order to achieve a smooth user experience, two major key requirements are sub-second response time and data freshness. Cluster computing frameworks such as Hadoop or Hive/Hbase work well for storing large volumes of data, although they are not optimized for ingesting streaming data and making it available for queries in realtime. Also, long query latencies make these systems sub-optimal choices for powering interactive dashboards and BI use-cases.
In this talk we will present Druid as a complementary solution to existing hadoop based technologies. Druid is an open-source analytics data store, designed from scratch, for OLAP and business intelligence queries over massive data streams. It provides low latency realtime data ingestion and fast sub-second adhoc flexible data exploration queries.
Many large companies are switching to Druid for analytics, and we will cover how druid is able to handle massive data streams and why it is a good fit for BI use cases.
Agenda -
1) Introduction and Ideal Use cases for Druid
2) Data Architecture
3) Streaming Ingestion with Kafka
4) Demo using Druid, Kafka and Superset.
5) Recent Improvements in Druid moving from lambda architecture to Exactly once Ingestion
6) Future Work
Securing Hadoop in an Enterprise Context
Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
OGCE TeraGrid 2010 Science Gateway Tutorial Intro
The document summarizes a tutorial presentation about the Open Grid Computing Environments (OGCE) software tools for building science gateways. The OGCE tools include a gadget container, workflow composer called XBaya, and application factory service called GFac. The presentation demonstrates how these tools can be used to build portals and compose workflows to access resources like the TeraGrid.
LEGO: Data Driven Growth Hacking Powered by Big Data
This document describes LEGO's data-driven growth hacking powered by big data. It discusses how LEGO uses a data lake and log processing engine to process hundreds of billions of log lines from various sources at scale. The data is then profiled, cleansed, transformed and loaded into data warehouses to enable self-service analysis and predictive data apps for executives, analysts and product managers.
Apache Accumulo 1.8.0 Overview
This talk will be an overview of the new features and improvements currently implemented for the Apache Accumulo 1.8.0 release. This will be a discussion about some of these exciting changes with a focus on what is of the most importance for users.
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
This document discusses enabling Apache Zeppelin and Spark for data science in the enterprise. It outlines current issues with Zeppelin and Spark integration including secure data access, multi-tenancy, and fault tolerance. It then describes how using Livy Server as a session management service solves these issues by providing secure, isolated sessions for each user. The document concludes by covering near term improvements like session management and long term goals like controlled sharing and model deployment.
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
The amount of information coming from a Cloud deployment, that could be used to have a better situational awareness, and operate it efficiently is huge. Tools as the ones provided by Apache foundation can be used to build a solution to that challenge.
Nowadays Cloud deployments are pervasive in businesses, with scalability and multi tenancy as their core capabilities. This means that these deployments can grow easily beyond 1000 nodes and efficient operation of these huge clusters requires real time log analysis, metrics, events and configuration data. Performing correlation and finding patterns, not just to get to root causes but also to predict failures and reduce risk requires tools that go beyond current solutions.
In the prototype developed by Red Hat and KEEDIO (keedio.com), we managed to address the above challenges with the use of Big Data tools like Apache NiFi, Apache Kafka and Apache Flink, that enabled us to process the constant stream of syslog messages (RFC5424) produced by the Infrastructure as a Service, provided by OpenStack services, and also detect common failure patterns that could arise and generate alerts as needed.
This session is an (Intermediate) talk in our Apache Nifi and Data Science track. It focuses on Apache Flink, Apache Nifi, Apache Kafka and is geared towards Architect, Data Scientist, Data Analyst, Developer / Engineer audiences.
Speaker
Miguel Perez Colino, Senior Design Product Manager, Red Hat
Suneel Marthi, Senior Principal Engineer, Red Hat
Spark Security
This document discusses security requirements and solutions for Apache Spark production deployments. It covers authenticating users with Kerberos/AD, authorizing access to Spark jobs and data with Ranger, auditing access, and encrypting data at rest and in motion. It provides examples of configuring Kerberos authentication for Spark, using Ranger to control authorization to HDFS and SparkSQL, and demonstrates dynamic row filtering and masking of sensitive data in SparkSQL queries based on user policies.
IoT Applications and Patterns using Apache Spark & Apache Bahir
The Internet of Things (IoT) is all about connected devices that produce and exchange data, and building applications that produce insights from these high volumes of data is very challenging and require a understanding of multiple protocols, platforms and other components. On this session, we will start by providing a quick introduction to IoT, some of the common analytic patterns used on IoT, and also touch on the MQTT protocol and how it is used by IoT solutions some of the quality of services tradeoffs to be considered when building an IoT application. We will also discuss some of the Apache Spark platform components, the ones utilized by IoT applications to process devices streaming data.
We will also talk about Apache Bahir and some of its IoT connectors available for the Apache Spark platform. We will also go over the details on how to build, test and deploy an IoT application for Apache Spark using the MQTT data source for the new Apache Spark Structure Streaming functionality.
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
As Hadoop becomes the defacto big data platform, enterprises deploy HDP across wide range of physical and virtual environments spanning private and public clouds. This session will cover key considerations for cloud deployment and showcase Cloudbreak for simple and consistent deployment across cloud providers of choice.
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
http://flink-forward.org/kb_sessions/multi-tenant-flink-as-a-service-on-yarn/
Since June 2016, Flink-as-a-service has been available to researchers and companies in Sweden from the Swedish ICT SICS Data Center at www.hops.site using the HopsWorks platform. Flink applications can be either deployed as jobs (batch or streaming) or written and run directly from Apache Zeppelin on YARN. Flink applications are run within a project on a YARN cluster with the novel property that Flink applications are metered and charged to projects. Projects are also securely isolated from each other and include support for project-specific Kafka topics that are protected from access by users that are not members of the project. Hopsworks is entirely UI-driven, is open-source, and Flink applications that include Kafka topics can be created in a few mouse clicks. In this talk we will discuss the challenges in building a metered version of Flink-as-a-Service for YARN, experiences with Flink-on-YARN, and some of the possibilities that Hopsworks opens up for building secure, multi-ten
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...
During the last year, the team at IBM Research at Ireland has been using Apache Spark to perform analytics on large volumes of sensor data. These applications need to be executed on a daily basis, therefore, it was essential for them to understand Spark resource utilization. They found it cumbersome to manually consume and efficiently inspect the CSV files for the metrics generated at the Spark worker nodes.
Although using an external monitoring system like Ganglia would automate this process, they were still plagued with the inability to derive temporal associations between system-level metrics (e.g. CPU utilization) and job-level metrics (e.g. job or stage ID) as reported by Spark. For instance, they were not able to trace back the root cause of a peak in HDFS Reads or CPU usage to the code in their Spark application causing the bottleneck.
To overcome these limitations, they developed SparkOScope. Taking advantage of the job-level information available through the existing Spark Web UI and to minimize source-code pollution, they use the existing Spark Web UI to monitor and visualize job-level metrics of a Spark application (e.g. completion time). More importantly, they extend the Web UI with a palette of system-level metrics of the server/VM/container that each of the Spark job’s executor ran on. Using SparkOScope, you can navigate to any completed application and identify application-logic bottlenecks by inspecting the various plots providing in-depth timeseries for all relevant system-level metrics related to the Spark executors, while also easily associating them with stages, jobs and even source code lines incurring the bottleneck.
They have made Sparkoscope available as a standalone module, and also extended the available Sinks (mongodb, mysql).
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
The document discusses Hopsworks, an open source platform for self-service Apache Spark, Flink, Kafka, and Hadoop clusters that addresses issues with usability, security, and operations that have hindered Hadoop's adoption. Hopsworks provides a web-based interface and uses a distributed database to store metadata externally for improved scalability. It introduces new abstractions like projects and datasets to simplify cluster management and data sharing.
Introduction to Hortonworks Data Cloud for AWS
Hortonworks Data Cloud is a new cloud product from Hortonworks that offers pay-as-you-go pricing for launching and managing Hadoop clusters on AWS. It handles common big data use cases and focuses on ease of use by providing prescriptive cluster types. The product aims to improve enterprise readiness in the cloud by providing scalable storage, security and governance features, and reliability through auto-recovery of unhealthy nodes. It also matches Hadoop with cloud capabilities like scalable storage, customizability, and cost-effective compute.
Extending the Yahoo Streaming Benchmark + MapR Benchmarks
The document summarizes benchmark tests that were performed to compare the throughput of Apache Storm and Apache Flink for processing streaming data. The original Yahoo! benchmark showed Storm outperforming Flink. However, the author repeated the tests and was able to achieve much higher throughput with Flink by addressing bottlenecks. When deployed on a high-performance MapR cluster, Flink processed over 72 million messages per second, significantly outperforming the original Storm results. The document concludes by noting Flink's compatibility features that allow reuse of existing Storm applications and components.
Apache Hive 2.0: SQL, Speed, Scale
Apache Hive 2.0 provides major new features for SQL on Hadoop such as:
- HPLSQL which adds procedural SQL capabilities like loops and branches.
- LLAP which enables sub-second queries through persistent daemons and in-memory caching.
- Using HBase as the metastore which speeds up query planning times for queries involving thousands of partitions.
- Improvements to Hive on Spark and the cost-based optimizer.
- Many bug fixes and under-the-hood improvements were also made while maintaining backwards compatibility where possible.
Data science lifecycle with Apache Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It provides an overview of Zeppelin's history and architecture, including how interpreters and notebook storage are pluggable. The document also outlines Zeppelin's roadmap for improving enterprise support through features like multi-tenancy, impersonation, job management and frontend performance.
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
This document discusses Apache Zeppelin, an open-source notebook for interactive data analytics. It provides an overview of Zeppelin's features, including interactive notebooks, multiple backends, interpreters, and a display system. The document also covers Zeppelin's adoption timeline, from its origins as a commercial product in 2012 to becoming an Apache Incubator project in 2014. Future projects involving Zeppelin like Helium and Z-Manager are also briefly described.
More Related Content
What's hot
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
In this talk I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets and facebook posts.
In the same flow I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, Email and Slack.
I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy and Python TextBlob.
This talk will be a deep dive into how to manage complex dataflow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning and business rules.
Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in stream and
storing data into Apache Phoenix on HBase and Apache Hive as ORC tables.
In this talk, I will walk through each step in the process from ingest of each source, applying filters, performing transformations, converting types, picking and converting fields and finally storing data to Apache Phoenix on HBase.
A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x.
This is based on a few talks I have given at the Future of Data - Princeton meetup on various ingestion and processing patterns with Apache NiFi.
Speaker:
TImothy Spann, Solutions Engineer, Hortonworks
Zeppelin at Twitter
Zeppelin at Twitter - Prasad Wagle, Technical Lead in the Data Platform team - Twitter
Prasad will talk about how Zeppelin is used at Twitter, the development work they did before release and the features and enhancements they are working on to increase adoption.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
Big Data is all about being to access and process data in various formats, and from various sources. Apache Bahir provides extensions to distributed analytic platforms providing them access to different data sources. In this talk we will introduce you to Apache Bahir and its various connectors that are available for Apache Spark and Apache Flink. We will also go over the details of how to build, test and deploy an Spark Application using the MQTT data source for the new Apache Spark 2.0 Structure Streaming functionality.
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
When interacting with analytics dashboards in order to achieve a smooth user experience, two major key requirements are sub-second response time and data freshness. Cluster computing frameworks such as Hadoop or Hive/Hbase work well for storing large volumes of data, although they are not optimized for ingesting streaming data and making it available for queries in realtime. Also, long query latencies make these systems sub-optimal choices for powering interactive dashboards and BI use-cases.
In this talk we will present Druid as a complementary solution to existing hadoop based technologies. Druid is an open-source analytics data store, designed from scratch, for OLAP and business intelligence queries over massive data streams. It provides low latency realtime data ingestion and fast sub-second adhoc flexible data exploration queries.
Many large companies are switching to Druid for analytics, and we will cover how druid is able to handle massive data streams and why it is a good fit for BI use cases.
Agenda -
1) Introduction and Ideal Use cases for Druid
2) Data Architecture
3) Streaming Ingestion with Kafka
4) Demo using Druid, Kafka and Superset.
5) Recent Improvements in Druid moving from lambda architecture to Exactly once Ingestion
6) Future Work
Securing Hadoop in an Enterprise Context
Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
OGCE TeraGrid 2010 Science Gateway Tutorial Intro
The document summarizes a tutorial presentation about the Open Grid Computing Environments (OGCE) software tools for building science gateways. The OGCE tools include a gadget container, workflow composer called XBaya, and application factory service called GFac. The presentation demonstrates how these tools can be used to build portals and compose workflows to access resources like the TeraGrid.
LEGO: Data Driven Growth Hacking Powered by Big Data
This document describes LEGO's data-driven growth hacking powered by big data. It discusses how LEGO uses a data lake and log processing engine to process hundreds of billions of log lines from various sources at scale. The data is then profiled, cleansed, transformed and loaded into data warehouses to enable self-service analysis and predictive data apps for executives, analysts and product managers.
Apache Accumulo 1.8.0 Overview
This talk will be an overview of the new features and improvements currently implemented for the Apache Accumulo 1.8.0 release. This will be a discussion about some of these exciting changes with a focus on what is of the most importance for users.
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
This document discusses enabling Apache Zeppelin and Spark for data science in the enterprise. It outlines current issues with Zeppelin and Spark integration including secure data access, multi-tenancy, and fault tolerance. It then describes how using Livy Server as a session management service solves these issues by providing secure, isolated sessions for each user. The document concludes by covering near term improvements like session management and long term goals like controlled sharing and model deployment.
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
The amount of information coming from a Cloud deployment, that could be used to have a better situational awareness, and operate it efficiently is huge. Tools as the ones provided by Apache foundation can be used to build a solution to that challenge.
Nowadays Cloud deployments are pervasive in businesses, with scalability and multi tenancy as their core capabilities. This means that these deployments can grow easily beyond 1000 nodes and efficient operation of these huge clusters requires real time log analysis, metrics, events and configuration data. Performing correlation and finding patterns, not just to get to root causes but also to predict failures and reduce risk requires tools that go beyond current solutions.
In the prototype developed by Red Hat and KEEDIO (keedio.com), we managed to address the above challenges with the use of Big Data tools like Apache NiFi, Apache Kafka and Apache Flink, that enabled us to process the constant stream of syslog messages (RFC5424) produced by the Infrastructure as a Service, provided by OpenStack services, and also detect common failure patterns that could arise and generate alerts as needed.
This session is an (Intermediate) talk in our Apache Nifi and Data Science track. It focuses on Apache Flink, Apache Nifi, Apache Kafka and is geared towards Architect, Data Scientist, Data Analyst, Developer / Engineer audiences.
Speaker
Miguel Perez Colino, Senior Design Product Manager, Red Hat
Suneel Marthi, Senior Principal Engineer, Red Hat
Spark Security
This document discusses security requirements and solutions for Apache Spark production deployments. It covers authenticating users with Kerberos/AD, authorizing access to Spark jobs and data with Ranger, auditing access, and encrypting data at rest and in motion. It provides examples of configuring Kerberos authentication for Spark, using Ranger to control authorization to HDFS and SparkSQL, and demonstrates dynamic row filtering and masking of sensitive data in SparkSQL queries based on user policies.
IoT Applications and Patterns using Apache Spark & Apache Bahir
The Internet of Things (IoT) is all about connected devices that produce and exchange data, and building applications that produce insights from these high volumes of data is very challenging and require a understanding of multiple protocols, platforms and other components. On this session, we will start by providing a quick introduction to IoT, some of the common analytic patterns used on IoT, and also touch on the MQTT protocol and how it is used by IoT solutions some of the quality of services tradeoffs to be considered when building an IoT application. We will also discuss some of the Apache Spark platform components, the ones utilized by IoT applications to process devices streaming data.
We will also talk about Apache Bahir and some of its IoT connectors available for the Apache Spark platform. We will also go over the details on how to build, test and deploy an IoT application for Apache Spark using the MQTT data source for the new Apache Spark Structure Streaming functionality.
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
As Hadoop becomes the defacto big data platform, enterprises deploy HDP across wide range of physical and virtual environments spanning private and public clouds. This session will cover key considerations for cloud deployment and showcase Cloudbreak for simple and consistent deployment across cloud providers of choice.
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
http://flink-forward.org/kb_sessions/multi-tenant-flink-as-a-service-on-yarn/
Since June 2016, Flink-as-a-service has been available to researchers and companies in Sweden from the Swedish ICT SICS Data Center at www.hops.site using the HopsWorks platform. Flink applications can be either deployed as jobs (batch or streaming) or written and run directly from Apache Zeppelin on YARN. Flink applications are run within a project on a YARN cluster with the novel property that Flink applications are metered and charged to projects. Projects are also securely isolated from each other and include support for project-specific Kafka topics that are protected from access by users that are not members of the project. Hopsworks is entirely UI-driven, is open-source, and Flink applications that include Kafka topics can be created in a few mouse clicks. In this talk we will discuss the challenges in building a metered version of Flink-as-a-Service for YARN, experiences with Flink-on-YARN, and some of the possibilities that Hopsworks opens up for building secure, multi-ten
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...
During the last year, the team at IBM Research at Ireland has been using Apache Spark to perform analytics on large volumes of sensor data. These applications need to be executed on a daily basis, therefore, it was essential for them to understand Spark resource utilization. They found it cumbersome to manually consume and efficiently inspect the CSV files for the metrics generated at the Spark worker nodes.
Although using an external monitoring system like Ganglia would automate this process, they were still plagued with the inability to derive temporal associations between system-level metrics (e.g. CPU utilization) and job-level metrics (e.g. job or stage ID) as reported by Spark. For instance, they were not able to trace back the root cause of a peak in HDFS Reads or CPU usage to the code in their Spark application causing the bottleneck.
To overcome these limitations, they developed SparkOScope. Taking advantage of the job-level information available through the existing Spark Web UI and to minimize source-code pollution, they use the existing Spark Web UI to monitor and visualize job-level metrics of a Spark application (e.g. completion time). More importantly, they extend the Web UI with a palette of system-level metrics of the server/VM/container that each of the Spark job’s executor ran on. Using SparkOScope, you can navigate to any completed application and identify application-logic bottlenecks by inspecting the various plots providing in-depth timeseries for all relevant system-level metrics related to the Spark executors, while also easily associating them with stages, jobs and even source code lines incurring the bottleneck.
They have made Sparkoscope available as a standalone module, and also extended the available Sinks (mongodb, mysql).
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
The document discusses Hopsworks, an open source platform for self-service Apache Spark, Flink, Kafka, and Hadoop clusters that addresses issues with usability, security, and operations that have hindered Hadoop's adoption. Hopsworks provides a web-based interface and uses a distributed database to store metadata externally for improved scalability. It introduces new abstractions like projects and datasets to simplify cluster management and data sharing.
Introduction to Hortonworks Data Cloud for AWS
Hortonworks Data Cloud is a new cloud product from Hortonworks that offers pay-as-you-go pricing for launching and managing Hadoop clusters on AWS. It handles common big data use cases and focuses on ease of use by providing prescriptive cluster types. The product aims to improve enterprise readiness in the cloud by providing scalable storage, security and governance features, and reliability through auto-recovery of unhealthy nodes. It also matches Hadoop with cloud capabilities like scalable storage, customizability, and cost-effective compute.
Extending the Yahoo Streaming Benchmark + MapR Benchmarks
The document summarizes benchmark tests that were performed to compare the throughput of Apache Storm and Apache Flink for processing streaming data. The original Yahoo! benchmark showed Storm outperforming Flink. However, the author repeated the tests and was able to achieve much higher throughput with Flink by addressing bottlenecks. When deployed on a high-performance MapR cluster, Flink processed over 72 million messages per second, significantly outperforming the original Storm results. The document concludes by noting Flink's compatibility features that allow reuse of existing Storm applications and components.
Apache Hive 2.0: SQL, Speed, Scale
Apache Hive 2.0 provides major new features for SQL on Hadoop such as:
- HPLSQL which adds procedural SQL capabilities like loops and branches.
- LLAP which enables sub-second queries through persistent daemons and in-memory caching.
- Using HBase as the metastore which speeds up query planning times for queries involving thousands of partitions.
- Improvements to Hive on Spark and the cost-based optimizer.
- Many bug fixes and under-the-hood improvements were also made while maintaining backwards compatibility where possible.
What's hot (20)
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
Druid: Sub-Second OLAP queries over Petabytes of Streaming Data
LEGO: Data Driven Growth Hacking Powered by Big Data
LEGO: Data Driven Growth Hacking Powered by Big Data
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
Enabling Apache Zeppelin and Spark for Data Science in the Enterprise
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
Cloud Operations with Streaming Analytics using Apache NiFi and Apache Flink
IoT Applications and Patterns using Apache Spark & Apache Bahir
IoT Applications and Patterns using Apache Spark & Apache Bahir
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...
Extending the Yahoo Streaming Benchmark + MapR Benchmarks
Extending the Yahoo Streaming Benchmark + MapR Benchmarks
Similar to Apache Zeppelin Helium and Beyond
Data science lifecycle with Apache Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It provides an overview of Zeppelin's history and architecture, including how interpreters and notebook storage are pluggable. The document also outlines Zeppelin's roadmap for improving enterprise support through features like multi-tenancy, impersonation, job management and frontend performance.
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
This document discusses Apache Zeppelin, an open-source notebook for interactive data analytics. It provides an overview of Zeppelin's features, including interactive notebooks, multiple backends, interpreters, and a display system. The document also covers Zeppelin's adoption timeline, from its origins as a commercial product in 2012 to becoming an Apache Incubator project in 2014. Future projects involving Zeppelin like Helium and Z-Manager are also briefly described.
Unified Batch and Real-Time Stream Processing Using Apache Flink
This talk was given at Capital One on September 15, 2015 at the launch of the Washington DC Area Apache Flink Meetup. Apache flink is positioned at the forefront of 2 major trends in Big Data Analytics:
- Unification of Batch and Stream processing
- Multi-purpose Big Data Analytics frameworks
In these slides, we will also find answers to the burning question: Why Apache Flink? You will also learn more about how Apache Flink compares to Hadoop MapReduce, Apache Spark and Apache Storm.
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
This document discusses Apache Zeppelin and Apache Flink integration. It describes how the Flink interpreter allows users to run Flink jobs within Zeppelin notebooks, accessing features like dynamic forms, angular displays, and progress monitoring. The roadmap includes improving multi-tenancy with authentication and containers, and developing Helium as a platform for packaging and distributing analytics applications on Zeppelin.
Apache Zeppelin and Helium @ApacheCon 2017 may, FL
Presented by
- Moon soo Lee (moon@zepl.com)
- Hoon Park (1ambda@zepl.com)
- Ahyoung Ryu (ahyoungryu@zepl.com)
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
This introductory level talk is about Apache Flink: a multi-purpose Big Data analytics framework leading a movement towards the unification of batch and stream processing in the open source.
With the many technical innovations it brings along with its unique vision and philosophy, it is considered the 4 G (4th Generation) of Big Data Analytics frameworks providing the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases: batch, streaming, relational queries, machine learning and graph processing.
In this talk, you will learn about:
1. What is Apache Flink stack and how it fits into the Big Data ecosystem?
2. How Apache Flink integrates with Hadoop and other open source tools for data input and output as well as deployment?
3. Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark.
4. Who is using Apache Flink?
5. Where to learn more about Apache Flink?
spark_v1_2
Spark is an open-source cluster computing framework that can run analytics applications much faster than Hadoop by keeping data in memory rather than on disk. While Spark can access Hadoop's HDFS storage system and is often used as a replacement for Hadoop's MapReduce, Hadoop remains useful for batch processing and Spark is not expected to fully replace it. Spark provides speed, ease of use, and integration of SQL, streaming, and machine learning through its APIs in multiple languages.
Spark SQL | Apache Spark
In this era of ever growing data, the need for analyzing it for meaningful business insights becomes more and more significant. There are different Big Data processing alternatives like Hadoop, Spark, Storm etc. Spark, however is unique in providing batch as well as streaming capabilities, thus making it a preferred choice for lightening fast Big Data Analysis platforms.
Big Data Processing With Spark
In this era of ever growing data, the need for analyzing it for meaningful business insights becomes more and more significant. There are different Big Data processing alternatives like Hadoop, Spark, Storm etc. Spark, however is unique in providing batch as well as streaming capabilities, thus making it a preferred choice for lightening fast Big Data Analysis platforms.
Extending DevOps to Big Data Applications with Kubernetes
DevOps, continuous delivery and modern architectural trends can incredibly speed up the software development process. Big Data applications cannot be an exception and need to keep the same pace.
Flink in action
Artsem Semianenko (Adform) - "Flink in action или как приручить белочку"
Slides for presentation: https://www.youtube.com/watch?v=YSI5_RFlcPE
Source: https://github.com/art4ul/flink-demo
Adding Spark support to Kylin at Bay Area Spark Meetup
Quick talk about Kylin, existing challenges and potential area to leverage Spark Ecosystem at Bay Area Spark Meetup on March 25.
H2O PySparkling Water
Michal Malohlava talks about the PySparkling Water package for Spark and Python users.
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Started with-apache-spark
In the past, emerging technologies took years to mature. In the case of big data, while effective tools are still emerging, the analytics requirements are changing rapidly resulting in businesses to either make it or be left behind
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.
Intro to Spark with Zeppelin
This document provides an introduction to Apache Spark and Zeppelin. It describes Spark as an open source cluster computing framework, and its APIs for Scala, Java, Python and R. Key Spark components are outlined like Spark Core, Spark SQL, MLlib and GraphX. RDDs are defined as Spark's primary abstraction, and DataFrames/Datasets are presented as higher-level APIs built on RDDs. The benefits of Spark SQL for structured data are highlighted. Examples demonstrate basic Spark and SQL usage. Finally, Apache Zeppelin and the Hortonworks sandbox are introduced as tools for interactive data analytics on Spark and Hadoop clusters.
Big Data Processing with Spark and Scala
In this era of ever growing data, the need for analyzing it for meaningful business insights becomes more and more significant. There are different Big Data processing alternatives like Hadoop, Spark, Storm etc. Spark, however is unique in providing batch as well as streaming capabilities, thus making it a preferred choice for lightening fast Big Data Analysis platforms.
ApacheCon NA 2011 report
The document summarizes the speaker's experience at ApacheCon NA 2011. It discusses several keynotes and sessions attended, including talks on building secure software, the success of Hadoop, Watson's use of Apache technologies, and new features in Lucene 4.0 like improved performance through UTF8 encoding and faster querying through new query types and indexing approaches. The document also mentions several projects that build user interfaces and experiences on top of Solr, like Prism, Blacklight, TwigKit, and Ajax Solr.
Similar to Apache Zeppelin Helium and Beyond (20)
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Unified Batch and Real-Time Stream Processing Using Apache Flink
Unified Batch and Real-Time Stream Processing Using Apache Flink
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
Apache Zeppelin and Helium @ApacheCon 2017 may, FL
Apache Zeppelin and Helium @ApacheCon 2017 may, FL
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with Kubernetes
Adding Spark support to Kylin at Bay Area Spark Meetup
Adding Spark support to Kylin at Bay Area Spark Meetup
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
More from DataWorks Summit/Hadoop Summit
State of Security: Apache Spark & Apache Zeppelin
This document discusses Spark security and provides an overview of authentication, authorization, encryption, and auditing in Spark. It describes how Spark leverages Kerberos for authentication and uses services like Ranger and Sentry for authorization. It also outlines how communication channels in Spark are encrypted and some common issues to watch out for related to Spark security.
Unleashing the Power of Apache Atlas with Apache Ranger
The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include:
- The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies.
- Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer.
- Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance.
- An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared
Enabling Digital Diagnostics with a Data Science Platform
This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.
Revolutionize Text Mining with Spark and Zeppelin
This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.
Double Your Hadoop Performance with Hortonworks SmartSense
This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.
Hadoop Crash Course
The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Data Science Crash Course
This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.
Apache Spark Crash Course
This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
Dataflow with Apache NiFi
This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Schema Registry - Set you Data Free
Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.Mool - Automated Log Analysis using Data Science and ML
QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
How Hadoop Makes the Natixis Pack More Efficient
Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase in Practice
HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.Backup and Disaster Recovery in Hadoop
While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
Scaling HDFS to Manage Billions of Files with Distributed Storage Schemes
Scaling HDFS to Manage Billions of Files with Distributed Storage SchemesDataWorks Summit/Hadoop Summit
Hadoop Distributed File System (HDFS) evolves from a MapReduce-centric storage system to a generic, cost-effective storage infrastructure where HDFS stores all data of inside the organizations. The new use case presents a new sets of challenges to the original HDFS architecture. One challenge is to scale the storage management of HDFS - the centralized scheme within NameNode becomes a main bottleneck which limits the total number of files stored. Although a typical large HDFS cluster is able to store several hundred petabytes of data, it is inefficient to handle large amounts of small files under the current architecture.
In this talk, we introduce our new design and in-progress work that re-architects HDFS to attack this limitation. The storage management is enhanced to a distributed scheme. A new concept of storage container is introduced for storing objects. HDFS blocks are stored and managed as objects in the storage containers instead of being tracked only by NameNode. Storage containers are replicated across DataNodes using a newly-developed high-throughput protocol based on the Raft consensus algorithm. Our current prototype shows that under the new architecture the storage management of HDFS scales 10x better, demonstrating that HDFS is capable of storing billions of files.More from DataWorks Summit/Hadoop Summit (20)
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
Scaling HDFS to Manage Billions of Files with Distributed Storage Schemes
Scaling HDFS to Manage Billions of Files with Distributed Storage Schemes
Recently uploaded
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Trusted Execution Environment for Decentralized Process Mining
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Recently uploaded (20)
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Apache Zeppelin Helium and Beyond
- 1. Apache Zeppelin Moon soo Lee moon@nflabs.com He lium And BeyondHe 2
- 3. Zeppelin 2012. 12 Data analytics solution based on AMP Lab Spark/Shark
- 4. ? Cloudera-ML ML-base MRQL Shark ? Zeppelin We thought there’re missing piece in Hadoop landscape. An analytics env.
- 5. Zeppelin 2012. 12 Data analytics solution based on AMP Lab Spark/Shark 2013. 10 Opensource interactive analytics feature as ‘Zeppelin’ 2013. 10 2014. 08
- 6. Zeppelin 2012. 12 Data analytics solution based on AMP Lab Spark/Shark 2013. 10 Opensource interactive analytics feature as ‘Zeppelin’ 2014. 12 ASF incubation Incubation Status http://incubator.apache.org/projects/zeppelin.html
- 7. Zeppelin 2012. 12 Data analytics solution based on AMP Lab Spark/Shark 2013. 10 Opensource interactive analytics feature as ‘Zeppelin’ 2014. 12 ASF incubation 2016. 03 110 Contributors world wide 1355 Stars on github repo 3 Releases One of the most popular project in ASF
- 8. Zeppelin A web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.
- 9. Let’s take a look
- 10. Zeppelin JDBC Markdown > _ Shell Interpreter : pluggable layer for language / processing backend integration 20+ interpreters are supported officially 2016. 03. Interpreters in Zeppelin source tree. Does not include 3rd party interpreters
- 11. Zeppelin Interpreter : pluggable layer for language / processing backend integration
- 12. Zeppelin Interpreter : Easy to extend public abstract class Interpreter { public void open(); public void close(); public InterpreterResult interpret(String st, InterpreterContext context); public void cancel(InterpreterContext context); public int getProgress(InterpreterContext context); public List<String> completion(String buf, int cursor); public FormType getFormType(); public Scheduler getScheduler(); } {Must have {Good to have Advanced {
- 13. Zeppelin Notebook Repo : pluggable layer for notebook persistence 4+ Notebook repos are supported officially 2016. 03. Notebook repos in Zeppelin source tree. Does not include 3rd party interpreters
- 14. Zeppelin Notebook Repo : Easy to extend public interface NotebookRepo { public List<NoteInfo> list() throws IOException; public Note get(String noteId) throws IOException; public void save(Note note) throws IOException; public void remove(String noteId) throws IOException; public void checkpoint(String noteId, String checkPointName) throws IOException; public void close(); }
- 15. Zeppelin Visualizations : 6 Built-in visualizations comes with pivot Table Bar Pie Area Line Scatter Free to draw any customized visualizations inside of notebook …
- 16. He liumHe 2 Platform for data analytics application that makes visualization pluggable and more. http://issues.apache.org/jira/browse/ZEPPELIN-533 https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal https://github.com/apache/incubator-zeppelin/pull/836 Proposal Umbrella issue Pull request Makes Zeppelin fly!
- 17. He liumHe 2 RESTful API Websocket Interpreter Notebook Storage Spark Flink Geode JDBC … FileSystem AmazonS3 Git … ZeppelinServer Interpreters and Notebook storage are pluggable
- 18. He liumHe 2 Interpreter Notebook Storage Spark Flink Geode JDBC … FileSystem AmazonS3 Git … ZeppelinServer Visualizations Map WordCloud … We want visualization be pluggable
- 19. He liumHe 2 Interpreter Notebook Storage Spark Flink Geode JDBC … FileSystem AmazonS3 Git … Application Visualizations Map WordCloud … Resource Pool SparkContext Flink Environment JDBC connection … Analytics ML … User object Extend pluggable visualization to pluggable analytics application
- 20. He liumHe 2 Helium application is interaction between view, algorithm and resources = + View Algorithm Zeppelin provided Resources Application
- 21. He liumHe 2 Zeppelin Server Web browser View Interpreter Process Algorithm Resource pool Resource pool Resource pools are connected Helium application runs where resource exists
- 22. Helium Application: Easy to extend public abstract class Application { public Application(ApplicationContext context); public abstract void run(ResourceSet args); public abstract void unload(); } He liumHe 2
- 23. He liumHe 2 Interpreter Notebook Storage Application Resource Pool SparkContext Flink Environment JDBC connection …User object Spark Flink Geode JDBC … FileSystem AmazonS3 Git … Map WordCloud …Maven Download and load on the fly Online repository for pluggable modules
- 24. He liumHe 2 Helium Registry zeppelin-packages my company + Add XX VisualizationWordcloud Make your table output to word cloud Install R Interpreter R is a free software environment for statistical computing and graphics. It compiles and runs on a wide variety of UNIX platforms, Windows and MacOS Install ZeppelinHub Notebook Storage Save your notebook in ZeppelinHub. You can access control and share your notebook online Install Registry for pluggable modules
- 25. He liumHe 2 Conclusion Helium trying to bring Zeppelin from notebook to analytics application platform. You can build and distribute not only your visualizations but also analytics application that uses cluster resources provided by Zeppelin interpreters. Next challenge is enrich the Helium registry.
- 26. Zeppelin Enterprise Ready Multi-tenancy Job scheduler HA Usability Improvement UX improvement Table data support Add yours! More details on https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap Roadmap
- 27. z Zeppelin and Friends Z-Manager ZeppelinHub Collaboration/Sharing Packaging & Deployment Zeppelin + Full stack on a cloud We want to grow community around Zeppelin, more!
- 28. Zeppelin Homepage http://zeppelin.incubator.apache.org/ Mailing list users@zeppelin.incubator.apache.org dev@zeppelin.incubator.apache.org Issue tracker https://issues.apache.org/jira/browse/ZEPPELIN Github repository http://github.com/apache/incubator-zeppelin Join the community
- 29. Thank you Moon soo Lee moon@nflabs.com moon@apache.org https://twitter.com/issuefreaks